Bondlayer → Matriz import pipeline (runbook)

Status: active · Updated: 2026-07-26 · Owner: import pipeline

The single map of how a Bondlayer site becomes a live Matriz site, the tools involved, the exact run sequence, and how each piece is verified. Companion docs: editor JSON capture · Site JSON v1 format · xlsx export format · mission blockers · the review.

Path correction (2026-07-26). Every Python path in this doc previously read archive/2026-07-13-bondlayer-tooling/bondlayer-import/. That directory does not exist — archive/2026-07-13-bondlayer-tooling/ contains only the JS bondlayer-converter. The Python importer is live and maintained at apps/bondlayer-worker/ and deploys as a Cloudflare Container. Paths below are corrected; the JS converter genuinely is frozen under archive/.

Deployment (2026-09-02, D:importer-containers). The same Dockerfile that used to deploy to Render now builds the matriz-import-runtime Cloudflare Container (apps/bondlayer-worker/edge/ — a verbatim-relay Worker with an edge token gate; .github/workflows/deploy-import-runtime.yml deploys it on merge). skyfall-api picks the host with IMPORT_RUNTIME (workers/src/lib/import-runtime.ts, the sole reader of the importer origin vars); the cutover ran 2026-09-02 and "container" is the live value — "render" remains only as an inert rollback path (the Render service itself is retired; see plans/OWNER-OPS.md for the decommission runbook). The HTTP contract (/import, /import/jobs/:id, /materialize, /trimscan, the finalize_url + finalize_token callbacks) is byte-identical on both; the wizard and this doc’s pipeline are unaffected. Spec: plans/IMPORTER-CONTAINERS-MIGRATION-SPEC.md. Mentions of “Render” below are historical — the current host is the Cloudflare Container above, not part of the contract either way.

Known fidelity gaps found by the 2026-07-26 live capture — read before trusting a repeater-heavy import: ../Research/BONDLAYER-SLIDER-REPEATERS.md §3 (slider repeaters expand wrong; dot nav never wired; every Glide layout setting dropped) and ../Research/BONDLAYER-REPEATER-QUERY-SEMANTICS.md §6 (orphan repeaters/forms/styles records must be ignored, not imported).

The flow

 Bondlayer exports                    transform → interchange              persist → live
 ──────────────────                   ───────────────────────              ──────────────
 editor.json  ──▶ apps/bondlayer-worker/src/ (Python) ─┐
 (structure,        parse → style → element →        │
  styles, schemas)  cms → site → slot_values →       ├─▶  Matriz Site JSON v1  ─┐
                    api_writer.to_site_json           │    (camelCase, DTCG,      │
                                                      │     canonical slots)      │
 export_*.xlsx ──▶ archive/2026-07-13-bondlayer-tooling/bondlayer-converter/ (JS) ──┘            │              │
 (CMS records)      xlsx-collection-parser →                      │              │
                    (mergeCmsIntoSiteJson)  ───────────────────────┘              │
                                                                                  ▼
                                          POST /api/targets (provision)  →  POST /api/import/site/:siteId
                                          (site + design system,           (validate → batch insert,
                                           default_language)                atomic) → Neon
                                                                                  │
                                                                                  ▼
                                                          editor opens the site from Neon (Workers API)

Two complementary input arms target one interchange format (Matriz Site JSON v1), which a Neon-native two-step API persists.

The two arms

Arm	Tool	Produces	Why
Editor JSON	`apps/bondlayer-worker/src/` (Python)	site, pages, dom-node tree, component styles, design tokens, fonts, content types (from schemas), canonical `slot_values`	the structure + styling + the bindings that reference content
xlsx CMS	`archive/2026-07-13-bondlayer-tooling/bondlayer-converter/` (JS)	content types + items from `export_<id>.xlsx`	the actual CMS records
Merge	`site-json-merge.js`	one unified payload	items re-pointed to the editor-JSON type ids the bindings use

Run sequence (end to end)

# 1. Editor JSON → base Site JSON v1  (provision+ingest, or --output to inspect)
cd apps/bondlayer-worker/src
python3 import_bondlayer.py --editor-json ./site.editor.json \
  --default-language pt --output base.json
#   or, to push live:  --api-url https://<workers-host> --api-token <jwt>

# 2. Merge the xlsx CMS records into the base → unified payload
cd ../../../archive/2026-07-13-bondlayer-tooling/bondlayer-converter
node xlsx-to-sitejson.mjs ./xlsx-dir --merge-base <path-to>/base.json \
  --out unified.json --default-language pt

# 3. Provision + ingest (live)  — what --api-url automates, by hand:
#   POST /api/targets { name, slug, defaultLanguage }    → { site, designSystemId }
#   POST /api/import/site/:siteId   (body = unified.json) → { success, stats, warnings }

xlsx exports silently omit fields (`D:cms-export-missing-fields`)

Bondlayer’s CMS xlsx exporter drops every schema field flagged excludeExport: true — the rows arrive, the columns do not, and nothing says so. On the 12x12 capture, export_users.xlsx (416 rows, 27 columns) has no boolean_email_publico column although the users schema in nrkdmrhhrk.editor.json declares it, so every member lost their “email público” consent flag. The same export also omits text_bio, image_image, image_profile_picture, multiRef_editions and text_custom_link on users, and image_image_blur on Eventos (cmfvtqbukRz9aF9S).

Detect + report (always on). The xlsx arm (xlsx_cms.py) now runs in Python: parse_xlsx_directory returns each absorbed workbook’s raw header row (headers), and merge_cms_into_site_json diffs every collection matched to an editor-JSON content type against it (missing_export_fields: header bases via parse_header, so text_x.pt and ref_x.title count as present). Excluded: the system fields (_title, _slug, _created_at, _updated_at) and sensitive fields (sensitive_field_ids, D:cms-field-fidelity 1a — a password’s absence is the goal, not a loss). xlsx-only collections are never diffed (their schema is their header). The result — per collection {contentTypeId, name, bondlayerCollectionId, missingFields, excludeExportFlagged} — is:

printed by import_bondlayer.run_import as a WARNING … (cms_export_missing_fields) block beside the reconcile report, [excludeExport] marking fields whose schema carries the flag (the known cause); and
attached to the owner-facing findings as importFindings.cmsExportMissingFields (one finding per collection, counted in totalFindings; see import_findings.py’s INCLUDED list).

Only collections that actually reach the CMS slice are diffed — users only when the members opt-in admits it (D:users-collection-public).

Backfill (opt-in): --backfill-from-live <bondlayerProjectId>. The published site still carries those values: it serves one content bundle per screen at https://cdn.bndlyr.com/<projectId>/_s/content.<screenId>_<n>.js (n = 0, 1, … until the first non-200; screen ids from json.loads(editor["design"])["web"]["screens"]). Each is window.BndLyrContent = {…}; with full records ({"id": "u-…", "_collection": "users", …}) nested anywhere. live_backfill.py walks every screen, indexes matching records, and fills only keys absent from an item’s data (an xlsx value is never overwritten), matched through the item’s data._bondlayer.id provenance and coerced to the xlsx arm’s conventions (scalars as-is; {"all": v} → v, extra locales → a {"default", <lang>} bag; ref → {"ref": uuid5_bl(id), "bondlayerId": id}; multiRef → a list of those; empty values skipped). Counts (recordsMatched/itemCount/ filledByField) join the WARNING block and the findings entry’s backfill. On 12x12 (nrkdmrhhrk) it recovers boolean_email_publico for 158 of 416 users (101 true / 57 false) — coverage is partial by nature: only records some published screen renders are in the bundles.

Privacy. The bundles carry everything on a record, including password hashes (text_password) and phone numbers. The indexer keeps, per record, only the fields being backfilled (the missing-field set, which never contains a sensitive field); bundles are parsed one at a time and never written to disk; log lines carry URLs, counts and exception class names — never a record or a value.

The default fetcher sends an explicit User-Agent (the CDN 403s urllib’s default) and falls back to curl when urllib’s TLS/proxy path fails. Tests (test_cms_export_missing_fields.py) inject a fetcher — no network.

Members: seeding the audience after an import (`D:member-seed`)

Two ways to opt in, same underlying mapping. The import wizard (frontend/src/components/import/ImportWizard.tsx) now has a Members step (S11, D:member-optin) that runs automatically as part of the normal import flow, for captures whose xlsx export has a users sheet: a counts-only pre-flight census (record count, linked-item count, field names, duplicate-email count — never an email/name/phone value) renders BEFORE any consent exists, with a default-OFF “Import members — no emails are sent” checkbox. Checking it seeds site_visitors/site_members in the SAME request that ingests the rest of the site (POST /api/import/site/:targetId, via runMemberBulkSeed — the exact write core POST /targets/:targetId/members/bulk below uses, not a second implementation). Leaving it unchecked (the default) reproduces today’s exact hard-refuse behavior byte-for-byte: no member data reaches the payload beyond the counts-only census. Consent is per-import and never persists across a re-analyze or a re-import (each import is a fresh target, D169).

The manual path below — the standalone users_to_members.py CLI (D182) plus a hand-run POST /members/bulk — remains the out-of-band route for a capture that was already imported before this consent step existed, or for member rows that came from somewhere other than the workbooks (the --members-dir override). Both paths funnel through the identical mapping (member_mapper.py) and the identical validated write (runMemberBulkSeed) — there is exactly one implementation of each, not a wizard copy and a CLI copy.

Members are not part of the Site JSON payload — they are people, not content, and they land in site_visitors/site_members rather than content_items. The import produces the artifacts; a separate, builder-authed call loads them.

1. The import writes the artifacts (automatic when the xlsx export has a users sheet — D122). Both files contain personal data; they are import working files and are .gitignored:

File	Shape
`<members-dir>/members.json`	`{ field_mapping, sensitive_fields_excluded, source, skipped, members: [{ bondlayer_id, email, name, role, role_resolved, profile }] }`
`<members-dir>/member_items.json`	`{ target_model, gap, collections: { <collectionId>: { ownerField, records: [...] } } }`

There is no password column in either file, by construction — the mapper hard-refuses sensitive-flagged fields (8.3(a) passwordless cutover). Nothing downstream can import a credential because nothing upstream emits one.

2. Seed them into the target — one call does both passes:

# members only
curl -sS -X POST "$API/api/targets/$TARGET_ID/members/bulk" \
  -H "Authorization: Bearer $CLERK_JWT" -H 'Content-Type: application/json' \
  --data-binary @members.json

# members + their member-owned records, in one two-pass request
jq -n --slurpfile m members.json --slurpfile i member_items.json \
  '{members: $m[0].members, items: $i[0]}' \
| curl -sS -X POST "$API/api/targets/$TARGET_ID/members/bulk" \
    -H "Authorization: Bearer $CLERK_JWT" -H 'Content-Type: application/json' \
    --data-binary @-

The body may be the members.json artifact verbatim, its members array, a bare array, or { members, items, defaultStatus, defaultRole }. defaultStatus is active (or pending for an approve-on-join site) and applies only to rows being created.

The reply is a per-row outcome summary — counts plus a skipped list keyed by payload index/bondlayerId. No skip reason ever echoes an email, name or phone number; look the row up by index in the source artifact.

{ "data": {
  "target":  { "id": "…", "membersEnabled": false, "joinMode": "open" },
  "members": { "received": 416, "created": 414, "updated": 0,
               "skipped": [{ "index": 12, "bondlayerId": "u_…", "reason": "malformed email (value withheld …)" }] },
  "items":   { "received": 331, "created": 331, "updated": 0, "skipped": [], "rejected": [],
               "collections": { "users_folders": { "contentTypeId": "…", "created": 331, "updated": 0 } } },
  "notes":   ["Member area is DISABLED for this target — …"] } }

Re-running is safe. Identity is (target_id, lower(email)) — the same key the magic-link redeem upserts on — so a second run updates and never duplicates. site_members.status and session_generation are never overwritten on conflict (a re-seed cannot un-block a member or sign everyone out); role and profile are refreshed, and profile merges so a member’s own form-edited keys survive.

3. Turn the member area on, otherwise seeded members cannot request a link at all (/members/request-code 404s while enabled is false). For a migrated audience invite is the right join mode — only members that exist may sign in:

curl -sS -X PATCH "$API/api/targets/$TARGET_ID/member-settings" \
  -H "Authorization: Bearer $CLERK_JWT" -H 'Content-Type: application/json' \
  -d '{"enabled":true,"joinMode":"invite","signInPath":"/sign-in","afterSignInPath":"/account"}'

How a seeded member actually signs in (D73, unchanged — the seed adds no auth path of its own):

POST /api/m/:targetId/members/request-code   { email }      → constant 200, magic link emailed
GET  /api/m/:targetId/members/session?token=…              → atomic burn → site_visitors upsert
                                                              (stamps email_verified_at)
                                                            → finds the SEEDED site_members row
                                                            → __matriz_member cookie → 302
GET  /api/m/:targetId/members/me/items?collection=…        → their seeded records

Two gates to know: MEMBER_SESSION_SIGNING_KEY must be set on the Worker or the redeem 503s, and the member must be status: 'active' (memberMeetsGate) to read gated content. The seed deliberately leaves email_verified_at NULL — seeding is not proof that anyone controls an address; the first redeem stamps it.

Member-item contract (verified against workers/src/routes/modules/member-data.ts):

the collection is resolved by (content_types.project_id, name) — not by target — so the seed reuses an existing same-named project collection rather than creating a second one that would make that lookup ambiguous;
rows need is_current = true, and ownership is data->>'visitorId', which is always server-derived (a payload-supplied visitorId is stripped);
seeded rows are status: 'draft' on purpose. Publish selects status = 'published' (lib/page-render.ts), so a member-private row seeded as published would become publicly renderable the moment any repeater bound that collection — the D122 leak class. /members/me/items applies no default status filter, so its owner still reads it.

Why the importer does not POST this itself. api_writer.py runs inside the importer container (matriz-import-runtime) with the server-to-server IMPORT_WORKER_TOKEN, not a builder session. Teaching the bulk endpoint a second, non-Clerk auth mode — on the one route that writes hundreds of people’s personal data — is a bigger security surface than it saves keystrokes, so the seed stays an explicit, builder-authed step and member_mapper.py prints the artifact paths instead.

Component map

Python (apps/bondlayer-worker/src/)

parser.py → bl_types.py (dataclasses) — parse editor JSON.
style_transformer.py, element_mapper.py, cms_mapper.py, site_mapper.py — map to Matriz shapes.
slot_mapper.py — canonical slot_values (localized text, structured image/link). [G5]
api_writer.py — to_site_json() (snake→camel, DTCG coercion, drift reconciliation) + MatrizApiWriter (provision + ingest over HTTP). Replaces the deleted db_writer.py (Supabase).
gen_sample_payload.py — emits the golden contract fixture.
xlsx_cms.py — the Python xlsx arm + merge; missing_export_fields reports schema fields a workbook omits (D:cms-export-missing-fields).
live_backfill.py — opt-in --backfill-from-live: refills those fields from the published site’s content bundles (wanted-fields-only index; see the privacy note above).

JS (archive/2026-07-13-bondlayer-tooling/bondlayer-converter/)

src/parsers/xlsx-collection-parser.js — export_*.xlsx → v1 content types/items (UUIDv5 id remap). [G2]
src/parsers/site-json-merge.js — unify the two arms.
xlsx-to-sitejson.mjs — CLI (--merge-base, --out).
(retired: import-to-supabase.ts deleted; database-writer.ts stubbed.)

Workers (workers/src/)

routes/sites.ts POST / — provisioning (site + design system, default_language). [C]
import/site-json.ts — the canonical v1 types.
import/validate.ts — structural validation (422 before any write).
import/ingest.ts — v1 → Drizzle rows (tenancy/audit stamping).
routes/import.ts POST /site/:siteId — auth → access → validate → atomic db.batch insert.

Verify it offline

# Python
cd apps/bondlayer-worker/src && python3 test_slot_mapper.py && python3 api_writer.py >/dev/null
# JS
cd archive/2026-07-13-bondlayer-tooling/bondlayer-converter && node verify-xlsx.mjs && node verify-merge.mjs
# Workers (validator, ingest mapper, routes, cross-arm contract)
cd workers && npx tsc --noEmit && npx vitest run --config vitest.node.config.ts

The cross-arm contract test (workers/test/node/python-emitter-contract.test.ts) feeds the actual Python-emitter output to the Workers validator — drift on either side fails there.

Decisions

D31 wire-format = camelCase at the Workers/Drizzle boundary.
D32 design tokens = DTCG {$type,$value}.
D33 provisioning implemented as POST /api/sites; import is a two-step provision-then-ingest flow.
D50 1:1 conversion fidelity (2026-07-02):
- Symbols → components. Symbol definitions become components catalog rows (new v1 components section; inserted via the service-role connection — the table’s RLS is SELECT-only for authenticated). Instances are EXPANDED into full dom-node subtrees (definition clone with instance-scoped ids, intra-symbol trigger targets remapped, nested symbols recursive with a cycle guard) and their roots stamped with component_id/component_instance/instance_status. Both enrich paths (publish + editor preview) skip compiling instance nodes that have children, so the expanded subtree — not the weaker structure compile — renders.
- Desktop-first styles. Bondlayer breakpoints are max-width media queries; styles now emit as base + max<px> layers (exact pixel values), desktop states hoisted to top level (styles.hover, not base.hover), kebab-case property names, and private styles baked into style_overrides (the render path never reads the styles column). The serializer + editor preview render max<px>/min<px> layers and custom variant states (extra1-3/current/…) as .state-<name> class rules.
- Shared styles (D:import-shared-styles, default ON, 2026-09-23). A last pass (style_sharing.py, before the payload is assembled) moves repeated private bodies OUT of style_overrides: symbol clones of one definition element share one component_style, then any identical body used by ≥ 2 nodes is deduped (uuid5 ids, stable across re-imports); overrides equal to the linked style and max<px> values restating the next-wider max<px> layer are dropped; instance roots are linked unless their subtree differs from the symbol master. Render-identical: the pass compares every node’s resolved style and reverts itself on any divergence, and test_style_sharing_parity.py re-proves it on every capture. Bodies carrying an asset url stay on the node (materialization only rewrites node fields). --no-share-styles emits the legacy payload. Measured with tools/style-audit/audit_overrides.py: nodes with overrides 72–91% → 6–23%.
- Triggers → behavior runtime. Extraction reads the real event key (was triggerEvent — every interaction collapsed to click), keeps delay/targetValue/multi-targets, maps state actions to set/remove/toggle_state + urlRedirect→navigate. Published pages embed a small interactions runtime (repeater-clone-aware; scope:'item' honored) allowed via a per-page CSP sha256 hash — script-src never loosens to unsafe-inline; pages without behaviors keep 'none'.
- Unmapped Bondlayer element types are reported by the CLI (29 more types mapped: form-label, tabs/slider/cart/user families, lottie, jsonld); editor-disabled elements import hidden (display:none) with bondlayer.disabled provenance.

Done vs gated

Done & offline-verified: both arms, the interchange format + validator, atomic ingest, canonical slot_values, provisioning, the merge, and the Python↔TS contract lock.

Gated (needs live secrets / a real export — see MISSION-BLOCKERS.md):

G13 the live acceptance test (one real import lands in Neon, editor opens it styled).
G11 asset-binary upload to R2.
Confirm against a real editor.json: the static-image attr key (G5 residual) and the merge’s schema-id == xlsx-filename-id assumption (both have safe failure modes today).
G4 style-merge render path + the @matriz/* framework slices — sequenced after G13.