sc-site-sync/docs/architecture.md

site-sync architecture

site-sync is a one-way desired-state deployer. It delegates SERIALIZATION to Saltcorn's backup.create_pack_json and the bulk of APPLICATION to pack.install_pack, and adds three things of its own: a diff/plan/preview engine, a surgical policy-gated delete engine, and an opt-in reference-row layer. The peer transport, sealed secrets, config workflow, and per-tenant patterns are transplanted from the sibling dev-deploy plugin.

Module map (lib/)

Transport & identity. The peer protocol, peer store, at-rest sealing, admin helpers, registered-table bootstrap and sync-policy constants live in lib/common/ as generic infrastructure (they began as code shared with dev-deploy; site-sync now owns its copy). The site-sync modules below bind them:

Module	Responsibility
constants.js	Plugin name/version, SYNC_MODES, DELETE_POLICY, APPLY_POLICY, PLAN_STATUS, signed-request header/content-type names, the peer KEK salt/info (PEER_KEK_INFO) and table names; re-exports SYNCABLE_CONFIG_KEYS, ROW_SYNC_LOCKED_TABLES, DATA_MODES and the infra / lock / synced-mode predicates from common/syncConstants.js.
transport.js	API paths and size/time limits, and the outbound client on common/peerClient.js -- one signedRequest (signed headers, application/vnd.site-sync+json, timeout, byte-capped read, response-signature verification before use) behind peerJson (2xx JSON + responder env_id check) and peerBytes (verified file bytes).
peerAuth.js	Binding of common/peerServer.js: inbound requirePeerAuth (header check -> declared/streamed body cap -> skew -> peer lookup -> signature, host = Express req.host -> nonce replay -> JSON body -> signed TLS gate on req.protocol) and sendSigned/sendSignedJson, which sign every reply against the request's nonce.
peers.js	_ss_peers binding of common/peerStore.js: CRUD; secrets sealed at rest (common/aesGcm.js, KEK from common/sessionKey.js -- only an explicitly configured session_secret); rotate (generate or install the other side's new secret); peer_id allocation retries on collision; deleting a peer deletes its plans (operations.deletePeer first detaches the ones that can still be rolled back).
env.js	Env identity in _sc_config key site_sync_env via common/configStore.js (survives backup, restored before onLoad).
ids.js	randomUuid (env_id, plan ids, row identities).
schema.js	Idempotent table bootstrap: registered _ss_peers (fields from common/peerStore.js)/_ss_table_modes; raw regenerable _ss_seen_nonces (common/peerServer.js)/_ss_plans/_ss_rollback/_ss_apply_lock; race-tolerant column migrations (_ss_plans.claimed_from, _ss_rollback.restore_files) for workers booting together (and drops 0.0.1's unused _ss_peer_state).
render.js	Admin-UI helpers: the admin URL table, theme-aware sendWrap, nav; re-exports the shared admin gate, escape, csrf field and flash-and-redirect.

New desired-state code:

Module	Responsibility
capture.js	buildDesiredState = create_pack_json -> sanitizeDesired (parity-only plugins, config allow-list, exclusion of locked tables and of _-prefixed infra tables AND fields, plain-JSON normalization) -> order-insensitive hash. Also sanitizes packs received from a peer.
diff.js	Pure computePlan(desired, current, opts): per-kind create/update/delete (roles by id, isEqual with per-kind normalization), nested table field/constraint diff, structural TABLE-rename detection (views/pages: create + delete), code pages as a first-class kind, config overwrite set, destructive annotation.
plan.js	Live row-count gathering, can_install_pack warnings, reference-row preview, plan persistence (_ss_plans; retention never prunes a plan whose restore point is kept) + precondition hash, plan listing, the data-drop tables an apply must name, and preview/apply-form HTML.
siteLock.js	The site-wide apply lock: one _ss_apply_lock row per tenant schema under a renewed lease timed by the database clock; atomic acquire (insert, or take over a lapsed lease) that recovers plans a stopped holder left in flight to their claimed_from status; lease checkpoints (assertHeld) and finishClaimed (no status write after the lock was lost); read-only lockState for the dashboard / MCP.
apply.js	Orchestrator: site lock -> atomic plan claim -> precondition re-check -> metadata snapshot + restore point -> confirmed-rename pre-pass -> deep-clone+prune -> install_pack -> delete pass -> reference-row sync -> non-silent getState().refresh(), checking the lease before each mutating phase; collects every non-fatal problem into result.warnings.
deletes.js	applyRenames (confirmed renames only; data-preserving Table.rename, SQLite fallback) + applyDeletes (reverse-dependency prune incl. code pages, models, constraint removals; FK fields stripped before any table drop; live row re-count and the data-loss gates).
rowIdentity.js	Registered _ss_row_uuid field, ensureManagedSchema, uuid<->pk resolution (via pk_name), locked-table gates.
rowPayload.js	Portable row <-> local row (FK ids <-> referenced-row uuid; File fields <-> a content-hash placeholder; user-mode FKs nulled with a warning).
rows.js	Data-mode storage (_ss_table_modes), manifest build (pre-pass schema), the apply engine (insert fixpoint -> converge/FK-fix, writing only changed rows -> managed prune that keeps rows user data still references), file materialization (fetch-by-hash + dedup; a failed fetch keeps the existing reference), and the source-side content-hash file index.
layoutFiles.js	Always-on content-addressed transfer of files referenced inside page layouts / view configs / the site logo: placeholderize (capture), resolvePlaceholders (apply; image-mime allow-list + hash-verify + dedup; min_role = most permissive of the referencing pages/views), indexLocalFiles (extends the source file index).
rollback.js	Pre-apply restore point (metadata pack + exactly the rows the gate can drop/change, in _ss_rollback) and rollbackApply (under the site lock, once per apply: mirror-mode re-sync of metadata/config -- an apply run under the same lease --, inverse renames, removal of inserted rows, restore of captured rows by pk, starter un-marking; file refs revert via retained old bytes).
operations.js	The admin operations, transport-free -- the routes AND the MCP tools are thin adapters over them, so the rules live here: settings options + validation, peer resolution (id / env_id / label) and pairing, the pull orchestration (diffPluginsWithPeer parity check -> signed fetchPeerState -> buildPlan), plan listing and standing (can apply / stale / can roll back / interrupted), apply and rollback (the site lock is taken below, in apply.js / rollback.js, so push-apply shares it; a lock refusal is an OpError with no rollback offer), the lock's state for the dashboard and sitesync_status (idle / running / interrupted), table listing and data modes, this site's read-only state summary. Actionable failures throw OpError.
routes.js	Route table + admin handlers (rendering over operations.js) + machine handlers (wrapped by peerRoute: auth, then a signed reply even on error).
mcpTools.js	mcp_tools for the saltcorn-mcp server: 15 sitesync_* tools (read / write / admin) as adapters over operations.js; plan views trimmed (names, counts, gates) or one entity at a time, through ctx.redact; a generated peer secret returned once.

Apply order (why it is what it is)

1. Lock + claim + precondition -- take the site lock (siteLock.js, below: a second apply or a rollback started meanwhile, in any process, is refused), atomically claim the plan (a double-submit is refused; the status it left is kept in claimed_from), re-hash the target's current state, and refuse if it changed since preview (optimistic concurrency -- airtight against other applies, which the lock serializes). Row inserts don't change the metadata hash, so the delete engine additionally re-counts rows live.
2. Snapshot + restore point -- Saltcorn's Snapshot.take_if_changed (metadata) plus site-sync's own restore point (pre-apply metadata pack + raw rows of touched/dropped tables) so the apply is one-click revertible.
3. Renames (before upsert, only those the admin ticked on the apply form; never on a remote push-apply) -- Table.rename preserves data and makes install_pack update the renamed table rather than create a fresh empty one.
4. Upsert -- a deep-cloned, pruned pack fed to install_pack (name=undefined to skip the installed_packs append; plugins/event_logs/model_instances emptied; config re-intersected with the allow-list; existing desired-named tags deleted so the create path runs instead of the append-without-dedup path).
5. Deletes (after upsert) -- surgical per-entity prune in reverse-dependency order (page_groups -> pages -> views -> triggers -> library -> tags -> models -> code pages -> constraint removals + field drops on surviving tables, so no surviving FK column points at a doomed table -> FK fields + triggers of every doomed table -> the tables), each data-bearing drop gated by on_delete + the table name TYPED into the apply form's allow-list (never pre-filled; empty on a remote push-apply) + the confirm tick, with users/locked tables never dropped and no name-sweep.
6. Reference rows + files -- insert fixpoint (handles required cross-table FKs), converge (managed) / seed-once (starter), managed prune by uuid; File fields are materialized by fetching bytes from the source's GET /api/file/:hash (verified against the placeholder hash, deduped so unchanged files are not re-fetched; the response itself is signature-verified first).
7. Refresh -- one non-silent getState().refresh() so cluster workers converge.

Steps 3-6 each start by renewing the lease (assertHeld); see below.

One apply at a time: the site lock

Saltcorn forks several workers even on SQLite, and several servers may share one database, so the guard cannot live in process memory. _ss_apply_lock holds at most one row per tenant schema: the holder (a random token), the operation and plan, and expires_ms. Every time in it is the DATABASE's clock (julianday('now') on SQLite, clock_timestamp() on Postgres), never a server's.

• Acquire is one atomic statement each way: INSERT ... ON CONFLICT DO NOTHING RETURNING while the lock is free, else UPDATE ... WHERE expires_ms < now RETURNING (take over a lapsed lease). Of any number of racing callers exactly one wins; the rest get LockBusyError naming what runs. Nothing queues.
• Lease: 60 s, renewed every 15 s by a background timer while the operation runs, so a holder that dies (crash, kill, restart) loses the lock within a minute.
• Recovery: every apply and rollback runs under the lock, so its holder knows any plan still applying / rolling_back was abandoned. Acquiring therefore returns each to claimed_from (recorded atomically by the claim: SET status = $1, claimed_from = status), exactly what a failed apply or rollback restores -- an interrupted apply is pending again (its restore point rolls it back), an interrupted rollback can be retried. Plans claimed by 0.0.3 (no claimed_from) fall back to pending / applied. The operation's warnings name each recovered plan.
• Reporting is read-only: lockState + the in-flight plans give the dashboard and sitesync_status their idle / running / interrupted state, and a plan's standing says interrupted (judged by the status it will be recovered to) when it is in flight with no live lease.
• A stalled holder (event loop blocked or database unreachable past the lease) can lose the lock. assertHeld before each mutating phase and before the final status write throws LockLostError; finishClaimed then writes no plan status (the new holder recovered the plan and may be working on it).
• Nesting: a rollback's metadata step is an apply run under the rollback's lease (withSiteLock({ lease })), not a second acquire.

Storage lifecycle (backup-safe by construction)

• Identity -> _sc_config (restored before onLoad; no duplicate).
• Must-survive runtime state (peer secrets, table modes) -> registered Saltcorn tables (_ss_peers, _ss_table_modes) that ride backup.
• Regenerable state (nonce store, persisted plans, rollback restore points, the apply lock) -> raw portable DDL (TEXT/INTEGER/BIGINT, schema-prefixed), invisible to backup but safe to lose.
• A managed table's _ss_row_uuid is a registered field (never a raw ALTER) so it survives restore instead of rolling back the whole table.