sc-dev-deploy/README.md

dev-deploy

A Saltcorn plugin that migrates metadata changes across Dev, Test, and Prod environments.

dev-deploy treats your schema as something you promote, not something you hand-copy. As you edit metadata in one Saltcorn instance, the plugin records each change as an immutable entry in an append-only ops journal. You then promote those ops to a peer instance (Test, Prod, or another tenant) over an HMAC-authenticated HTTP endpoint. Each tracked entity carries a stable UUID so the same table/view/page resolves to the same identity on every instance, even though local integer ids differ. When a peer and the local instance have both touched the same entity since the last sync, the incoming op is held as a conflict for you to resolve (theirs / mine / per-field merge) rather than silently overwriting.

What it migrates

The plugin tracks these Saltcorn metadata entity kinds (see lib/constants.js, ENTITY_KINDS):

• table, field, constraint
• view, page, page_group, page_group_member
• trigger, workflow_step
• role
• library
• tag
• file

Children point at their parent via parent_uuid, so a field's identity is anchored to its table, and so on.

What it does NOT touch

User-table row data is left alone by default. A table's rows belong to the local environment and deploys never touch them unless you explicitly change the table's data_mode away from the default user:

• user (default) -- rows belong to the local environment; deploys never touch them. The only safe choice for end-user-entered data.
• starter -- rows ship to the target on first install, then the target owns them; later changes on the source side do not propagate.
• managed -- rows always sync from source; the source is canonical and the target's edits get overwritten or surface as conflicts.

Switching a table to managed or starter rewrites that table's schema (it adds a hidden _dd_row_uuid column for stable cross-environment row identity) and ships the current rows. The Saltcorn users table is locked to data_mode=user and cannot be changed. See docs/managed-rows.md.

How it works (at a glance)

1. Bootstrap. On first load the plugin creates its six _dd_* tables, assigns this instance an env_id (a UUID), and backfills stable UUIDs for all existing metadata entities. Two environments that bootstrap from the same metadata population assign identical UUIDs to entities with the same (kind, name), because UUIDs are derived from a frozen namespace (ID_NAMESPACE in lib/constants.js).
2. Capture. Saltcorn metadata mutations are wrapped so each change appends one op to _dd_ops (op types like update_*, set_table_mode, insert_row, etc.). History is append-only.
3. Promote / pull. From the admin UI you generate a plan (ops not yet sent to a peer, tracked by a per-peer outbound anchor) and promote them, or pull a peer's ops inbound. Transport is HMAC-signed.
4. Apply. The receiver applies each op, resolving UUIDs to its own local ids. Concurrent edits are detected and parked as conflicts.
5. Resolve. Pending conflicts are resolved from the admin UI as theirs / mine / per-field merge.
6. Revert. Reverting an op appends a compensating op rather than rewriting history.

Features and status

• Append-only ops journal with stable per-entity UUIDs -- done.
• HMAC-SHA256 authenticated peer transport (timestamp skew window, nonce, host-bound canonical) -- done.
• Plan / promote / pull with per-peer, per-direction anchors -- done.
• Concurrent-edit conflict detection with theirs / mine / per-field merge -- done.
• Revert via compensating ops -- done.
• Per-table data modes (user / starter / managed) with hidden _dd_row_uuid row identity -- done.
• File-entity sync (files identified by a stable id derived from disk location) -- done.
• Multi-tenant: each tenant schema is its own dev-deploy environment with its own env_id and schema-qualified _dd_* tables; HMAC canonical is host-bound so a request signed for one tenant is rejected against another on the same server -- done. See docs/multitenancy.md.
• Plugin-version drift warnings: promote/pull compare the local and peer plugin lists and surface mismatches as warnings (non-blocking) -- done.
• Engines: Node >=20. Saltcorn plugin API version 1.

The six _dd_* tables

All six are created idempotently in onLoad (lib/schema.js) and are schema-qualified per tenant. Strings, JSON payloads, and ISO 8601 timestamps are TEXT; booleans and surrogate ids are INTEGER.

Table	Purpose	Key columns
_dd_env	This instance's dev-deploy identity (singleton row).	env_id (PK), env_label, on_destructive_op (default confirm), require_tls, created_at, bootstrapped_at
_dd_peers	Configured peers and their encrypted shared secret.	peer_id (PK), env_id (UNIQUE), label, base_url, peer_secret_ciphertext, peer_secret_iv, peer_secret_tag, require_tls, created_at, last_seen_at
_dd_entity_ids	Maps stable UUID to the local entity.	uuid (PK), kind, current_name, current_id, parent_uuid, created_at, UNIQUE(kind, current_id)
_dd_ops	The append-only ops journal.	op_id (PK), source_env_id, op_type, entity_kind, entity_uuid, payload, parent_op_id, correlation_id, schema_version, created_at, applied_at, status (default committed), conflict_with_op_id
_dd_anchors	Per-peer, per-direction sync watermark.	(peer_id, direction) (PK), last_op_id, updated_at
_dd_table_modes	Per-table data_mode and ship state.	table_uuid (PK), data_mode, updated_at, starter_shipped_at

HTTP endpoints

Admin UI (session, admin role role_id === 1)

Method	Path	Purpose
GET	/admin/dev-deploy/	Dashboard: env identity, op/entity counts, pending conflicts.
GET	/admin/dev-deploy/ops	Journal viewer (supports ?limit=, ?since=op_id; JSON via Accept: application/json).
POST	/admin/dev-deploy/revert	Append a compensating op to revert an op.
GET	/admin/dev-deploy/peers	List peers; show this instance's env_id; add-peer form.
POST	/admin/dev-deploy/peers/add	Pair a peer (generates or accepts a 64-hex shared secret).
POST	/admin/dev-deploy/peers/rotate	Rotate a peer's shared secret.
POST	/admin/dev-deploy/peers/delete	Delete a peer.
GET	/admin/dev-deploy/plan	Preview the ops that would be promoted to a chosen peer.
POST	/admin/dev-deploy/promote	Promote outbound ops to a peer via signed ingest.
POST	/admin/dev-deploy/pull	Pull a peer's ops inbound via signed journal and apply them.
GET	/admin/dev-deploy/tables	List tracked tables and their data_mode.
POST	/admin/dev-deploy/tables/set	Set a table's data_mode.
GET	/admin/dev-deploy/conflicts	List pending conflicts.
POST	/admin/dev-deploy/conflicts/resolve	Resolve a conflict with theirs or mine.
GET	/admin/dev-deploy/conflicts/merge	Per-field merge view for a conflicting update_* op.
POST	/admin/dev-deploy/conflicts/merge/apply	Apply a per-field merge resolution.

Machine API (HMAC peer auth, CSRF-exempt)

These routes are registered with noCsrf: true; onLoad also registers /dev-deploy/api/ in Saltcorn's disable_csrf_routes.

Method	Path	Purpose
GET	/dev-deploy/api/journal?since=op_id	Return this env's ops since an op id (max 1000, oldest first).
POST	/dev-deploy/api/ingest	Apply a batch of ops from a peer; advances the inbound anchor.
GET	/dev-deploy/api/file/:uuid	Stream a file entity's bytes by UUID.
GET	/dev-deploy/api/health	Return this env's env_id, label, and installed plugin list.

Every machine-API request must carry these headers (see lib/peerAuth.js):

Header	Meaning
X-DD-Env-Id	Caller's env_id, looked up in _dd_peers.
X-DD-Timestamp	Milliseconds since epoch; rejected if skew > 5 min.
X-DD-Nonce	Random per-request bytes (replay padding).
X-DD-Signature	Hex HMAC-SHA256 over the canonical string.

The request body uses Content-Type: application/vnd.dev-deploy+json so Saltcorn's express.json() does not consume the stream before the HMAC is verified over the exact raw bytes. The canonical string binds the target host, so a request signed for one tenant will not verify if replayed against another tenant on the same server. See docs/peering.md.

Quick start

These steps assume the dual-instance dev layout where the project root holds your state dirs and the upstream Saltcorn lives in a sibling subfolder, with dev-deploy/ next to them.

1. Install the plugin into the dev instances. From the project root:

   ./reinstallDevDeploy.sh
   

   This installs dev-deploy into the MAIN (.dev-state), TEST (.dev-state-test), and Postgres (.dev-state-pg) instances. Because the plugins folder's node_modules symlinks are shared with the sibling saltcorn-idp plugin, re-run ./reinstallIdp.sh afterward to keep both installs consistent.

2. Run an instance. From the project root, start MAIN on :3000:

   ./startServer.sh
   

   For the second SQLite instance (TEST on :3001):

   ./startServerTest.sh
   

   For the multi-tenant Postgres instance (on :3002):

   ./startServerPg.sh
   

   On first load the plugin bootstraps: it creates the _dd_* tables, assigns an env_id, and backfills entity UUIDs. Open http://localhost:3000/admin/dev-deploy/ (as an admin) to reach the dashboard.

3. Run the gates. The plugin ships an end-to-end harness and four targeted gates:

   node test/e2e.js                  # full end-to-end run (npm test)
   node test/mtGate.js               # multi-tenant isolation (Postgres)
   node test/mixedTopologyGate.js    # standalone-to-tenant push peering + host-bound HMAC
   node test/managedRowsGate.js      # managed-row sync across tenants (Postgres)
   node test/pullGate.js             # reverse + tenant-to-tenant pull peering (Postgres)
   

   The Postgres gates require the :3002 instance to be running. See docs/testing.md.

Documentation

Doc	Covers
docs/architecture.md	The ops journal, stable UUIDs, entity model, and apply pipeline.
docs/peering.md	Pairing, the HMAC transport, anchors, and the machine API.
docs/multitenancy.md	Per-tenant environments, schema-qualified tables, host-bound auth.
docs/managed-rows.md	user / starter / managed data modes and _dd_row_uuid.
docs/operations.md	Day-to-day plan / promote / pull, conflict resolution, revert.
docs/testing.md	The e2e harness and the multi-tenant / managed-row gates.

License

MIT. Author: Scott Duensing.