sc-dev-deploy/README.md

# dev-deploy

A Saltcorn plugin that migrates **metadata** changes across Dev, Test, and
Prod environments.

dev-deploy treats your schema as something you promote, not something you
hand-copy. As you edit metadata in one Saltcorn instance, the plugin records
each change as an immutable entry in an append-only **ops journal**. You then
promote those ops to a peer instance (Test, Prod, or another tenant) over an
HMAC-authenticated HTTP endpoint. Each tracked entity carries a **stable UUID**
so the same table/view/page resolves to the same identity on every instance,
even though local integer ids differ. When a peer and the local instance have
both touched the same entity since the last sync, the incoming op is held as a
**conflict** for you to resolve (theirs / mine / per-field merge) rather than
silently overwriting.

## What it migrates

The plugin tracks these Saltcorn metadata entity kinds (see
`lib/constants.js`, `ENTITY_KINDS`):

- `table`, `field`, `constraint`
- `view`, `page`, `page_group`, `page_group_member`
- `trigger`, `workflow_step`
- `role`
- `library`
- `tag`
- `file`

Children point at their parent via `parent_uuid`, so a field's identity is
anchored to its table, and so on.

## What it does NOT touch

**User-table row data is left alone by default.** A table's rows belong to the
local environment and deploys never touch them unless you explicitly change the
table's `data_mode` away from the default `user`:

- `user` (default) -- rows belong to the local environment; deploys never
  touch them. The only safe choice for end-user-entered data.
- `starter` -- rows ship to the target on first install, then the target owns
  them; later changes on the source side do not propagate.
- `managed` -- rows always sync from source; the source is canonical and the
  target's edits get overwritten or surface as conflicts.

Switching a table to `managed` or `starter` rewrites that table's schema (it
adds a hidden `_dd_row_uuid` column for stable cross-environment row identity)
and ships the current rows. The Saltcorn `users` table is **locked** to
`data_mode=user` and cannot be changed. See
[docs/managed-rows.md](docs/managed-rows.md).

## How it works (at a glance)

1. **Bootstrap.** On first load the plugin creates its six `_dd_*` tables,
   assigns this instance an `env_id` (a UUID), and backfills stable UUIDs for
   all existing metadata entities. Two environments that bootstrap from the
   same metadata population assign identical UUIDs to entities with the same
   `(kind, name)`, because UUIDs are derived from a frozen namespace
   (`ID_NAMESPACE` in `lib/constants.js`).
2. **Capture.** Saltcorn metadata mutations are wrapped so each change appends
   one op to `_dd_ops` (op types like `update_*`, `set_table_mode`,
   `insert_row`, etc.). History is append-only.
3. **Promote / pull.** From the admin UI you generate a **plan** (ops not yet
   sent to a peer, tracked by a per-peer outbound **anchor**) and **promote**
   them, or **pull** a peer's ops inbound. Transport is HMAC-signed.
4. **Apply.** The receiver applies each op, resolving UUIDs to its own local
   ids. Concurrent edits are detected and parked as conflicts.
5. **Resolve.** Pending conflicts are resolved from the admin UI as
   theirs / mine / per-field merge.
6. **Revert.** Reverting an op appends a compensating op rather than rewriting
   history.

## Features and status

- **Append-only ops journal** with stable per-entity UUIDs -- done.
- **HMAC-SHA256 authenticated peer transport** (timestamp skew window, nonce,
  host-bound canonical) -- done.
- **Plan / promote / pull** with per-peer, per-direction anchors -- done.
- **Concurrent-edit conflict detection** with theirs / mine / per-field
  merge -- done.
- **Revert** via compensating ops -- done.
- **Per-table data modes** (`user` / `starter` / `managed`) with hidden
  `_dd_row_uuid` row identity -- done.
- **File-entity sync** (files identified by a stable id derived from disk
  location) -- done.
- **Multi-tenant**: each tenant schema is its own dev-deploy environment with
  its own `env_id` and schema-qualified `_dd_*` tables; HMAC canonical is
  host-bound so a request signed for one tenant is rejected against another on
  the same server -- done. See [docs/multitenancy.md](docs/multitenancy.md).
- **Plugin-version drift warnings**: promote/pull compare the local and peer
  plugin lists and surface mismatches as warnings (non-blocking) -- done.
- **Engines**: Node `>=20`. Saltcorn plugin API version 1.

## The six `_dd_*` tables

All six are created idempotently in `onLoad` (`lib/schema.js`) and are
schema-qualified per tenant. Strings, JSON payloads, and ISO 8601 timestamps
are `TEXT`; booleans and surrogate ids are `INTEGER`.

| Table | Purpose | Key columns |
| --- | --- | --- |
| `_dd_env` | This instance's dev-deploy identity (singleton row). | `env_id` (PK), `env_label`, `on_destructive_op` (default `confirm`), `require_tls`, `created_at`, `bootstrapped_at` |
| `_dd_peers` | Configured peers and their encrypted shared secret. | `peer_id` (PK), `env_id` (UNIQUE), `label`, `base_url`, `peer_secret_ciphertext`, `peer_secret_iv`, `peer_secret_tag`, `require_tls`, `created_at`, `last_seen_at` |
| `_dd_entity_ids` | Maps stable UUID to the local entity. | `uuid` (PK), `kind`, `current_name`, `current_id`, `parent_uuid`, `created_at`, `UNIQUE(kind, current_id)` |
| `_dd_ops` | The append-only ops journal. | `op_id` (PK), `source_env_id`, `op_type`, `entity_kind`, `entity_uuid`, `payload`, `parent_op_id`, `correlation_id`, `schema_version`, `created_at`, `applied_at`, `status` (default `committed`), `conflict_with_op_id` |
| `_dd_anchors` | Per-peer, per-direction sync watermark. | `(peer_id, direction)` (PK), `last_op_id`, `updated_at` |
| `_dd_table_modes` | Per-table `data_mode` and ship state. | `table_uuid` (PK), `data_mode`, `updated_at`, `starter_shipped_at` |

## HTTP endpoints

### Admin UI (session, admin role `role_id === 1`)

| Method | Path | Purpose |
| --- | --- | --- |
| GET | `/admin/dev-deploy/` | Dashboard: env identity, op/entity counts, pending conflicts. |
| GET | `/admin/dev-deploy/ops` | Journal viewer (supports `?limit=`, `?since=op_id`; JSON via `Accept: application/json`). |
| POST | `/admin/dev-deploy/revert` | Append a compensating op to revert an op. |
| GET | `/admin/dev-deploy/peers` | List peers; show this instance's `env_id`; add-peer form. |
| POST | `/admin/dev-deploy/peers/add` | Pair a peer (generates or accepts a 64-hex shared secret). |
| POST | `/admin/dev-deploy/peers/rotate` | Rotate a peer's shared secret. |
| POST | `/admin/dev-deploy/peers/delete` | Delete a peer. |
| GET | `/admin/dev-deploy/plan` | Preview the ops that would be promoted to a chosen peer. |
| POST | `/admin/dev-deploy/promote` | Promote outbound ops to a peer via signed `ingest`. |
| POST | `/admin/dev-deploy/pull` | Pull a peer's ops inbound via signed `journal` and apply them. |
| GET | `/admin/dev-deploy/tables` | List tracked tables and their `data_mode`. |
| POST | `/admin/dev-deploy/tables/set` | Set a table's `data_mode`. |
| GET | `/admin/dev-deploy/conflicts` | List pending conflicts. |
| POST | `/admin/dev-deploy/conflicts/resolve` | Resolve a conflict with `theirs` or `mine`. |
| GET | `/admin/dev-deploy/conflicts/merge` | Per-field merge view for a conflicting `update_*` op. |
| POST | `/admin/dev-deploy/conflicts/merge/apply` | Apply a per-field merge resolution. |

### Machine API (HMAC peer auth, CSRF-exempt)

These routes are registered with `noCsrf: true`; `onLoad` also registers
`/dev-deploy/api/` in Saltcorn's `disable_csrf_routes`.

| Method | Path | Purpose |
| --- | --- | --- |
| GET | `/dev-deploy/api/journal?since=op_id` | Return this env's ops since an op id (max 1000, oldest first). |
| POST | `/dev-deploy/api/ingest` | Apply a batch of ops from a peer; advances the inbound anchor. |
| GET | `/dev-deploy/api/file/:uuid` | Stream a file entity's bytes by UUID. |
| GET | `/dev-deploy/api/health` | Return this env's `env_id`, `label`, and installed plugin list. |

Every machine-API request must carry these headers (see `lib/peerAuth.js`):

| Header | Meaning |
| --- | --- |
| `X-DD-Env-Id` | Caller's `env_id`, looked up in `_dd_peers`. |
| `X-DD-Timestamp` | Milliseconds since epoch; rejected if skew > 5 min. |
| `X-DD-Nonce` | Random per-request bytes (replay padding). |
| `X-DD-Signature` | Hex HMAC-SHA256 over the canonical string. |

The request body uses `Content-Type: application/vnd.dev-deploy+json` so
Saltcorn's `express.json()` does not consume the stream before the HMAC is
verified over the exact raw bytes. The canonical string binds the target host,
so a request signed for one tenant will not verify if replayed against another
tenant on the same server. See [docs/peering.md](docs/peering.md).

## Quick start

These steps assume the dual-instance dev layout where the project root holds
your state dirs and the upstream Saltcorn lives in a sibling subfolder, with
`dev-deploy/` next to them.

1. **Install the plugin into the dev instances.** From the project root:

   ```
   ./reinstallDevDeploy.sh
   ```

   This installs `dev-deploy` into the MAIN (`.dev-state`), TEST
   (`.dev-state-test`), and Postgres (`.dev-state-pg`) instances. Because the
   plugins folder's `node_modules` symlinks are shared with the sibling
   `saltcorn-idp` plugin, re-run `./reinstallIdp.sh` afterward to keep both
   installs consistent.

2. **Run an instance.** From the project root, start MAIN on `:3000`:

   ```
   ./startServer.sh
   ```

   For the second SQLite instance (TEST on `:3001`):

   ```
   ./startServerTest.sh
   ```

   For the multi-tenant Postgres instance (on `:3002`):

   ```
   ./startServerPg.sh
   ```

   On first load the plugin bootstraps: it creates the `_dd_*` tables, assigns
   an `env_id`, and backfills entity UUIDs. Open
   `http://localhost:3000/admin/dev-deploy/` (as an admin) to reach the
   dashboard.

3. **Run the gates.** The plugin ships an end-to-end harness and four
   targeted gates:

   ```
   node test/e2e.js                  # full end-to-end run (npm test)
   node test/mtGate.js               # multi-tenant isolation (Postgres)
   node test/mixedTopologyGate.js    # standalone-to-tenant push peering + host-bound HMAC
   node test/managedRowsGate.js      # managed-row sync across tenants (Postgres)
   node test/pullGate.js             # reverse + tenant-to-tenant pull peering (Postgres)
   ```

   The Postgres gates require the `:3002` instance to be running. See
   [docs/testing.md](docs/testing.md).

## Documentation

| Doc | Covers |
| --- | --- |
| [docs/architecture.md](docs/architecture.md) | The ops journal, stable UUIDs, entity model, and apply pipeline. |
| [docs/peering.md](docs/peering.md) | Pairing, the HMAC transport, anchors, and the machine API. |
| [docs/multitenancy.md](docs/multitenancy.md) | Per-tenant environments, schema-qualified tables, host-bound auth. |
| [docs/managed-rows.md](docs/managed-rows.md) | `user` / `starter` / `managed` data modes and `_dd_row_uuid`. |
| [docs/operations.md](docs/operations.md) | Day-to-day plan / promote / pull, conflict resolution, revert. |
| [docs/testing.md](docs/testing.md) | The e2e harness and the multi-tenant / managed-row gates. |

## License

MIT. Author: Scott Duensing.