202 lines
9.1 KiB
Markdown
202 lines
9.1 KiB
Markdown
# saltcorn — dev environment for the dev-deploy plugin
|
|
|
|
This project root holds the `dev-deploy` Saltcorn plugin and the launcher
|
|
scripts + per-instance state needed to develop and test it against two local
|
|
Saltcorn instances. Upstream Saltcorn lives in `saltcorn/` (a sibling git
|
|
checkout managed independently from this project).
|
|
|
|
## Layout
|
|
|
|
```
|
|
saltcorn/ project root (this git repo)
|
|
├── dev-deploy/ the plugin (Saltcorn plugin, sc_plugin_api_version 1)
|
|
├── startServer.sh launch MAIN instance on :3000
|
|
├── startServerTest.sh launch TEST instance on :3001
|
|
├── devServer.sh launch MAIN under `saltcorn dev:serve` (tsc watch)
|
|
├── installSaltcorn.sh reproduce this dev environment from scratch
|
|
├── .dev-state/ MAIN instance: env.sh, saltcorn.sqlite, files/, sessions
|
|
├── .dev-state-test/ TEST instance: same shape, separate port + secret
|
|
└── saltcorn/ upstream Saltcorn checkout (its own .git, gitignored here)
|
|
```
|
|
|
|
The upstream subfolder, both `.dev-state*` directories, and `node_modules/`
|
|
are all gitignored — only the plugin + scripts + this README are tracked.
|
|
|
|
## Prerequisites
|
|
|
|
- `git`, `curl`
|
|
- `nvm` (auto-installed at `~/.nvm` if missing) → Node 20
|
|
- `git-lfs` for the `.gitattributes` LFS filters (`git lfs install` once per clone)
|
|
|
|
`installSaltcorn.sh` handles nvm + Node 20 install for you.
|
|
|
|
## Fresh install
|
|
|
|
```bash
|
|
./installSaltcorn.sh [destination] # default: ./saltcorn
|
|
```
|
|
|
|
Clones upstream Saltcorn into `<dest>/saltcorn/`, runs `npm install` + `npm
|
|
run tsc`, generates `<dest>/.dev-state/env.sh` with a fresh session secret,
|
|
initializes the SQLite schema, creates the admin user, and writes
|
|
`<dest>/startServer.sh`.
|
|
|
|
The script sets up the **MAIN** instance only. To add the **TEST** instance,
|
|
duplicate `.dev-state/` as `.dev-state-test/`, edit its `env.sh` to use a new
|
|
`SALTCORN_SESSION_SECRET` and add `export SALTCORN_PORT="3001"`, then:
|
|
|
|
```bash
|
|
source .dev-state-test/env.sh
|
|
saltcorn reset-schema -f
|
|
saltcorn create-user -a -e admin@local -p AdminP@ss1
|
|
```
|
|
|
|
The reference `.dev-state-test/env.sh` already in this repo shows the exact
|
|
pattern.
|
|
|
|
## Running
|
|
|
|
```bash
|
|
./startServer.sh # MAIN → http://localhost:3000/
|
|
./startServerTest.sh # TEST → http://localhost:3001/
|
|
./devServer.sh # MAIN under `saltcorn dev:serve` (nodemon + tsc watch)
|
|
```
|
|
|
|
Login on either instance: `admin@local` / `AdminP@ss1`.
|
|
|
|
Each `startServer*.sh` runs `saltcorn install-plugin -d ./dev-deploy` on every
|
|
boot, so edits to `dev-deploy/` go live on the next restart.
|
|
|
|
## The dev-deploy plugin
|
|
|
|
`dev-deploy/` migrates Saltcorn metadata (tables, fields, views, pages,
|
|
triggers, roles, library, tags, constraints, files, page groups, workflow
|
|
steps, plus selected config keys and plugin configuration) across Dev/Test/Prod
|
|
environments via an append-only ops journal with stable cross-environment
|
|
UUIDs and HMAC-authenticated peer endpoints. Concurrent edits surface as
|
|
conflicts in the admin UI with theirs/mine/per-field-merge resolution. User
|
|
row data is left alone unless an admin explicitly marks a table as `managed`
|
|
or `starter`.
|
|
|
|
Admin UI: `/admin/dev-deploy/` (logged in as an admin). Machine API:
|
|
`/dev-deploy/api/{journal,ingest,file/:uuid,health}` (HMAC-signed peer requests).
|
|
|
|
### Pairing two instances
|
|
|
|
Do this once per pair (e.g. MAIN ↔ TEST). Each side needs the other's
|
|
`env_id` (a UUID shown on its `/admin/dev-deploy/` dashboard) and base URL.
|
|
|
|
1. **On instance A** — go to `/admin/dev-deploy/peers`. Under "Add peer", fill
|
|
in B's `env_id`, a label, B's base URL, and leave "Existing secret" blank.
|
|
Click **Pair**. A's response shows a 64-hex shared secret **once** — copy it.
|
|
2. **On instance B** — `/admin/dev-deploy/peers` → "Add peer". Fill in A's
|
|
`env_id` + base URL, paste the secret from step 1 into "Existing secret",
|
|
click **Pair**.
|
|
|
|
Both sides now have a peer row with a sealed copy of the shared secret. The
|
|
plain secret is never persisted on either side — only the AES-256-GCM
|
|
ciphertext (KEK derived from `SALTCORN_SESSION_SECRET`). Use **Rotate** on
|
|
either side to roll the secret (the peer page surfaces the new one once;
|
|
paste it on the other side via Rotate).
|
|
|
|
### Promoting changes (source → target)
|
|
|
|
1. Make metadata edits on the source (e.g. create a table on MAIN).
|
|
2. Go to `/admin/dev-deploy/plan`, pick the target peer, click **Show plan**.
|
|
Lists the ops that would be sent (everything since the last outbound
|
|
anchor for that peer).
|
|
3. Click **Promote N ops**. The target applies them, advances the outbound
|
|
anchor, and returns per-op results (applied / error / conflict counts in
|
|
the redirect flash).
|
|
|
|
Plugin-list mismatches between source and target are surfaced as warnings in
|
|
the same flash message (e.g. "peer missing plugin X", "plugin version mismatch
|
|
on Y"). They don't block the promote — they're advisory.
|
|
|
|
### Pulling changes (target ← source)
|
|
|
|
On the target, `/admin/dev-deploy/peers` → **Pull** on the source's peer row.
|
|
Fetches everything since the last inbound anchor and applies it. If any op
|
|
conflicts with a local op on the same entity since the last sync, the incoming
|
|
op is journaled with `status='conflict'` and **not** applied; the flash
|
|
redirects to `/admin/dev-deploy/conflicts`.
|
|
|
|
### Resolving conflicts
|
|
|
|
A conflict means both sides edited the same entity since the last sync. On
|
|
the target side at `/admin/dev-deploy/conflicts`, each pending conflict shows
|
|
the incoming op (theirs) next to the local op (mine):
|
|
|
|
- **Use theirs** — apply the incoming op now, overwriting the local change.
|
|
The local op stays in the journal but its effect is gone.
|
|
- **Use mine** — mark the incoming op `rejected`. Local state stands. The
|
|
peer keeps trying to ship the op on future pulls; we skip it by op_id.
|
|
- **Merge per field** (only for `update_X` vs `update_X` conflicts) — opens
|
|
a form with each diverging field. Pick **keep current**, **take incoming**,
|
|
or type a custom value per field. Submitting marks the conflict `merged`.
|
|
|
|
### Per-table data mode
|
|
|
|
By default, table **rows** are local-only (`user` mode) and dev-deploy only
|
|
migrates the table's schema. Change this per-table at `/admin/dev-deploy/tables`:
|
|
|
|
- **user** — rows belong to each environment; never synced. The Saltcorn
|
|
`users` table is hard-locked to this mode.
|
|
- **starter** — on the first promote, ships current rows once. The target
|
|
then owns them; future row changes on the source don't propagate.
|
|
- **managed** — rows always sync from source. Source is canonical; target's
|
|
row edits get overwritten or surface as row conflicts.
|
|
|
|
Switching to `managed` or `starter` adds a hidden `_dd_row_uuid` column to
|
|
the table via raw `ALTER` (not registered in `_sc_fields`, so Saltcorn's
|
|
table builder doesn't show it) and ships current rows in the next promote.
|
|
Switching back to `user` drops the column (best-effort; older SQLite without
|
|
DROP COLUMN support will error). FKs to user-mode tables are NULL'd on the
|
|
target with a warning attached to the op.
|
|
|
|
### Reverting an op
|
|
|
|
The journal at `/admin/dev-deploy/ops` shows every op with a **Revert** button.
|
|
Revert appends a compensating op (`create_X` → drop; `drop_X` → recreate from
|
|
the captured before-snapshot; `update_X` → re-apply before-snapshot as a
|
|
patch) rather than rewriting history. The inverse op promotes to peers like
|
|
any other op on the next push. Reverting a `drop` produces a new entity with
|
|
a fresh UUID — content is restored, identity is not.
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
# Both servers must be running first.
|
|
./startServer.sh & ./startServerTest.sh &
|
|
cd dev-deploy && npm test
|
|
```
|
|
|
|
Runs `dev-deploy/test/e2e.js` — ~50+ end-to-end tests covering pairing,
|
|
promote, pull, conflict resolution, constraints, files, page groups,
|
|
workflow steps, config propagation, managed/starter row data, revert, and
|
|
machine-endpoint security. Tests run in order and share state; don't reorder.
|
|
|
|
`test/sc-exec.js` is a shim used by the tests to run JS against Saltcorn's
|
|
models with full `require()` access (saltcorn's built-in `run-js` uses a vm
|
|
sandbox that hides Field/TableConstraint/File/etc.).
|
|
|
|
## Notes
|
|
|
|
- **Per-instance `sessions.sqlite`.** Saltcorn's SQLite session store writes
|
|
`sessions.sqlite` at process cwd (`packages/server/routes/utils.js`).
|
|
`startServer.sh` and `startServerTest.sh` `cd` into their respective state
|
|
directory before `exec saltcorn serve`, so each instance gets its own
|
|
sessions DB. `devServer.sh` cannot do this (its `dev:serve` runs `npm run
|
|
tsc` which needs cwd=upstream root), so its sessions land in
|
|
`saltcorn/sessions.sqlite` — don't run `devServer.sh` and `startServer.sh`
|
|
against the MAIN instance simultaneously.
|
|
|
|
- **`SALTCORN_SESSION_SECRET` is the pairing identity.** dev-deploy derives
|
|
its at-rest encryption key (KEK) for peer secrets via HKDF from this value.
|
|
Rotating it invalidates every peer pairing on the instance. The secret is
|
|
generated once per instance and persisted in that instance's `env.sh`.
|
|
|
|
- **Upstream Saltcorn updates.** Pull from upstream via
|
|
`git -C saltcorn pull` — the project root is a separate repo and doesn't
|
|
see those commits. After a pull, you may need `npm install && npm run tsc`
|
|
inside `saltcorn/`.
|