DVX_GUI/server/README.md

# DVX package server

The catalogue of add-on packages (`.DPK` files) that DVX's Software program browses, installs, and
updates from, and where publishers upload them. It is a [calog](../../calog) application over
SQLite, built on [singemaster](../../singemaster)'s shared modules: accounts, sessions, the log,
mail, keys, the blob store, and `lib/service.lua`'s route wrapper, account routes, and operator
checks. The two services can run in one calog process.

    cp dvx.conf.example dvx.conf           # and edit it
    calog dvxserver.lua -- dvx.conf        # in this folder

The server finds singemaster at `../../singemaster` and calog at `../../calog` from its own folder,
wherever calog is started; `singe=DIR` and `calog=DIR` arguments say where they are when they are
elsewhere.

## Both services in one process

    calog ../../singemaster/master.lua dvxserver.lua -- singe=../../singemaster master=master.conf dvx=dvx.conf

`calogArgs()` is one list for every script in a run, so each service takes its own configuration
by name (`master=`, `dvx=`). Each listens on its own `bindPort` with its own database, and the
shared modules are loaded once per service. `test/run.sh` runs both this way.

## How it is deployed

As singemaster is: plain HTTP bound to loopback, behind a reverse proxy that answers for the public
name. One difference matters: DVX's web library speaks plain HTTP only (there is no TLS on a DOS
machine), so the proxy must answer DVX's requests over plain HTTP as well as HTTPS. That costs no
trust, because no package is trusted for the way it arrived: DVX installs a package only when the
repository's signature of it checks against the repository key DVX was installed with.

A client that opens a connection and then stalls holds one serving context until it gives up, so
the proxy should time out slow requests (`client_header_timeout` and `client_body_timeout` in
nginx) and buffer responses, which also frees a context as soon as a slow download is handed to
the proxy.

## Trust

* **The repository key stays offline.** The service holds only the repository's public key
  (`repoKey`). The operator signs an approved package on their own computer
  (`tools/review.lua`), and the service checks that signature with the package tool before it
  publishes the version. A stolen server can withhold or withdraw packages, but it cannot make one
  DVX will install.
* **Every upload is checked by the package tool** (`dvxpkg check`, the same code DVX installs
  with): the manifest, every file against it, and the rules for where each kind of package may put
  its files. Nothing a manifest says is taken on the uploader's word.
* **Publishers may sign their own packages** with their own keys (`dvxpkg keygen`, or the Package
  Builder). The catalogue shows each version's publisher key, if it has one, so a user can tell two
  publishers apart.
* **An id belongs to the account that first uploaded it.** Another account's upload of the same id
  is refused, as is a second upload of a version already taken. Ids of the vendor `kpunch` are
  DVX's own and only the operator uploads them.

## The API

All JSON except the downloads. Every reply carries `X-Request-Id`, which is also in the log.

| Method | Path | What it does |
|---|---|---|
| GET  | `/v1/health` | Liveness and the schema version. |
| GET  | `/v1/catalogue` | Every published package at its newest approved version: id, version, name, kind, summary, publisher, author, licence, homepage, dvx, abi, size, sha256, signer, signed, and depends (`[{id, min}]`). No session. |
| GET  | `/v1/package/<id>` | One package's approved versions, newest first. |
| GET  | `/v1/download/<id>/<version>` | The bytes, with `X-Content-Sha256`. Resumable: `Range` is answered. |
| GET  | `/v1/signature/<id>/<version>` | The repository's signature of that version (a `NAME.SIG` file). |
| POST | `/v1/updates` | `{dvx, abi, installed: [{id, version}]}`, the client's own list: answers `updates` (the newer versions) and `unknown` (ids the catalogue does not have). Nothing of it is kept. |
| POST | `/v1/upload` | The `.DPK` as the body, from a signed-in account. Taken in as a pending version for review. DVX's Software sends it (File > Publish a Package) with the account its settings give. |
| GET  | `/v1/mine` | The account's own packages, every version with its review state and note. |
| GET  | `/v1/admin/pending` | Operator. Versions waiting for a review, oldest first. |
| GET  | `/v1/admin/download/<id>/<version>` | Operator. Any version's bytes, to review and sign. |
| POST | `/v1/admin/version/state` | Operator. `{package, version, state: approved or rejected, signature, note}`. Approving needs the repository's signature of the version. |
| POST | `/v1/admin/package/state` | Operator. `{package, state: published or withdrawn}`. |
| GET  | `/v1/admin/log` | Operator. The log, searched by any field (as singemaster's). |
| POST | `/v1/admin/key/grant`, `/v1/admin/key/revoke` | Operator. `{account, key}`; the key `admin` makes an administrator. |

The account routes (`/v1/account/register`, `verify`, `login`, `logout`, `recover`, `reset`, and
`GET /v1/account`) are singemaster's, with mail that names DVX.

The operator is the console token (`adminToken`) or the session of an account holding the `admin`
key. Withdrawing a package stops it being listed and downloaded; the copies installed keep working
and simply stop being offered updates.

## Serving contexts

A DOS machine downloads slowly. The main context listens and does the housekeeping (queued mail,
expired accounts); `serveContexts` serving contexts, each with its own database connection and
routes, accept on its listener, so one slow download holds up only the context sending it. Each
connection goes to one of them; with every one busy, new connections wait in the system's queue
until one is free.

## Reviewing uploads

`tools/review.lua` runs on the operator's own computer, the one place the repository's secret key
is kept. It needs singemaster's `lib/config.lua` (`singe=DIR` when singemaster is not beside this
repository) and the package tool from a DVX build or the SDK.

    calog tools/review.lua -- review.conf pending
    calog tools/review.lua -- review.conf fetch acme.cards 1.0.0
    calog tools/review.lua -- review.conf approve acme.cards 1.0.0 "looks fine"
    calog tools/review.lua -- review.conf reject acme.cards 1.1.0 "the help file is missing"

`review.conf` holds `server` (the service's public address), `token` (the console token or an
administrator's session), `dvxpkg`, `secret` (the repository's secret key), and `work` (where
fetched packages and their signatures are kept). `fetch` checks the download against the digest the
service sent and shows the manifest and files as the package tool reads them. `approve` signs the
fetched file itself, so what is approved is exactly what was reviewed; if that file has changed
since, the service refuses the signature. `withdraw` and `publish` take a package off the catalogue
and put it back.

## Publishing DVX itself

`make` in the DVX tree writes `dist/DVXUPD.DPK`: DVX itself as the package `kpunch.dvx` (kind
`system`), every file of the release but the optional fonts and DVX's optional parts, at this DVX's
version and ABI. Packages
of the vendor `kpunch` are the operator's, so it is uploaded with the console token and approved
like any other:

    curl -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/octet-stream" \
         --data-binary @dist/DVXUPD.DPK https://packages.example.com/v1/upload
    calog tools/review.lua -- review.conf approve kpunch.dvx 1.0.1

Software then lists it first on its Updates page. DVX installs it only when the repository's
signature checks against the key in its own `SYSTEM\KEYS`, and names the add-ons a new ABI would
leave unable to load before anything is installed. Setup installed DVX with its record listing every
file (the release build writes it into Setup's package), so the update also removes the files the
new version dropped.

The base set of fonts is a package too, `kpunch.basefont` ("DVX Fonts", `src/fonts/PACKAGE.INI`):
`make` writes it as `dist/BASEFONT.DPK`, at DVX's version. Setup installs and records it with DVX
unless told to install no fonts, so publishing a newer one updates every DVX that has the fonts,
and offers them to those that do not:

    curl -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/octet-stream" \
         --data-binary @dist/BASEFONT.DPK https://packages.example.com/v1/upload
    calog tools/review.lua -- review.conf approve kpunch.basefont 1.0.1

DVX's optional parts (`src/packages.ini`: DVX BASIC, Databases, Networking, Communications, Media,
Developer Tools, and Demos) are packages as well, one per module, each part a bundle that needs its
members. `make` writes them all to `dist/packages/` (`BASIC.DPK`, `DVXBASIC.DPK`, `BASDEV.DPK`, ...),
at DVX's version. Setup installs and records the parts chosen, so publishing newer ones updates
every DVX that has them, and offers the parts to those that do not; publish them with the DVX they
were built with:

    for dpk in dist/packages/*.DPK; do
        curl -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/octet-stream" \
             --data-binary @"$dpk" https://packages.example.com/v1/upload
    done
    calog tools/review.lua -- review.conf approve kpunch.basic 1.0.1    # and each of the others

The same build writes DVX's own add-ons to `dist/addons/` (`src/addons/README.md`): More Fonts and
the General MIDI SoundFont, kept out of the base install (the CD carries them too, signed when the
media are signed: with `DVX_SIGN_KEY=KEY.SEC`, else `~/.ssh/KPUNCH.SEC` when it is there). They are
uploaded and approved the same way:

    for dpk in dist/addons/*.DPK; do
        curl -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/octet-stream" \
             --data-binary @"$dpk" https://packages.example.com/v1/upload
    done
    calog tools/review.lua -- review.conf approve kpunch.morefont 1.0.0
    calog tools/review.lua -- review.conf approve kpunch.dvxgm 1.0.0

## Configuration

`dvx.conf.example` lists every key with a comment. Beyond singemaster's (`bindHost`, `bindPort`,
`publicOrigin`, `dataDir`, `database`, `pbkdf2Iterations`, `mailFrom`, the relay settings, and
`adminToken`), the server needs `dvxpkg` (the package tool), `repoKey` (the repository's public
key), `serveContexts`, and `uploadMax` (the largest upload in bytes).

## Tests

    test/run.sh

Needs `bin/host/dvxpkg` from a DVX build, calog, and singemaster beside this repository. It makes
its own keys and packages with the package tool, starts the service on a scratch database, and runs
two suites: `testPackages` (uploads checked and refused for the right reasons, review with the
repository's signature, the catalogue, resumed downloads that DVX would trust, updates, withdrawing,
an account's own packages, and a slow client not holding up the others) and `testReview` (the review
tool end to end, including a changed file whose signature is refused). Then it starts singemaster
and this service in one calog process and checks that both answer.

    test/serve.sh DATA PORT FIXTURES DVXPKG

The package server for DVX's DOS tests: the DOS test harness runs it for a script that says
`# REQUIRES: PKGSERVER`. It makes a test repository key, gives the test DVX that key, a record of
DVX itself, and a theme to update (installed with the package tool), uploads and approves the
fixture packages and the next version of DVX with `tools/review.lua`, and serves them on
`127.0.0.1:PORT`, which the emulator's guest reaches as `10.0.2.2`.