calog/API.md

# calog script API reference

Every function below is a **native** that calog registers into each engine, so a script in
any language can call it. This reference covers the natives the `calog` runner and its
libraries expose. (For the C embedding API -- `calogCreate`, `calogRegister`, `calogFnInvoke`,
etc. -- see the README and `src/calog.h`.)

## Conventions

- **Calling syntax per engine.** Most engines call a native by its bare name --
  `cryptoUuid()`, `kvSet("k", 1)`. Two differ:
  - **Wren**: `Calog.call("name", [args])`, e.g. `Calog.call("kvSet", ["k", 1])`.
  - **Scheme (s7)**: s-expression form, `(kvSet "k" 1)`.
- **Output & exit** (provided by the `calog` runner): `calogPrint(...)` writes to stdout;
  `calogExit([code])` tears everything down and exits. calog is event-driven, so a script
  must call `calogExit` (or be interrupted) to end -- a finished top level does not exit.
- **Values.** Arguments and results marshal through one canonical type: `nil`, `bool`, `int`,
  `real`, `string`, `list`, and `map` (keyed record). Strings are **binary-safe** (may contain
  embedded NULs) everywhere the underlying library allows it.
- **Signatures.** Every entry is written `name(param: type, ...) -> returnType`, using the
  value types above plus `handle` (an opaque int64 resource handle), `fn` (a function value),
  and `any` (a value of any type). `[x]` marks an optional argument and `...x` a variadic one.
  A signature with no `->` returns `nil`. A return type given as `T | nil` may be `nil` (e.g.
  a lookup miss or end of stream).
- **Handles** (db connections, sockets, ssh sessions, tasks) are opaque int64 values from a
  process-wide table, so a handle can be **passed to another context and used there** -- e.g.
  lent to a function in another script via `calogCall`, which uses it and returns while the
  caller is blocked (a safe synchronous hand-off). What is *not* safe is using one handle from
  two contexts at the same instant, or closing it while another context is mid-operation: at
  most one context may touch a given handle at any moment, and exactly one closes it. (Tasks are
  the exception -- a task is owner-scoped: only its spawner may `taskEval`/`taskClose` it.)
- **Callbacks.** Natives that take a `fn` argument (`psSubscribe`, `timerAfter`, `timerEvery`,
  `calogExport`) accept a first-class function value in every engine. A function value handed
  *to* a script (say, returned by `calogCall`) is invoked directly -- except in my-basic, which
  calls it with `calogInvoke(fn, ...args)`.
- **Binary data in my-basic.** my-basic strings are text (NUL-terminated), so a calog string that
  carries an embedded NUL arrives as a distinct **byte-buffer** value instead (a NUL-free string stays an
  ordinary string, unchanged). Byte buffers are length-carrying and binary-safe; work with them via
  `byteLen(b) -> int`, `byteAt(b, i: int) -> int`, `byteSlice(b, start: int, count: int) -> bytes`,
  `byteConcat(...parts) -> bytes` (each part a byte buffer or a string), `strToByte(s: string) -> bytes`,
  and `byteToStr(b) -> string`. `+` concatenates byte buffers (and `string + bytes`), and `=`/`<>` compare
  them by content. A byte buffer egresses back to a native as a full-length binary string. Every other
  engine's strings are already binary-safe, so this applies to my-basic only.
- **Sandboxing (my-basic).** A my-basic context honors the same per-context limits as Lua/JS -- the native
  allow-list, a wall-clock time budget, and a memory cap (an over-budget or runaway script is retired).
  `INPUT` never reads host stdin: it yields an empty line, so a script takes input through natives like
  every other engine.
- **Availability.** Every library in this reference is compiled into `bin/calog`: crypto,
  json, kv, fs, time, timer, export, pubsub, task, net (TCP / UDP / ENet), db (SQLite /
  PostgreSQL / MySQL), http, and ssh. ssh needs a reachable server; http needs a reachable
  endpoint.

---

## Runner (`calog` binary)

| Function | Description |
|---|---|
| `calogPrint(...values: any)` | Write each argument to stdout, space-separated, with a trailing newline. |
| `calogExit([code: int])` | Tear down the runtime and exit the process with `code` (default `0`). Does not return. |

## archive

Compression and archives over libarchive. All data is binary-safe. One-shot codecs work whole-buffer
in memory; archive read/write use an opaque integer handle. A per-transfer 64 MiB cap guards against a
decompression bomb.

| Function | Description |
| --- | --- |
| `compress(data: string, filter: string [, level: int]) -> string` | Compress `data`. `filter`: `"gzip"`, `"bzip2"`, `"xz"`, `"zstd"`, `"lz4"`, `"compress"`, `"none"`. |
| `decompress(data: string [, filter: string]) -> string` | Decompress; the filter is auto-detected from the magic bytes if omitted. |
| `archiveReadOpen(bytes: string) -> handle` | Open an archive held in memory. |
| `archiveReadOpenFile(path: string) -> handle` | Open an archive streamed from a file. |
| `archiveReadNext(handle) -> map \| nil` | Advance one entry: `{ name, size, mode, mtime, type[, linkname] }`; `nil` at the end. |
| `archiveReadData(handle) -> string` | The current entry's bytes (whole, capped). |
| `archiveReadClose(handle)` | Close and free the reader. |
| `archiveWriteOpen(format: string, filter: string [, level: int]) -> handle` | Start an in-memory archive. `format` is a container name (see Container formats below); `filter` compresses the whole archive, as for `compress`. |
| `archiveWriteEntry(handle, name: string, data: string [, opts: map])` | Add one entry. `opts`: `{ mode: int, mtime: int, type: "file"\|"dir" }`. |
| `archiveWriteFinish(handle) -> string` | Close and return the whole archive; frees the handle. |

**Container formats.** `archiveReadOpen`/`archiveReadOpenFile` auto-detect the format from the byte
stream, so extracting takes no format argument. `archiveWriteOpen` takes one of the names below.
Compression is orthogonal to the container: a `.tar.zst`, for example, is `format` `"tar"` with
`filter` `"zstd"`.

| Format | Read | Write | Notes |
| --- | --- | --- | --- |
| `tar` / `ustar` / `pax` / `gnutar` / `v7tar` | yes | yes | POSIX and GNU tar variants. |
| `cpio` (aliases `newc`, `odc`) | yes | yes | |
| `zip` | yes | yes | |
| `7zip` | yes | yes | |
| `iso9660` | yes | yes | ISO CD/DVD filesystem image. |
| `ar` (alias `arbsd`) | yes | yes | Unix `.a` archive. |
| `mtree` | yes | yes | File-hierarchy listing: entry metadata, no file contents. |
| `warc` | yes | yes | Web archive. |
| `xar` | yes | yes | Extensible archive (Apple `.pkg`); needs vendored libxml2 + OpenSSL. |
| `shar` | no | yes | Self-extracting shell archive. |
| `cab` | yes | no | Microsoft Cabinet; extract only. |
| `lha` / `lzh` | yes | no | LHA; extract only. |
| `rar` / `rar5` | yes | no | RAR is proprietary -- libarchive can extract but not create it. |

Formats marked write `no` are decode-only: the encoder is proprietary or absent. `xar` read and
write are backed by the vendored libxml2 (its TOC is XML) plus OpenSSL (its entry checksums are
MD5/SHA1); both are already in the build. A single unnamed stream can be written with `format`
`"raw"`, but for one-shot compression prefer `compress`/`decompress`.

## crypto

Binary-safe cryptographic primitives over OpenSSL.

| Function | Description |
|---|---|
| `cryptoHashSha256(data: string) -> string` | SHA-256 as a 64-char lowercase hex digest. |
| `cryptoHashSha1(data: string) -> string` | SHA-1 as a 40-char lowercase hex digest. |
| `cryptoHmacSha256(key: string, data: string) -> string` | HMAC-SHA-256 as 64-char lowercase hex. |
| `cryptoRandomBytes(count: int) -> string` | `count` cryptographically-random bytes. |
| `cryptoBase64Encode(data: string) -> string` | Base64-encode. |
| `cryptoBase64Decode(text: string) -> string` | Base64-decode (trailing whitespace tolerated). |
| `cryptoHexEncode(data: string) -> string` | Lowercase hex encode. |
| `cryptoHexDecode(hexText: string) -> string` | Hex decode (case-insensitive). |
| `cryptoUuid() -> string` | A random RFC 4122 version-4 UUID string. |

## csv

Parse and format RFC 4180 CSV. Cells are strings; quoted cells may contain the delimiter, newlines,
and doubled quotes. Binary-safe.

| Function | Description |
| --- | --- |
| `csvParse(text: string [, delimiter: string]) -> list(list(string))` | Parse CSV into rows of string cells. The optional delimiter is one byte (default `,`). |
| `csvFormat(rows: list(list) [, delimiter: string]) -> string` | Format rows of cells (string/int/real/bool/nil) into CSV (records joined by CRLF); cells with the delimiter, a quote, or a newline are quoted. |

## db

SQL over SQLite, PostgreSQL, and MySQL/MariaDB. Parameters are always **bound** (never
string-spliced), so queries are injection-safe. Values marshal as: NULL <-> nil, integers/
reals as-is, text and BLOBs as binary-safe strings.

| Function | Description |
|---|---|
| `dbOpen(driver: string, conn: string) -> handle` | Open a connection. `driver` is `"sqlite"`, `"postgres"`, or `"mysql"`. `conn` is a SQLite path or `":memory:"`, a libpq conninfo, or a MySQL `key=value ...` string. |
| `dbExec(handle: handle, sql: string, ...params: any) -> int` | Run a non-query statement with bound params; returns rows affected. |
| `dbQuery(handle: handle, sql: string, ...params: any) -> list` | Run a query; returns a list of `{column: value}` row maps. |
| `dbClose(handle: handle)` | Close the connection. |

## export

Share a function by name across contexts and engines.

| Function | Description |
|---|---|
| `calogExport(name: string, fn: fn)` | Publish function `fn` under a global `name`. |
| `calogUnexport(name: string)` | Remove an exported name. |
| `calogCall(name: string, ...args: any) -> any` | Call an exported function by name -- works in **every** engine. (An export is also reachable by its bare name -- `exportedFn(args)` -- on Lua/JS/Squirrel/s7, on Ruby at top level (via `method_missing`), on Tcl (via its `unknown` handler), and on my-basic case-insensitively (it uppercases identifiers); Wren, Berry, and Janet always need `calogCall`.) |

## fs

POSIX filesystem access. A failed operation raises a catchable script error carrying
`strerror(errno)`.

| Function | Description |
|---|---|
| `fsRead(path: string) -> string` | Read a whole file (binary-safe). |
| `fsWrite(path: string, data: string)` | Create/truncate and write `data`. |
| `fsAppend(path: string, data: string)` | Create if absent, append at the end. |
| `fsExists(path: string) -> bool` | Whether the path exists. |
| `fsRemove(path: string)` | Unlink a file. |
| `fsMkdir(path: string)` | Create one directory level (existing dir is OK). |
| `fsList(path: string) -> list` | Entry name strings, excluding `.` and `..`. |
| `fsStat(path: string) -> map \| nil` | `{size, isDir, isFile, mtime}`, or nil if the path is absent. |

## http

Minimal HTTP/1.1 client over `http://` and `https://`. Each call is its own connection
(`Connection: close`). 3xx redirects are followed (up to 16, to break loops). `https://`
verifies the server certificate against the host's native trust store by default: the Windows
system certificate stores, the macOS keychain (on a build made with the Apple SDK), or the
well-known CA-bundle files on Linux/BSD. Overrides, highest priority first: `CALOG_CA_BUNDLE`
(authoritative -- trust exactly that PEM bundle, so it can pin to a private CA), then the
OpenSSL-standard `SSL_CERT_FILE` / `SSL_CERT_DIR`. If no trust anchors can be found, a verified
request fails with a clear error rather than a misleading per-certificate failure.

| Function | Description |
|---|---|
| `httpGet(url: string) -> map` | GET a URL, following redirects. Returns `{status, body, headers}` (headers keyed by lowercased name). |
| `httpRequest(opts: map) -> map` | `opts` is `{method (default "GET"), url, headers (map), body, insecure (bool), maxRedirects (int, default 16; 0 = don't follow)}`. `insecure=true` skips TLS verification. Returns `{status, body, headers}`. |

## httpd (a script, not a native library)

The HTTP/1.1 + WebSocket server lives entirely in **`examples/httpd.lua`** -- there is no C httpd. It
is built on the generic `tcp*` transport above (with the `tls` option for HTTPS/WSS) plus the crypto
natives (`cryptoHashSha1` + `cryptoBase64Encode` for the WebSocket handshake), which is the calog
thesis: systems primitives in C, protocol in a script. It runs as-is on any engine with binary-safe
strings (all but my-basic, whose strings are text); a my-basic port would use the byte-buffer type and
its `byte*` helpers for the masked/binary frames instead of string operations.

```lua
local httpd = dofile("examples/httpd.lua")   -- or paste/require the module
local s = httpd.new()
s:route("GET", "/hi", function(req) return "hello " .. req.path end)          -- string body => 200
s:route("GET", "/made", function(req) return { status = 201, body = "x" } end) -- map => custom status
s:websocket("/ws", function(msg) return "echo: " .. msg.message end)          -- reply string, or nil
s:serve(8080)                                    -- opts: { tls, cert, key, keep = fn, acceptTimeout }
```

A route handler receives `{ method, path, query, headers (lowercased), body }` and returns a map
`{ status, headers, body }`, a bare string (=> 200), or nil (=> 204). The server does HTTP/1.1
keep-alive; re-registering a path hot-swaps its handler. A WebSocket handler receives
`{ path, message }` per inbound text/binary frame and returns a text reply or nil (request/reply
model; no server-push). Concurrency follows the actor model: one context per accept loop (serial),
or share the listener handle across several contexts for parallelism.

## json

| Function | Description |
|---|---|
| `jsonParse(text: string) -> any` | Parse JSON: object -> map, array -> list, number -> int or real, string, true/false, null -> nil. |
| `jsonStringify(value: any) -> string` | Serialize a value to compact JSON text. |

## kv

A process-wide, thread-safe store shared by every context and engine. Holds **data only** (a
function value is rejected). Keys are binary-safe.

| Function | Description |
|---|---|
| `kvSet(key: string, value: any)` | Store a deep copy of `value` under `key` (replaces any existing). |
| `kvGet(key: string) -> any \| nil` | A deep copy of the stored value, or nil if absent. |
| `kvHas(key: string) -> bool` | Whether the key is present. |
| `kvDelete(key: string)` | Remove the key (no error if absent). |
| `kvKeys() -> list` | A list of the stored keys (strings). |

## net

Three first-class transports -- **TCP**, **UDP**, and **ENet** (reliable/ordered delivery
over UDP) -- all always available. Payloads are binary-safe strings. Blocking calls
(`tcpAccept`/`tcpRecv`/`udpRecvFrom`/`enetService`) stall only the calling context's thread.

TCP and UDP:

| Function | Description |
|---|---|
| `tcpConnect(host: string, port: int) -> handle` | Connect to a TCP server. |
| `tcpListen(port: int [, opts: map]) -> handle` | Listen on a TCP port. `opts`: `{ tls (bool), cert (PEM path), key (PEM path) }` -- with `tls`, every accepted connection is a TLS server session (used by the HTTPS/WSS server script). |
| `tcpAccept(handle: handle [, timeoutMs: int]) -> handle \| nil` | Block for a client (completing the TLS handshake for a TLS listener); returns a connection handle. With `timeoutMs`, returns nil if none arrives in time, so an accept loop can re-check its own stop condition. |
| `tcpSend(handle: handle, data: string) -> int` | Send all of `data`; returns bytes sent. |
| `tcpRecv(handle: handle, maxBytes: int) -> string \| nil` | Read up to `maxBytes`; nil at end of stream. |
| `tcpClose(handle: handle)` | Close a socket. |
| `udpOpen(port: int) -> handle` | Open a UDP socket (`port` 0 = ephemeral). |
| `udpSendTo(handle: handle, host: string, port: int, data: string) -> int` | Send a datagram; returns bytes sent. |
| `udpRecvFrom(handle: handle, maxBytes: int) -> map` | Receive one datagram: `{data, host, port}`. |
| `udpClose(handle: handle)` | Close a UDP socket. |

ENet (reliable UDP -- ordered, reliable channels over UDP):

| Function | Description |
|---|---|
| `enetHost(port: int, maxPeers: int) -> handle` | Create an ENet host. |
| `enetConnect(hostHandle: handle, host: string, port: int, channels: int) -> handle` | Initiate a connection to a peer; returns a peer handle. |
| `enetService(hostHandle: handle, timeoutMs: int) -> map` | Poll for one event within `timeoutMs`. Returns `{type, ...}` where `type` is `"none"`, `"connect"`, `"receive"` (with `peer`, `channel`, `data`), or `"disconnect"`. |
| `enetSend(peerHandle: handle, channel: int, data: string, reliable: bool)` | Queue a packet on a channel. |
| `enetDisconnect(peerHandle: handle)` | Begin disconnecting a peer. |
| `enetClose(hostHandle: handle)` | Destroy an ENet host. |

## proc

Run a subprocess and capture its output (POSIX only; started with `posix_spawn`, not `fork`).

| Function | Description |
| --- | --- |
| `procRun(argv: list(string) [, opts: map]) -> map` | Spawn `argv[0]`, wait, and return `{ exit: int, stdout: string, stderr: string }` (on POSIX, exit is the negative signal number if the child was killed; on Windows it is the process exit code). `opts`: `{ stdin: string, cwd: string, env: map (string->string; replaces the environment) }`. Blocks the caller's context thread; stdin is fed while stdout/stderr are drained, so a large transfer cannot deadlock. POSIX spawns with `posix_spawn`; Windows spawns with `CreateProcess`. |

## pubsub

Deliver a message to every subscriber of a topic, across contexts and engines. Delivery is
synchronous; each subscriber runs on its own context's thread and gets a deep copy of the
message. Keep publish graphs acyclic.

| Function | Description |
|---|---|
| `psSubscribe(topic: string, fn: fn) -> int` | Register `fn` to receive messages published on `topic`; returns a subscription id. |
| `psUnsubscribe(id: int)` | Drop the subscription with that id. |
| `psPublish(topic: string, msg: any) -> int` | Deliver a copy of `msg` to every subscriber; returns how many were invoked. |

## regex

Perl-compatible regular expressions over PCRE2. `flags` is a string of single letters: `i`
case-insensitive, `m` multiline, `s` dot-matches-newline, `x` extended, `g` replace-all (for
`regexReplace`). A non-participating capture group is `nil`. Binary-safe.

| Function | Description |
| --- | --- |
| `regexMatch(pattern, subject [, flags]) -> nil \| list` | First match, as `[fullMatch, group1, group2, ...]`; `nil` if no match. |
| `regexSearch(pattern, subject [, flags]) -> nil \| map` | First match, as `{ match, start, end, groups: list }` (byte offsets); `nil` if no match. |
| `regexReplace(pattern, subject, replacement [, flags]) -> string` | Replace matches; `replacement` uses `$1` / `${name}`. The `g` flag replaces all. |
| `regexSplit(pattern, subject [, flags]) -> list(string)` | Split `subject` on matches of `pattern`. |

## ssh

SSH/SFTP over libssh2. Requires a reachable SSH server. Payloads are binary-safe. A connection
handle can be passed to and used by another context (see Handles above), but a libssh2 session
is **not thread-safe**: never operate on one handle from two contexts at once -- calog does not
serialize it for you.

| Function | Description |
|---|---|
| `sshConnect(host: string[, port: int]) -> handle` | Connect (port defaults to 22). |
| `sshAuthPassword(handle: handle, user: string, password: string) -> bool` | Password authentication. |
| `sshAuthKey(handle: handle, user: string, privateKeyPath: string[, publicKeyPath: string, passphrase: string]) -> bool` | Public-key authentication. |
| `sshExec(handle: handle, command: string) -> map` | Run a remote command: `{stdout, stderr, exitCode}`. |
| `sshClose(handle: handle)` | Close the session. |
| `sftpGet(handle: handle, remotePath: string) -> string` | Read a remote file whole (binary-safe). |
| `sftpPut(handle: handle, remotePath: string, data: string)` | Create/truncate a remote file (mode 0644). |
| `sftpList(handle: handle, path: string) -> list` | `[{name, size, isDir}, ...]`. |
| `sftpStat(handle: handle, path: string) -> map \| nil` | `{size, isDir}`, or nil if the path is missing. |
| `sftpRemove(handle: handle, path: string)` | Remove a remote file. |
| `sftpMkdir(handle: handle, path: string)` | Create a remote directory (mode 0755). |

## task

Launch and manage other calog script contexts. Tasks are fire-and-forget: a spawned context
runs on its own thread; results come back through host natives or the shared kv/pubsub. A
task is owned by the context that spawned it, and only that owner may `taskEval`/`taskClose`.
`taskClose` is cooperative -- a task busy in a pure loop must poll `taskActive()` and break
out, or the close blocks until the task next returns to the runtime.

| Function | Description |
|---|---|
| `taskSpawn(engine: string, code: string) -> handle` | Run a code string on a named engine (`"lua"`, `"js"`, `"squirrel"`, `"mybasic"`, `"berry"`, `"s7"`, `"wren"`, `"mruby"`, `"tcl"`, `"janet"`). |
| `taskLoad(baseName: string) -> handle` | Launch a script *file* (engine chosen by extension). |
| `taskEval(handle: handle, code: string)` | Feed more code into a running task (runs on its thread). |
| `taskClose(handle: handle)` | Ask a task to stop (cooperative: sets its shutdown flag, then waits for its thread to exit). |
| `taskActive() -> bool` | Called *inside a task*: `false` once its owner has asked it to stop. A long loop polls this -- `while taskActive() do ... end` -- so `taskClose` need not block. |
| `taskActive(handle: handle) -> bool` | Called *in the owner*: whether the task spawned under `handle` is still open. |
| `taskExit()` | Called *inside a task*: retire this task's **own** context. Deferred -- the current code finishes normally, then the context stops and the runtime reaps it (no owner action needed). |
| `taskSelf() -> int` | The calling script's own context id. |
| `taskCount() -> int` | Number of tasks this library currently holds open. |

## time

| Function | Description |
|---|---|
| `timeNow() -> real` | Wall-clock epoch seconds, fractional (CLOCK_REALTIME). |
| `timeMonotonic() -> real` | Seconds from an unspecified origin (CLOCK_MONOTONIC); use for intervals. |
| `timeSleep(ms: int)` | Block the calling context for `ms` milliseconds. |

## timer

One background thread drives every timer; each callback runs on the context that created the
timer.

| Function | Description |
|---|---|
| `timerAfter(ms: int, fn: fn) -> int` | Fire `fn` once, `ms` milliseconds from now; returns a timer id. |
| `timerEvery(ms: int, fn: fn) -> int` | Fire `fn` every `ms` milliseconds; returns a timer id. |
| `timerCancel(id: int)` | Stop a pending or repeating timer. |

## xml

Parse and produce XML over vendored libxml2. An element node is a map `{ tag, attr, children }`:
`tag` is the element name; `attr` (present only when the element has attributes) maps attribute names
to string values; and `children` is an ordered list whose items are child element maps or plain
strings (text nodes). Mixed content and all character data are preserved, so `xmlParse` then
`xmlStringify` round-trips the element structure; comments, processing instructions, and the XML
declaration are dropped. Parsing loads no external entities and does no network I/O (no XXE), and
nesting is bounded by the same maximum depth as `json`. Marshalling is binary-safe (UTF-8).

| Function | Description |
|---|---|
| `xmlParse(text: string) -> map` | Parse an XML document; returns its root element node. Malformed XML raises a catchable error. |
| `xmlStringify(node: map) -> string` | Serialize an element node (the shape `xmlParse` returns) back to XML text. |