AI_Slop/calog
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Several issues fixed that were found while dogfooding.

Browse source
This commit is contained in:
Scott Duensing 2026-08-04 18:50:26 -05:00
parent f02f62a61d
commit 1f6eb5cfe8
30 changed files with 1334 additions and 174 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
API.md
View file

@ -144,6 +144,7 @@ of them registers a native for it -- which is the point, because then it is poli
| Function | Description |
|---|---|
| `calogPrint(...values: any)` | Write each argument to stdout, space-separated, with a trailing newline. |
| `calogArgs() -> list[string]` | The arguments given after `--` on the command line (`calog build.lua -- release v2`), the same list for every script in the run, empty when none were given. This is the sanctioned way to parameterize a run: the engines' own `getenv`/`argv` are removed, so without it every project would invent its own convention. |
| `calogExit([code: int])` | Tear down the runtime and exit the process with `code` (default `0`). **Does not return** -- it unwinds the calling script at the call site, so nothing after it runs, and stops every other live script at its next native call, or at its interpreter's next hook if it calls none. The first caller's `code` is the one reported; a script that catches the unwind still cannot call another native. |
## archive
@ -155,7 +156,7 @@ decompression bomb.
| Function | Description |
| --- | --- |
| `compress(data: string, filter: string [, level: int]) -> string` | Compress `data`. `filter`: `"gzip"`, `"bzip2"`, `"xz"`, `"zstd"`, `"lz4"`, `"compress"`, `"none"`. |
| `decompress(data: string [, filter: string]) -> string` | Decompress; the filter is auto-detected from the magic bytes if omitted. |
| `decompress(data: string [, filter: string]) -> string` | Decompress; the filter is auto-detected from the magic bytes if omitted. Naming one ENFORCES it: data in any other codec (or none) is rejected rather than decoded, so `decompress(blob, "gzip")` is a usable codec restriction for untrusted input. Names are the same as `compress` accepts. |
| `archiveReadOpen(bytes: string) -> handle` | Open an archive held in memory. |
| `archiveReadOpenFile(path: string) -> handle` | Open an archive streamed from a file. |
| `archiveReadNext(handle) -> map \| nil` | Advance one entry: `{ name, size, mode, mtime, type[, linkname] }`; `nil` at the end. |
@ -282,7 +283,7 @@ strings (all but my-basic, whose strings are text); a my-basic port would use th
its `byte*` helpers for the masked/binary frames instead of string operations.
```lua
local httpd = dofile("examples/httpd.lua") -- or paste/require the module
local httpd = load(fsRead("examples/httpd.lua"))() -- fsRead + load: dofile/require are not available
local s = httpd.new()
s:route("GET", "/hi", function(req) return "hello " .. req.path end) -- string body => 200
s:route("GET", "/made", function(req) return { status = 201, body = "x" } end) -- map => custom status
@ -380,7 +381,7 @@ case-insensitive, `m` multiline, `s` dot-matches-newline, `x` extended, `g` repl
| `regexMatch(pattern, subject [, flags]) -> nil \| list` | First match, as `[fullMatch, group1, group2, ...]`; `nil` if no match. |
| `regexSearch(pattern, subject [, flags]) -> nil \| map` | First match, as `{ match, start, end, groups: list }` (byte offsets); `nil` if no match. |
| `regexReplace(pattern, subject, replacement [, flags]) -> string` | Replace matches; `replacement` uses `$1` / `${name}`. The `g` flag replaces all. |
| `regexSplit(pattern, subject [, flags]) -> list(string)` | Split `subject` on matches of `pattern`. |
| `regexSplit(pattern, subject [, flags]) -> list(string)` | Split `subject` on matches of `pattern`. An EMPTY match is not a separator -- the scan steps over it and the byte stays in the current piece -- so `regexSplit('x*', 'abc')` is `['abc']` while `regexSplit('[0-9]*', 'a1b2c')` is `['a','b','c']`. |
## ssh

321
AUDIT.md
View file

@ -2,9 +2,13 @@
Original audit: 15 scoped reviewers + 1 adversarial verifier per finding (118 raw, 4 refuted, 114 confirmed).
A **follow-on audit** is appended at the end of this file: 23 findings, IDs prefixed `S`, from the
2026-07-24/27 session. Different provenance (one reported defect and what pulling on it uncovered),
same format. It is a separate body of work -- the counts in this section cover the original 114 only.
A **follow-on audit** is appended at the end of this file: 36 findings, IDs prefixed `S`, from the
2026-07-24 to 2026-08-04 session. Same format, different provenance, and it arrived in four passes:
one reported defect and everything pulling on it uncovered (S1-S17); three open items re-examined
after being written off as accepted limitations (S18-S20); a six-lens review sweep with adversarial
verification (S21-S23); and the leads that sweep raised but never verified, plus the items its
verification confirmed (S24-S36). It is a separate body of work -- the counts in this section cover
the original 114 only.
## Remediation status
@ -1398,7 +1402,7 @@ s->ast is an _ls_create() sentinel list whose `prev` always points at the tail:
---
# Follow-on audit -- 2026-07-24/27 session (23 findings)
# Follow-on audit -- 2026-07-24 to 2026-08-04 session (36 findings)
Provenance differs from the audit above, and it matters when reading these. They were not produced by
a scoped reviewer sweep. They came out of **one reported defect** -- `calogExit` did not behave as
@ -1418,13 +1422,13 @@ inherits, leaving `calogEnd` a third verb for a narrow slice. See design.md sec
## Remediation status
**22 FIXED, 1 KEPT.** Verified: `make test` **959 checks / 0 failed** (with the new zig-free
**35 FIXED, 1 KEPT.** Verified: `make test` **971 checks / 0 failed** (with the new zig-free
`cross-lint` running inside it); ThreadSanitizer clean on the actor core, the libraries, the JS path
and my-basic; `tools/crossBuild.sh` **39 ok / 0 failed**, all ten engines built for four targets and
**run** statically on musl (151 checks across 12 binaries); Windows PE and macOS x86_64/arm64 full
CLIs build; every shipped example runs.
New regression tests: `tests/testExit.c` (66 checks, all ten engines), `tests/testTeardown.c` (29
New regression tests: `tests/testExit.c` (76 checks, all ten engines), `tests/testTeardown.c` (34
checks, JavaScript on purpose -- QuickJS is the only VM that reports a stranded handle), plus cases
added to `tests/testHooks.c` and `tests/testTask.c`.
@ -2012,3 +2016,308 @@ time -- it costs a handful of -O2 objects and a link, and OpenSSL was already re
`bin/testNet`.
---
## Fourth pass -- clearing the open list (13)
Everything left open after the third pass, plus the seven leads the earlier audit had reported but
never verified (each re-verified independently and adversarially before being touched; all seven were
real, every one demonstrated by running something rather than by reading).
---
### S24. [bug] libs/calogTimer.c:387, libs/calogPubsub.c:146 --- FIXED
**Status: FIXED**
**A timer or pubsub callback that FAILED was discarded entirely: no stderr, no error handler, no exit code -- it simply stopped working.**
Both invoke a script callable from a thread with no caller to return a status to, and both inspected
that status only for `calogErrDeadE` (to auto-cancel a dead timer). Every other failure was dropped
on the floor. `psPublish` returns a delivery COUNT, not a status, so a broken subscriber was invisible
to everyone: the publisher saw a number, the handler's author saw nothing. Fixed by exposing the
error-posting path the actor layer already used for a failed eval (`calogPostError`) and calling it
from both, which routes the failure to the runtime's error handler on the host thread exactly like any
other script error.
*Verifier:* A script arming a failing timer callback now reports
`calog: script error: ... timer callback blew up` and exits 1; a failing subscriber likewise. Both
produced nothing at all before.
---
### S25. [bug] src/calogMain.c:279, src/context.c (contextDrainQueue) --- FIXED
**Status: FIXED**
**A failure could not set the exit code if the runner had not launched the context itself, and even when it could, whether it won was a race.**
Three defects in one path. (1) `onError` only flagged contexts in `gLaunched`, so a task a script
spawned could fail, print, and leave the run reporting success. (2) The exit code was
first-writer-wins, so a task that died just before its parent's `calogExit(0)` had its failure
recorded second and thrown away -- whether the run reported the failure depended on which thread got
there first. (3) An error message a dying context posted during teardown was freed undelivered by
`contextDrainQueue`, losing both the diagnostic and the code.
Fixed respectively by: claiming the exit code for a context the runner did not launch (without
closing it -- it belongs to whoever spawned it); replacing first-writer-wins with **first non-zero
wins, and zero stores nothing**, so ordering can no longer decide whether a failure is reported; and
dispatching a pending error at drain instead of freeing it, which is safe because every context
thread is joined by then. An aborted script is not reported as an error at all (S19), so what reaches
that drain is a genuine failure.
*Verifier:* `taskSpawn` of a failing script followed by the parent's `calogExit()` reported **0 in
7 of 8 runs** before; **1 in 13 of 13** after, deterministically. Explicit `calogExit(3)` still
reports 3, and a clean run still reports 0.
---
### S26. [bug] libs/calogRegex.c:368 --- FIXED
**Status: FIXED**
**`regexSplit` silently discarded subject bytes whenever the pattern could match empty.**
One variable served as both the search position and the start of the piece being accumulated. On an
empty match the code advanced it by one byte to guarantee progress -- which also moved the piece
start, dropping that byte from the output entirely. Fixed by splitting the two roles: `searchFrom`
advances, `pieceStart` does not.
*Verifier:* Demonstrated before the fix -- `regexSplit('x*', 'abc')` returned `['']` (the whole
subject gone), `regexSplit('[0-9]*', 'a1b2c')` returned `['', '', '']`. After: `['abc']` and
`['a','b','c']`. Three cases added to `tests/testUtil.c`, and the documented rule (an empty match is
not a separator) is now stated in API.md.
---
### S27. [bug] libs/calogDb.c:749 --- FIXED
**Status: FIXED**
**`mysqlRowValue` wrote its NUL terminator at the connector's UNTRUNCATED length, one byte past a heap allocation.**
Column buffers are sized from the field's `max_length`, but the connector reports the value's full
length even when it only copied `buffer_length` bytes -- and `max_length` under-reports for some
types (a zerofill `BIGINT` caps at its 19-digit display width but renders 20). The terminator then
landed outside the allocation, and `mysqlColumn` read past it as well. Fixed by clamping to the
capacity actually allocated, taken from calog's own `binds` array: the connector mutates its private
copy's `buffer_length` during conversion but never writes back to ours.
*Verifier:* AddressSanitizer `heap-buffer-overflow ... WRITE of size 1` at `mysqlRowValue`, reproduced
against the shipped `bin/calog` through an ordinary Lua `dbQuery`, using a purpose-started local
MariaDB and a `bigint(19) unsigned zerofill` column. Reported as high, corrected to medium on the
grounds that it needs a specific column type -- though the verifier also noted a hostile or
MITM'd server can push the overflow much further via the TIME codec, and calog's default `sslmode`
does not verify certificates.
---
### S28. [bug] libs/calogProc.c (env block, pipe guard) --- FIXED
**Status: FIXED**
**`procRun` handed the child a truncated environment and leaked the strings past the hole; separately, a failed `pipe()` leaked the descriptors already opened.**
The environment vector was written at the pair's index, so any pair the loop skipped (a non-string
key or value, a failed malloc) left a NULL in the middle. `execve` stops at the first NULL, so one
skipped entry silently truncated -- often emptied -- the child's environment, and both cleanup loops
stopped at the same hole and leaked the rest. Fixed by writing through a compacted index. The pipe
guard was `pipe(a) != 0 || pipe(b) != 0 || pipe(c) != 0`: a short-circuit that abandoned whatever the
earlier calls had opened, so a process near its fd limit lost two descriptors per call until nothing
in it could open anything again. Fixed by staging the three calls, each failure closing its
predecessors, with the duplicated teardown factored into one helper.
*Verifier:* Both demonstrated on the real binary -- the environment case with a numeric value in the
map (child environment empty, LeakSanitizer reporting the four leaked strings), the pipe case with an
LD_PRELOAD interposer forcing the second `pipe()` to fail. After the fix the child receives all four
valid variables and nothing leaks.
---
### S29. [bug] libs/calogXml.c:435 --- FIXED
**Status: FIXED**
**Every failed parse of XML containing an internal entity leaked libxml2's entity table.**
An internal entity declaration makes libxml2 build a hidden "SAX compatibility mode" document to hold
the entity table. libxml2 frees it itself only when the parse finishes cleanly; a fatal error halts
the parser first, and `xmlFreeParserCtxt` never owns it. Fixed by reclaiming `ctxt->myDoc` explicitly.
*Verifier:* 847 bytes leaked per failed parse, perfectly linear (500 parses -> 423,500 bytes), and
~876 KB for a single document declaring 2000 entities. LeakSanitizer-clean after the fix across 200
iterations.
---
### S30. [gap] Makefile (cross-lint) --- FIXED
**Status: FIXED**
**The lint added to prevent an ABI-define divergence did not check the file the divergence came from.**
`cross-lint` verified that every ABI define reached `tools/crossWinFull.sh` and
`tools/crossMacFull.sh`, but `tools/crossDeps.sh` -- which builds the cross `libmruby.a`, and is where
the original `MRB_USE_DEBUG_HOOK` mismatch (S11) actually occurred -- was never inspected. Fixed with
a separately scoped check, because folding it into the existing loops would demand that a script
compiling no calog sources contain `_SQ64` and every `CALOG_WITH_*` selector, flooding the lint with
false failures.
*Verifier:* Teeth confirmed both ways: with `MRB_USE_DEBUG_HOOK` stripped from a copy of
`tools/crossDeps.sh` the lint now fails and names the file; restored, it passes.
---
### S31. [bug] libs/calogArchive.c:278 --- FIXED
**Status: FIXED**
**`decompress` accepted a `filter` argument, never type-checked it, and ignored it completely.**
The reader always sniffs every codec libarchive supports, so naming one changed nothing: a script
pinning `"gzip"` to reject other codecs on untrusted input silently accepted zstd, xz, bzip2 and plain
uncompressed bytes alike, and `decompress(data, 12345)` was accepted without complaint. Fixed by
enforcing it -- the filter actually used is compared against the caller's name after the header is
read, using libarchive's own names, which are exactly the ones `compress` accepts. Enforcing was
chosen over deleting the argument because it makes a documented feature true rather than removing a
capability scripts have reason to want.
*Verifier:* Before: `decompress(gzipData, 'zstd')` returned the data. After: mismatched filter, bogus
name, non-string filter, and plain data named as gzip are all rejected, while the correct name and the
no-argument auto-detect still work. Four cases added to `tests/testArchive.c`.
---
### S32. [bug] 13 library Register functions --- FIXED
**Status: FIXED**
**Thirteen libraries returned `calogOkE` while discarding the status of every `calogRegisterInline` call.**
A failed registration (the OOM path `broker.c` already handles) left the runtime booting with a
native missing, so the failure surfaced much later and far from its cause as "no such function" --
instead of the clean startup abort `main` is written to produce. Six libraries already checked.
Fixed by factoring the pattern rather than adding a thirteenth copy of it: a shared
`CalogNativeEntryT` table plus `calogRegisterBatch`, which stops at the first failure and returns its
status. `calogArchive` had already grown a private version of exactly this table; the shared one
replaces the need for twelve more.
*Verifier:* Demonstrated with a `--wrap=strdup` probe failing one name allocation:
`calogCryptoRegister` returned success while `cryptoHmacSha256` was absent. Every library now returns
the failure. `make test` 966 checks / 0 failed.
---
### S33. [dead-code] src/calog.h (calogLastTrace) --- FIXED
**Status: FIXED (removed)**
**`calogLastTrace` could not do what its own documentation described, and nothing used it.**
Documented as formatting "the CALLING thread's current cross-context call chain" for use from inside a
native. The trace stack is thread-local and pushed by the *calling* thread before it marshals, so a
native running on the callee's thread -- which is where a script's native call actually executes --
always sees an empty chain. Measured: a native invoked from inside a cross-context callable reported
length 0. It also had a real defect (on truncation the buffer held more text than the returned length
accounted for), zero callers, zero tests and no mention in API.md. Removed, like
`calogAbortCurrent` before it. The post-mortem chain carried in error messages -- which IS used and
tested -- remains the supported way to see a boundary.
*Verifier:* Confirmed empirically before removing: a test written specifically to exercise it from
inside the cross-engine call path measured an empty trace, which is what turned this from "untested
API" into "API that cannot work as described".
---
### S34. [bug] src/context.c (contextDrainQueue) --- FIXED
**Status: FIXED**
**A blocking cross-context call still queued at teardown was freed without ever waking its caller.**
`contextDrainQueue` `messageFree`d a pending `messageCallE` without signalling its reply box, leaving
a foreign thread parked on a condition variable nothing could ever signal -- and since teardown joins
that thread, one stuck call became a process-wide hang. This is the concrete, verifiable half of a
reported three-way teardown deadlock. Fixed by replying `calogErrDeadE` instead of freeing, which
every caller of a cross-context native already handles.
*Verifier:* The reported deadlock repro (a repeating timer whose callback calls a host-thread native,
interrupted mid-chunk) stops cleanly in ~1.0 s across 8 consecutive runs, and all TSan targets stay at
zero warnings. The broader structural claim behind that report -- that the before-contexts destroy
hooks join foreign threads while the host has stopped pumping -- is NOT closed by this and remains
open; see the note in the remaining-work list.
---
### S35. [bug] src/context.c (contextReclaimCallables / calogContextUntrackFn) --- FIXED
**Status: FIXED**
**A foreign thread dropping a callable's last reference during a context's teardown stranded the engine handle inside an interpreter about to be destroyed -- the original QuickJS abort, from a direction the sweep did not cover.**
`threadMain` closes the context's queue when `serveLoop` stops, then runs the per-context shutdown
hooks, and only then sweeps its callables. A last-drop landing anywhere in that gap cannot be
marshalled to the dying thread (the queue is closed), so `calogFnFinalizeForeign` runs it -- and that
path is deliberately barred from touching the interpreter, so it frees the adapter's struct WITHOUT
the engine release. The handle is then still live at `destroyInterpreter`, which is exactly what
QuickJS aborts on.
Two distinct windows, and only one of them is what the report described. If the finalize has not yet
reached `calogContextUntrackFn`, the sweep still sees the entry -- with a refcount of zero, so
`calogFnRetainIfLive` refuses it and it was silently skipped. If the finalize gets through untrack
first, the entry is gone from the list and the sweep never sees it at all. The second is far wider
(as wide as the shutdown hooks take) and is the one a test can actually hit.
Both are closed by the same mechanism, decided under the one lock they serialize on. `untrack` now
returns whether the finalize may proceed: while the owner has stopped serving but has not yet
reclaimed, it leaves the callable TRACKED and reports "adopted", so the finalize stops and hands its
shell over. The sweep then finds it and runs the engine release on the only thread permitted to. To
make the handoff safe in the first window too, `CalogFnT` gained a shell-holder count separate from
its reference count: the reference count reaching zero commits someone to finalizing, the holder
count says who may free the memory afterwards. The sweep takes a holder for an entry whose finalize
is still in flight, inherits the one handed across for an adopted entry, and the last party out
frees.
*Verifier:* Reproduced DETERMINISTICALLY rather than by churn, which is what made this worth doing --
the existing 20-round close-while-firing test never once reached the path (instrumented and counted:
0 hits in 30 runs). `tests/testDropInsideTheReclaimWindow` uses a per-context shutdown hook, which by
construction runs inside the window on the dying context's own thread, to park there until a separate
thread has dropped the host's last reference to a JavaScript closure. With the adoption branch
disabled the binary aborts on
`quickjs.c:2682: JS_FreeRuntime: Assertion 'list_empty(&rt->gc_obj_list)' failed` (SIGABRT, exit 134)
-- the same signature as S2. With it, 32 checks / 0 failed, 30 consecutive stress runs clean, and all
TSan targets at zero warnings.
---
### S36. [bug] src/context.c:576 (calogDestroy) --- FIXED
**Status: FIXED**
**`calogDestroy` deadlocked outright when a background thread was mid-callback into a context: measured hanging on 4 runs in 5.**
The `calogDestroyBeforeContextsE` hooks run on the host thread and stop background threads by
JOINING them. But the host, being inside `calogDestroy`, never calls `calogPump` again -- so any
host-directed CALL outstanding at that moment can never be served. With a timer callback in flight
that calls a host-thread native, the cycle closes on itself: the host waits on the timer thread, the
timer thread waits on the context serving its callback, and that context waits for a host reply
nobody is left to deliver.
Fixed by closing host-directed CALLs and failing the ones already queued *before* the first hook
runs. The refusal is deliberately narrow -- CALLs only, since a CALL is the one kind whose sender
waits, and a fire-and-forget script error must still reach the handler that S25 restored. It is also
not a policy choice but a statement of fact: once the host thread is inside `calogDestroy` it will
never pump again, so refusing is the only honest answer to a call that would otherwise block forever.
*Verifier:* Reproduced first (a 1 ms timer whose callback calls a non-inline native, then
`calogDestroy`): **4 of 5 runs hung**. After the fix, **0 of 10**. The regression test in
`tests/testTeardown.c` carries a watchdog so a recurrence is a named failure rather than a wedged
`make test`, and it catches the bug on **5 runs out of 5** when the fix is bypassed.
**A process note worth more than the fix.** Several intermediate "the test does not reproduce it"
measurements in this work were wrong, and wrong in the same way: removing the fix left a static
function unused, `-Werror` failed the build, and each run silently exercised the STALE binary. Three
separate conclusions were drawn from that before the compiler error was noticed. When disabling a
fix to prove a test has teeth, the disable must keep the build green -- gate it at runtime, or check
the build output -- because a failed rebuild looks exactly like a test that passes.
---

15
Makefile
View file

@ -730,6 +730,7 @@ bin/testKv: obj/testKv.o obj/calogKv.o lib/libcalog.a lib/liblua.a | bin
bin/testTimer: obj/testTimer.o obj/calogTimer.o lib/libcalog.a lib/liblua.a | bin
$(CC) $(LDFLAGS) -pthread -o $@ $^ $(LUALIBS)
bin/testPubsub: obj/testPubsub.o obj/calogPubsub.o lib/libcalog.a lib/liblua.a | bin
$(CC) $(LDFLAGS) -pthread -o $@ $^ $(LUALIBS)
@ -969,6 +970,15 @@ CROSSENV = ZIG="$(ZIGBIN)" CALOG_ZIG="$(ZIGBIN)"
CROSSFULL = tools/crossWinFull.sh tools/crossMacFull.sh
CROSSDEFS = _SQ64 SQUSEDOUBLE MB_DOUBLE_FLOAT MRB_USE_DEBUG_HOOK
# Defines that must ALSO reach the pinned-dep builder. tools/crossDeps.sh -- not the full-CLI
# scripts -- compiles the cross libmruby.a, and mruby records cc.defines in no generated header, so
# an adapter built with the define and an archive built without it link cleanly and then disagree
# about the layout of mrb_state at runtime. This is checked separately from CROSSDEFS because
# crossDeps.sh compiles no calog sources and knows nothing about _SQ64 or the CALOG_WITH_* selectors:
# folding it into CROSSFULL would flood the lint with false failures.
CROSSDEPFILES = tools/crossDeps.sh src/mruby/build_config.rb
CROSSDEPDEFS = MRB_USE_DEBUG_HOOK
# Zig-free, hermetic, sub-second. Every calog source the native build compiles must also be compiled
# by both full-CLI cross scripts, and every ABI define + engine selector must reach them. Text-level
# on purpose: it needs no toolchain, so `make test` can run it.
@ -985,6 +995,11 @@ cross-lint:
grep -q -- "$$d" $$s || { echo "cross-lint: -D$$d is missing from $$s" >&2; bad=1; }; \
done; \
done; \
for d in $(CROSSDEPDEFS); do \
for s in $(CROSSDEPFILES); do \
grep -q -- "$$d" $$s || { echo "cross-lint: -D$$d is missing from $$s" >&2; bad=1; }; \
done; \
done; \
if [ $$bad -eq 0 ]; then echo "== cross-lint ok: cross scripts cover every calog source, ABI define and engine =="; \
else echo "== cross-lint FAILED: tools/cross*Full.sh drifted from the native build; fix them or run 'make cross' ==" >&2; exit 1; fi

135
design.md
View file

@ -1661,10 +1661,40 @@ What ends one script is therefore `taskExit` (deferred, and the only mechanism t
`tests/testTeardown.c` drives the early-context-death reclaim path through `calogCurrentRetire` --
the same call `taskExit` makes -- rather than through an API nothing else used.
### The window the sweep does not see by itself
`threadMain` closes the queue when `serveLoop` stops, runs the per-context shutdown hooks, and only
then sweeps. A foreign last-drop landing in that gap cannot be marshalled to the dying thread, so it
finalizes on the dropping thread -- which is barred from touching the interpreter and therefore skips
the engine release, stranding the handle in a VM about to be destroyed. That is the sec 25 crash
again, reached from a direction the sweep did not cover.
Two windows, not one. If the finalize has not yet reached `calogContextUntrackFn`, the sweep sees the
entry with a refcount of zero and used to skip it (`calogFnRetainIfLive` refuses, correctly -- see
sec 28). If the finalize gets through untrack first, the entry is gone and the sweep never sees it.
The second is as wide as the shutdown hooks take, and is the one that reproduces.
Both close on the single lock they serialize on. `calogContextUntrackFn` now reports whether the
finalize may proceed: while the owner has stopped serving and has not yet reclaimed, it leaves the
callable TRACKED and answers no, so the finalize stops and hands the callable to the sweep -- the one
thread allowed to run that release. For the first window, where a finalize is still in flight and
will arrive later, `CalogFnT` gained a **shell-holder count separate from its reference count**: the
reference count reaching zero commits someone to finalizing, the holder count says who may free the
memory afterwards. The sweep takes a holder when a finalize is still coming, inherits the one handed
across when the callable was adopted, and whoever is last frees. Without that split there is no safe
answer -- either the sweep frees the shell while the finalize is about to read it, or the reverse.
### Test
`tests/testTeardown.c` uses JavaScript throughout, because on any other engine a stranded handle is
invisible -- there, surviving the teardown IS the assertion. It covers a subscriber, an export and a
invisible -- there, surviving the teardown IS the assertion.
The window above is tested deterministically rather than by churn, and that distinction earned its
keep: the existing 20-round close-while-firing case never reached the path once (instrumented and
counted -- 0 hits in 30 runs). A per-context shutdown hook runs, by construction, inside the window
on the dying context's own thread, so parking there until another thread drops the host's last
reference reproduces it every time. With the adoption branch disabled the binary aborts on the
familiar `list_empty(&rt->gc_obj_list)`. It covers a subscriber, an export and a
timer callback left registered at `calogDestroy`; each of those whose script instead errors out
first; a script that ends itself; a closure handed to a Lua script
whose owner then dies (invoking it afterwards must fail cleanly, not reach into a destroyed VM); and
@ -2065,3 +2095,106 @@ Makefile path was missed because nothing built it. So `static` is now a prerequi
It costs a handful of -O2 objects and a link -- the vendored archives are shared with the rest of the
tree and OpenSSL is already required by `bin/testNet` -- and it is the only thing that stops this
particular rot recurring a third time.
---
## 31. Failures with nowhere to go, a teardown that could not finish, and a parameter channel
### Failures that had nowhere to go
Three of the defects closed here were the same bug in different clothes, and the same one this whole
stretch of work started with: a failure that reaches nobody.
- A **timer or pubsub callback** that failed was discarded outright. Both invoke a script callable
from a thread with no caller to return a status to, and both looked at that status only to notice a
dead context. `psPublish` returns a delivery COUNT, so a broken subscriber was invisible from every
direction at once. Both now route the failure through `calogPostError`, the same path a failed eval
already used.
- A **spawned task's** failure could not set the exit code at all: the runner only flagged contexts it
had launched itself, and a task belongs to the script that spawned it.
- An **error posted during teardown** was freed undelivered by `contextDrainQueue`, taking the
diagnostic and the exit code with it.
The exit-code rule changed with them. It was "whoever asks first names it", which sounds like it
protects a failure and does not: a task that died just before its parent's `calogExit(0)` had its
failure recorded second and dropped, so whether the run reported it depended on which thread won a
race. It is now **first non-zero wins, and zero stores nothing** -- a failure cannot be masked, and
ordering no longer decides. This is safe precisely because an aborted script is not reported as an
error (sec 29), so what arrives during a teardown is genuine.
The same drain had a second victim: a blocking cross-context CALL still queued when its target died
was freed without signalling its reply box, leaving a foreign thread parked forever on a condition
nothing could raise -- and teardown joins that thread, so one stuck call hung the process. It now
replies `calogErrDeadE`.
### Teardown that cannot join what it is waiting on
`calogDestroy` runs the before-contexts hooks on the host thread, and those hooks stop background
threads by JOINING them. The host is inside `calogDestroy` by then, so it never calls `calogPump`
again -- which makes every host-directed CALL outstanding at that moment unservable. With a timer
callback in flight that calls a host-thread native, the cycle closes: the host waits on the timer
thread, the timer thread waits on the context serving its callback, and that context waits for a host
reply nobody is left to deliver. Measured at 4 hangs in 5 runs.
The fix is to close host-directed CALLs, and fail the ones already queued, before the first hook
runs. Narrow on purpose: CALLs only, because a CALL is the one message kind whose sender waits, and a
fire-and-forget script error must still reach the handler. It is less a policy than an admission --
once the host thread enters `calogDestroy` it will never pump again, so a call arriving after that
was never going to be answered, and failing it immediately is the only honest response.
### A parameter channel
Sealing the engines (sec 30) removed `getenv` along with everything else, which left scripts with no
sanctioned way to receive a parameter -- every project would have invented its own convention out of
files or the kv store. `bin/calog` now takes `--` on the command line: everything before it is a
script, everything after is data, readable from any engine as `calogArgs()`. Registered inline, since
it reads immutable startup state and has nothing to serialize on the host thread.
That the gap existed at all is the honest cost of sec 30, and worth stating plainly: removing a
capability from ten engines is only half a decision. The other half is providing the one thing the
removal actually took away, in a form the host controls.
---
## 32. What the unverified leads turned out to be
The review sweep behind sec 30 produced seventeen candidates and a verification budget of eight. That
cap was a choice, and it deferred nine findings rather than dismissing them. When the remaining seven
were finally put through the same adversarial verification, **all seven were real, and every one was
demonstrated by running something rather than by reading**. The lesson is not that the reviewers were
right -- it is that "unverified" had been quietly filed as "probably nothing", and it was not.
None of them were in the actor core, which is the other thing worth noticing. They were in the
libraries, where a defect is quieter: nothing asserts, no sanitizer necessarily fires, and the
symptom is a leak, a truncated environment, or an argument that silently does nothing.
- **A heap overflow reachable from an ordinary query.** `mysqlRowValue` wrote its NUL at the length
the connector reported, which is the value's UNTRUNCATED length even when only `buffer_length`
bytes were copied -- and the buffer is sized from `max_length`, which under-reports for some types
(a zerofill `BIGINT` caps at its 19-digit display width but renders 20). The clamp uses calog's own
`binds` array, not the connector's: the connector mutates its private copy's `buffer_length` during
conversion and never writes back to ours.
- **A child process with an empty environment.** `procRun` wrote its environment vector at each
pair's index, so any pair it skipped left a NULL in the middle. `execve` stops at the first NULL.
One non-string value emptied the child's environment and leaked every string past the hole.
- **Two leaks and a lost diagnostic**, each small and each permanent: libxml2's hidden entity-table
document on every failed parse, the pipe descriptors a short-circuited `pipe(a) || pipe(b)` guard
abandoned, and `regexSplit` dropping a subject byte per empty match.
- **A documented argument that did nothing.** `decompress(data, filter)` accepted a filter, never
type-checked it, and ignored it -- so a script pinning `"gzip"` against untrusted input accepted
every codec libarchive supports. This one is where the usual rule (leave it out rather than ship a
partial) pointed the other way: the capability is documented and genuinely wanted, so making it
true beat removing it. The filter is now compared against the one actually used, with libarchive's
own names, which are exactly the ones `compress` accepts.
- **The lint that did not check the file the bug came from.** `cross-lint` verified that every
ABI-visible define reached both full-CLI cross scripts, but not `tools/crossDeps.sh` -- which is
where the `MRB_USE_DEBUG_HOOK` divergence of sec 27 actually happened. It needed a separately
scoped check, since that script compiles no calog sources and folding it into the existing loops
would demand `_SQ64` and every `CALOG_WITH_*` selector be present in it.
- **Thirteen libraries reporting success on a partial registration.** Each discarded the status of
every `calogRegisterInline` call, so a failed registration surfaced much later as "no such
function" instead of the clean startup abort `main` is written to produce. The fix was to stop
writing the pattern: a shared `CalogNativeEntryT` table plus `calogRegisterBatch`, which stops at
the first failure and returns it. `calogArchive` had already grown a private version of exactly
that table -- a good sign the shared one was overdue, and that twelve more copies were the wrong
direction.

3
examples/scripts/README.md
View file

@ -25,6 +25,9 @@ Conventions every example follows:
not return: the statement after it never runs, and the first code asked for is the one the
process reports. `Ctrl-C` ends a run the same way, aborting the scripts rather than waiting
for them to finish.
- **`calogArgs()`** returns the arguments given after `--` (`bin/calog build.lua -- release v2`),
the same list for every script in the run. It is the only parameter channel: the engines' own
`getenv`/`argv` are not available.
- Extensions map to engines: `.lua .js .nut .bas .be .scm .wren`.
## `languages/` -- one guided tour per engine

17
libs/calogArchive.c
View file

@ -278,6 +278,9 @@ static int32_t arcDecompress(CalogValueT *args, int32_t argCount, CalogValueT *r
if (argCount < 1 || argCount > 2 || args[0].type != calogStringE) {
return calogFail(result, calogErrArgE, "decompress expects (data [, filter])");
}
if (argCount == 2 && args[1].type != calogStringE) {
return calogFail(result, calogErrArgE, "decompress: filter must be a string");
}
archive = archive_read_new();
if (archive == NULL) {
return calogFail(result, calogErrOomE, "decompress: out of memory");
@ -295,6 +298,20 @@ static int32_t arcDecompress(CalogValueT *args, int32_t argCount, CalogValueT *r
archive_read_free(archive);
return status;
}
// A named filter is ENFORCED, not decoration. The reader always sniffs every codec libarchive
// supports, so naming one used to change nothing at all: a script pinning "gzip" to reject other
// codecs silently accepted zstd, xz, bzip2 and plain uncompressed bytes alike. The names are
// libarchive's own, so they are exactly the ones compress() accepts.
if (argCount == 2) {
const char *actual;
actual = archive_filter_name(archive, 0);
if (actual == NULL || strcmp(actual, args[1].as.s.bytes) != 0) {
status = calogFail(result, calogErrArgE, "decompress: data is not in the named filter's format");
archive_read_free(archive);
return status;
}
}
memset(&out, 0, sizeof(out));
status = calogOkE;
for (;;) {

26
libs/calogCrypto.c
View file

@ -30,19 +30,25 @@ static int32_t cryptoUuidNative(CalogValueT *args, int32_t argCount, CalogValueT
// The single source of truth for lowercase hex digit rendering, shared by cryptoBytesToHex
// and cryptoUuidNative.
static const char cryptoHexDigits[] = "0123456789abcdef";
// Every inline native this library exposes. One table so registration is a single
// checked call (calogRegisterBatch) rather than a run of calls whose status was dropped.
static const CalogNativeEntryT gCryptoNatives[] = {
{ "cryptoHashSha256", cryptoHashSha256Native },
{ "cryptoHashSha1", cryptoHashSha1Native },
{ "cryptoHmacSha256", cryptoHmacSha256Native },
{ "cryptoRandomBytes", cryptoRandomBytesNative },
{ "cryptoBase64Encode", cryptoBase64EncodeNative },
{ "cryptoBase64Decode", cryptoBase64DecodeNative },
{ "cryptoHexEncode", cryptoHexEncodeNative },
{ "cryptoHexDecode", cryptoHexDecodeNative },
{ "cryptoUuid", cryptoUuidNative },
};
int32_t calogCryptoRegister(CalogT *calog) {
calogRegisterInline(calog, "cryptoHashSha256", cryptoHashSha256Native, NULL);
calogRegisterInline(calog, "cryptoHashSha1", cryptoHashSha1Native, NULL);
calogRegisterInline(calog, "cryptoHmacSha256", cryptoHmacSha256Native, NULL);
calogRegisterInline(calog, "cryptoRandomBytes", cryptoRandomBytesNative, NULL);
calogRegisterInline(calog, "cryptoBase64Encode", cryptoBase64EncodeNative, NULL);
calogRegisterInline(calog, "cryptoBase64Decode", cryptoBase64DecodeNative, NULL);
calogRegisterInline(calog, "cryptoHexEncode", cryptoHexEncodeNative, NULL);
calogRegisterInline(calog, "cryptoHexDecode", cryptoHexDecodeNative, NULL);
calogRegisterInline(calog, "cryptoUuid", cryptoUuidNative, NULL);
return calogOkE;
return calogRegisterBatch(calog, gCryptoNatives, (int64_t)(sizeof(gCryptoNatives) / sizeof(gCryptoNatives[0])), NULL);
}

17
libs/calogDb.c
View file

@ -382,6 +382,7 @@ typedef union MysqlScalarT {
// Row context bound to mysqlRowName/mysqlRowValue for one mysqlQuery call.
typedef struct MysqlRowCtxT {
MYSQL_FIELD *fields;
MYSQL_BIND *binds; // OUR array: buffer_length here is exactly what was allocated
char **buffers;
unsigned long *lengths;
my_bool *isNull;
@ -706,6 +707,7 @@ static int32_t mysqlQuery(MYSQL *conn, const CalogValueT *sqlValue, const CalogV
return calogFail(result, status, "dbQuery: out of memory");
}
rowCtx.fields = fields;
rowCtx.binds = binds;
rowCtx.buffers = buffers;
rowCtx.lengths = lengths;
rowCtx.isNull = isNull;
@ -741,13 +743,24 @@ static void mysqlRowName(void *ctx, int32_t column, const char **nameOut, int64_
static int32_t mysqlRowValue(void *ctx, int32_t column, CalogValueT *out) {
MysqlRowCtxT *rowCtx;
unsigned long length;
rowCtx = (MysqlRowCtxT *)ctx;
if (rowCtx->isNull[column]) {
return calogOkE;
}
rowCtx->buffers[column][rowCtx->lengths[column]] = '\0';
return mysqlColumn(rowCtx->fields[column].type, rowCtx->buffers[column], rowCtx->lengths[column], out);
// The connector reports the value's UNTRUNCATED length even when it only copied buffer_length
// bytes, and the buffer is sized from the field's max_length -- which under-reports for some
// types (a zerofill BIGINT caps max_length at its 19-digit display width but renders 20). The
// terminator then lands one byte past the allocation, and mysqlColumn reads past it too. Clamp
// to the capacity WE allocated. Our own binds array is the right source: the connector mutates
// its private copy's buffer_length during conversion, but never writes back to this one.
length = rowCtx->lengths[column];
if (length >= rowCtx->binds[column].buffer_length) {
length = (unsigned long)(rowCtx->binds[column].buffer_length - 1);
}
rowCtx->buffers[column][length] = '\0';
return mysqlColumn(rowCtx->fields[column].type, rowCtx->buffers[column], length, out);
}
#endif

26
libs/calogExport.c
View file

@ -44,17 +44,29 @@ static int32_t exportRemove(CalogValueT *args, int32_t argCount, CalogValueT *re
static int32_t exportResolve(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static int32_t exportResolveByFind(CalogValueT *args, int32_t argCount, CalogValueT *result, int32_t (*find)(const char *, int64_t));
static int32_t exportResolveFold(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
// Every inline native this library exposes. One table so registration is a single
// checked call (calogRegisterBatch) rather than a run of calls whose status was dropped.
static const CalogNativeEntryT gExportNatives[] = {
{ "calogExport", exportPublish },
{ "calogUnexport", exportRemove },
{ "calogCall", exportCall },
{ "__calogExportResolve", exportResolve },
// Case-insensitive variant: my-basic uppercases every identifier at parse time, so its
// bare-name resolver matches an export regardless of the case it was registered with.
{ "__calogExportResolveFold", exportResolveFold },
};
int32_t calogExportRegister(CalogT *calog) {
int32_t status;
calogRegistryRetain(&gInitMutex, &gRefCount);
calogRegisterInline(calog, "calogExport", exportPublish, NULL);
calogRegisterInline(calog, "calogUnexport", exportRemove, NULL);
calogRegisterInline(calog, "calogCall", exportCall, NULL);
calogRegisterInline(calog, "__calogExportResolve", exportResolve, NULL);
// Case-insensitive variant: my-basic uppercases every identifier at parse time, so its
// bare-name resolver matches an export regardless of the case it was registered with.
calogRegisterInline(calog, "__calogExportResolveFold", exportResolveFold, NULL);
status = calogRegisterBatch(calog, gExportNatives, (int64_t)(sizeof(gExportNatives) / sizeof(gExportNatives[0])), NULL);
if (status != calogOkE) {
return status;
}
return calogAtDestroy(calog, calogExportShutdown, calogDestroyAfterContextsE);
}

24
libs/calogFs.c
View file

@ -55,18 +55,24 @@ static int32_t fsReadNative(CalogValueT *args, int32_t argCount, CalogValueT *re
static int32_t fsRemoveNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static int32_t fsStatNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static int32_t fsWriteNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
// Every inline native this library exposes. One table so registration is a single
// checked call (calogRegisterBatch) rather than a run of calls whose status was dropped.
static const CalogNativeEntryT gFsNatives[] = {
{ "fsRead", fsReadNative },
{ "fsWrite", fsWriteNative },
{ "fsAppend", fsAppendNative },
{ "fsExists", fsExistsNative },
{ "fsRemove", fsRemoveNative },
{ "fsMkdir", fsMkdirNative },
{ "fsList", fsListNative },
{ "fsStat", fsStatNative },
};
int32_t calogFsRegister(CalogT *calog) {
calogRegisterInline(calog, "fsRead", fsReadNative, NULL);
calogRegisterInline(calog, "fsWrite", fsWriteNative, NULL);
calogRegisterInline(calog, "fsAppend", fsAppendNative, NULL);
calogRegisterInline(calog, "fsExists", fsExistsNative, NULL);
calogRegisterInline(calog, "fsRemove", fsRemoveNative, NULL);
calogRegisterInline(calog, "fsMkdir", fsMkdirNative, NULL);
calogRegisterInline(calog, "fsList", fsListNative, NULL);
calogRegisterInline(calog, "fsStat", fsStatNative, NULL);
return calogOkE;
return calogRegisterBatch(calog, gFsNatives, (int64_t)(sizeof(gFsNatives) / sizeof(gFsNatives[0])), NULL);
}

12
libs/calogHttp.c
View file

@ -141,6 +141,14 @@ static bool httpSameOrigin(const HttpUrlT *a, const HttpUrlT *b);
static void httpSetTimeouts(CalogSocketT fd, int timeoutSec);
static int32_t httpTlsHandshake(HttpConnT *conn, const HttpUrlT *url, CalogValueT *result);
static int httpWriteAuthority(char *out, size_t cap, const char *scheme, const HttpUrlT *base);
// Every inline native this library exposes. One table so registration is a single
// checked call (calogRegisterBatch) rather than a run of calls whose status was dropped.
static const CalogNativeEntryT gHttpNatives[] = {
{ "httpGet", httpGetNative },
{ "httpRequest", httpRequestNative },
};
int32_t calogHttpRegister(CalogT *calog) {
@ -156,9 +164,7 @@ int32_t calogHttpRegister(CalogT *calog) {
// Windows has no SIGPIPE (a reset/closed peer surfaces as a normal send/recv error instead).
signal(SIGPIPE, SIG_IGN);
#endif
calogRegisterInline(calog, "httpGet", httpGetNative, NULL);
calogRegisterInline(calog, "httpRequest", httpRequestNative, NULL);
return calogOkE;
return calogRegisterBatch(calog, gHttpNatives, (int64_t)(sizeof(gHttpNatives) / sizeof(gHttpNatives[0])), NULL);
}

12
libs/calogJson.c
View file

@ -50,12 +50,18 @@ static int32_t jsonParseValue(JsonParseT *p, CalogValueT *out, int32_t depth);
static int32_t jsonPutUtf8(JsonBufT *buf, uint32_t cp);
static void jsonSkipWs(JsonParseT *p);
static int32_t jsonStringifyNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
// Every inline native this library exposes. One table so registration is a single
// checked call (calogRegisterBatch) rather than a run of calls whose status was dropped.
static const CalogNativeEntryT gJsonNatives[] = {
{ "jsonParse", jsonParseNative },
{ "jsonStringify", jsonStringifyNative },
};
int32_t calogJsonRegister(CalogT *calog) {
calogRegisterInline(calog, "jsonParse", jsonParseNative, NULL);
calogRegisterInline(calog, "jsonStringify", jsonStringifyNative, NULL);
return calogOkE;
return calogRegisterBatch(calog, gJsonNatives, (int64_t)(sizeof(gJsonNatives) / sizeof(gJsonNatives[0])), NULL);
}

22
libs/calogKv.c
View file

@ -47,15 +47,27 @@ static int32_t kvGet(CalogValueT *args, int32_t argCount, CalogValueT *result, v
static int32_t kvHas(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static int32_t kvKeys(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static int32_t kvSet(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
// Every inline native this library exposes. One table so registration is a single
// checked call (calogRegisterBatch) rather than a run of calls whose status was dropped.
static const CalogNativeEntryT gKvNatives[] = {
{ "kvSet", kvSet },
{ "kvGet", kvGet },
{ "kvHas", kvHas },
{ "kvDelete", kvDelete },
{ "kvKeys", kvKeys },
};
int32_t calogKvRegister(CalogT *calog) {
int32_t status;
calogRegistryRetain(&gInitMutex, &gRefCount);
calogRegisterInline(calog, "kvSet", kvSet, NULL);
calogRegisterInline(calog, "kvGet", kvGet, NULL);
calogRegisterInline(calog, "kvHas", kvHas, NULL);
calogRegisterInline(calog, "kvDelete", kvDelete, NULL);
calogRegisterInline(calog, "kvKeys", kvKeys, NULL);
status = calogRegisterBatch(calog, gKvNatives, (int64_t)(sizeof(gKvNatives) / sizeof(gKvNatives[0])), NULL);
if (status != calogOkE) {
return status;
}
return calogAtDestroy(calog, calogKvShutdown, calogDestroyAfterContextsE);
}

74
libs/calogProc.c
View file

@ -48,6 +48,9 @@ typedef struct ProcWinStdinT {
#endif
static int32_t procBufAppend(ProcBufT *buffer, const void *bytes, size_t length);
#ifndef _WIN32
static void procFreeSpawn(char **argv, char **builtEnv);
#endif
static CalogValueT *procOpt(CalogAggT *opts, const char *name);
static int32_t procResult(CalogValueT *result, int32_t exitCode, ProcBufT *out, ProcBufT *err);
static int32_t procRun(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
@ -83,6 +86,26 @@ static int32_t procBufAppend(ProcBufT *buffer, const void *bytes, size_t length)
}
#ifndef _WIN32
// Release the argv vector and, when procRun built one, the replacement environment block. The
// terminating NULL is the vector's own, so this is only correct because the build loop compacts --
// a hole would stop it early and leak the tail.
static void procFreeSpawn(char **argv, char **builtEnv) {
int64_t index;
free(argv);
if (builtEnv != NULL) {
for (index = 0; builtEnv[index] != NULL; index++) {
free(builtEnv[index]);
}
free(builtEnv);
}
}
#endif
static CalogValueT *procOpt(CalogAggT *opts, const char *name) {
CalogValueT key;
CalogValueT *field;
@ -514,9 +537,11 @@ static int32_t procRun(CalogValueT *args, int32_t argCount, CalogValueT *result,
if (field != NULL && field->type == calogAggE && calogAggIsKeyed(field->as.agg)) {
CalogAggT *envMap;
int64_t envCount;
int64_t envUsed;
int64_t e;
envMap = field->as.agg;
envCount = envMap->pairCount;
envUsed = 0;
builtEnv = (char **)calloc((size_t)envCount + 1, sizeof(char *));
if (builtEnv == NULL) {
free(argv);
@ -534,26 +559,43 @@ static int32_t procRun(CalogValueT *args, int32_t argCount, CalogValueT *result,
v = envMap->pairs[e].value.as.s.bytes;
klen = (size_t)envMap->pairs[e].key.as.s.length;
vlen = (size_t)envMap->pairs[e].value.as.s.length;
builtEnv[e] = (char *)malloc(klen + vlen + 2);
if (builtEnv[e] == NULL) {
// Written through envUsed, NOT e: a pair this loop skips (a non-string key or
// value, or a failed malloc) must not leave a NULL hole in the middle of the
// vector. execve stops at the first NULL, so one skipped entry silently handed the
// child a truncated -- often empty -- environment, and the cleanup loops below stop
// at the same hole and leaked everything past it.
builtEnv[envUsed] = (char *)malloc(klen + vlen + 2);
if (builtEnv[envUsed] == NULL) {
continue;
}
memcpy(builtEnv[e], k, klen);
builtEnv[e][klen] = '=';
memcpy(builtEnv[e] + klen + 1, v, vlen);
builtEnv[e][klen + 1 + vlen] = '\0';
memcpy(builtEnv[envUsed], k, klen);
builtEnv[envUsed][klen] = '=';
memcpy(builtEnv[envUsed] + klen + 1, v, vlen);
builtEnv[envUsed][klen + 1 + vlen] = '\0';
envUsed++;
}
envp = builtEnv;
}
}
if (pipe(inPipe) != 0 || pipe(outPipe) != 0 || pipe(errPipe) != 0) {
free(argv);
if (builtEnv != NULL) {
for (i = 0; builtEnv[i] != NULL; i++) {
free(builtEnv[i]);
// Staged, not short-circuited: `pipe(a) || pipe(b) || pipe(c)` leaked whatever the earlier
// calls had already opened, so a process at its fd limit lost two descriptors per procRun until
// nothing in the process could open anything again.
if (pipe(inPipe) != 0) {
procFreeSpawn(argv, builtEnv);
return calogFail(result, calogErrArgE, "procRun: could not create pipes");
}
free(builtEnv);
if (pipe(outPipe) != 0) {
close(inPipe[0]);
close(inPipe[1]);
procFreeSpawn(argv, builtEnv);
return calogFail(result, calogErrArgE, "procRun: could not create pipes");
}
if (pipe(errPipe) != 0) {
close(inPipe[0]);
close(inPipe[1]);
close(outPipe[0]);
close(outPipe[1]);
procFreeSpawn(argv, builtEnv);
return calogFail(result, calogErrArgE, "procRun: could not create pipes");
}
posix_spawn_file_actions_init(&actions);
@ -574,13 +616,7 @@ static int32_t procRun(CalogValueT *args, int32_t argCount, CalogValueT *result,
close(inPipe[0]);
close(outPipe[1]);
close(errPipe[1]);
free(argv);
if (builtEnv != NULL) {
for (i = 0; builtEnv[i] != NULL; i++) {
free(builtEnv[i]);
}
free(builtEnv);
}
procFreeSpawn(argv, builtEnv);
if (spawnRc != 0) {
close(inPipe[1]);
close(outPipe[0]);

24
libs/calogPubsub.c
View file

@ -52,13 +52,25 @@ static void pubsubFreeAll(void);
static int32_t pubsubPublish(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static int32_t pubsubSubscribe(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static int32_t pubsubUnsubscribe(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
// Every inline native this library exposes. One table so registration is a single
// checked call (calogRegisterBatch) rather than a run of calls whose status was dropped.
static const CalogNativeEntryT gPubsubNatives[] = {
{ "psSubscribe", pubsubSubscribe },
{ "psUnsubscribe", pubsubUnsubscribe },
{ "psPublish", pubsubPublish },
};
int32_t calogPubsubRegister(CalogT *calog) {
int32_t status;
calogRegistryRetain(&gInitMutex, &gRefCount);
calogRegisterInline(calog, "psSubscribe", pubsubSubscribe, NULL);
calogRegisterInline(calog, "psUnsubscribe", pubsubUnsubscribe, NULL);
calogRegisterInline(calog, "psPublish", pubsubPublish, NULL);
status = calogRegisterBatch(calog, gPubsubNatives, (int64_t)(sizeof(gPubsubNatives) / sizeof(gPubsubNatives[0])), NULL);
if (status != calogOkE) {
return status;
}
return calogAtDestroy(calog, calogPubsubShutdown, calogDestroyAfterContextsE);
}
@ -144,6 +156,12 @@ static int32_t pubsubPublish(CalogValueT *args, int32_t argCount, CalogValueT *r
continue;
}
status = calogFnInvoke(collected[index], &messageCopy, 1, &callResult);
// A subscriber that FAILED is reported. publish returns a delivery COUNT, not a status, so
// without this a broken handler was invisible to everyone: the publisher saw a number, the
// handler's own author saw nothing at all. Read the owner before the release below.
if (status != calogOkE && status != calogErrDeadE && status != calogErrNotFoundE) {
calogPostError(calogFnRuntime(collected[index]), calogFnOwner(collected[index]), &callResult);
}
calogValueFree(&callResult);
calogValueFree(&messageCopy);
calogFnRelease(collected[index]);

25
libs/calogRegex.c
View file

@ -315,7 +315,8 @@ static int32_t regexSplit(CalogValueT *args, int32_t argCount, CalogValueT *resu
const char *flags;
int64_t subjectLen;
int64_t flagsLen;
PCRE2_SIZE offset;
PCRE2_SIZE pieceStart;
PCRE2_SIZE searchFrom;
bool global;
int32_t status;
@ -346,14 +347,20 @@ static int32_t regexSplit(CalogValueT *args, int32_t argCount, CalogValueT *resu
}
subject = args[1].as.s.bytes;
subjectLen = args[1].as.s.length;
offset = 0;
// Two distinct positions, which used to be one variable and was the bug: pieceStart is where the
// piece being accumulated begins, searchFrom is where the next match is looked for. They differ
// only after an empty match, where the search must advance to make progress but the piece must
// NOT -- sharing them threw away one subject byte per empty match, so any pattern that can match
// empty (`x*`, `[0-9]*`) silently lost most of the subject.
pieceStart = 0;
searchFrom = 0;
while (status == calogOkE) {
PCRE2_SIZE *ovector;
PCRE2_SIZE matchStart;
PCRE2_SIZE matchEnd;
CalogValueT piece;
int rc;
rc = pcre2_match(code, (PCRE2_SPTR)subject, (PCRE2_SIZE)subjectLen, offset, 0, matchData, NULL);
rc = pcre2_match(code, (PCRE2_SPTR)subject, (PCRE2_SIZE)subjectLen, searchFrom, 0, matchData, NULL);
if (rc < 0) {
break;
}
@ -361,25 +368,27 @@ static int32_t regexSplit(CalogValueT *args, int32_t argCount, CalogValueT *resu
matchStart = ovector[0];
matchEnd = ovector[1];
if (matchEnd == matchStart) {
// Empty match: advance one byte so the split terminates rather than looping.
// An empty match is not a separator: advance the SEARCH one byte so the loop terminates,
// and leave the piece where it started so the byte we stepped over is still part of it.
if (matchStart >= (PCRE2_SIZE)subjectLen) {
break;
}
offset = matchStart + 1;
searchFrom = matchStart + 1;
continue;
}
status = calogValueString(&piece, subject + offset, (int64_t)(matchStart - offset));
status = calogValueString(&piece, subject + pieceStart, (int64_t)(matchStart - pieceStart));
if (status == calogOkE) {
status = calogAggPush(pieces, &piece);
if (status != calogOkE) {
calogValueFree(&piece);
}
}
offset = matchEnd;
pieceStart = matchEnd;
searchFrom = matchEnd;
}
if (status == calogOkE) {
CalogValueT tail;
status = calogValueString(&tail, subject + offset, (int64_t)(subjectLen - (int64_t)offset));
status = calogValueString(&tail, subject + pieceStart, (int64_t)(subjectLen - (int64_t)pieceStart));
if (status == calogOkE) {
status = calogAggPush(pieces, &tail);
if (status != calogOkE) {

34
libs/calogSsh.c
View file

@ -94,9 +94,28 @@ static ssize_t sshReadSftpFile(void *handle, int streamId, char *buffer, size_t
static int32_t sshResolve(SshLibT *lib, int64_t handle, CalogValueT *result, SshConnT **out);
static void sshSessionDestroy(LIBSSH2_SESSION *session, CalogSocketT sock, const char *reason);
static void sshWaitSocket(CalogSocketT sock, LIBSSH2_SESSION *session);
// Every inline native this library exposes. One table so registration is a single
// checked call (calogRegisterBatch) rather than a run of calls whose status was dropped.
static const CalogNativeEntryT gSshNatives[] = {
{ "sshConnect", sshConnect },
{ "sshAuthPassword", sshAuthPassword },
{ "sshAuthKey", sshAuthKey },
{ "sshExec", sshExec },
{ "sshClose", sshClose },
{ "sftpGet", sftpGet },
{ "sftpPut", sftpPut },
{ "sftpList", sftpList },
{ "sftpStat", sftpStat },
{ "sftpRemove", sftpRemove },
{ "sftpMkdir", sftpMkdir },
};
int32_t calogSshRegister(CalogT *calog) {
int32_t status;
pthread_mutex_lock(&gSshLibMutex);
if (gSshLib == NULL) {
SshLibT *lib;
@ -130,17 +149,10 @@ int32_t calogSshRegister(CalogT *calog) {
}
gSshLib->refCount++;
pthread_mutex_unlock(&gSshLibMutex);
calogRegisterInline(calog, "sshConnect", sshConnect, gSshLib);
calogRegisterInline(calog, "sshAuthPassword", sshAuthPassword, gSshLib);
calogRegisterInline(calog, "sshAuthKey", sshAuthKey, gSshLib);
calogRegisterInline(calog, "sshExec", sshExec, gSshLib);
calogRegisterInline(calog, "sshClose", sshClose, gSshLib);
calogRegisterInline(calog, "sftpGet", sftpGet, gSshLib);
calogRegisterInline(calog, "sftpPut", sftpPut, gSshLib);
calogRegisterInline(calog, "sftpList", sftpList, gSshLib);
calogRegisterInline(calog, "sftpStat", sftpStat, gSshLib);
calogRegisterInline(calog, "sftpRemove", sftpRemove, gSshLib);
calogRegisterInline(calog, "sftpMkdir", sftpMkdir, gSshLib);
status = calogRegisterBatch(calog, gSshNatives, (int64_t)(sizeof(gSshNatives) / sizeof(gSshNatives[0])), gSshLib);
if (status != calogOkE) {
return status;
}
return calogAtDestroy(calog, calogSshShutdown, calogDestroyAfterContextsE);
}

28
libs/calogTask.c
View file

@ -97,9 +97,25 @@ static int32_t taskLoad(CalogValueT *args, int32_t argCount, CalogVa
static int32_t taskSelf(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static int32_t taskSpawn(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static int64_t taskWrapContext(TaskLibT *lib, CalogContextT *context, CalogValueT *result, const char *label);
// Every inline native this library exposes. One table so registration is a single
// checked call (calogRegisterBatch) rather than a run of calls whose status was dropped.
static const CalogNativeEntryT gTaskNatives[] = {
{ "taskSpawn", taskSpawn },
{ "taskLoad", taskLoad },
{ "taskEval", taskEval },
{ "taskClose", taskClose },
{ "taskActive", taskActive },
{ "taskExit", taskExit },
{ "taskSelf", taskSelf },
{ "taskCount", taskCount },
};
int32_t calogTaskRegister(CalogT *calog) {
int32_t status;
pthread_mutex_lock(&gTaskLibMutex);
if (gTaskLib == NULL) {
TaskLibT *lib;
@ -118,14 +134,10 @@ int32_t calogTaskRegister(CalogT *calog) {
}
gTaskLib->refCount++;
pthread_mutex_unlock(&gTaskLibMutex);
calogRegisterInline(calog, "taskSpawn", taskSpawn, gTaskLib);
calogRegisterInline(calog, "taskLoad", taskLoad, gTaskLib);
calogRegisterInline(calog, "taskEval", taskEval, gTaskLib);
calogRegisterInline(calog, "taskClose", taskClose, gTaskLib);
calogRegisterInline(calog, "taskActive", taskActive, gTaskLib);
calogRegisterInline(calog, "taskExit", taskExit, gTaskLib);
calogRegisterInline(calog, "taskSelf", taskSelf, gTaskLib);
calogRegisterInline(calog, "taskCount", taskCount, gTaskLib);
status = calogRegisterBatch(calog, gTaskNatives, (int64_t)(sizeof(gTaskNatives) / sizeof(gTaskNatives[0])), gTaskLib);
if (status != calogOkE) {
return status;
}
return calogAtDestroy(calog, calogTaskShutdown, calogDestroyAfterContextsE);
}

18
libs/calogTime.c
View file

@ -7,6 +7,8 @@
#include "calogTime.h"
#include "calogInternal.h"
#include <errno.h>
#include <time.h>
@ -20,11 +22,19 @@ static int32_t timeSleepNative(CalogValueT *args, int32_t argCount, CalogValueT
static double timeToSeconds(const struct timespec *ts);
// Every inline native this library exposes. One table so registration is a single
// checked call (calogRegisterBatch) rather than a run of calls whose status was dropped.
static const CalogNativeEntryT gTimeNatives[] = {
{ "timeNow", timeNowNative },
{ "timeMonotonic", timeMonotonicNative },
{ "timeSleep", timeSleepNative },
};
int32_t calogTimeRegister(CalogT *calog) {
calogRegisterInline(calog, "timeNow", timeNowNative, NULL);
calogRegisterInline(calog, "timeMonotonic", timeMonotonicNative, NULL);
calogRegisterInline(calog, "timeSleep", timeSleepNative, NULL);
return calogOkE;
return calogRegisterBatch(calog, gTimeNatives, (int64_t)(sizeof(gTimeNatives) / sizeof(gTimeNatives[0])), NULL);
}

25
libs/calogTimer.c
View file

@ -76,13 +76,25 @@ static int64_t timerNowNs(void);
static void timerPruneLocked(void);
static int32_t timerSchedule(CalogValueT *args, int32_t argCount, CalogValueT *result, bool periodic);
static void *timerThreadMain(void *arg);
// Every inline native this library exposes. One table so registration is a single
// checked call (calogRegisterBatch) rather than a run of calls whose status was dropped.
static const CalogNativeEntryT gTimerNatives[] = {
{ "timerAfter", timerAfterNative },
{ "timerEvery", timerEveryNative },
{ "timerCancel", timerCancelNative },
};
int32_t calogTimerRegister(CalogT *calog) {
int32_t status;
calogRegistryRetain(&gInitMutex, &gRefCount);
calogRegisterInline(calog, "timerAfter", timerAfterNative, NULL);
calogRegisterInline(calog, "timerEvery", timerEveryNative, NULL);
calogRegisterInline(calog, "timerCancel", timerCancelNative, NULL);
status = calogRegisterBatch(calog, gTimerNatives, (int64_t)(sizeof(gTimerNatives) / sizeof(gTimerNatives[0])), NULL);
if (status != calogOkE) {
return status;
}
return calogAtDestroy(calog, calogTimerShutdown, calogDestroyBeforeContextsE);
}
@ -385,6 +397,13 @@ static void *timerThreadMain(void *arg) {
pthread_mutex_unlock(&gTimerMutex);
status = calogFnInvoke(cb, NULL, 0, &res);
// A callback that FAILED is reported. Nothing else observes this thread, so dropping it
// here meant a broken timer callback produced no stderr, no error handler call and no
// failing exit code -- it simply stopped working. Read the owner before the release
// below, which may free the callable.
if (status != calogOkE && status != calogErrDeadE) {
calogPostError(calogFnRuntime(cb), calogFnOwner(cb), &res);
}
calogValueFree(&res);
if (status == calogErrDeadE) {
// The owning context is gone: auto-cancel the timer (find it again by id, since

20
libs/calogXml.c
View file

@ -65,15 +65,21 @@ static int32_t xmlSetStr(CalogAggT *map, const char *key, const char *bytes, in
static void xmlStartElement(void *userData, const xmlChar *name, const xmlChar **atts);
static void xmlStop(XmlParseT *state, int32_t status);
static int32_t xmlStringifyNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
// Every inline native this library exposes. One table so registration is a single
// checked call (calogRegisterBatch) rather than a run of calls whose status was dropped.
static const CalogNativeEntryT gXmlNatives[] = {
{ "xmlParse", xmlParseNative },
{ "xmlStringify", xmlStringifyNative },
};
int32_t calogXmlRegister(CalogT *calog) {
// libxml2 self-initializes, thread-safely (mutex-guarded), when a parser context is created, so
// there is no explicit init to do and nothing process-global to tear down: its per-thread state
// is released when each context thread exits.
calogRegisterInline(calog, "xmlParse", xmlParseNative, NULL);
calogRegisterInline(calog, "xmlStringify", xmlStringifyNative, NULL);
return calogOkE;
return calogRegisterBatch(calog, gXmlNatives, (int64_t)(sizeof(gXmlNatives) / sizeof(gXmlNatives[0])), NULL);
}
@ -433,6 +439,14 @@ static int32_t xmlParseNative(CalogValueT *args, int32_t argCount, CalogValueT *
// XXE-safe. NOCDATA delivers CDATA content through the characters callback.
xmlCtxtUseOptions(ctxt, XML_PARSE_NONET | XML_PARSE_NOCDATA | XML_PARSE_NOENT | XML_PARSE_NO_XXE);
xmlParseChunk(ctxt, args[0].as.s.bytes, (int)args[0].as.s.length, 1);
// An internal entity declaration makes libxml2 build a hidden "SAX compatibility mode" document
// to hold the entity table. libxml2 frees that itself only when the parse finishes cleanly; a
// fatal error halts the parser first, and xmlFreeParserCtxt never owns it -- so every failed
// parse of a document with an internal entity leaked the whole entity table.
if (ctxt->myDoc != NULL) {
xmlFreeDoc(ctxt->myDoc);
ctxt->myDoc = NULL;
}
xmlFreeParserCtxt(ctxt);
if (state.status == calogOkE && !state.haveRoot) {

14
src/broker.c
View file

@ -55,6 +55,20 @@ int32_t calogCall(CalogT *broker, const char *name, CalogValueT *args, int32_t a
}
int32_t calogRegisterBatch(CalogT *broker, const CalogNativeEntryT *entries, int64_t count, void *userData) {
int64_t index;
int32_t status;
for (index = 0; index < count; index++) {
status = calogRegisterInline(broker, entries[index].name, entries[index].fn, userData);
if (status != calogOkE) {
return status;
}
}
return calogOkE;
}
CalogT *calogBrokerCreate(void) {
CalogT *broker;

6
src/calog.h
View file

@ -169,12 +169,6 @@ int32_t calogAtDestroy(CalogT *calog, CalogDestroyHookFnT fn, CalogDestroyPhaseE
// Register per-context hooks; either may be NULL. userData is passed through to both. Setup-only.
int32_t calogAtContext(CalogT *calog, CalogContextHookFnT init, CalogContextHookFnT shutdown, void *userData);
// Format the CALLING thread's current cross-context call chain (newest first, e.g.
// "lua ctx 2 <- janet ctx 1") into buffer, returning the bytes written. Call it from within a
// native to see how a nested cross-engine call reached it. (Post-mortem, a failed cross-context
// call's error message already carries the same chain as "[engine ctx N] ..." tags -- see above.)
int32_t calogLastTrace(char *buffer, size_t size);
// ---- natives ----
// A name beginning with "__" is INTERNAL: it is callable via calogCall but is NOT exposed
// into any engine's global namespace, so scripts can neither see nor shadow it.

33
src/calogInternal.h
View file

@ -154,6 +154,18 @@ CalogEntryT *calogLookup(CalogT *calog, const char *name);
// Safe only while the registry is frozen -- i.e. before any context starts.
void calogForEach(CalogT *calog, void (*visit)(const CalogEntryT *entry, void *ud), void *ud);
// One inline native to register: the script-visible name and the C function behind it.
typedef struct CalogNativeEntryT {
const char *name;
CalogNativeFnT fn;
} CalogNativeEntryT;
// Register every entry as an inline native, stopping at the FIRST failure and returning its status.
// A library's Register must not report success on a partial registration: the runtime then boots
// with a native missing and the script gets "no such function" at run time, far from the cause. One
// userData for the whole batch, which is what every library actually wants.
int32_t calogRegisterBatch(CalogT *calog, const CalogNativeEntryT *entries, int64_t count, void *userData);
// ---- abort bookkeeping (calogAbortAll is the public half; see calog.h) ----
// Mark the calling context: the abort error is being raised into its VM right now. Call it wherever
// the abort is injected into running script code.
@ -163,6 +175,12 @@ void calogAbortRaise(void);
// error, say) still gets its diagnostic while some other script is tearing the runtime down.
bool calogAbortRaised(CalogT *calog);
// Report a script failure to the runtime's error handler, delivered on the host thread. A library
// that invokes a script callback from its OWN thread (the timer thread, a publish) uses this: that
// thread has no caller to return a status to, so without it a callback that failed vanishes -- no
// stderr, no handler, and under bin/calog no failing exit code either.
void calogPostError(CalogT *calog, uint64_t contextId, const CalogValueT *result);
// ---- callable lifecycle (driven by the engine adapters) ----
int32_t calogFnCreate(CalogFnT **out, CalogT *runtime, CalogNativeFnT fn, void *userData, CalogReleaseFnT release, uint64_t ownerCtxId);
void calogFnFinalize(CalogFnT *fn);
@ -181,6 +199,16 @@ void calogFnMarkDead(CalogFnT *fn);
// Take a reference only if the callable has not already dropped to zero (a finalize in flight).
// Call with the owner context's queueMutex held; see the definition for why that is required.
bool calogFnRetainIfLive(CalogFnT *fn);
// Shell lifetime, independent of the reference count: see CalogFnT in value.c. Only the owner's
// reclaim sweep takes a second holder, and only under the owner's queueMutex.
void calogFnShellHold(CalogFnT *fn);
void calogFnShellDrop(CalogFnT *fn);
// Engine release for a callable whose refcount already hit zero on another thread, run on the
// owner's thread by its reclaim sweep. Drops the sweep's shell holder.
void calogFnReclaimOrphan(CalogFnT *fn);
// Whether the owner's teardown adopted this callable rather than letting its finalize complete.
bool calogFnAdopted(const CalogFnT *fn);
void calogFnMarkAdopted(CalogFnT *fn);
uint64_t calogFnOwner(const CalogFnT *fn);
CalogT *calogFnRuntime(const CalogFnT *fn);
void *calogFnUserData(const CalogFnT *fn);
@ -198,7 +226,10 @@ bool calogContextRegistered(CalogT *runtime, uint64_t ctxId);
// that can no longer be resolved. Track failing (OOM) fails the create: an untracked callable would
// be exactly the leak this exists to prevent.
int32_t calogContextTrackFn(CalogT *runtime, uint64_t ownerCtxId, CalogFnT *fn);
void calogContextUntrackFn(CalogT *runtime, uint64_t ownerCtxId, CalogFnT *fn);
// Remove a callable from its owner's list. Returns false when the owner's teardown ADOPTED it
// instead -- the callable stays tracked and the owner's reclaim sweep takes over both the engine
// release and the shell -- in which case the caller must stop and touch it no further.
bool calogContextUntrackFn(CalogT *runtime, uint64_t ownerCtxId, CalogFnT *fn);
// ---- per-context resource limits (sandboxing) ----
// The mutable state a limited context enforces on its OWN thread. memUsed is charged by the engine's

105
src/calogMain.c
View file

@ -83,6 +83,7 @@ static BOOL WINAPI consoleHandler(DWORD ctrlType);
#endif
static const CalogEngineT *engineForExtension(const char *ext);
static const char *extensionOf(const char *arg);
static int32_t nativeCalogArgs(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static int32_t nativeCalogExit(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static int32_t nativeCalogPrint(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static void noteExitCode(int32_t code);
@ -94,11 +95,9 @@ static char *readFile(const char *path);
static void requestShutdown(int32_t code);
static bool resolveArg(const char *arg, const CalogEngineT **outEngine, char **outSource);
// Requested by a script (calogExit), a signal, or the last live context erroring out; the host
// pump loop watches gShutdown and exits with gExitCode. gExitRequested latches the first request of
// any kind (see noteExitCode) so nothing that follows can rewrite the code it named.
// Requested by a script (calogExit), a signal, or a context erroring out; the host pump loop watches
// gShutdown and exits with gExitCode. See noteExitCode for how the code is arbitrated.
static _Atomic bool gShutdown = false;
static _Atomic bool gExitRequested = false;
static _Atomic int32_t gExitCode = 0;
// Set by the signal handler, consumed by main: a signal means STOP, so the runtime is latched
// aborting exactly as calogExit latches it. The handler cannot do that itself -- calogAbortAll is
@ -108,6 +107,13 @@ static _Atomic int32_t gExitCode = 0;
// good and ready.
static _Atomic bool gAbortRequested = false;
// The arguments after "--" on the command line: the sanctioned way to parameterize a run. Written
// once before any context starts and never mutated, so calogArgs reads them without a lock. Scripts
// have no other channel -- the engines' own getenv/argv are removed (see API.md) -- and a documented
// one beats every project inventing its own convention out of files or the kv store.
static char **gScriptArgs = NULL;
static int32_t gScriptArgCount = 0;
// One entry per script context the runner launched. onError flags `failed` for a context whose
// script errors; the pump loop then closes that context and drops the live count. When the live
// count reaches zero -- every launched context has errored out -- the runner exits.
@ -234,6 +240,42 @@ static const char *extensionOf(const char *arg) {
}
// calogArgs() -> the list of strings given after "--" on the command line, the same list for every
// script in the run, empty when none were given. Registered inline: it reads immutable startup data,
// so there is nothing to serialize on the host thread.
static int32_t nativeCalogArgs(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData) {
CalogAggT *agg;
int32_t index;
int32_t status;
(void)args;
(void)argCount;
(void)userData;
calogValueNil(result);
status = calogAggCreate(&agg, calogListE);
if (status != calogOkE) {
return calogFail(result, status, "calogArgs: out of memory");
}
for (index = 0; index < gScriptArgCount; index++) {
CalogValueT item;
status = calogValueString(&item, gScriptArgs[index], (int64_t)strlen(gScriptArgs[index]));
if (status != calogOkE) {
calogAggFree(agg);
return calogFail(result, status, "calogArgs: out of memory");
}
status = calogAggPush(agg, &item);
if (status != calogOkE) {
calogValueFree(&item);
calogAggFree(agg);
return calogFail(result, status, "calogArgs: out of memory");
}
}
calogValueAgg(result, agg);
return calogOkE;
}
// calogExit([code]) -- a script asks the runner to tear everything down and exit. Registered inline
// so it runs on the CALLING script's thread: that is what lets calogAbortAll unwind this very script
// (an error raised on the host thread could not), and everything it touches is thread-safe anyway.
@ -284,29 +326,52 @@ static int32_t nativeCalogPrint(CalogValueT *args, int32_t argCount, CalogValueT
}
// Record what the process will report. Whoever asks FIRST names it -- a script's calogExit, a signal,
// or a script erroring out -- and everything after is ignored. That is the only rule under which a
// failure cannot be masked: neither by a sibling's later calogExit(0), nor by a context erroring out
// while the runtime tears down after a deliberate exit.
// Record what the process will report. The rule is that a FAILURE can never be masked, and that
// ordering must not be what decides whether it is:
//
// - the first NON-ZERO code wins outright and cannot be displaced, so one failure's status is
// never rewritten by a second failure, by a sibling's calogExit(0), or by a signal arriving
// during the teardown that failure started;
// - zero stores nothing. It is already the default, so it stands exactly when nothing failed.
//
// The earlier rule was "whoever asks first names it", which sounds equivalent and is not: a task
// that died just before its parent's calogExit(0) had its failure recorded second and thrown away,
// so whether the run reported the failure depended on which thread got there first. An aborted
// script is not reported as an error at all (see calogAbortRaised), so what reaches here during a
// teardown is a genuine failure and deserves to be reported.
static void noteExitCode(int32_t code) {
if (!atomic_exchange(&gExitRequested, true)) {
atomic_store(&gExitCode, code);
int32_t expected;
if (code == 0) {
return;
}
expected = 0;
atomic_compare_exchange_strong(&gExitCode, &expected, code);
}
static void onError(uint64_t contextId, const char *message, void *userData) {
int32_t index;
bool launched;
(void)userData;
fprintf(stderr, "calog: script error: %s\n", (message != NULL) ? message : "(unknown)");
// Flag the failing context so the pump loop retires it (a script error closes its context).
launched = false;
for (index = 0; index < gLaunchedCount; index++) {
if (gLaunched[index].id == contextId) {
atomic_store(&gLaunched[index].failed, true);
launched = true;
break;
}
}
// A context the RUNNER did not launch -- a task the script spawned. The pump loop only reaps
// what it launched, so nothing else would ever record this failure and the run would report
// success with an error printed above it. Its context belongs to whoever spawned it, so it is
// not closed here; only the exit code is claimed, and first-writer-wins keeps it honest.
if (!launched) {
noteExitCode(1);
}
}
@ -336,9 +401,10 @@ static BOOL WINAPI consoleHandler(DWORD ctrlType) {
static void printUsage(FILE *stream, const char *program) {
size_t index;
fprintf(stream, "usage: %s <script> [script ...]\n", program);
fprintf(stream, "usage: %s <script> [script ...] [-- arg ...]\n", program);
fprintf(stream, "\n");
fprintf(stream, "Run each script through the calog multi-language runtime.\n");
fprintf(stream, "Arguments after -- are passed to every script, readable with calogArgs().\n");
fprintf(stream, "A name with a known extension runs on that engine:\n");
fprintf(stream, " ");
// Derived from gEngines so the help text can never drift from the resolver's own truth.
@ -530,7 +596,23 @@ int main(int argc, char **argv) {
}
}
// "--" separates the scripts to run from the arguments handed to them (calogArgs). Everything
// before it is a script; everything after is data. Without it, every argument is a script, so
// existing invocations are unaffected.
scriptCount = argc - 1;
for (index = 1; index < argc; index++) {
if (strcmp(argv[index], "--") == 0) {
scriptCount = index - 1;
gScriptArgs = &argv[index + 1];
gScriptArgCount = argc - index - 1;
break;
}
}
if (scriptCount <= 0) {
fprintf(stderr, "calog: no script given\n");
printUsage(stderr, program);
return 2;
}
engines = (const CalogEngineT **)malloc((size_t)scriptCount * sizeof(*engines));
sources = (char **)malloc((size_t)scriptCount * sizeof(*sources));
if (engines == NULL || sources == NULL) {
@ -557,6 +639,7 @@ int main(int argc, char **argv) {
calogSetErrorHandler(calog, onError, NULL);
if (calogRegister(calog, "calogPrint", nativeCalogPrint, NULL) != calogOkE ||
calogRegisterInline(calog, "calogArgs", nativeCalogArgs, NULL) != calogOkE ||
calogRegisterInline(calog, "calogExit", nativeCalogExit, NULL) != calogOkE) {
fprintf(stderr, "calog: failed to register the runner natives\n");
status = 1;

142
src/context.c
View file

@ -123,6 +123,8 @@ struct CalogContextT {
_Atomic bool abortRaised; // the abort error was raised INTO this VM (see calogAbortRaised)
bool closing; // guarded by broker->ctxMutex: a close already owns this context
bool queueClosed; // guarded by queueMutex: serveLoop has stopped serving, refuse new messages
bool reclaimDone; // guarded by queueMutex: the callable sweep has run, stop adopting
bool callsClosed; // guarded by queueMutex: refuse CALLs only (errors still land)
// Every callable this context created, so it can reclaim their engine handles before its
// interpreter is destroyed (contextReclaimCallables). Guarded by queueMutex -- a finalize can
// reach it from a foreign thread on the best-effort path in actorReleaseCallable.
@ -176,6 +178,7 @@ static int32_t contextSendBlocking(CalogT *calog, uint64_t targetId, Mes
static bool dispatchCommon(CalogContextT *context, MessageT *message);
static bool enqueueRaw(CalogContextT *context, MessageT *message);
static int32_t hookGrow(void **array, int64_t count, int64_t *cap, size_t elemSize);
static void hostCallsClose(CalogT *calog);
static void hostDispatch(CalogT *calog, MessageT *message);
static uint64_t idCompose(int64_t index, uint32_t generation);
static uint32_t idGeneration(uint64_t id);
@ -184,7 +187,6 @@ static int limitNameCompare(const void *a, const void *b);
static MessageT *messageDequeue(CalogContextT *context);
static void messageFree(MessageT *message);
static bool onOwnerThread(CalogT *runtime, uint64_t owner);
static void postError(CalogT *calog, uint64_t contextId, const CalogValueT *result);
static int32_t pumpUntil(CalogContextT *context, uint64_t token, int32_t *outStatus, CalogValueT *result);
static void registryFreePush(CalogT *calog, int64_t index);
static CalogContextT *registryResolveLocked(CalogT *calog, uint64_t id);
@ -292,31 +294,6 @@ static void traceEnrich(CalogT *runtime, uint64_t ctxId, CalogValueT *result) {
}
int32_t calogLastTrace(char *buffer, size_t size) {
int32_t written;
int32_t depth;
int32_t i;
if (size == 0) {
return 0;
}
buffer[0] = '\0';
written = 0;
depth = traceDepth < CALOG_TRACE_MAX ? traceDepth : CALOG_TRACE_MAX;
for (i = depth - 1; i >= 0; i--) {
const char *engine;
int n;
engine = traceEngineName(traceFrames[i].runtime, traceFrames[i].ctxId);
n = snprintf(buffer + written, size - (size_t)written, "%s%s ctx %lld", written > 0 ? " <- " : "", engine, (long long)idIndex(traceFrames[i].ctxId));
if (n < 0 || (size_t)n >= size - (size_t)written) {
break;
}
written += n;
}
return written;
}
static int32_t actorInvokeCallable(CalogFnT *callable, CalogValueT *args, int32_t argCount, CalogValueT *result) {
CalogNativeFnT fn;
CalogT *runtime;
@ -587,6 +564,9 @@ void calogDestroy(CalogT *calog) {
if (calog == NULL) {
return;
}
// FIRST, before any hook runs: see hostCallsClose. The hooks below join background threads, and
// a thread waiting on a host reply that can no longer come would never be joinable.
hostCallsClose(calog);
// Before-context destroy hooks: a library with a background thread that invokes context
// callbacks stops it here, while those contexts are still alive. Reverse registration order.
for (index = calog->destroyHookCount - 1; index >= 0; index--) {
@ -1171,7 +1151,7 @@ static void contextDispatchEval(CalogContextT *context, MessageT *message) {
free(message->source);
free(message);
if (status != calogOkE) {
postError(context->broker, context->id, &result);
calogPostError(context->broker, context->id, &result);
}
calogValueFree(&result);
}
@ -1196,6 +1176,33 @@ static void contextDrainQueue(CalogContextT *context) {
if (message->kind == messageReleaseE) {
calogFnFinalize(message->callable);
free(message);
} else if (message->kind == messageCallE) {
CalogValueT dead;
int32_t argIndex;
// A BLOCKING caller is parked on this message's reply box. Freeing the message would
// leave that thread waiting on a condition nothing can ever signal -- and teardown
// joins that thread, so the wait becomes a process-wide hang rather than one stuck
// call. Fail the call instead: the caller gets calogErrDeadE, which every caller of a
// cross-context native already handles.
if (message->args != NULL) {
for (argIndex = 0; argIndex < message->argCount; argIndex++) {
calogValueFree(&message->args[argIndex]);
}
free(message->args);
message->args = NULL;
}
calogValueNil(&dead);
calogFail(&dead, calogErrDeadE, "calog: the target context is gone");
contextReply(context->broker, message, calogErrDeadE, &dead);
calogValueFree(&dead);
} else if (message->kind == messageErrorE) {
// An error a dying context posted just before its thread ended. Freeing it would throw
// away the one diagnostic the run has for that failure -- and, under bin/calog, the
// failing exit code with it, so a task that died during teardown made the run report
// success. Delivering is safe here: every context thread has been joined by the time
// this runs, so the handler is called on the host thread with nothing left to race.
contextDispatchError(context->broker, message);
} else {
messageFree(message);
}
@ -1352,27 +1359,44 @@ int32_t calogContextTrackFn(CalogT *runtime, uint64_t ownerCtxId, CalogFnT *fn)
// Drop a callable from its owner's list as it is finalized. Order within the list carries no
// meaning, so the hole is filled from the end. A callable the owner already reclaimed is not in any
// list (the sweep took the whole array), which is why this can run for one and simply find nothing.
void calogContextUntrackFn(CalogT *runtime, uint64_t ownerCtxId, CalogFnT *fn) {
bool calogContextUntrackFn(CalogT *runtime, uint64_t ownerCtxId, CalogFnT *fn) {
CalogContextT *owner;
int64_t index;
int64_t found;
bool mayFinalize;
if (runtime == NULL || ownerCtxId == CALOG_HOST_ID) {
return;
return true;
}
mayFinalize = true;
pthread_mutex_lock(&runtime->ctxMutex);
owner = registryResolveLocked(runtime, ownerCtxId);
if (owner != NULL) {
pthread_mutex_lock(&owner->queueMutex);
found = -1;
for (index = 0; index < owner->ownedCount; index++) {
if (owner->ownedFns[index] == fn) {
owner->ownedCount--;
owner->ownedFns[index] = owner->ownedFns[owner->ownedCount];
found = index;
break;
}
}
// The owner has stopped serving but has not yet reclaimed (threadMain runs the per-context
// shutdown hooks in between, so this window is as long as those take). Finalizing now would
// happen on the wrong thread, which means skipping the engine release -- and the handle
// would then still be live inside an interpreter that is about to be destroyed. Leave the
// callable TRACKED instead: the sweep is the one thread that may run that release, and it
// is about to walk this very list. The finalize's shell holder travels with it.
if (found >= 0 && owner->queueClosed && !owner->reclaimDone) {
calogFnMarkAdopted(fn);
mayFinalize = false;
} else if (found >= 0) {
owner->ownedCount--;
owner->ownedFns[found] = owner->ownedFns[owner->ownedCount];
}
pthread_mutex_unlock(&owner->queueMutex);
}
pthread_mutex_unlock(&runtime->ctxMutex);
return mayFinalize;
}
@ -1431,7 +1455,10 @@ void calogSetErrorHandler(CalogT *calog, CalogErrorFnT fn, void *userData) {
// which most VMs quietly tolerate and QuickJS aborts on. See design.md sec 26.
static void contextReclaimCallables(CalogContextT *context) {
CalogFnT **owned;
CalogFnT **orphans;
int64_t total;
int64_t count;
int64_t orphanCount;
int64_t index;
// Take the whole list first: an engine release can cascade (a VM finalizer dropping another of
@ -1448,17 +1475,40 @@ static void contextReclaimCallables(CalogContextT *context) {
// untrack needs this same mutex, so inspecting the refcount here is safe.
pthread_mutex_lock(&context->queueMutex);
owned = context->ownedFns;
total = context->ownedCount;
count = 0;
for (index = 0; index < context->ownedCount; index++) {
orphanCount = 0;
// An entry whose count already hit zero cannot be retained -- a finalize is committed for it on
// another thread. It must not be skipped either: that thread is NOT this one, so it is barred
// from touching the interpreter and finalizes without running the engine release, stranding the
// handle inside a VM that is about to assert it owns nothing. Those go on a second list and get
// their release run here, on the thread that may. A shell holder is taken for each so the
// committed finalize cannot free the memory before this sweep is done with it.
orphans = (total > 0) ? (CalogFnT **)malloc((size_t)total * sizeof(*orphans)) : NULL;
for (index = 0; index < total; index++) {
if (calogFnRetainIfLive(owned[index])) {
owned[count] = owned[index]; // compact: keep only what this sweep now owns
count++;
} else if (orphans != NULL) {
// An adopted entry's finalize already handed its shell holder over (calogContextUntrackFn);
// one that has not been adopted still has a finalize in flight that will drop its own, so
// this sweep takes a holder of its own to keep the shell alive until it is done.
if (!calogFnAdopted(owned[index])) {
calogFnShellHold(owned[index]);
}
orphans[orphanCount] = owned[index];
orphanCount++;
}
}
context->ownedFns = NULL;
context->ownedCount = 0;
context->ownedCap = 0;
context->reclaimDone = true; // from here a finalize completes normally: no more adopting
pthread_mutex_unlock(&context->queueMutex);
for (index = 0; index < orphanCount; index++) {
calogFnReclaimOrphan(orphans[index]);
}
free(orphans);
if (owned == NULL) {
return;
}
@ -1608,7 +1658,10 @@ static bool dispatchCommon(CalogContextT *context, MessageT *message) {
// the refusal; the caller owns the message either way.
static bool enqueueRaw(CalogContextT *context, MessageT *message) {
pthread_mutex_lock(&context->queueMutex);
if (context->queueClosed) {
// callsClosed refuses CALLs but still accepts everything else -- notably a script error, which
// is fire-and-forget and gets delivered by the drain. A CALL is the one kind whose sender WAITS,
// so it is the only kind that can turn "nobody will serve this" into a hang.
if (context->queueClosed || (context->callsClosed && message->kind == messageCallE)) {
pthread_mutex_unlock(&context->queueMutex);
return false;
}
@ -1625,6 +1678,25 @@ static bool enqueueRaw(CalogContextT *context, MessageT *message) {
}
// Stop the host serving CALLs, and fail everything already queued for it.
//
// Once the host thread is inside calogDestroy it never calls calogPump again, so every host-directed
// CALL from that moment on is guaranteed never to be served -- its sender would block forever. That
// is not hypothetical: the before-contexts destroy hooks stop background threads by JOINING them,
// and a timer thread parked mid-callback on a context that is itself waiting for a host reply
// produces a three-way cycle (host -> timer thread -> context -> host) that hangs teardown outright.
// Refusing new calls and failing the parked ones breaks it before the first join.
static void hostCallsClose(CalogT *calog) {
if (calog->hostContext == NULL) {
return;
}
pthread_mutex_lock(&calog->hostContext->queueMutex);
calog->hostContext->callsClosed = true;
pthread_mutex_unlock(&calog->hostContext->queueMutex);
contextDrainQueue(calog->hostContext);
}
static void hostDispatch(CalogT *calog, MessageT *message) {
if (message->kind == messageErrorE) {
contextDispatchError(calog, message);
@ -1708,7 +1780,9 @@ static void messageFree(MessageT *message) {
// Post a fire-and-forget script error to the host thread's error handler.
static void postError(CalogT *calog, uint64_t contextId, const CalogValueT *result) {
// Deliver a script failure to the runtime's error handler (on the host thread). Also the way a
// library's own background thread reports a callback that failed -- see calogInternal.h.
void calogPostError(CalogT *calog, uint64_t contextId, const CalogValueT *result) {
MessageT *message;
const char *text;

71
src/value.c
View file

@ -32,6 +32,17 @@ struct CalogFnT {
// at create and read by other threads (actorInvokeCallable marshals them), so mutating those
// would be a data race against an invoke already in flight.
_Atomic bool reclaimed;
// How many parties still need the SHELL itself to exist, independent of refCount. refCount
// reaching zero commits someone to finalizing; this says who may free the memory afterwards.
// Normally one (the finalize). The owner's teardown sweep takes a second when it finds a
// callable whose count already hit zero: it must still run the engine release on its own
// thread, and the committed finalize on another thread would otherwise free the shell out from
// under it. Last one out frees. See contextReclaimCallables.
_Atomic int32_t shellHolders;
// Set by calogContextUntrackFn when the owner's teardown took this callable over instead of
// letting the finalize complete. It tells the sweep that the finalize already handed its shell
// holder across, so the sweep inherits it rather than taking another.
_Atomic bool adopted;
};
static int32_t aggregateCopyDepth(CalogAggT **out, const CalogAggT *src, int32_t depth);
@ -211,6 +222,8 @@ int32_t calogFnCreate(CalogFnT **out, CalogT *runtime, CalogNativeFnT fn, void *
atomic_init(&callable->refCount, CALLABLE_INITIAL_REFCOUNT);
atomic_init(&callable->alive, true);
atomic_init(&callable->reclaimed, false);
atomic_init(&callable->shellHolders, 1);
atomic_init(&callable->adopted, false);
// List it against the owning context, which reclaims the engine handle if it is torn down while
// something else still holds this callable (design.md sec 26). No list, no reclaim -- so a
// failure here fails the create rather than handing back a callable that could strand its handle.
@ -234,6 +247,13 @@ void calogFnFinalize(CalogFnT *callable) {
if (callable == NULL) {
return;
}
// Untrack reports whether this finalize may proceed. It says no when the owner is tearing down
// and its reclaim sweep has not run yet: the callable stays tracked, and the sweep -- which is
// the only thread allowed to touch that interpreter -- runs the engine release and frees the
// shell. This holder is handed across with it, so nothing is dropped here.
if (!calogContextUntrackFn(callable->runtime, callable->ownerCtxId, callable)) {
return;
}
// Last reference is gone. If the owner context is still alive, run the engine's closure
// release (luaL_unref / sq_release / ...), which also frees the engine's per-callable
// struct; the actor layer routes this to the owner's thread (design.md sec 10) so the
@ -248,7 +268,6 @@ void calogFnFinalize(CalogFnT *callable) {
// A callable the owner already RECLAIMED (calogFnReclaim, as its interpreter was torn down)
// has no release hook left, so this frees only the shell -- the engine handle went with the
// interpreter that owned it.
calogContextUntrackFn(callable->runtime, callable->ownerCtxId, callable);
if (callable->release != NULL && !atomic_exchange_explicit(&callable->reclaimed, true, memory_order_acq_rel)) {
if (callable->ownerCtxId == CALOG_HOST_ID || calogContextRegistered(callable->runtime, callable->ownerCtxId)) {
callable->release(callable);
@ -256,7 +275,7 @@ void calogFnFinalize(CalogFnT *callable) {
free(callable->userData);
}
}
free(callable);
calogFnShellDrop(callable);
}
@ -326,11 +345,57 @@ void calogFnFinalizeForeign(CalogFnT *callable) {
if (callable == NULL) {
return;
}
calogContextUntrackFn(callable->runtime, callable->ownerCtxId, callable);
// As calogFnFinalize: the owner's teardown may take this over instead (see there).
if (!calogContextUntrackFn(callable->runtime, callable->ownerCtxId, callable)) {
return;
}
if (callable->release != NULL && !atomic_exchange_explicit(&callable->reclaimed, true, memory_order_acq_rel)) {
free(callable->userData);
}
calogFnShellDrop(callable);
}
// Did the owner's teardown adopt this callable (see calogContextUntrackFn)? If so its finalize
// handed a shell holder across, and the sweep must not take a second one.
bool calogFnAdopted(const CalogFnT *callable) {
return atomic_load_explicit(&callable->adopted, memory_order_acquire);
}
void calogFnMarkAdopted(CalogFnT *callable) {
atomic_store_explicit(&callable->adopted, true, memory_order_release);
}
// Drop one shell holder; the last one out frees the memory. Every path that used to free(callable)
// goes through this, so a sweep holding a second reference cannot be freed out from under.
void calogFnShellDrop(CalogFnT *callable) {
if (atomic_fetch_sub_explicit(&callable->shellHolders, 1, memory_order_acq_rel) == 1) {
free(callable);
}
}
// Take a shell holder. Only the owner's reclaim sweep does this, and only while holding the owner's
// queueMutex -- which is what proves the shell is still there to be taken, since every path that
// frees one untracks first and untrack needs that same lock.
void calogFnShellHold(CalogFnT *callable) {
atomic_fetch_add_explicit(&callable->shellHolders, 1, memory_order_acq_rel);
}
// Run the engine release for a callable whose last reference is already gone, then drop the sweep's
// shell holder. Used only by contextReclaimCallables: a finalize is committed for this callable on
// some other thread, but that thread may not touch the interpreter, so the engine release has to
// happen here -- this is the owner's own thread and the interpreter is still alive. The reclaimed
// exchange keeps it to exactly once whichever party gets here first.
void calogFnReclaimOrphan(CalogFnT *callable) {
calogFnMarkDead(callable);
if (callable->release != NULL && !atomic_exchange_explicit(&callable->reclaimed, true, memory_order_acq_rel)) {
callable->release(callable);
}
calogFnShellDrop(callable);
}

6
tests/testArchive.c
View file

@ -35,6 +35,12 @@ static const char *SCRIPT =
"end\n"
"check(#compress(data,'gzip',9) <= #compress(data,'gzip',1), 'gzip level 9 <= level 1')\n"
"check(decompress(compress(data,'gzip')) == data, 'decompress auto-detects filter')\n"
// A named filter is enforced, not decoration: it used to be accepted and then ignored entirely,
// so a script pinning a codec against untrusted input was pinning nothing.
"check(decompress(compress(data,'gzip'), 'gzip') == data, 'decompress accepts the correct named filter')\n"
"check(select(1, pcall(decompress, compress(data,'gzip'), 'zstd')) == false, 'decompress rejects a mismatched filter')\n"
"check(select(1, pcall(decompress, data, 'gzip')) == false, 'decompress rejects uncompressed data named as gzip')\n"
"check(select(1, pcall(decompress, compress(data,'gzip'), 42)) == false, 'decompress rejects a non-string filter')\n"
"local w = archiveWriteOpen('tar','gzip')\n"
"archiveWriteEntry(w, 'a.txt', 'alpha')\n"
"archiveWriteEntry(w, 'b.txt', 'beta\\0gamma')\n"

219
tests/testTeardown.c
View file

@ -23,11 +23,13 @@
#include "calogTimer.h"
#include "calogInternal.h" // calogPubsubShutdown/calogExportShutdown: internal, driven by hand here
#include <pthread.h>
#include <stdatomic.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <time.h>
#include <unistd.h>
#define CHECK(cond, msg) checkImpl((cond), (msg), __LINE__)
@ -41,7 +43,23 @@ static _Atomic int32_t errorCount = 0;
static int32_t testsRun = 0;
static int32_t testsFailed = 0;
// State for testDropInsideTheReclaimWindow: the host's own reference to a JS closure, plus the
// handshake that parks the dying context inside the exact window under test.
static CalogFnT *heldFn = NULL;
static uint64_t windowCtxId = 0;
static _Atomic bool inWindow = false;
static _Atomic bool dropDone = false;
// testDestroyJoinsWithCallInFlight: a regression here HANGS rather than failing a check, so a
// watchdog turns it back into a reportable failure. 15 s against a teardown that takes milliseconds.
#define DESTROY_WATCHDOG_SECONDS 15
static _Atomic bool destroyDone = false;
static void checkImpl(bool condition, const char *message, int32_t line);
static void *dropperThread(void *arg);
static int32_t nativeHold(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static int32_t nativeHostTick(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static void windowShutdownHook(CalogContextT *context, void *userData);
static void *destroyWatchdogThread(void *arg);
static int32_t nativeEndSelf(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static int32_t nativeReady(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
static int32_t nativeReport(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData);
@ -50,6 +68,8 @@ static void pumpUntilReady(void);
static void startRuntime(void);
static void testCrossEngineValueOutlivesOwner(void);
static void testReclaimUnderConcurrentDrops(void);
static void testDropInsideTheReclaimWindow(void);
static void testDestroyJoinsWithCallInFlight(void);
static void testGuardsAfterShutdown(void);
static void testHeldCallableSurvivesTeardown(const char *what, const char *source);
static void testOwnerDiesBeforeTheRuntime(const char *what, const char *source, bool expectError);
@ -78,6 +98,85 @@ static int32_t nativeEndSelf(CalogValueT *args, int32_t argCount, CalogValueT *r
}
// Drops the host's reference from a thread that is NOT the dying context's, which is what makes the
// finalize unable to run the engine release itself.
static void *dropperThread(void *arg) {
struct timespec tick = { 0, PUMP_INTERVAL_NS };
(void)arg;
while (!atomic_load(&inWindow)) {
nanosleep(&tick, NULL);
}
calogFnRelease(heldFn); // last reference: commits a finalize on THIS thread
heldFn = NULL;
atomic_store(&dropDone, true);
return NULL;
}
// The host keeps its own reference to a closure the script hands over.
static int32_t nativeHold(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData) {
(void)userData;
calogValueNil(result);
if (argCount == 1 && args[0].type == calogFnE) {
heldFn = args[0].as.fn;
calogFnRetain(heldFn);
}
return calogOkE;
}
// If calogDestroy has not returned within the budget it is deadlocked and never will. Report the
// failure and end the process here: leaving it wedged would hang `make test` with no explanation,
// which is a far worse signal than a named failing check.
static void *destroyWatchdogThread(void *arg) {
struct timespec tick = { 0, PUMP_INTERVAL_NS };
int64_t waited;
(void)arg;
for (waited = 0; waited < (int64_t)DESTROY_WATCHDOG_SECONDS * 1000000000 / PUMP_INTERVAL_NS; waited++) {
if (atomic_load(&destroyDone)) {
return NULL;
}
nanosleep(&tick, NULL);
}
printf("FAIL testTeardown.c calogDestroy deadlocked with a callback in flight (waited %ds)\n",
DESTROY_WATCHDOG_SECONDS);
fflush(stdout);
_exit(1);
return NULL;
}
// Registered NON-inline on purpose: a script calling this marshals to the host thread and blocks
// waiting for the reply, which is the shape that used to deadlock teardown.
static int32_t nativeHostTick(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData) {
(void)args;
(void)argCount;
(void)userData;
atomic_fetch_add(&reportValue, 1);
calogValueNil(result);
return calogOkE;
}
// Runs on the dying context's own thread, AFTER serveLoop has stopped serving (its queue is closed)
// and BEFORE contextReclaimCallables -- precisely the window where a foreign last-drop cannot be
// marshalled to this thread. Parking here holds that window open for as long as the test needs.
static void windowShutdownHook(CalogContextT *context, void *userData) {
struct timespec tick = { 0, PUMP_INTERVAL_NS };
(void)userData;
if (calogContextId(context) != windowCtxId) {
return;
}
atomic_store(&inWindow, true);
while (!atomic_load(&dropDone)) {
nanosleep(&tick, NULL);
}
}
static int32_t nativeReady(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData) {
(void)args;
(void)argCount;
@ -146,6 +245,124 @@ static void startRuntime(void) {
// has to fail: re-growing a registry that nothing will ever free again would strand the callable it
// holds (the very leak the phase fix exists to prevent). Reads stay safe, which is what the pubsub
// and export tests already rely on.
// The window contextReclaimCallables exists for, held open deliberately rather than raced for.
//
// A JavaScript context hands the HOST a closure. The host then drops its last reference from a
// different thread at the exact moment the context has stopped serving its queue but has not yet
// reclaimed -- the gap threadMain spends running the per-context shutdown hooks. That drop cannot be
// marshalled to the dying context's thread (its queue is closed), so the finalize doing it is not
// allowed to touch the interpreter. If it simply gave up there, the JS function would still be live
// inside a runtime that is about to assert it owns nothing, and this binary would abort.
//
// The shutdown hook itself is what makes this deterministic: it runs inside the window, on the
// context's own thread, and parks there until the drop has happened.
static void testDropInsideTheReclaimWindow(void) {
CalogContextT *ctx;
pthread_t dropper;
heldFn = NULL;
windowCtxId = 0;
atomic_store(&inWindow, false);
atomic_store(&dropDone, false);
atomic_store(&readyFlag, false);
calog = calogCreate();
if (calog == NULL) {
CHECK(false, "reclaim window: runtime create failed");
return;
}
calogSetErrorHandler(calog, onError, NULL);
calogRegisterInline(calog, "ready", nativeReady, NULL);
calogRegisterInline(calog, "hold", nativeHold, NULL);
calogAtContext(calog, NULL, windowShutdownHook, NULL);
ctx = calogContextOpen(calog, &calogJsEngine);
if (ctx == NULL) {
CHECK(false, "reclaim window: context open failed");
calogDestroy(calog);
return;
}
windowCtxId = calogContextId(ctx);
calogContextEval(ctx, "hold(function () { return 1; }); ready();");
pumpUntilReady();
CHECK(heldFn != NULL, "reclaim window: the host holds a reference to the JS closure");
if (pthread_create(&dropper, NULL, dropperThread, NULL) != 0) {
CHECK(false, "reclaim window: could not start the dropper thread");
calogContextClose(ctx);
calogDestroy(calog);
return;
}
// Blocks: stops the context serving, runs the hook (which parks until the drop lands), and only
// then reclaims and destroys the interpreter.
calogContextClose(ctx);
pthread_join(dropper, NULL);
CHECK(atomic_load(&dropDone), "reclaim window: the last reference was dropped inside the window");
calogDestroy(calog);
CHECK(true, "reclaim window: the interpreter was destroyed with no handle stranded inside it");
printf(" reclaim window: a foreign last-drop landed between queue-close and reclaim\n");
}
// calogDestroy must not hang when a background thread is mid-callback into a context.
//
// The before-contexts destroy hooks stop background threads by JOINING them, and they run on the
// host thread -- which, being inside calogDestroy, will never pump again. A timer callback in flight
// that calls a HOST-thread native therefore produced a three-way cycle: the host waits on the timer
// thread, the timer thread waits on the context serving its callback, and that context waits for a
// host reply nobody is left to deliver. Measured before the fix: 4 of 5 runs hung outright.
//
// A regression does not fail an assertion, it wedges the binary, so a watchdog turns it back into a
// named failure -- `make test` reports something actionable instead of hanging with no explanation.
// Measured with the fix bypassed: caught on 5 runs out of 5.
//
// The loop below must run to completion and nothing may come between its last pump and calogDestroy:
// the cycle needs a callback genuinely in flight, and any pause lets the outstanding call drain.
static void testDestroyJoinsWithCallInFlight(void) {
CalogContextT *ctx;
pthread_t watchdog;
struct timespec tick = { 0, PUMP_INTERVAL_NS };
int32_t index;
atomic_store(&reportValue, 0);
calog = calogCreate();
if (calog == NULL) {
CHECK(false, "destroy-with-call-in-flight: runtime create failed");
return;
}
calogSetErrorHandler(calog, onError, NULL);
calogRegister(calog, "hostTick", nativeHostTick, NULL); // host-thread native, on purpose
calogTimerRegister(calog);
ctx = calogContextOpen(calog, &calogLuaEngine);
if (ctx == NULL) {
CHECK(false, "destroy-with-call-in-flight: context open failed");
calogDestroy(calog);
return;
}
calogContextEval(ctx, "timerEvery(1, function() hostTick() end)");
// Pump the FULL budget rather than stopping at the first tick: the cycle needs a callback
// actually in flight when calogDestroy runs, so the timer has to be firing steadily into the
// host at that moment. Stopping early lands teardown in a quiet gap and the hang does not
// reproduce -- measured, which is why this loop has no early exit.
for (index = 0; index < PUMP_LIMIT; index++) {
calogPump(calog);
nanosleep(&tick, NULL);
}
CHECK(atomic_load(&reportValue) > 0, "destroy-with-call-in-flight: the timer callback reached the host");
atomic_store(&destroyDone, false);
if (pthread_create(&watchdog, NULL, destroyWatchdogThread, NULL) != 0) {
CHECK(false, "destroy-with-call-in-flight: could not start the watchdog");
}
calogDestroy(calog);
atomic_store(&destroyDone, true);
pthread_join(watchdog, NULL);
CHECK(true, "destroy-with-call-in-flight: calogDestroy returned instead of hanging on the join");
printf(" destroy with a timer callback mid-call into the host\n");
}
static void testGuardsAfterShutdown(void) {
CalogContextT *ctx;
@ -390,6 +607,8 @@ int main(void) {
"psSubscribe('t', function () {}); ready(); endSelf();", false);
testCrossEngineValueOutlivesOwner();
testReclaimUnderConcurrentDrops();
testDropInsideTheReclaimWindow();
testDestroyJoinsWithCallInFlight();
testGuardsAfterShutdown();
printf("\n%d checks, %d failed\n", testsRun, testsFailed);

5
tests/testUtil.c
View file

@ -36,6 +36,11 @@ static const char *SCRIPT =
"check(regexReplace('a', 'banana', 'X') == 'bXnana', 'regexReplace first only')\n"
"check(regexReplace('([a-z]+)@([a-z]+)', 'user@host', '${2}.${1}') == 'host.user', 'regexReplace group refs')\n"
"check(table.concat(regexSplit(',', 'a,b,c'), '|') == 'a|b|c', 'regexSplit')\n"
// A pattern that CAN match empty used to lose a subject byte per empty match, because one
// variable served as both the search position and the start of the piece being accumulated.
"check(table.concat(regexSplit('x*', 'abc'), '|') == 'abc', 'regexSplit keeps the subject when only empty matches occur')\n"
"check(table.concat(regexSplit('[0-9]*', 'a1b2c'), '|') == 'a|b|c', 'regexSplit splits on real matches around empty ones')\n"
"check(table.concat(regexSplit('-*', 'a-b'), '|') == 'a|b', 'regexSplit loses no bytes around an empty match')\n"
"local s = regexSearch('([0-9]+)', 'abc123def')\n"
"check(s.match == '123' and s.start == 3 and s['end'] == 6, 'regexSearch offsets')\n"
"check(s.groups[1] == '123', 'regexSearch capture group')\n"