diff --git a/API.md b/API.md index 7b3f230a..7026aaa8 100644 --- a/API.md +++ b/API.md @@ -53,7 +53,9 @@ etc. -- see the README and `src/calog.h`.) native allow-list (every engine), a wall-clock time budget, and a memory cap (an over-budget or runaway script is retired). The time budget and memory cap hold for all engines except s7 -- Lua, JavaScript, my-basic, Berry, Tcl, mruby, Squirrel, Wren, and Janet -- each enforced by the cleanest - mechanism its VM allows (see design.md sec 24); s7 is allow-list-only, a documented limit. Bound + mechanism its VM allows (see design.md sec 24); s7 is allow-list-only, a documented limit, and a + context asking for a time budget or a memory cap on s7 is refused (`calogContextOpenLimited` returns + NULL, and a capped script's `taskSpawn` onto s7 is an error) rather than opened without them. Bound granularity varies: exact for Lua/JS/Berry/mruby, otherwise allocation-, loop-, or statement-granular (a single operation may transiently overshoot the memory cap before the next check). For my-basic, `INPUT` never reads host stdin: it yields an empty line, so a script takes input through natives like @@ -101,7 +103,8 @@ refusal, a counting allocator, an instruction or heartbeat hook, the bytecode lo or -- for Janet -- a watchdog thread); design.md sec 24 has the per-engine details and the two accepted limitations (a script that deliberately catches the sandbox error and loops can pin its thread; a single allocation of script-controlled size can transiently overshoot the cap). **s7 is allow-list -only** -- its unchecked allocator and optimizer-collapsed loops leave no safe hook point. +only** -- its unchecked allocator and optimizer-collapsed loops leave no safe hook point -- so a +time budget or memory cap on s7 is refused, never silently dropped. Value-model edges that follow from each language (not calog limits): Tcl and s7 have no true `nil` (it marshals out as `""` / `#`) and collapse `bool` to `0`/`1`; a hybrid list+map aggregate @@ -205,6 +208,7 @@ be raised later without a migration. Compare the result with `cryptoEquals`. | Function | Description | |---|---| | `cryptoHashSha256(data: string) -> string` | SHA-256 as a 64-char lowercase hex digest. | +| `cryptoHashFileSha256(path: string) -> string` | SHA-256 of a file's contents, read a piece at a time, so a file of any size is hashed in constant memory. | | `cryptoHashSha1(data: string) -> string` | SHA-1 as a 40-char lowercase hex digest. | | `cryptoHmacSha256(key: string, data: string) -> string` | HMAC-SHA-256 as 64-char lowercase hex. | | `cryptoRandomBytes(count: int) -> string` | `count` cryptographically-random bytes. | @@ -256,11 +260,12 @@ POSIX filesystem access. A failed operation raises a catchable script error carr | Function | Description | |---|---| -| `fsRead(path: string) -> string` | Read a whole file (binary-safe). | +| `fsRead(path: string [, offset: int [, length: int]]) -> string` | Read a whole file (binary-safe), or `length` bytes from `offset` (to the end without a length); short at the end of the file, empty past it. How a file too large to hold is read a piece at a time. | | `fsWrite(path: string, data: string)` | Create/truncate and write `data`. | | `fsAppend(path: string, data: string)` | Create if absent, append at the end. | | `fsExists(path: string) -> bool` | Whether the path exists. | | `fsRemove(path: string)` | Unlink a file. | +| `fsRename(from: string, to: string)` | Move a file, replacing `to` if it exists. | | `fsMkdir(path: string)` | Create one directory level (existing dir is OK). | | `fsList(path: string) -> list` | Entry name strings, excluding `.` and `..`. | | `fsStat(path: string) -> map \| nil` | `{size, isDir, isFile, mtime}`, or nil if the path is absent. | diff --git a/AUDIT.md b/AUDIT.md index 8334ca60..f186bd88 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -2,12 +2,13 @@ Original audit: 15 scoped reviewers + 1 adversarial verifier per finding (118 raw, 4 refuted, 114 confirmed). -A **follow-on audit** is appended at the end of this file: 36 findings, IDs prefixed `S`, from the +A **follow-on audit** is appended at the end of this file: 39 findings, IDs prefixed `S`, from the 2026-07-24 to 2026-08-04 session. Same format, different provenance, and it arrived in four passes: one reported defect and everything pulling on it uncovered (S1-S17); three open items re-examined after being written off as accepted limitations (S18-S20); a six-lens review sweep with adversarial verification (S21-S23); and the leads that sweep raised but never verified, plus the items its -verification confirmed (S24-S36). It is a separate body of work -- the counts in this section cover +verification confirmed (S24-S36); and a fifth pass on 2026-09-29 (S37-S39) that closed the one +open note left behind. It is a separate body of work -- the counts in this section cover the original 114 only. ## Remediation status @@ -2244,7 +2245,7 @@ every caller of a cross-context native already handles. interrupted mid-chunk) stops cleanly in ~1.0 s across 8 consecutive runs, and all TSan targets stay at zero warnings. The broader structural claim behind that report -- that the before-contexts destroy hooks join foreign threads while the host has stopped pumping -- is NOT closed by this and remains -open; see the note in the remaining-work list. +open; see the note in the remaining-work list. (Closed 2026-09-29 by S37.) --- @@ -2321,3 +2322,58 @@ fix to prove a test has teeth, the disable must keep the build green -- gate it the build output -- because a failed rebuild looks exactly like a test that passes. --- + +## Fifth pass -- the open note, and two found beside it (3), 2026-09-29 + +### S37. [bug] libs/calogTimer.c (calogTimerShutdown) --- FIXED + +**`calogDestroy` hung whenever a timer callback re-armed itself during teardown: 10 runs in 10.** + +The other leg of S34's structural note. `calogTimerShutdown` went through `calogRegistryRelease`, +which holds `gInitMutex` for the whole of `timerFreeAll` -- and that joins the timer thread. The +thread is inside `calogFnInvoke`, waiting on the context running its callback; the callback calls +`timerAfter`, an inline native, whose `timerSchedule` waits on `gInitMutex`. Host waits on thread, +thread on context, context on the host's lock. A callback that schedules another timer is ordinary. + +Fixed by dropping the count under `gInitMutex` and tearing down after it is released +(`calogRegistryReleaseLast`, in value.c beside `calogRegistryRelease`): the waiting schedule then +takes the lock, sees zero, and fails with `calogErrDeadE`, the callback returns, and the join +completes. A new `gTeardownMutex`, held by a register and by the whole shutdown but never by a +schedule, keeps a runtime registering from racing the last one's teardown, which the old lock had +covered. Pubsub, export, and kv use the same release but join no thread, so they are unchanged. + +*Verifier:* `testTeardown` gained `testDestroyJoinsWithRearmInFlight` (a Lua `timerEvery` whose +callback works a while and then calls `timerAfter`). With the old shutdown restored it hit the +watchdog 3 runs in 3; with the fix, 35 checks, 0 failed, 3 runs in 3, under ASan/UBSan. + +### S38. [bug] src/context.c (contextOpenWithSandbox) --- FIXED + +**A capped script could escape its memory cap and deadline by spawning an s7 task.** + +s7 cannot enforce either limit (its loops bypass its begin-hook; its allocator is unchecked), which +was documented -- but `calogContextOpenLimited` still opened an s7 context with a cap it would ignore, +and `calogContextOpenInheriting` let a capped Lua script `taskSpawn('s7', ...)` a child that shared +the sandbox in name only. Sec 30 ("a task no longer escapes its parent's sandbox") did not cover it. + +Fixed by an `ignoresLimits` field on `CalogEngineT` (true for s7, false for the nine engines that +enforce) and a refusal in `contextOpenWithSandbox` when the engine ignores limits and the sandbox is +`metered`: the limited open returns NULL and the spawn is an error. An allow-list alone still opens +on s7, since the broker enforces that for every engine. + +*Verifier:* `testSandbox` gained five checks (a capped and a time-limited s7 open refused, an +allow-list-only one opening, a capped script's s7 spawn refused and never run, and a control spawning +s7 under an allow-list alone). With the flag cleared on s7, four of them fail; with it, 54 checks, +0 failed. + +### S39. [build] tools/crossWinFull.sh, tools/crossMacFull.sh, Makefile (RELOBJ) --- FIXED + +**`make test` could not run: `libs/calogTrust.c` never reached the cross scripts or the static build.** + +`calogTrust.c` was split out of `calogHttp.c` (the platform root loaders, shared with the TLS +transport) and wired into the native build only. `cross-lint` failed first, which stopped `make +test` before any suite ran; behind it, `bin/calogStatic` failed to link (`calogNet` needs +`calogTrustLoad`). Added to both cross scripts -- the macOS Keychain define moves to the file that +now holds the Keychain code -- and to `RELOBJ`. + +*Verifier:* `make cross-lint` ok; `make static` links and runs; `make test` exits 0. + diff --git a/Makefile b/Makefile index 5cd7ac1e..3e518bc0 100644 --- a/Makefile +++ b/Makefile @@ -673,7 +673,7 @@ release: # examples/staticDemo.c; the dlopen/getaddrinfo link warnings are expected and harmless # for a binary that uses neither.) RELFLAGS = -std=c11 $(WARN) $(DEP) -O2 -RELOBJ = obj/rel/value.o obj/rel/broker.o obj/rel/context.o obj/rel/luaEngine.o obj/rel/luaAdapter.o obj/rel/calogHandle.o obj/rel/calogDb.o obj/rel/calogNet.o +RELOBJ = obj/rel/value.o obj/rel/broker.o obj/rel/context.o obj/rel/luaEngine.o obj/rel/luaAdapter.o obj/rel/calogHandle.o obj/rel/calogDb.o obj/rel/calogNet.o obj/rel/calogTrust.o obj/rel: mkdir -p obj/rel @@ -761,7 +761,7 @@ bin/testTrace: obj/testTrace.o obj/calogExport.o lib/libcalog.a lib/liblua.a lib # The httpd-as-a-script (examples/httpd.lua): protocol in Lua over the tcp* transport (calogNet, which # needs libenet) + the crypto natives (calogCrypto, which needs OpenSSL). -bin/testHttpdLua: obj/testHttpdLua.o obj/calogNet.o obj/calogTrust.o obj/calogCrypto.o obj/calogHandle.o lib/libcalog.a lib/liblua.a lib/libenet.a $(SSLARCH) | bin +bin/testHttpdLua: obj/testHttpdLua.o obj/calogNet.o obj/calogTrust.o obj/calogCrypto.o obj/calogFs.o obj/calogHandle.o lib/libcalog.a lib/liblua.a lib/libenet.a $(SSLARCH) | bin $(CC) $(LDFLAGS) -pthread -o $@ $(filter-out $(SSLARCH),$^) $(LUALIBS) $(SSLARCH) bin/testHttps: obj/testHttps.o obj/calogHttp.o obj/calogTrust.o lib/libcalog.a lib/liblua.a $(SSLARCH) | bin diff --git a/design.md b/design.md index 0fd4d4dd..a4f1a61d 100644 --- a/design.md +++ b/design.md @@ -619,6 +619,8 @@ last drop -- are preserved). `testEngineLua` captures a Lua closure on its conte thread, then invokes and releases it from the main thread; both marshal to the owner, ASan/TSan-clean. Limit: releasing a callable whose owner *context* has been destroyed is the deferred non-quiescent-teardown case (sec 9) -- best-effort inline finalize for now. +(Superseded: secs 26 and 28 finalize orphaned releases in the drain and adopt drops landing in the +reclaim window; what remains is the deliberate leak on a failed post, which beats corrupting a VM.) **JavaScript adapter (Duktape), the fourth engine.** Vendored Duktape 2.7.0 (the single amalgamated `duktape.c`/`duktape.h`/`duk_config.h`) in `vendor/duktape`, built @@ -820,7 +822,9 @@ returns the slot to the freelist; the next reuse bumps the generation. A stale i recycler -- `testActor`'s generation test proves it. The registry lock is held across enqueue, so a foreign enqueue cannot race a destroy onto a freed queue mutex. Still quiescence-assuming (no call to the context in flight at teardown); in-flight -reference draining is the remaining sec 9 hardening. +reference draining is the remaining sec 9 hardening. (Superseded: queued CALLs are answered +`calogErrDeadE` in the drain and enqueues onto a closed queue are refused -- AUDIT.md S34 and S36 -- +and secs 26 and 28 drain the references.) **Engine on a thread (the EngineT vtable).** `EngineT` gained `runSource`; `contextEval(context, source, result)` marshals a script run onto the context's own @@ -1345,7 +1349,8 @@ Cross-cutting design points: before the VM is torn down. Memory still needs the allocator (a 5 ms watchdog poll cannot bound an exponential loop), so the counting allocator interrupts on an over-cap allocation. -**s7 -- the documented remaining limit.** s7's `Malloc/Calloc/Realloc` are unchecked macros with no +**s7 -- the documented remaining limit** (and, since 2026-09-29, refused rather than ignored: a +limited open or a capped script's `taskSpawn` on s7 fails, AUDIT.md S38). s7's `Malloc/Calloc/Realloc` are unchecked macros with no allocator hook (a single large allocation deref-crashes), its only heap control is a *cell-count* `(*s7* 'max-heap-size)` (not byte-accurate, and it does not stop a single big allocation), and its `begin_hook` does not fire inside s7's optimizer-collapsed loops -- the sandbox-critical case. Both diff --git a/examples/httpd.lua b/examples/httpd.lua index 0896cc0d..947f4b5a 100644 --- a/examples/httpd.lua +++ b/examples/httpd.lua @@ -15,6 +15,8 @@ -- local s = httpd.new() -- s:route("GET", "/hi", function(req) return "hello " .. req.path end) -- s:route("GET", "/made", function(req) return { status = 201, body = "created" } end) +-- s:route("POST", "/up", fn, { spool = { dir = "spool", max = 8e9 } }) -- body to a file: req.bodyFile, req.bodySize +-- return { file = { path = p, offset = 0, length = n } } -- a response streamed from a file -- s:websocket("/ws", function(msg) return "echo: " .. msg.message end) -- returns a text reply or nil -- s:serve(8080) -- opts: { keep = fn, acceptTimeout = ms } @@ -23,32 +25,42 @@ httpd.__index = httpd local WS_MAGIC = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11" local HEADER_MAX = 64 * 1024 -local BODY_MAX = 16 * 1024 * 1024 +local BODY_MAX = 16 * 1024 * 1024 -- a body held in memory; a spooled route sets its own cap local RECV_CHUNK = 8192 +local SPOOL_CHUNK = 65536 -- read this much at a time into a spooled body +local SPOOL_FLUSH = 1024 * 1024 -- and write it out once this much has gathered +local FILE_CHUNK = 65536 -- a file response is sent this much at a time local ACCEPT_POLL = 200 -- ms; the accept loop wakes this often to re-check its keep predicate local REASON = { - [101] = "Switching Protocols", [200] = "OK", [201] = "Created", [204] = "No Content", + [101] = "Switching Protocols", [200] = "OK", [201] = "Created", [202] = "Accepted", [204] = "No Content", + [206] = "Partial Content", [413] = "Content Too Large", [416] = "Range Not Satisfiable", [301] = "Moved Permanently", [302] = "Found", [400] = "Bad Request", [401] = "Unauthorized", [403] = "Forbidden", [404] = "Not Found", [405] = "Method Not Allowed", [500] = "Internal Server Error", } function httpd.new() - return setmetatable({ routes = {}, prefixes = {}, wsRoutes = {} }, httpd) + return setmetatable({ routes = {}, routeOpts = {}, prefixes = {}, wsRoutes = {} }, httpd) end -function httpd:route(method, path, handler) - self.routes[method:upper() .. " " .. path] = handler +-- opts.spool = { dir = folder, max = bytes } writes the request body to a file in `dir` instead of +-- holding it in memory: the handler gets req.bodyFile and req.bodySize, the file is removed after +-- the handler returns unless it moved it, and a body over `max` is answered 413 unread. +-- opts.onRefused(req, status), if given, is told of a request answered that way without its handler, +-- so a server that logs every request can log that one too. +function httpd:route(method, path, handler, opts) + self.routes[method:upper() .. " " .. path] = handler + self.routeOpts[method:upper() .. " " .. path] = opts end -- A route for everything under a path. The handler gets req.prefix (what matched) and req.rest (the -- segments after it), so "/v1/download/" can serve "/v1/download/dragon/3" without every id needing -- its own registration. An exact route always wins, and between prefixes the longest one does. -function httpd:routePrefix(method, prefix, handler) - self.prefixes[#self.prefixes + 1] = { method = method:upper(), prefix = prefix, handler = handler } +function httpd:routePrefix(method, prefix, handler, opts) + self.prefixes[#self.prefixes + 1] = { method = method:upper(), prefix = prefix, handler = handler, opts = opts } table.sort(self.prefixes, function(a, b) return #a.prefix > #b.prefix end) end @@ -58,10 +70,12 @@ function httpd:websocket(path, handler) end --- Read one request off the connection, starting from any bytes `buf` already holds from a previous --- (pipelined) read. Returns (request, leftoverBuffer) or nil on close/malformed. request is --- { method, path, query, version, headers, body }; leftoverBuffer feeds the next keep-alive read. -local function readRequest(conn, buf) +-- Read one request's head off the connection, starting from any bytes `buf` already holds from a +-- previous (pipelined) read. Returns (request, rest) -- request is { method, path, query, version, +-- headers, length } with the body still unread, rest the bytes already received past the head -- or +-- nil on close or a malformed head. The body is read by readBody or spoolBody once the route is +-- known, so a route can take it as a file and a body over its cap is refused before it is read. +local function readHead(conn, buf) buf = buf or "" local headerEnd = nil while true do @@ -91,25 +105,74 @@ local function readRequest(conn, buf) -- the leftover-buffer keep-alive path (a TE.CL smuggle), so reject it rather than mis-frame it. if headers["transfer-encoding"] then return nil end -- Content-Length is DIGIT-only per RFC 7230; reject the hex/scientific/signed/float forms tonumber - -- would otherwise accept (a body-length desync / smuggling risk), and reject anything over the cap. + -- would otherwise accept (a body-length desync / smuggling risk). The cap is the route's. local clen = 0 local clRaw = headers["content-length"] if clRaw then - if not clRaw:match("^%d+$") then return nil end - local n = tonumber(clRaw) - if not n or n > BODY_MAX then return nil end - clen = math.tointeger(n) or 0 + if not clRaw:match("^%d+$") or #clRaw > 18 then return nil end + clen = math.tointeger(tonumber(clRaw)) or 0 end - while #rest < clen do + return { method = method:upper(), path = path, query = query, version = version, headers = headers, length = clen }, rest +end + + +-- The body into memory: gathered as pieces and joined once, since growing one string by each piece +-- copies all of it every time. Returns (body, leftover) or nil on close. +local function readBody(conn, rest, clen) + local parts, have = { rest }, #rest + while have < clen do local chunk = tcpRecv(conn, RECV_CHUNK) if not chunk then return nil end - rest = rest .. chunk + parts[#parts + 1] = chunk + have = have + #chunk end - local req = { - method = method:upper(), path = path, query = query, version = version, - headers = headers, body = rest:sub(1, clen), - } - return req, rest:sub(clen + 1) -- leftover: bytes of the next pipelined request, if any + local all = table.concat(parts) + return all:sub(1, clen), all:sub(clen + 1) +end + + +-- The body into a new file in `dir`, a piece at a time, written out in larger runs. Returns (path, +-- leftover) or nil on close, with any partial file removed. +local function spoolBody(conn, rest, clen, dir) + fsMkdir(dir) + local path = dir .. "/body-" .. cryptoHexEncode(cryptoRandomBytes(8)) .. ".part" + local parts, pending, have = {}, 0, 0 + local leftover = "" + local function flush() + if pending > 0 then + fsAppend(path, table.concat(parts)) + parts, pending = {}, 0 + end + end + fsWrite(path, "") + local function take(chunk) + local need = clen - have + if #chunk > need then + leftover = chunk:sub(need + 1) + chunk = chunk:sub(1, need) + end + parts[#parts + 1] = chunk + pending = pending + #chunk + have = have + #chunk + if pending >= SPOOL_FLUSH then flush() end + end + -- A dropped connection raises out of tcpRecv: the partial file goes before the error does. + local ok, closed = pcall(function() + take(rest) + while have < clen do + local chunk = tcpRecv(conn, SPOOL_CHUNK) + if not chunk then return true end + take(chunk) + end + flush() + return false + end) + if not ok or closed then + fsRemove(path) + if not ok then error(closed, 0) end + return nil + end + return path, leftover end @@ -124,7 +187,8 @@ end -- Turn a handler's return value into an HTTP response and send it. nil -> 204; a string -> 200 with --- that body; a table -> { status = 200, headers = {}, body = "" }. +-- that body; a table -> { status = 200, headers = {}, body = "" }, or with file = { path, offset, +-- length } in place of a body, that piece of a file sent a chunk at a time rather than held whole. local function writeResponse(conn, resp, keepAlive) local status, body, extra = 200, "", nil if resp == nil then @@ -136,14 +200,27 @@ local function writeResponse(conn, resp, keepAlive) body = resp.body or "" extra = resp.headers end + local file = type(resp) == "table" and resp.file or nil + local length = file and file.length or #body local out = { string.format("HTTP/1.1 %d %s\r\nContent-Length: %d\r\nConnection: %s\r\n", - status, REASON[status] or "Status", #body, keepAlive and "keep-alive" or "close") } + status, REASON[status] or "Status", length, keepAlive and "keep-alive" or "close") } if extra then for k, v in pairs(extra) do out[#out + 1] = k .. ": " .. v .. "\r\n" end end out[#out + 1] = "\r\n" - out[#out + 1] = body - tcpSend(conn, table.concat(out)) -- raises on a broken connection; httpd:handle's pcall catches it + if not file then + out[#out + 1] = body + tcpSend(conn, table.concat(out)) -- raises on a broken connection; httpd:handle's pcall catches it + return + end + tcpSend(conn, table.concat(out)) + local offset, sent = file.offset or 0, 0 + while sent < length do + local piece = fsRead(file.path, offset + sent, math.min(FILE_CHUNK, length - sent)) + if #piece == 0 then error("httpd: the file ended before its length", 0) end + tcpSend(conn, piece) + sent = sent + #piece + end end @@ -266,7 +343,7 @@ function httpd:handle(conn) local buf = "" while true do local req - req, buf = readRequest(conn, buf) + req, buf = readHead(conn, buf) if not req then break end local upgrade = (req.headers["upgrade"] or ""):lower():find("websocket", 1, true) local connhdr = (req.headers["connection"] or ""):lower():find("upgrade", 1, true) @@ -278,25 +355,51 @@ function httpd:handle(conn) return -- the connection is now a (closed) WebSocket, not keep-alive HTTP end local keepAlive = wantsKeepAlive(req) - local handler = self.routes[req.method .. " " .. req.path] or self.routes["* " .. req.path] + local key = req.method .. " " .. req.path + local handler = self.routes[key] or self.routes["* " .. req.path] + local opts = self.routeOpts[key] or self.routeOpts["* " .. req.path] if not handler then for _, entry in ipairs(self.prefixes) do if (entry.method == req.method or entry.method == "*") and req.path:sub(1, #entry.prefix) == entry.prefix then req.prefix = entry.prefix req.rest = req.path:sub(#entry.prefix + 1) handler = entry.handler + opts = entry.opts break end end end + -- The body, now that the route says how to take it. One over its cap is answered 413 + -- unread, and the connection closed, since the unread body is still on the wire. + local spool = opts and opts.spool + local cap = spool and spool.max or BODY_MAX + if req.length > cap then + if opts and opts.onRefused then + pcall(opts.onRefused, req, 413) + end + writeResponse(conn, { status = 413, body = "Content Too Large" }, false) + break + end + if spool then + req.bodyFile, buf = spoolBody(conn, buf, req.length, spool.dir) + if not req.bodyFile then break end + req.bodySize, req.body = req.length, "" + else + req.body, buf = readBody(conn, buf, req.length) + if not req.body then break end + end if handler then local ok, resp = pcall(handler, req) + if req.bodyFile and fsExists(req.bodyFile) then + fsRemove(req.bodyFile) + end if ok then writeResponse(conn, resp, keepAlive) else writeResponse(conn, { status = 500, body = "Internal Server Error" }, keepAlive) end else + if req.bodyFile then fsRemove(req.bodyFile) end writeResponse(conn, { status = 404, body = "Not Found" }, keepAlive) end if not keepAlive then break end diff --git a/libs/calogCrypto.c b/libs/calogCrypto.c index e858fe8c..c91b2fb9 100644 --- a/libs/calogCrypto.c +++ b/libs/calogCrypto.c @@ -7,9 +7,12 @@ #include "calogCrypto.h" #include "calogInternal.h" +#include #include #include +#include #include +#include #include #include @@ -21,6 +24,7 @@ static int32_t cryptoBase64EncodeNative(CalogValueT *args, int32_t argCount, Cal static int32_t cryptoBytesToHex(CalogValueT *result, const unsigned char *bytes, size_t length); static int32_t cryptoDigestHex(CalogValueT *args, int32_t argCount, CalogValueT *result, const EVP_MD *md, const char *usage); static int32_t cryptoEqualsNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); +static int32_t cryptoHashFileSha256Native(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); static int32_t cryptoHashSha1Native(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); static int32_t cryptoHashSha256Native(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); static int32_t cryptoHexDecodeNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); @@ -45,10 +49,14 @@ static const char cryptoHexDigits[] = "0123456789abcdef"; #define CRYPTO_PBKDF2_ITERATIONS_MAX 10000000 #define CRYPTO_PBKDF2_LENGTH_MIN 16 #define CRYPTO_PBKDF2_LENGTH_MAX 1024 +// cryptoHashFileSha256 reads the file this much at a time, so a file of any size is hashed in +// constant memory. +#define CRYPTO_FILE_CHUNK 65536 // Every inline native this library exposes. One table so registration is a single // checked call (calogRegisterBatch) rather than a run of calls whose status was dropped. static const CalogNativeEntryT gCryptoNatives[] = { { "cryptoHashSha256", cryptoHashSha256Native }, + { "cryptoHashFileSha256", cryptoHashFileSha256Native }, { "cryptoHashSha1", cryptoHashSha1Native }, { "cryptoHmacSha256", cryptoHmacSha256Native }, { "cryptoRandomBytes", cryptoRandomBytesNative }, @@ -248,6 +256,55 @@ static int32_t cryptoEqualsNative(CalogValueT *args, int32_t argCount, CalogValu } +// The SHA-256 of a file's contents, as hex, read a piece at a time: the digest of a file too large +// to hold as a string, which cryptoHashSha256 would need whole. +static int32_t cryptoHashFileSha256Native(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData) { + unsigned char digest[EVP_MAX_MD_SIZE]; + unsigned char *chunk; + EVP_MD_CTX *context; + FILE *file; + unsigned int digestLen; + size_t got; + bool good; + + (void)userData; + calogValueNil(result); + if (argCount != 1 || args[0].type != calogStringE) { + return calogFail(result, calogErrArgE, "cryptoHashFileSha256 expects (path)"); + } + file = fopen(args[0].as.s.bytes, "rb"); + if (file == NULL) { + return calogFail(result, calogErrNotFoundE, strerror(errno)); + } + chunk = (unsigned char *)malloc(CRYPTO_FILE_CHUNK); + context = EVP_MD_CTX_new(); + if (chunk == NULL || context == NULL) { + free(chunk); + EVP_MD_CTX_free(context); + fclose(file); + return calogFail(result, calogErrOomE, "cryptoHashFileSha256: out of memory"); + } + good = (EVP_DigestInit_ex(context, EVP_sha256(), NULL) == 1); + while (good && (got = fread(chunk, 1, CRYPTO_FILE_CHUNK, file)) > 0) { + good = (EVP_DigestUpdate(context, chunk, got) == 1); + } + if (good && ferror(file)) { + good = false; + } + digestLen = 0; + if (good) { + good = (EVP_DigestFinal_ex(context, digest, &digestLen) == 1); + } + free(chunk); + EVP_MD_CTX_free(context); + fclose(file); + if (!good) { + return calogFail(result, calogErrNotFoundE, "cryptoHashFileSha256: reading or hashing the file failed"); + } + return cryptoBytesToHex(result, digest, (size_t)digestLen); +} + + static int32_t cryptoHashSha1Native(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData) { (void)userData; return cryptoDigestHex(args, argCount, result, EVP_sha1(), "cryptoHashSha1 expects (data)"); diff --git a/libs/calogFs.c b/libs/calogFs.c index 31e5ca2b..048a132e 100644 --- a/libs/calogFs.c +++ b/libs/calogFs.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -35,9 +36,13 @@ #ifdef _WIN32 typedef struct _stat64 CalogStatT; #define calogStat _stat64 +#define calogFstat _fstat64 +#define calogSeek _lseeki64 #else typedef struct stat CalogStatT; #define calogStat stat +#define calogFstat fstat +#define calogSeek lseek #endif // fsRead's starting buffer capacity when fstat's st_size reports 0 (procfs/sysfs/FIFO-style @@ -52,7 +57,9 @@ static int32_t fsMapSet(CalogAggT *agg, const char *keyName, CalogValueT *value) static int32_t fsMkdirNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); static int32_t fsPutFile(const char *path, const char *bytes, int64_t length, bool append, CalogValueT *result); static int32_t fsReadNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); +static int32_t fsReadRange(const char *path, int64_t start, int64_t length, CalogValueT *result); static int32_t fsRemoveNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); +static int32_t fsRenameNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); static int32_t fsStatNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); static int32_t fsWriteNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); // Every inline native this library exposes. One table so registration is a single @@ -63,6 +70,7 @@ static const CalogNativeEntryT gFsNatives[] = { { "fsAppend", fsAppendNative }, { "fsExists", fsExistsNative }, { "fsRemove", fsRemoveNative }, + { "fsRename", fsRenameNative }, { "fsMkdir", fsMkdirNative }, { "fsList", fsListNative }, { "fsStat", fsStatNative }, @@ -264,10 +272,18 @@ static int32_t fsReadNative(CalogValueT *args, int32_t argCount, CalogValueT *re (void)userData; calogValueNil(result); - if (argCount != 1 || args[0].type != calogStringE) { - return calogFail(result, calogErrArgE, "fsRead expects (path)"); + if (argCount < 1 || argCount > 3 || args[0].type != calogStringE || (argCount >= 2 && args[1].type != calogIntE) || (argCount == 3 && args[2].type != calogIntE)) { + return calogFail(result, calogErrArgE, "fsRead expects (path [, offset [, length]])"); } path = args[0].as.s.bytes; + // A piece of the file: from offset, length bytes or to the end. A file too large to hold is + // read this way a piece at a time. + if (argCount >= 2) { + if (args[1].as.i < 0 || (argCount == 3 && args[2].as.i < 0)) { + return calogFail(result, calogErrRangeE, "fsRead: offset and length must not be negative"); + } + return fsReadRange(path, args[1].as.i, (argCount == 3) ? args[2].as.i : -1, result); + } fd = open(path, O_RDONLY | O_BINARY); if (fd < 0) { return fsFail(result, errno); @@ -323,6 +339,70 @@ static int32_t fsReadNative(CalogValueT *args, int32_t argCount, CalogValueT *re } +// length bytes of path from start, or to the end when length is negative; fewer when the file ends +// first, and "" from at or past the end. The buffer is the size of what will be read, never of +// what was asked for, so a bogus length cannot ask for memory the file does not justify. +static int32_t fsReadRange(const char *path, int64_t start, int64_t length, CalogValueT *result) { + CalogStatT st; + char *data; + int64_t available; + int64_t have; + ssize_t got; + int fd; + int saved; + int32_t status; + + fd = open(path, O_RDONLY | O_BINARY); + if (fd < 0) { + return fsFail(result, errno); + } + if (calogFstat(fd, &st) != 0) { + saved = errno; + close(fd); + return fsFail(result, saved); + } + available = ((int64_t)st.st_size > start) ? (int64_t)st.st_size - start : 0; + if (length < 0 || length > available) { + length = available; + } + if (length == 0) { + close(fd); + return calogValueString(result, "", 0); + } + if (calogSeek(fd, start, SEEK_SET) < 0) { + saved = errno; + close(fd); + return fsFail(result, saved); + } + data = (char *)malloc((size_t)length); + if (data == NULL) { + close(fd); + return calogFail(result, calogErrOomE, "fsRead: out of memory"); + } + have = 0; + while (have < length) { + got = read(fd, data + have, (size_t)(length - have)); + if (got < 0) { + if (errno == EINTR) { + continue; + } + saved = errno; + free(data); + close(fd); + return fsFail(result, saved); + } + if (got == 0) { + break; + } + have += (int64_t)got; + } + close(fd); + status = calogValueString(result, data, have); + free(data); + return status; +} + + static int32_t fsRemoveNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData) { (void)userData; calogValueNil(result); @@ -348,6 +428,27 @@ static int32_t fsRemoveNative(CalogValueT *args, int32_t argCount, CalogValueT * } +// Moves from to to, replacing to when it exists (Windows refuses to rename over a file, so there it +// is removed first). Within one filesystem this is how a finished file takes its place whole. +static int32_t fsRenameNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData) { + (void)userData; + calogValueNil(result); + if (argCount != 2 || args[0].type != calogStringE || args[1].type != calogStringE) { + return calogFail(result, calogErrArgE, "fsRename expects (from, to)"); + } +#ifdef _WIN32 + // Windows rename refuses an existing target, and _unlink refuses a read-only one. + if (_chmod(args[1].as.s.bytes, _S_IWRITE) == 0) { + unlink(args[1].as.s.bytes); + } +#endif + if (rename(args[0].as.s.bytes, args[1].as.s.bytes) != 0) { + return fsFail(result, errno); + } + return calogOkE; +} + + static int32_t fsStatNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData) { CalogStatT st; CalogValueT value; diff --git a/libs/calogTimer.c b/libs/calogTimer.c index 54565d92..a7309d70 100644 --- a/libs/calogTimer.c +++ b/libs/calogTimer.c @@ -65,6 +65,10 @@ static bool gThreadStarted = false; static bool gShutdown = false; static pthread_mutex_t gInitMutex = PTHREAD_MUTEX_INITIALIZER; static int32_t gRefCount = 0; +// Held by a register and by the whole of a shutdown, never by a schedule: it keeps a runtime +// registering from racing the last one's teardown, without the teardown's join holding the lock a +// timer callback's own schedule is waiting on. +static pthread_mutex_t gTeardownMutex = PTHREAD_MUTEX_INITIALIZER; static int32_t timerAfterNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); static int32_t timerCancelNative(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); @@ -90,7 +94,9 @@ static const CalogNativeEntryT gTimerNatives[] = { int32_t calogTimerRegister(CalogT *calog) { int32_t status; + pthread_mutex_lock(&gTeardownMutex); calogRegistryRetain(&gInitMutex, &gRefCount); + pthread_mutex_unlock(&gTeardownMutex); status = calogRegisterBatch(calog, gTimerNatives, (int64_t)(sizeof(gTimerNatives) / sizeof(gTimerNatives[0])), NULL); if (status != calogOkE) { return status; @@ -100,10 +106,16 @@ int32_t calogTimerRegister(CalogT *calog) { void calogTimerShutdown(void) { - // calogRegistryRelease holds gInitMutex for the whole call, including timerFreeAll below, - // so a timerSchedule that checks gRefCount under gInitMutex (see timerSchedule) can never - // observe a mid-teardown state: it either runs entirely before this or entirely after. - calogRegistryRelease(&gInitMutex, &gRefCount, timerFreeAll); + // The count drops under gInitMutex, so a timerSchedule (which checks it under the same lock) + // runs entirely before this or sees zero and fails. The teardown -- which joins the timer + // thread -- runs after gInitMutex is dropped: the thread may be inside a callback whose own + // timerAfter is waiting for gInitMutex, and holding it across the join deadlocked the three + // (AUDIT.md S34). gTeardownMutex keeps a new registration out until the teardown is done. + pthread_mutex_lock(&gTeardownMutex); + if (calogRegistryReleaseLast(&gInitMutex, &gRefCount)) { + timerFreeAll(); + } + pthread_mutex_unlock(&gTeardownMutex); } @@ -185,8 +197,8 @@ static int32_t timerFindByIdLocked(int64_t id) { } -// Invoked by calogRegistryRelease while gInitMutex is held, exactly once, when the last -// registered runtime releases the timer library. Stops the thread (if it ever started), then +// Invoked by calogTimerShutdown, under gTeardownMutex but not gInitMutex, exactly once, when the +// last registered runtime releases the timer library. Stops the thread (if it ever started), then // releases every remaining callback and frees the list. Contexts are still alive here, so a // callback release routes a finalize to its owner thread just like calogExport. static void timerFreeAll(void) { diff --git a/src/berry/berryEngine.c b/src/berry/berryEngine.c index d1b546e3..fdb097fa 100644 --- a/src/berry/berryEngine.c +++ b/src/berry/berryEngine.c @@ -20,7 +20,8 @@ const CalogEngineT calogBerryEngine = { berryExtensions, berryEngineCreate, berryEngineDestroy, - berryEngineRun + berryEngineRun, + false }; diff --git a/src/calog.h b/src/calog.h index b6833490..372d2504 100644 --- a/src/calog.h +++ b/src/calog.h @@ -137,6 +137,10 @@ typedef struct CalogEngineT { int32_t (*createInterpreter)(CalogContextT *context, void **interpOut); void (*destroyInterpreter)(void *interp); int32_t (*runSource)(void *interp, const char *source, CalogValueT *result); + // True for an engine that cannot enforce a memory cap or a wall-clock limit (s7: its loops + // bypass its begin-hook). A context with either limit is refused on it rather than opened + // unlimited. An embedder's own engine sets it false unless it has the same gap. + bool ignoresLimits; } CalogEngineT; // ---- runtime ---- @@ -220,18 +224,20 @@ CalogContextT *calogContextOpen(CalogT *calog, const CalogEngineT *engine); // // wallClockMillis -- enforced for ALL engines EXCEPT s7: Lua/JS/my-basic/mruby via a periodic VM // hook, Berry via its instruction heartbeat, Squirrel/Wren from the bytecode loop, // Tcl via its built-in time limit, Janet via an out-of-band watchdog thread. A -// script blocked in a long-running C native is not preempted (inherent). s7 ignores -// it -- its begin-hook does not fire inside its optimized loops (design.md sec 24). +// script blocked in a long-running C native is not preempted (inherent). s7 cannot +// enforce it -- its begin-hook does not fire inside its optimized loops (design.md +// sec 24) -- so a limited open or a capped script's taskSpawn on s7 is REFUSED +// (NULL / an error) rather than handed a context that ignores the limit. // memoryBytes -- enforced for the same nine engines: a per-VM allocator (Lua/JS), or a counting -// allocator / VM-loop check charged to the running context (the rest). s7 ignores -// it. Bound granularity varies (exact for Lua/JS/Berry/mruby, else allocation- or +// allocator / VM-loop check charged to the running context (the rest). Refused on s7, +// as above. Bound granularity varies (exact for Lua/JS/Berry/mruby, else allocation- or // loop- or statement-granular: a single operation may transiently overshoot). // Cooperative model: retirement is serviced after the eval returns, so a script that DELIBERATELY // catches the sandbox error and loops can pin its context thread (every engine, incl. Lua/JS -- see // design.md sec 24). A merely runaway script (no catch) is always retired. typedef struct CalogLimitsT { - int64_t memoryBytes; // 0 = unlimited (all engines but s7) - int64_t wallClockMillis; // 0 = unlimited (all engines but s7) + int64_t memoryBytes; // 0 = unlimited; non-zero is refused on s7 + int64_t wallClockMillis; // 0 = unlimited; non-zero is refused on s7 const char *const *allowList; // NULL = all natives permitted; else a NULL-terminated list of allowed names int32_t maxContexts; // 0 = unbounded; else the most contexts this sandbox may contain, counting the first } CalogLimitsT; diff --git a/src/calogInternal.h b/src/calogInternal.h index aefd3c53..484f843f 100644 --- a/src/calogInternal.h +++ b/src/calogInternal.h @@ -315,6 +315,7 @@ int32_t calogMapSetBool(CalogAggT *map, const char *key, bool flag); // reference drops. void calogRegistryRetain(pthread_mutex_t *initMutex, int32_t *refCount); void calogRegistryRelease(pthread_mutex_t *initMutex, int32_t *refCount, void (*freeAll)(void)); +bool calogRegistryReleaseLast(pthread_mutex_t *initMutex, int32_t *refCount); // ---- library shutdown hooks ---- // Each is defined in its libs/calog.c and registered with the runtime via calogAtDestroy, so diff --git a/src/context.c b/src/context.c index 59400534..e75f5473 100644 --- a/src/context.c +++ b/src/context.c @@ -726,6 +726,12 @@ static CalogContextT *contextOpenWithSandbox(CalogT *broker, const CalogEngineT int64_t index; uint32_t generation; + // A limit the engine would silently ignore is refused instead: a limited open, or a sandboxed + // script's taskSpawn, would otherwise hand back a context free of the caps it was asked for. + if (engine != NULL && engine->ignoresLimits && sandbox != NULL && sandbox->metered) { + sandboxRelease(sandbox); + return NULL; + } context = (CalogContextT *)calloc(1, sizeof(*context)); if (context == NULL) { sandboxRelease(sandbox); diff --git a/src/janet/janetEngine.c b/src/janet/janetEngine.c index efa554f7..dc88773e 100644 --- a/src/janet/janetEngine.c +++ b/src/janet/janetEngine.c @@ -20,7 +20,8 @@ const CalogEngineT calogJanetEngine = { janetExtensions, janetEngineCreate, janetEngineDestroy, - janetEngineRun + janetEngineRun, + false }; diff --git a/src/js/jsEngine.c b/src/js/jsEngine.c index 51d1f8e2..426aa4a6 100644 --- a/src/js/jsEngine.c +++ b/src/js/jsEngine.c @@ -20,7 +20,8 @@ const CalogEngineT calogJsEngine = { jsExtensions, jsEngineCreate, jsEngineDestroy, - jsEngineRun + jsEngineRun, + false }; diff --git a/src/lua/luaEngine.c b/src/lua/luaEngine.c index 323c11e4..ec3f0cc5 100644 --- a/src/lua/luaEngine.c +++ b/src/lua/luaEngine.c @@ -21,7 +21,8 @@ const CalogEngineT calogLuaEngine = { luaExtensions, luaEngineCreate, luaEngineDestroy, - luaEngineRun + luaEngineRun, + false }; diff --git a/src/mruby/mrubyEngine.c b/src/mruby/mrubyEngine.c index f17266d0..7e3bea9f 100644 --- a/src/mruby/mrubyEngine.c +++ b/src/mruby/mrubyEngine.c @@ -20,7 +20,8 @@ const CalogEngineT calogMrubyEngine = { mrubyExtensions, mrubyEngineCreate, mrubyEngineDestroy, - mrubyEngineRun + mrubyEngineRun, + false }; diff --git a/src/mybasic/mybasicEngine.c b/src/mybasic/mybasicEngine.c index 9dcd54f5..4deada11 100644 --- a/src/mybasic/mybasicEngine.c +++ b/src/mybasic/mybasicEngine.c @@ -31,7 +31,8 @@ const CalogEngineT calogMyBasicEngine = { mybasicExtensions, mybasicEngineCreate, mybasicEngineDestroy, - mybasicEngineRun + mybasicEngineRun, + false }; diff --git a/src/s7/s7Engine.c b/src/s7/s7Engine.c index e7c83003..0c40c5b0 100644 --- a/src/s7/s7Engine.c +++ b/src/s7/s7Engine.c @@ -20,7 +20,8 @@ const CalogEngineT calogS7Engine = { s7Extensions, s7EngineCreate, s7EngineDestroy, - s7EngineRun + s7EngineRun, + true }; diff --git a/src/squirrel/squirrelEngine.c b/src/squirrel/squirrelEngine.c index 74ceddfa..13d40357 100644 --- a/src/squirrel/squirrelEngine.c +++ b/src/squirrel/squirrelEngine.c @@ -20,7 +20,8 @@ const CalogEngineT calogSquirrelEngine = { squirrelExtensions, squirrelEngineCreate, squirrelEngineDestroy, - squirrelEngineRun + squirrelEngineRun, + false }; diff --git a/src/tcl/tclEngine.c b/src/tcl/tclEngine.c index aa39020d..8047ae62 100644 --- a/src/tcl/tclEngine.c +++ b/src/tcl/tclEngine.c @@ -21,7 +21,8 @@ const CalogEngineT calogTclEngine = { tclExtensions, tclEngineCreate, tclEngineDestroy, - tclEngineRun + tclEngineRun, + false }; diff --git a/src/value.c b/src/value.c index 1441422b..bfdef8be 100644 --- a/src/value.c +++ b/src/value.c @@ -771,6 +771,24 @@ void calogRegistryRelease(pthread_mutex_t *initMutex, int32_t *refCount, void (* } +// The release for a library whose teardown must not run under initMutex: its freeAll joins a +// thread that may itself be waiting for initMutex (a timer callback scheduling another timer). +// Answers whether this was the 1 -> 0 transition, so the caller tears down after the lock is +// dropped; a schedule that then takes the lock sees refcount 0 and fails instead of blocking. +bool calogRegistryReleaseLast(pthread_mutex_t *initMutex, int32_t *refCount) { + bool last; + + last = false; + pthread_mutex_lock(initMutex); + if (*refCount > 0) { + (*refCount)--; + last = (*refCount == 0); + } + pthread_mutex_unlock(initMutex); + return last; +} + + void calogRegistryRetain(pthread_mutex_t *initMutex, int32_t *refCount) { pthread_mutex_lock(initMutex); (*refCount)++; diff --git a/src/wren/wrenEngine.c b/src/wren/wrenEngine.c index 0f0f73cc..4a4a10d8 100644 --- a/src/wren/wrenEngine.c +++ b/src/wren/wrenEngine.c @@ -20,7 +20,8 @@ const CalogEngineT calogWrenEngine = { wrenExtensions, wrenEngineCreate, wrenEngineDestroy, - wrenEngineRun + wrenEngineRun, + false }; diff --git a/tests/testCrypto.c b/tests/testCrypto.c index 45616f25..1c27b0f1 100644 --- a/tests/testCrypto.c +++ b/tests/testCrypto.c @@ -14,7 +14,13 @@ #define CHECK(cond, msg) checkImpl((cond), (msg), __FILE__, __LINE__) #define PUMP_LIMIT 4000 -#define RESULT_SLOTS 24 +#define RESULT_SLOTS 27 +// Files for cryptoHashFileSha256; the large one spans several of its 64 KiB reads. +#define CRYPTO_TEST_SMALL "/tmp/calogCryptoSmall.bin" +#define CRYPTO_TEST_LARGE "/tmp/calogCryptoLarge.bin" +#define CRYPTO_TEST_LARGE_BYTES 200000 +#define CRYPTO_TEST_TEXT_(x) #x +#define CRYPTO_TEST_TEXT(x) CRYPTO_TEST_TEXT_(x) static CalogT *calog = NULL; static _Atomic int64_t results[RESULT_SLOTS]; @@ -115,6 +121,25 @@ int main(void) { atomic_store(&results[i], -1); } + // Two files for cryptoHashFileSha256: a known vector, and one larger than its read chunk. + { + FILE *file; + int32_t n; + + file = fopen(CRYPTO_TEST_SMALL, "wb"); + if (file != NULL) { + fputs("abc", file); + fclose(file); + } + file = fopen(CRYPTO_TEST_LARGE, "wb"); + if (file != NULL) { + for (n = 0; n < CRYPTO_TEST_LARGE_BYTES; n++) { + fputc('x', file); + } + fclose(file); + } + } + ctx = calogContextOpen(calog, &calogLuaEngine); calogContextEval(ctx, "report(1, #cryptoHashSha256('abc'))\n" // 64 hex chars @@ -145,6 +170,10 @@ int main(void) { "report(21, cryptoEquals('abc', 'abd') and 0 or 1)\n" "report(22, cryptoEquals('abc', 'abcd') and 0 or 1)\n" // differing lengths are unequal, not an error "report(23, cryptoEquals('a\\0b', 'a\\0b') and 1 or 0)\n" // binary-safe over embedded NULs + "report(24, cryptoHashFileSha256('" CRYPTO_TEST_SMALL "') == cryptoHashSha256('abc') and 1 or 0)\n" + "report(25, cryptoHashFileSha256('" CRYPTO_TEST_LARGE "') == cryptoHashSha256(string.rep('x', " CRYPTO_TEST_TEXT(CRYPTO_TEST_LARGE_BYTES) ")) and 1 or 0)\n" + "local missing = pcall(function() cryptoHashFileSha256('" CRYPTO_TEST_SMALL ".none') end)\n" + "report(26, missing and 0 or 1)\n" "done()"); pumpUntilDone(1); @@ -171,6 +200,11 @@ int main(void) { CHECK(atomic_load(&results[21]) == 1, "cryptoEquals is false for strings of equal length that differ"); CHECK(atomic_load(&results[22]) == 1, "cryptoEquals is false for strings of different lengths"); CHECK(atomic_load(&results[23]) == 1, "cryptoEquals is binary-safe over embedded NULs"); + CHECK(atomic_load(&results[24]) == 1, "cryptoHashFileSha256 matches the known vector"); + CHECK(atomic_load(&results[25]) == 1, "cryptoHashFileSha256 hashes a file larger than its chunk as the whole string"); + CHECK(atomic_load(&results[26]) == 1, "cryptoHashFileSha256 of a missing file raises a catchable error"); + remove(CRYPTO_TEST_SMALL); + remove(CRYPTO_TEST_LARGE); CHECK(atomic_load(&errorCount) == 0, "no uncaught errors"); calogDestroy(calog); diff --git a/tests/testFs.c b/tests/testFs.c index d2a09099..bc8ad5e9 100644 --- a/tests/testFs.c +++ b/tests/testFs.c @@ -15,7 +15,7 @@ #define CHECK(cond, msg) checkImpl((cond), (msg), __FILE__, __LINE__) #define PUMP_LIMIT 4000 -#define RESULT_SLOTS 12 +#define RESULT_SLOTS 19 static CalogT *calog = NULL; static _Atomic int64_t results[RESULT_SLOTS]; @@ -146,6 +146,19 @@ int main(void) { "fsRemove(path)\n" "report(10, fsExists(path) and 0 or 1)\n" // 1, exists went false "report(11, fsStat(path) == nil and 1 or 0)\n" // 1, stat of a missing path is nil + "fsWrite(path, '0123456789')\n" + "report(12, fsRead(path, 0, 4) == '0123' and 1 or 0)\n" // 1, a piece from the start + "report(13, fsRead(path, 6) == '6789' and 1 or 0)\n" // 1, from an offset to the end + "report(14, (fsRead(path, 8, 100) == '89' and fsRead(path, 20, 4) == '') and 1 or 0)\n" // 1, short at the end, empty past it + "local neg = pcall(function() fsRead(path, -1, 2) end)\n" + "report(15, neg and 0 or 1)\n" // 1, a negative offset is refused + "local other = dir .. '/other.bin'\n" + "fsWrite(other, 'old')\n" + "fsRename(path, other)\n" + "report(16, (fsExists(path) == false and fsRead(other) == '0123456789') and 1 or 0)\n" // 1, renamed over the old file + "local gone = pcall(function() fsRename(path, other) end)\n" + "report(17, gone and 0 or 1)\n" // 1, renaming a missing file raises + "fsRemove(other)\n" "fsRemove(dir)\n" "done()", dir); @@ -164,6 +177,12 @@ int main(void) { CHECK(atomic_load(&results[9]) == 1, "reading a missing file raises a catchable error"); CHECK(atomic_load(&results[10]) == 1, "fsRemove deletes the file (fsExists transitions to false)"); CHECK(atomic_load(&results[11]) == 1, "fsStat of a missing path returns nil"); + CHECK(atomic_load(&results[12]) == 1, "fsRead with an offset and a length reads that piece"); + CHECK(atomic_load(&results[13]) == 1, "fsRead with an offset alone reads to the end"); + CHECK(atomic_load(&results[14]) == 1, "fsRead past the end is short, then empty"); + CHECK(atomic_load(&results[15]) == 1, "fsRead refuses a negative offset"); + CHECK(atomic_load(&results[16]) == 1, "fsRename moves a file over an existing one"); + CHECK(atomic_load(&results[17]) == 1, "fsRename of a missing file raises a catchable error"); CHECK(atomic_load(&errorCount) == 0, "no uncaught errors"); calogDestroy(calog); diff --git a/tests/testHttpdLua.c b/tests/testHttpdLua.c index b82fea56..424e1220 100644 --- a/tests/testHttpdLua.c +++ b/tests/testHttpdLua.c @@ -9,6 +9,7 @@ #include "calog.h" #include "calogCrypto.h" +#include "calogFs.h" #include "calogNet.h" #include @@ -17,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -26,6 +28,14 @@ #define PORT 38910 #define HTTPS_PORT 38911 #define PUMP_LIMIT 4000 +// The spooled-body and file-response cases: a body larger than the server's read chunks, a cap just +// above it, and where the files go. +#define SPOOL_DIR "/tmp/calogHttpdSpool" +#define SPOOL_BYTES 200000 +#define SPOOL_CAP 300000 +#define FILE_PATH "/tmp/calogHttpdFile.bin" +#define HTTPD_TEST_TEXT_(x) #x +#define HTTPD_TEST_TEXT(x) HTTPD_TEST_TEXT_(x) static CalogT *calog = NULL; static _Atomic bool serving = true; @@ -43,6 +53,7 @@ static char *loadScript(const char *path); static int32_t nativeKeepServing(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); static int32_t nativeReady(CalogValueT *args, int32_t argCount, CalogValueT *result, void *userData); static bool readResponse(int fd, int *statusOut, char *body, size_t bodyCap); +static bool spoolDirEmpty(void); static void checkImpl(bool condition, const char *message, int32_t line) { @@ -129,6 +140,27 @@ static bool readResponse(int fd, int *statusOut, char *body, size_t bodyCap) { } +// Whether the spool folder holds no files (a spooled body's file is removed after its handler). +static bool spoolDirEmpty(void) { + DIR *dir; + struct dirent *entry; + bool empty; + + dir = opendir(SPOOL_DIR); + if (dir == NULL) { + return true; + } + empty = true; + while ((entry = readdir(dir)) != NULL) { + if (strcmp(entry->d_name, ".") != 0 && strcmp(entry->d_name, "..") != 0) { + empty = false; + } + } + closedir(dir); + return empty; +} + + static char *loadScript(const char *path) { FILE *f; char *buffer; @@ -273,6 +305,7 @@ int main(void) { } calogNetRegister(calog); calogCryptoRegister(calog); + calogFsRegister(calog); calogRegisterInline(calog, "keepServing", nativeKeepServing, NULL); calogRegisterInline(calog, "ready", nativeReady, NULL); @@ -288,6 +321,9 @@ int main(void) { "s:route('GET', '/hi', function(req) return 'hello ' .. req.path end)\n" "s:route('GET', '/made', function(req) return { status = 201, body = 'created' } end)\n" "s:route('GET', '/boom', function(req) return nil .. 'x' end)\n" /* runtime error -> 500 */ + "s:route('POST', '/up', function(req) return req.bodySize .. ' ' .. ((cryptoHashFileSha256(req.bodyFile) == cryptoHashSha256(string.rep('x', " HTTPD_TEST_TEXT(SPOOL_BYTES) "))) and 'ok' or 'bad') end, { spool = { dir = '" SPOOL_DIR "', max = " HTTPD_TEST_TEXT(SPOOL_CAP) " } })\n" + "s:route('POST', '/small', function(req) return tostring(#req.body) end)\n" + "s:route('GET', '/file', function(req) return { file = { path = '" FILE_PATH "', offset = 2, length = 5 } } end)\n" "s:websocket('/ws', function(msg) return 'echo: ' .. msg.message end)\n" "s:serve(38910, { keep = keepServing, onReady = ready })\n"; source = (char *)malloc(strlen(script) + strlen(setup) + 64); @@ -297,6 +333,17 @@ int main(void) { sprintf(source, "httpd = (function()\n%s\nend)()\n%s", script, setup); free(script); + // The file the file-response route serves a piece of. + { + FILE *file; + + file = fopen(FILE_PATH, "wb"); + if (file != NULL) { + fputs("0123456789", file); + fclose(file); + } + } + lua = calogContextOpen(calog, &calogLuaEngine); calogContextEval(lua, source); free(source); @@ -336,6 +383,59 @@ int main(void) { close(fd); } + // --- A spooled body: written to a file the handler reads, then removed; one over the route's cap + // answered 413 unread; an ordinary body in memory; a response streamed from a piece of a file --- + fd = clientConnect(PORT); + if (fd >= 0) { + char head[128]; + char *payload; + size_t sent; + ssize_t n; + + snprintf(head, sizeof(head), "POST /up HTTP/1.1\r\nHost: x\r\nContent-Length: %d\r\nConnection: close\r\n\r\n", SPOOL_BYTES); + payload = (char *)malloc(SPOOL_BYTES); + if (payload != NULL) { + memset(payload, 'x', SPOOL_BYTES); + send(fd, head, strlen(head), 0); + sent = 0; + while (sent < SPOOL_BYTES) { + n = send(fd, payload + sent, SPOOL_BYTES - sent, 0); + if (n <= 0) { + break; + } + sent += (size_t)n; + } + free(payload); + CHECK(readResponse(fd, &status, body, sizeof(body)) && status == 200 && strcmp(body, HTTPD_TEST_TEXT(SPOOL_BYTES) " ok") == 0, + "a spooled body reaches the handler as a file of the right size and contents"); + } + close(fd); + CHECK(spoolDirEmpty(), "the spooled body's file is removed after the handler"); + } + fd = clientConnect(PORT); + if (fd >= 0) { + char head[128]; + + snprintf(head, sizeof(head), "POST /up HTTP/1.1\r\nHost: x\r\nContent-Length: %d\r\nConnection: close\r\n\r\n", SPOOL_CAP + 1); + send(fd, head, strlen(head), 0); + CHECK(readResponse(fd, &status, body, sizeof(body)) && status == 413, "a body over the route's cap is answered 413 before it is sent"); + close(fd); + } + fd = clientConnect(PORT); + if (fd >= 0) { + const char *req = "POST /small HTTP/1.1\r\nHost: x\r\nContent-Length: 10\r\nConnection: close\r\n\r\n0123456789"; + send(fd, req, strlen(req), 0); + CHECK(readResponse(fd, &status, body, sizeof(body)) && status == 200 && strcmp(body, "10") == 0, "an ordinary body arrives whole in memory"); + close(fd); + } + fd = clientConnect(PORT); + if (fd >= 0) { + const char *req = "GET /file HTTP/1.1\r\nHost: x\r\nConnection: close\r\n\r\n"; + send(fd, req, strlen(req), 0); + CHECK(readResponse(fd, &status, body, sizeof(body)) && status == 200 && strcmp(body, "23456") == 0, "a file response sends the piece it names"); + close(fd); + } + // --- HTTP/1.1 keep-alive: two requests on ONE connection --- fd = clientConnect(PORT); if (fd >= 0) { @@ -554,6 +654,8 @@ int main(void) { } calogDestroy(calog); + remove(FILE_PATH); + rmdir(SPOOL_DIR); printf("\n%d checks, %d failed\n", testsRun, testsFailed); fflush(stdout); return testsFailed == 0 ? 0 : 1; diff --git a/tests/testSandbox.c b/tests/testSandbox.c index 3cb21ecb..4d7597c9 100644 --- a/tests/testSandbox.c +++ b/tests/testSandbox.c @@ -38,6 +38,15 @@ static void runLimited(const CalogEngineT *engine, const char *source, const static void runLimitedSettle(const CalogEngineT *engine, const char *source, const CalogLimitsT *limits); +// s7 allocates permanent strings once per process and never frees them (an s7 design +// characteristic, not a calog defect); the s7 checks below start its interpreter. Suppress exactly +// that allocation site so the leak check stays meaningful. LSan calls this weak hook automatically. +const char *__lsan_default_suppressions(void); +const char *__lsan_default_suppressions(void) { + return "leak:make_permanent_string\n"; +} + + static void checkImpl(bool condition, const char *message, int32_t line) { testsRun++; if (!condition) { @@ -380,6 +389,33 @@ int main(void) { "local b = pcall(taskSpawn, 'lua', 'reached()') " "report(a and not b)", &spawnLimits); CHECK(atomic_load(&reportValue) == 1, "spawn: maxContexts refuses the spawn past the bound"); + + // (e) An engine that cannot enforce a cap (s7) is refused a capped context rather than + // handed one that silently ignores it, whether the host asks directly or a capped script + // spawns a task on it. An allow-list alone is still fine: s7 enforces that. + { + CalogContextT *s7Ctx; + + s7Ctx = calogContextOpenLimited(calog, &calogS7Engine, &memLimits); + CHECK(s7Ctx == NULL, "s7: a memory-capped context is refused"); + s7Ctx = calogContextOpenLimited(calog, &calogS7Engine, &timeLimits); + CHECK(s7Ctx == NULL, "s7: a time-limited context is refused"); + s7Ctx = calogContextOpenLimited(calog, &calogS7Engine, &allowLimits); + CHECK(s7Ctx != NULL, "s7: an allow-list-only context still opens"); + if (s7Ctx != NULL) { + calogContextClose(s7Ctx); + } + } + memset(&spawnLimits, 0, sizeof(spawnLimits)); + spawnLimits.allowList = spawnAllow; + resetFlags(); + runLimitedSettle(&calogLuaEngine, "local ok = pcall(taskSpawn, 's7', '(reached)') report(ok)", &spawnLimits); + CHECK(atomic_load(&reportValue) == 1 && atomic_load(&reachedFlag), "spawn: an allow-list-only script spawns an s7 child"); + spawnLimits.memoryBytes = 8 * 1024 * 1024; + resetFlags(); + runLimitedSettle(&calogLuaEngine, "local ok = pcall(taskSpawn, 's7', '(reached)') report(ok)", &spawnLimits); + CHECK(atomic_load(&reportValue) == 0, "spawn: a capped script cannot spawn an s7 child"); + CHECK(!atomic_load(&reachedFlag), "spawn: the refused s7 child never ran"); } calogDestroy(calog); diff --git a/tests/testTeardown.c b/tests/testTeardown.c index 46711dbc..f690ab93 100644 --- a/tests/testTeardown.c +++ b/tests/testTeardown.c @@ -70,6 +70,7 @@ static void testCrossEngineValueOutlivesOwner(void); static void testReclaimUnderConcurrentDrops(void); static void testDropInsideTheReclaimWindow(void); static void testDestroyJoinsWithCallInFlight(void); +static void testDestroyJoinsWithRearmInFlight(void); static void testGuardsAfterShutdown(void); static void testHeldCallableSurvivesTeardown(const char *what, const char *source); static void testOwnerDiesBeforeTheRuntime(const char *what, const char *source, bool expectError); @@ -363,6 +364,50 @@ static void testDestroyJoinsWithCallInFlight(void) { } +// calogDestroy must not hang when a timer callback re-arms itself while the timer library shuts down. +// +// The other leg of the cycle above, through the timer library's own lock rather than the host: the +// shutdown held gInitMutex while it joined the timer thread, the timer thread waited on the context +// running its callback, and that callback's timerAfter waited on gInitMutex (AUDIT.md S34). A +// callback that schedules another timer is ordinary, and it hung 10 runs in 10 before the fix. The +// busy loop widens the window the callback spends in flight; without it the race is rarely hit. +static void testDestroyJoinsWithRearmInFlight(void) { + CalogContextT *ctx; + pthread_t watchdog; + struct timespec tick = { 0, PUMP_INTERVAL_NS }; + int32_t index; + + calog = calogCreate(); + if (calog == NULL) { + CHECK(false, "destroy-with-rearm-in-flight: runtime create failed"); + return; + } + calogSetErrorHandler(calog, onError, NULL); + calogTimerRegister(calog); + + ctx = calogContextOpen(calog, &calogLuaEngine); + if (ctx == NULL) { + CHECK(false, "destroy-with-rearm-in-flight: context open failed"); + calogDestroy(calog); + return; + } + calogContextEval(ctx, "timerEvery(1, function() local x = 0 for i = 1, 300000 do x = x + i end timerAfter(100000, function() end) end)"); + for (index = 0; index < PUMP_LIMIT; index++) { + calogPump(calog); + nanosleep(&tick, NULL); + } + atomic_store(&destroyDone, false); + if (pthread_create(&watchdog, NULL, destroyWatchdogThread, NULL) != 0) { + CHECK(false, "destroy-with-rearm-in-flight: could not start the watchdog"); + } + calogDestroy(calog); + atomic_store(&destroyDone, true); + pthread_join(watchdog, NULL); + CHECK(true, "destroy-with-rearm-in-flight: calogDestroy returned while a callback was scheduling"); + printf(" destroy with a timer callback re-arming itself\n"); +} + + static void testGuardsAfterShutdown(void) { CalogContextT *ctx; @@ -609,6 +654,7 @@ int main(void) { testReclaimUnderConcurrentDrops(); testDropInsideTheReclaimWindow(); testDestroyJoinsWithCallInFlight(); + testDestroyJoinsWithRearmInFlight(); testGuardsAfterShutdown(); printf("\n%d checks, %d failed\n", testsRun, testsFailed); diff --git a/tools/crossMacFull.sh b/tools/crossMacFull.sh index 1bf58614..680a970f 100755 --- a/tools/crossMacFull.sh +++ b/tools/crossMacFull.sh @@ -27,9 +27,9 @@ ZIG=${ZIG:-${CALOG_ZIG:-/home/scott/zig/current/zig}} export CALOG_ZIG="$ZIG" # the build/cross/bin wrappers resolve zig via $CALOG_ZIG AR="$R/build/cross/bin/zar" -# macOS Keychain trust for the HTTPS client (calogHttp httpLoadMacRoots). It needs the real Apple SDK -# framework headers (Security, CoreFoundation, libDER), which zig does NOT ship. When an SDK is -# present, calogHttp is compiled with -DCALOG_MAC_KEYCHAIN_TRUST against those headers and linked +# macOS Keychain trust for the HTTPS client and the TLS transport (calogTrust, shared by calogHttp and +# calogNet). It needs the real Apple SDK framework headers (Security, CoreFoundation, libDER), which +# zig does NOT ship. When an SDK is present, calogTrust is compiled with -DCALOG_MAC_KEYCHAIN_TRUST against those headers and linked # against the minimal tools/macStubs/*.tbd (zig's Mach-O linker segfaults on the real multi-target # SDK .tbd, so we link tiny hand-written stubs that export only the few symbols we call; the binary # still imports the real system frameworks by install-name at runtime). Without an SDK the build @@ -167,7 +167,8 @@ build_arch(){ cc calogDbFull libs/calogDb.c -Ivendor/sqlite -Ivendor/postgres/src/interfaces/libpq -Ivendor/postgres/src/include -I"$M/postgres-build/src/include" -Ivendor/mariadb/include -I"$M/mariadb/include" -DCALOG_WITH_SQLITE -DCALOG_WITH_PG -DCALOG_WITH_MYSQL cc calogExport libs/calogExport.c cc calogFs libs/calogFs.c - cc calogHttp libs/calogHttp.c -I"$M/openssl-src/include" $MACTRUST + cc calogHttp libs/calogHttp.c -I"$M/openssl-src/include" + cc calogTrust libs/calogTrust.c -I"$M/openssl-src/include" $MACTRUST cc calogJson libs/calogJson.c cc calogKv libs/calogKv.c cc calogNet libs/calogNet.c -Ivendor/enet/include -I"$M/openssl-src/include" diff --git a/tools/crossWinFull.sh b/tools/crossWinFull.sh index c7f5cf03..2019bfbd 100644 --- a/tools/crossWinFull.sh +++ b/tools/crossWinFull.sh @@ -89,6 +89,7 @@ cc calogDbFull libs/calogDb.c $BASE -Ivendor/sqlite -I"$W/postgres/include" -Ive cc calogExport libs/calogExport.c $BASE cc calogFs libs/calogFs.c $BASE cc calogHttp libs/calogHttp.c $BASE -I"$W/openssl-src/include" +cc calogTrust libs/calogTrust.c $BASE -I"$W/openssl-src/include" cc calogJson libs/calogJson.c $BASE cc calogKv libs/calogKv.c $BASE cc calogNet libs/calogNet.c $BASE -Ivendor/enet/include -I"$W/openssl-src/include"