calog/PORTING.md

# Porting calog

calog targets **Linux, macOS, and Windows**. Its core -- values, broker, the actor/threading
model, and every engine adapter -- is portable C11. The only operating-system-specific surface is
small and centralized:

- **Threads**: POSIX `pthreads`. Native on Linux and macOS; on Windows via mingw-w64's `winpthreads`.
- **Sockets + DNS**: BSD sockets + `getaddrinfo`, used only by the net/ssh/http libraries. BSD
  sockets on Linux/macOS, Winsock on Windows. All of this is behind `src/calogPlatform.h`.

There is no `epoll`/`eventfd`/`kqueue`, no `fork`/`exec`, and no `dlopen` in calog's own code (the
actor model uses blocking sockets on per-context threads), so a port is a thin shim, not a rewrite.

## The platform abstraction: `src/calogPlatform.h`

The net/ssh/http libraries use these instead of raw socket calls, so the same source compiles on all
three platforms:

| Abstraction | POSIX (Linux/macOS) | Windows |
|---|---|---|
| `CalogSocketT` | `int` | `SOCKET` |
| `CALOG_INVALID_SOCKET` | `-1` | `INVALID_SOCKET` |
| `calogSockClose(fd)` | `close` | `closesocket` |
| `calogSockErrno()` / `calogSockErrStr()` | `errno` / `strerror` | `WSAGetLastError` / `FormatMessage` |
| `calogSockSetNonblock(fd, on)` | `fcntl(O_NONBLOCK)` | `ioctlsocket(FIONBIO)` |
| `calogSockInProgress(err)` | `EINPROGRESS` | `WSAEWOULDBLOCK`/`WSAEINPROGRESS` |
| `calogPoll(...)` | `poll` | `WSAPoll` |
| `calogPlatformNetInit()` / `Shutdown()` | no-op | `WSAStartup` / `WSACleanup` |
| `CALOG_MSG_NOSIGNAL` | `MSG_NOSIGNAL` | `0` (Windows has no `SIGPIPE`) |

A socket `fd` is unsigned on Windows, so the code compares against `CALOG_INVALID_SOCKET` (never
`fd < 0`). The abstraction is behavior-identical on POSIX (it maps to the same names), so the Linux
and macOS builds are unchanged by it.

## Build matrix

| Target | Command | Runtime dependencies | DNS |
|---|---|---|---|
| Linux dev | `make` | glibc, libstdc++, libgcc, ASan/UBSan (dev only) | yes |
| Linux release | `make release` | **glibc only** (libstdc++/libgcc linked static) | yes |
| Linux static | `make static` on Alpine/musl | **none** (fully static) | yes (musl resolver) |
| Linux static | `make static` on glibc | none, but see caveat | **no** (glibc NSS needs dlopen) |
| macOS | `make` (Darwin auto-detected) | libSystem (always present) | yes |
| Windows | mingw-w64 build | core OS DLLs (`kernel32`, `ws2_32`) | yes |

The Makefile auto-detects the platform (`PLATFORM` = `linux`/`darwin`/`windows`/`posix`) and adjusts
the link line: `DLLIB` is `-ldl` on Linux (empty on macOS, where `dlopen` is in libSystem) and
`SOCKETLIBS` is `-lws2_32` on Windows.

### Feature and verification parity

The **full `calog` CLI -- all ten engines and every capability library** (net/tls/http-https/db
[SQLite+PostgreSQL+MariaDB]/archive/regex/ssh/xml + the core libs) -- builds for all three operating
systems. What differs across ports is not features but how thoroughly each is verified, and the scope
of the fully-static route.

| Port | Build path | Full CLI (10 engines + all libs) | Verified |
|---|---|---|---|
| Linux glibc (dev / release) | `make` / `make release` | yes | **runtime** (the test suite runs) |
| Windows x86_64 (PE) | `tools/crossWinFull.sh` (zig cross) | yes | build + link only |
| macOS x86_64 | `tools/crossMacFull.sh mac-x64` (zig cross) | yes | build + link only |
| macOS arm64 | `tools/crossMacFull.sh mac-arm64` (zig cross) | yes | build + link only |
| Linux static musl (Alpine) | `make static` on Alpine | no -- Lua-only demo (`examples/staticDemo.c`) | runtime (per component) |
| Linux static glibc | `make static` on glibc | no -- Lua-only demo | build only |

So engine and library parity across Linux, macOS, and Windows is **complete**. The one open gap is
**run-verification on Windows and macOS**: both are cross-built and linked with a single zig toolchain
but have not yet been executed on the target OS (a real Windows/wine or Mac run is the remaining step).
The fully-static route is a deliberately narrower, minimal-dependency demo (Lua only, not the full
CLI), and static **glibc loses DNS** (its NSS resolver needs `dlopen`, which a static binary cannot do)
-- so Alpine/musl is the path to a zero-dependency binary that still resolves names.

## Linux

- **`make`** -- the sanitized development build (ASan + UBSan + strict warnings).
- **`make release`** -- the distribution build: sanitizers off, `-static-libstdc++ -static-libgcc`,
  glibc dynamic. `ldd` collapses to just `libc`/`libm` + the loader. Because glibc stays dynamic,
  `getaddrinfo` can still `dlopen` its NSS modules, so **DNS works**. Runs on any mainstream
  glibc distro with nothing to install.
- **`make static` on Alpine (musl)** -- a fully static binary with **zero** runtime dependencies and
  **working DNS** (musl has a built-in resolver, no NSS/dlopen). This is the minimal-dependency
  artifact. Because calog builds every dependency from source, and Alpine's default toolchain is
  musl, no per-dependency changes are needed -- build it in an Alpine container.
- **`make static` on glibc** -- fully static, but name resolution and `dlopen`-based Lua C modules
  do not work (glibc NSS requires runtime `dlopen`). Connect by IP, or use the musl route above.

## macOS (Darwin)

The C is already POSIX-compatible (pthreads, BSD sockets, `getaddrinfo` are all native), so the port
is build configuration, not code -- confirmed by `tools/crossBuild.sh`, which links valid Mach-O
executables for **both x86_64 (Intel) and arm64 (Apple Silicon)** using zig's bundled `libSystem`
stub (calog only needs `libSystem` -- libc/pthreads/sockets -- so no Apple SDK/frameworks are
required for the core). Apple does not support statically linking libc, but `libSystem` is part of
the OS and always present, so "static everything except libSystem" is the natural minimal-deps build.
Each vendored dependency builds its own way and needs a macOS configuration:

- OpenSSL: `./Configure darwin64-<arch>-cc`
- Tcl: its `configure` detects Darwin
- mruby / Lua / SQLite / ENet / QuickJS / Squirrel / Berry / s7 / Wren: compile as-is with clang
- libssh2 / MariaDB: CMake with the macOS toolchain

Link against dynamic `libSystem` (the default); `-ldl` is dropped automatically (`DLLIB` is empty on
Darwin) and the C++ runtime is `libc++` (`CXXLIB = -lc++`). SIGPIPE is ignored at CLI startup, since
macOS `send()` has no `MSG_NOSIGNAL`. ENet's `unix.c` backend is correct on macOS (BSD sockets).

## Windows (mingw-w64)

calog's pthread code compiles unchanged on Windows against **winpthreads** (POSIX threads over the
Win32 API), which is **vendored** at `vendor/winpthreads/` and built from source like every other
dependency (see its `NOTICE.calog`). `src/calogPlatform.h` handles the Winsock differences. Building
uses a **mingw-w64** toolchain (not MSVC) for its POSIX-ish environment; `tools/crossBuild.sh` cross-
compiles from Linux with a single zig toolchain and is the tested path. Remaining work is build
configuration:

- Build each vendored dependency for mingw (OpenSSL `mingw64`, Tcl's `win/` makefile or configure,
  libssh2/MariaDB via CMake with the mingw toolchain, mruby with a mingw build_config).
- ENet selects its `win32.c` backend automatically (`ENETBACKEND`); the link adds `-lws2_32 -lwinmm`
  (handled by `SOCKETLIBS`).
- Static-link the runtime (`-static -static-libgcc -static-libstdc++`) so the binary depends only on
  core Windows DLLs, which are always present. DNS works through Winsock's `getaddrinfo`.

Remaining code review for Windows: `calogFs.c` and `calogHttp.c` use POSIX headers (`dirent.h`,
`sys/stat.h`, `sys/time.h`) that mingw-w64 mostly provides, but path handling (`/` vs `\`) and a few
calls may need adjustment; `calogSsh.c` already guards `sys/select.h` (Winsock supplies
`fd_set`/`select`), and `httpSetTimeouts` already branches on the Winsock `SO_RCVTIMEO` DWORD form.

## Cross-compiling from Linux with zig (`tools/crossBuild.sh`)

A single [zig](https://ziglang.org/download/) toolchain cross-compiles C and C++ to musl-static and
Windows without installing per-target toolchains, so the ports can be exercised on a plain Linux box:

```
ZIG=/path/to/zig ./tools/crossBuild.sh
```

It builds a representative set -- core + one engine of each kind (Lua, QuickJS, Squirrel/C++,
my-basic) + the socket library (`calogNet`) -- because the crypto/db/http/ssh libraries pull in
OpenSSL/MariaDB/PostgreSQL/libssh2, each needing its own per-target build. What it does:

- **musl**: fully static Linux binaries. These **run on the build host**, so the script builds AND
  runs them -- verifying the core, all four engine kinds, C++, and the socket abstraction on musl.
- **Windows**: builds `vendor/winpthreads` into `libwinpthreads.a`, then links real `.exe`s
  (including the Winsock + ENet-win32 socket path). Verified as valid PE executables; running them
  needs wine.

Verified results (this repo): musl -- Lua, QuickJS, my-basic, Squirrel, and `testNet` all build and
**pass** (fully static); Windows -- `testEngineLua.exe` and `testNet.exe` build against vendored
winpthreads. Since the Windows binaries are the *same source* that passes on musl, the build is strong
evidence; full run-verification needs a Windows host or wine.

## Cross-compiling the archive stack with zig (`tools/crossArchive.sh`)

The compression/archive library (libarchive + its five codecs: zlib, bzip2, lz4, zstd, xz/liblzma)
is the one vendored stack that leans on CMake, so it gets its own cross-build script. Same idea as
`crossBuild.sh`, same single zig toolchain:

```
ZIG=/path/to/zig ./tools/crossArchive.sh
```

It builds all five codecs plus libarchive for **musl** (fully static, RUN on the host) and **Windows
x64** (a `testArchive.exe` verified as a PE), driving each through calog's real `testArchive`
(compress/decompress round-trips through every codec). CMake cross-compilation with zig works by
wrapping `zig cc` as a single-binary compiler (`build/cross/bin/zcc-*`) plus a toolchain file. The
key asymmetry:

- **musl toolchain does NOT set `CMAKE_SYSTEM_NAME`.** A musl-static binary runs on this Linux host,
  so the build stays "native" (`CMAKE_CROSSCOMPILING` false) and CMake's `try_run` feature-detection
  executes normally. zig supplies its own musl sysroot, so no host headers leak in.
- **Windows toolchain sets `CMAKE_SYSTEM_NAME Windows`** -> cross mode. libarchive's single
  `CHECK_C_SOURCE_RUNS` is already guarded on `CMAKE_CROSSCOMPILING` (it degrades to a warning) and
  xz has no run-checks, so nothing needs pre-seeding.

Two target-specific gotchas the script handles:

- **zlib on musl** needs `-DHAVE_UNISTD_H` so it includes `<unistd.h>` for `lseek`. zlib's
  `./configure` normally sets this; the object-rule build (no configure) must pass it, and musl --
  unlike glibc -- does not declare `lseek` without the include. Only zlib is affected.
- **libarchive on Windows** pulls in system libraries the POSIX build does not: CNG crypto
  (`-lbcrypt`) for its AES/PBKDF2, and XmlLite + OLE (`-lxmllite -lole32`) for XAR, which libarchive
  auto-substitutes for the libxml2/expat we disable. The link adds
  `-lbcrypt -lxmllite -lole32 -ladvapi32 -lcrypt32 -lshlwapi`.

Everything lands under `build/cross/` (gitignored); the vendored sources and the native
`vendor/*/build` / `_cmk` trees the normal `make` uses are never touched.

## Status

- **Platform abstraction** (`calogPlatform.h`) and the net/ssh/http conversion: **done**, verified
  behavior-identical on Linux (`make test` 30/30, socket + HTTP loopback smoke, ASan- and TSan-clean).
- **Linux release** target: **done and verified** -- `ldd bin/calog` shows only `libc`/`libm` + loader.
- **Linux musl-static**: **run-verified** via `tools/crossBuild.sh` (zig) -- core + Lua + QuickJS +
  my-basic + Squirrel + `testNet` all build fully static and pass on the host. Also works via
  `make static` in an Alpine container (musl default toolchain).
- **Windows**: **build-verified** via `tools/crossBuild.sh` -- `testEngineLua.exe` and `testNet.exe`
  (core + engine + Winsock socket abstraction) link against vendored `winpthreads` into real PE
  executables. Run-verification needs a Windows host or wine; the binaries are the same source that
  passes on musl.
- **Build system**: OS-aware (`PLATFORM`, `DLLIB`, `CXXLIB`, `SOCKETLIBS`, `ENETBACKEND`, vendored
  `winpthreads`); the Linux values are identical to before, so the dev build is unchanged.
- **macOS**: **build-verified** via `tools/crossBuild.sh` -- core + Lua + `testNet` link into valid
  Mach-O executables for both **x86_64 and arm64** (zig's libSystem stub; no Apple SDK needed for
  calog's libc surface). Run-verification needs a Mac. The C code and Makefile hooks are all in place
  (`libc++`, no `-ldl`, SIGPIPE, native pthreads + BSD sockets).
- **Archive stack** (libarchive + zlib/bzip2/lz4/zstd/xz): **musl run-verified, Windows
  build-verified** via `tools/crossArchive.sh` (zig). The musl `testArchive` is fully static and
  passes all 22 checks on the host -- real compress/decompress through every codec **and an xar
  read+write round-trip**: the musl target now cross-builds OpenSSL (xar's MD5/SHA1) and libxml2 (its
  XML TOC), with iconv coming from musl's own libc, so its libarchive has full xar. The Windows build
  is a valid `testArchive.exe` (PE32+) linked against vendored winpthreads + the CNG/XmlLite system
  libs. This was the one CMake-heavy vendored stack, so it is the strongest cross-build evidence
  after the core. On **Windows**, xar needs no vendored XML/crypto/iconv at all: libarchive
  auto-uses the system **XmlLite** (its TOC) + **CNG/bcrypt** (its MD5/SHA1), both in the mingw
  sysroot, so the Windows `testArchive.exe` already has full xar. One musl gotcha worth noting:
  libarchive's `FIND_PATH(iconv.h)` returns the host `/usr/include` (glibc), whose headers break the
  musl compile, so the script pins `ICONV_INCLUDE_DIR` to an empty dir and lets zig cc resolve musl's
  own `iconv.h` implicitly.
- **Full Windows CLI cross-built (build-verified).** Every heavy vendored dependency now
  cross-compiles for `x86_64-windows-gnu` via zig -- OpenSSL, libxml2, PCRE2, libssh2, MariaDB, and
  the three hard ones **PostgreSQL/libpq, Tcl, mruby** (libpq needed a patched out-of-tree copy;
  see `tools/crossWinFull.sh`). The complete `calog.exe` links **all 10 engines + all 20 libraries**
  into a valid PE32+ that imports only Windows system DLLs (no vendored DLLs). Behavior-neutral
  `#ifdef _WIN32` guards in calog's own source cover the small OS differences -- `calogFs` (mingw
  1-arg `mkdir`, `O_BINARY`), `calogHttp` (no `SIGPIPE`), `calogInternal` (a `memmem` fallback),
  `calogPlatform` (`__declspec(thread)` -> `_Thread_local`, which also fixes a latent Windows
  thread-safety bug clang had been silently dropping), and `calogNet` (`SO_EXCLUSIVEADDRUSE` for
  listeners instead of the hijack-prone `SO_REUSEADDR`). Two libraries have a full Windows backend
  rather than a POSIX one: **`procRun`** spawns with `CreateProcess` + pipes (stdin fed on a helper
  thread while stdout/stderr drain, since anonymous pipes have no `poll`), and the **HTTPS client**
  reads trust anchors from the Windows `ROOT`/`CA` system certificate stores via CryptoAPI
  (`CertOpenSystemStore`) instead of a Unix CA-bundle path. All are behavior-neutral off Windows; the
  Linux suite stays green. Run-verification still needs a Windows host or wine.
- **Full macOS CLI cross-built for both archs (build-verified).** Every heavy vendored dependency
  now cross-compiles for `x86_64-apple-darwin` and `arm64-apple-darwin` via zig, whose libSystem
  stub supplies libc / pthreads / BSD sockets / getaddrinfo / dlopen / iconv -- so no Apple SDK is
  needed for calog's OS surface (see `tools/crossMacFull.sh`). The complete `calog` links **all 10
  engines + all 20 libraries** into a valid Mach-O per arch (`build/cross/mac-x64/calog` x86_64,
  `build/cross/mac-arm64/calog` arm64), importing only libSystem -- no vendored dylibs, no `-ldl`.
  Two additive `#if defined(__APPLE__)` guards are in calog's own source: `calogTimer` waits with a
  **relative** `pthread_cond_timedwait_relative_np` timeout on Darwin, because macOS pthreads have no
  `pthread_condattr_setclock` to pin the condvar to `CLOCK_MONOTONIC` (the relative wait preserves the
  monotonic scheduling intent across wall-clock changes); and `calogHttp`'s HTTPS client reads trust
  anchors from the **macOS keychain** via the Security framework (`SecTrustCopyAnchorCertificates`).
  Both are behavior-neutral off Darwin, and the Linux suite stays green. Run-verification still needs a
  Mac.
- **macOS Keychain HTTPS trust needs the Apple SDK (optional).** calog's OS surface (libc / pthreads /
  BSD sockets / getaddrinfo / dlopen / iconv) is all in zig's libSystem stub, so the *runner* needs no
  SDK. But the keychain trust path uses the **Security** + **CoreFoundation** frameworks, whose headers
  zig does not ship. When an SDK is present, `tools/crossMacFull.sh` compiles `calogHttp` with
  `-DCALOG_MAC_KEYCHAIN_TRUST` against the SDK framework headers; without one it omits keychain trust
  and HTTPS falls back to `SSL_CERT_FILE` / `CALOG_CA_BUNDLE`. Point `CALOG_MACSDK` at a `MacOSX*.sdk`
  directory to enable it. **Linker gotcha:** zig 0.16's Mach-O linker *segfaults* parsing the real
  multi-target SDK `.tbd` files, so calog links against hand-written minimal stubs
  (`tools/macStubs/{CoreFoundation,Security}.tbd`) that export only the handful of symbols the keychain
  code calls -- the binary still imports the real system frameworks by install-name at runtime. Both
  arch binaries are valid Mach-O importing `Security.framework` + `CoreFoundation.framework`
  (build-verified; run-verification still needs a Mac).
- **Reproducible dependency builds (`tools/crossDeps.sh`).** The full-CLI scripts above consume
  heavy pinned deps under `build/cross/<target>/`; `crossDeps.sh` rebuilds them from `vendor/` source
  with a single zig toolchain, so the cross build is reproducible from a clean checkout rather than
  depending on ad-hoc pins. Usage: `[ZIG=/path/to/zig] ./tools/crossDeps.sh <win|mac-x64|mac-arm64>
  [dep ...]` (it regenerates the zig wrappers + CMake toolchain file first, so nothing under
  `build/cross/` need pre-exist). It builds, in dependency order: the codecs (zlib/bzip2/lz4/zstd),
  xz, OpenSSL (mac also gets a BSD-`ar` **repack**, since zig's Mach-O linker cannot read OpenSSL's
  GNU-`ar` archives), libxml2, PCRE2, libssh2, MariaDB, PostgreSQL/libpq (the patched out-of-tree
  copy: mingw `.obj`->`.o`, a real `ar` rule for `libpq.a`, and `pg_pthread_*` symbol localization on
  Windows; libSystem pthreads and no patch on mac), Tcl, mruby, winpthreads (Windows only), and
  libarchive (mac = xar via the pinned OpenSSL+libxml2; Windows = codecs only, xar via system
  XmlLite/CNG). **VERIFIED build-clean from vendor source for ALL THREE targets, all 12 deps** (`OUT=`
  builds into a scratch dir without clobbering the pins): `mac-x64` and `mac-arm64` each build the full
  set in one clean invocation (OpenSSL confirmed Mach-O + `!<arch>` BSD repack); `win` builds the full
  set including the mingw-only bits -- winpthreads, the patched libpq (`.obj`->`.o`, real `ar` rule,
  `pg_pthread_*` localization, plus the Windows syslibs `-lws2_32 -lcrypt32 -lsecur32 -lbcrypt ...` so
  configure's `-lcrypto` link test passes), and the two `.rc` resources (libxml2's + libpq's
  `win32ver.rc`) compiled via a `zig rc` **windres shim** that handles both CMake's positional args and
  PostgreSQL's `-i/-o/--include-dir` form. The win CMake toolchain sets `CMAKE_RC_COMPILER` to that
  shim.
- **Tcl re-vendored to close the last gap (2026-07-06).** `vendor/tcl` had been trimmed to the native
  unix/linux build set (no `win/`, no `macosx/`), so cross-Tcl could not build from vendored source.
  The official tcl9.0.4 `win/` + `macosx/` subdirs are now restored, so all three ports build Tcl from
  `vendor/`; the native Linux build is unchanged (still `testEngineTcl` 13/0). All ports now match.