AI_Slop/calog
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
calog/PORTING.md

20 KiB
Raw Blame History

Porting calog

calog targets Linux, macOS, and Windows. Its core -- values, broker, the actor/threading model, and every engine adapter -- is portable C11. The only operating-system-specific surface is small and centralized:

  • Threads: POSIX pthreads. Native on Linux and macOS; on Windows via mingw-w64's winpthreads.
  • Sockets + DNS: BSD sockets + getaddrinfo, used only by the net/ssh/http libraries. BSD sockets on Linux/macOS, Winsock on Windows. All of this is behind src/calogPlatform.h.

There is no epoll/eventfd/kqueue, no fork/exec, and no dlopen in calog's own code (the actor model uses blocking sockets on per-context threads), so a port is a thin shim, not a rewrite.

The platform abstraction: src/calogPlatform.h

The net/ssh/http libraries use these instead of raw socket calls, so the same source compiles on all three platforms:

Abstraction POSIX (Linux/macOS) Windows
CalogSocketT int SOCKET
CALOG_INVALID_SOCKET -1 INVALID_SOCKET
calogSockClose(fd) close closesocket
calogSockErrno() / calogSockErrStr() errno / strerror WSAGetLastError / FormatMessage
calogSockSetNonblock(fd, on) fcntl(O_NONBLOCK) ioctlsocket(FIONBIO)
calogSockInProgress(err) EINPROGRESS WSAEWOULDBLOCK/WSAEINPROGRESS
calogPoll(...) poll WSAPoll
calogPlatformNetInit() / Shutdown() no-op WSAStartup / WSACleanup
CALOG_MSG_NOSIGNAL MSG_NOSIGNAL 0 (Windows has no SIGPIPE)

A socket fd is unsigned on Windows, so the code compares against CALOG_INVALID_SOCKET (never fd < 0). The abstraction is behavior-identical on POSIX (it maps to the same names), so the Linux and macOS builds are unchanged by it.

Build matrix

Target Command Runtime dependencies DNS
Linux dev make glibc, libstdc++, libgcc, ASan/UBSan (dev only) yes
Linux release make release glibc only (libstdc++/libgcc linked static) yes
Linux static make static on Alpine/musl none (fully static) yes (musl resolver)
Linux static make static on glibc none, but see caveat no (glibc NSS needs dlopen)
macOS make (Darwin auto-detected) libSystem (always present) yes
Windows mingw-w64 build core OS DLLs (kernel32, ws2_32) yes

The Makefile auto-detects the platform (PLATFORM = linux/darwin/windows/posix) and adjusts the link line: DLLIB is -ldl on Linux (empty on macOS, where dlopen is in libSystem) and SOCKETLIBS is -lws2_32 on Windows.

Feature and verification parity

The full calog CLI -- all ten engines and every capability library (net/tls/http-https/db [SQLite+PostgreSQL+MariaDB]/archive/regex/ssh/xml + the core libs) -- builds for all three operating systems. What differs across ports is not features but how thoroughly each is verified, and the scope of the fully-static route.

Port Build path Full CLI (10 engines + all libs) Verified
Linux glibc (dev / release) make / make release yes runtime (the test suite runs)
Windows x86_64 (PE) tools/crossWinFull.sh (zig cross) yes build + link only
macOS x86_64 tools/crossMacFull.sh mac-x64 (zig cross) yes build + link only
macOS arm64 tools/crossMacFull.sh mac-arm64 (zig cross) yes build + link only
Linux static musl (Alpine) make static on Alpine no -- Lua-only demo (examples/staticDemo.c) runtime (per component)
Linux static glibc make static on glibc no -- Lua-only demo build only

So engine and library parity across Linux, macOS, and Windows is complete. The one open gap is run-verification on Windows and macOS: both are cross-built and linked with a single zig toolchain but have not yet been executed on the target OS (a real Windows/wine or Mac run is the remaining step). The fully-static route is a deliberately narrower, minimal-dependency demo (Lua only, not the full CLI), and static glibc loses DNS (its NSS resolver needs dlopen, which a static binary cannot do) -- so Alpine/musl is the path to a zero-dependency binary that still resolves names.

Linux

  • make -- the sanitized development build (ASan + UBSan + strict warnings).
  • make release -- the distribution build: sanitizers off, -static-libstdc++ -static-libgcc, glibc dynamic. ldd collapses to just libc/libm + the loader. Because glibc stays dynamic, getaddrinfo can still dlopen its NSS modules, so DNS works. Runs on any mainstream glibc distro with nothing to install.
  • make static on Alpine (musl) -- a fully static binary with zero runtime dependencies and working DNS (musl has a built-in resolver, no NSS/dlopen). This is the minimal-dependency artifact. Because calog builds every dependency from source, and Alpine's default toolchain is musl, no per-dependency changes are needed -- build it in an Alpine container.
  • make static on glibc -- fully static, but name resolution and dlopen-based Lua C modules do not work (glibc NSS requires runtime dlopen). Connect by IP, or use the musl route above.

macOS (Darwin)

The C is already POSIX-compatible (pthreads, BSD sockets, getaddrinfo are all native), so the port is build configuration, not code -- confirmed by tools/crossBuild.sh, which links valid Mach-O executables for both x86_64 (Intel) and arm64 (Apple Silicon) using zig's bundled libSystem stub (calog only needs libSystem -- libc/pthreads/sockets -- so no Apple SDK/frameworks are required for the core). Apple does not support statically linking libc, but libSystem is part of the OS and always present, so "static everything except libSystem" is the natural minimal-deps build. Each vendored dependency builds its own way and needs a macOS configuration:

  • OpenSSL: ./Configure darwin64-<arch>-cc
  • Tcl: its configure detects Darwin
  • mruby / Lua / SQLite / ENet / QuickJS / Squirrel / Berry / s7 / Wren: compile as-is with clang
  • libssh2 / MariaDB: CMake with the macOS toolchain

Link against dynamic libSystem (the default); -ldl is dropped automatically (DLLIB is empty on Darwin) and the C++ runtime is libc++ (CXXLIB = -lc++). SIGPIPE is ignored at CLI startup, since macOS send() has no MSG_NOSIGNAL. ENet's unix.c backend is correct on macOS (BSD sockets).

Windows (mingw-w64)

calog's pthread code compiles unchanged on Windows against winpthreads (POSIX threads over the Win32 API), which is vendored at vendor/winpthreads/ and built from source like every other dependency (see its NOTICE.calog). src/calogPlatform.h handles the Winsock differences. Building uses a mingw-w64 toolchain (not MSVC) for its POSIX-ish environment; tools/crossBuild.sh cross- compiles from Linux with a single zig toolchain and is the tested path. Remaining work is build configuration:

  • Build each vendored dependency for mingw (OpenSSL mingw64, Tcl's win/ makefile or configure, libssh2/MariaDB via CMake with the mingw toolchain, mruby with a mingw build_config).
  • ENet selects its win32.c backend automatically (ENETBACKEND); the link adds -lws2_32 -lwinmm (handled by SOCKETLIBS).
  • Static-link the runtime (-static -static-libgcc -static-libstdc++) so the binary depends only on core Windows DLLs, which are always present. DNS works through Winsock's getaddrinfo.

Remaining code review for Windows: calogFs.c and calogHttp.c use POSIX headers (dirent.h, sys/stat.h, sys/time.h) that mingw-w64 mostly provides, but path handling (/ vs \) and a few calls may need adjustment; calogSsh.c already guards sys/select.h (Winsock supplies fd_set/select), and httpSetTimeouts already branches on the Winsock SO_RCVTIMEO DWORD form.

Cross-compiling from Linux with zig (tools/crossBuild.sh)

A single zig toolchain cross-compiles C and C++ to musl-static and Windows without installing per-target toolchains, so the ports can be exercised on a plain Linux box:

ZIG=/path/to/zig ./tools/crossBuild.sh

It builds a representative set -- core + one engine of each kind (Lua, QuickJS, Squirrel/C++, my-basic) + the socket library (calogNet) -- because the crypto/db/http/ssh libraries pull in OpenSSL/MariaDB/PostgreSQL/libssh2, each needing its own per-target build. What it does:

  • musl: fully static Linux binaries. These run on the build host, so the script builds AND runs them -- verifying the core, all four engine kinds, C++, and the socket abstraction on musl.
  • Windows: builds vendor/winpthreads into libwinpthreads.a, then links real .exes (including the Winsock + ENet-win32 socket path). Verified as valid PE executables; running them needs wine.

Verified results (this repo): musl -- Lua, QuickJS, my-basic, Squirrel, and testNet all build and pass (fully static); Windows -- testEngineLua.exe and testNet.exe build against vendored winpthreads. Since the Windows binaries are the same source that passes on musl, the build is strong evidence; full run-verification needs a Windows host or wine.

Cross-compiling the archive stack with zig (tools/crossArchive.sh)

The compression/archive library (libarchive + its five codecs: zlib, bzip2, lz4, zstd, xz/liblzma) is the one vendored stack that leans on CMake, so it gets its own cross-build script. Same idea as crossBuild.sh, same single zig toolchain:

ZIG=/path/to/zig ./tools/crossArchive.sh

It builds all five codecs plus libarchive for musl (fully static, RUN on the host) and Windows x64 (a testArchive.exe verified as a PE), driving each through calog's real testArchive (compress/decompress round-trips through every codec). CMake cross-compilation with zig works by wrapping zig cc as a single-binary compiler (build/cross/bin/zcc-*) plus a toolchain file. The key asymmetry:

  • musl toolchain does NOT set CMAKE_SYSTEM_NAME. A musl-static binary runs on this Linux host, so the build stays "native" (CMAKE_CROSSCOMPILING false) and CMake's try_run feature-detection executes normally. zig supplies its own musl sysroot, so no host headers leak in.
  • Windows toolchain sets CMAKE_SYSTEM_NAME Windows -> cross mode. libarchive's single CHECK_C_SOURCE_RUNS is already guarded on CMAKE_CROSSCOMPILING (it degrades to a warning) and xz has no run-checks, so nothing needs pre-seeding.

Two target-specific gotchas the script handles:

  • zlib on musl needs -DHAVE_UNISTD_H so it includes <unistd.h> for lseek. zlib's ./configure normally sets this; the object-rule build (no configure) must pass it, and musl -- unlike glibc -- does not declare lseek without the include. Only zlib is affected.
  • libarchive on Windows pulls in system libraries the POSIX build does not: CNG crypto (-lbcrypt) for its AES/PBKDF2, and XmlLite + OLE (-lxmllite -lole32) for XAR, which libarchive auto-substitutes for the libxml2/expat we disable. The link adds -lbcrypt -lxmllite -lole32 -ladvapi32 -lcrypt32 -lshlwapi.

Everything lands under build/cross/ (gitignored); the vendored sources and the native vendor/*/build / _cmk trees the normal make uses are never touched.

Status

  • Platform abstraction (calogPlatform.h) and the net/ssh/http conversion: done, verified behavior-identical on Linux (make test 30/30, socket + HTTP loopback smoke, ASan- and TSan-clean).
  • Linux release target: done and verified -- ldd bin/calog shows only libc/libm + loader.
  • Linux musl-static: run-verified via tools/crossBuild.sh (zig) -- core + Lua + QuickJS + my-basic + Squirrel + testNet all build fully static and pass on the host. Also works via make static in an Alpine container (musl default toolchain).
  • Windows: build-verified via tools/crossBuild.sh -- testEngineLua.exe and testNet.exe (core + engine + Winsock socket abstraction) link against vendored winpthreads into real PE executables. Run-verification needs a Windows host or wine; the binaries are the same source that passes on musl.
  • Build system: OS-aware (PLATFORM, DLLIB, CXXLIB, SOCKETLIBS, ENETBACKEND, vendored winpthreads); the Linux values are identical to before, so the dev build is unchanged.
  • macOS: build-verified via tools/crossBuild.sh -- core + Lua + testNet link into valid Mach-O executables for both x86_64 and arm64 (zig's libSystem stub; no Apple SDK needed for calog's libc surface). Run-verification needs a Mac. The C code and Makefile hooks are all in place (libc++, no -ldl, SIGPIPE, native pthreads + BSD sockets).
  • Archive stack (libarchive + zlib/bzip2/lz4/zstd/xz): musl run-verified, Windows build-verified via tools/crossArchive.sh (zig). The musl testArchive is fully static and passes all 22 checks on the host -- real compress/decompress through every codec and an xar read+write round-trip: the musl target now cross-builds OpenSSL (xar's MD5/SHA1) and libxml2 (its XML TOC), with iconv coming from musl's own libc, so its libarchive has full xar. The Windows build is a valid testArchive.exe (PE32+) linked against vendored winpthreads + the CNG/XmlLite system libs. This was the one CMake-heavy vendored stack, so it is the strongest cross-build evidence after the core. On Windows, xar needs no vendored XML/crypto/iconv at all: libarchive auto-uses the system XmlLite (its TOC) + CNG/bcrypt (its MD5/SHA1), both in the mingw sysroot, so the Windows testArchive.exe already has full xar. One musl gotcha worth noting: libarchive's FIND_PATH(iconv.h) returns the host /usr/include (glibc), whose headers break the musl compile, so the script pins ICONV_INCLUDE_DIR to an empty dir and lets zig cc resolve musl's own iconv.h implicitly.
  • Full Windows CLI cross-built (build-verified). Every heavy vendored dependency now cross-compiles for x86_64-windows-gnu via zig -- OpenSSL, libxml2, PCRE2, libssh2, MariaDB, and the three hard ones PostgreSQL/libpq, Tcl, mruby (libpq needed a patched out-of-tree copy; see tools/crossWinFull.sh). The complete calog.exe links all 10 engines + all 20 libraries into a valid PE32+ that imports only Windows system DLLs (no vendored DLLs). Behavior-neutral #ifdef _WIN32 guards in calog's own source cover the small OS differences -- calogFs (mingw 1-arg mkdir, O_BINARY), calogHttp (no SIGPIPE), calogInternal (a memmem fallback), calogPlatform (__declspec(thread) -> _Thread_local, which also fixes a latent Windows thread-safety bug clang had been silently dropping), and calogNet (SO_EXCLUSIVEADDRUSE for listeners instead of the hijack-prone SO_REUSEADDR). Two libraries have a full Windows backend rather than a POSIX one: procRun spawns with CreateProcess + pipes (stdin fed on a helper thread while stdout/stderr drain, since anonymous pipes have no poll), and the HTTPS client reads trust anchors from the Windows ROOT/CA system certificate stores via CryptoAPI (CertOpenSystemStore) instead of a Unix CA-bundle path. All are behavior-neutral off Windows; the Linux suite stays green. Run-verification still needs a Windows host or wine.
  • Full macOS CLI cross-built for both archs (build-verified). Every heavy vendored dependency now cross-compiles for x86_64-apple-darwin and arm64-apple-darwin via zig, whose libSystem stub supplies libc / pthreads / BSD sockets / getaddrinfo / dlopen / iconv -- so no Apple SDK is needed for calog's OS surface (see tools/crossMacFull.sh). The complete calog links all 10 engines + all 20 libraries into a valid Mach-O per arch (build/cross/mac-x64/calog x86_64, build/cross/mac-arm64/calog arm64), importing only libSystem -- no vendored dylibs, no -ldl. Two additive #if defined(__APPLE__) guards are in calog's own source: calogTimer waits with a relative pthread_cond_timedwait_relative_np timeout on Darwin, because macOS pthreads have no pthread_condattr_setclock to pin the condvar to CLOCK_MONOTONIC (the relative wait preserves the monotonic scheduling intent across wall-clock changes); and calogHttp's HTTPS client reads trust anchors from the macOS keychain via the Security framework (SecTrustCopyAnchorCertificates). Both are behavior-neutral off Darwin, and the Linux suite stays green. Run-verification still needs a Mac.
  • macOS Keychain HTTPS trust needs the Apple SDK (optional). calog's OS surface (libc / pthreads / BSD sockets / getaddrinfo / dlopen / iconv) is all in zig's libSystem stub, so the runner needs no SDK. But the keychain trust path uses the Security + CoreFoundation frameworks, whose headers zig does not ship. When an SDK is present, tools/crossMacFull.sh compiles calogHttp with -DCALOG_MAC_KEYCHAIN_TRUST against the SDK framework headers; without one it omits keychain trust and HTTPS falls back to SSL_CERT_FILE / CALOG_CA_BUNDLE. Point CALOG_MACSDK at a MacOSX*.sdk directory to enable it. Linker gotcha: zig 0.16's Mach-O linker segfaults parsing the real multi-target SDK .tbd files, so calog links against hand-written minimal stubs (tools/macStubs/{CoreFoundation,Security}.tbd) that export only the handful of symbols the keychain code calls -- the binary still imports the real system frameworks by install-name at runtime. Both arch binaries are valid Mach-O importing Security.framework + CoreFoundation.framework (build-verified; run-verification still needs a Mac).
  • Reproducible dependency builds (tools/crossDeps.sh). The full-CLI scripts above consume heavy pinned deps under build/cross/<target>/; crossDeps.sh rebuilds them from vendor/ source with a single zig toolchain, so the cross build is reproducible from a clean checkout rather than depending on ad-hoc pins. Usage: [ZIG=/path/to/zig] ./tools/crossDeps.sh <win|mac-x64|mac-arm64> [dep ...] (it regenerates the zig wrappers + CMake toolchain file first, so nothing under build/cross/ need pre-exist). It builds, in dependency order: the codecs (zlib/bzip2/lz4/zstd), xz, OpenSSL (mac also gets a BSD-ar repack, since zig's Mach-O linker cannot read OpenSSL's GNU-ar archives), libxml2, PCRE2, libssh2, MariaDB, PostgreSQL/libpq (the patched out-of-tree copy: mingw .obj->.o, a real ar rule for libpq.a, and pg_pthread_* symbol localization on Windows; libSystem pthreads and no patch on mac), Tcl, mruby, winpthreads (Windows only), and libarchive (mac = xar via the pinned OpenSSL+libxml2; Windows = codecs only, xar via system XmlLite/CNG). VERIFIED build-clean from vendor source for ALL THREE targets, all 12 deps (OUT= builds into a scratch dir without clobbering the pins): mac-x64 and mac-arm64 each build the full set in one clean invocation (OpenSSL confirmed Mach-O + !<arch> BSD repack); win builds the full set including the mingw-only bits -- winpthreads, the patched libpq (.obj->.o, real ar rule, pg_pthread_* localization, plus the Windows syslibs -lws2_32 -lcrypt32 -lsecur32 -lbcrypt ... so configure's -lcrypto link test passes), and the two .rc resources (libxml2's + libpq's win32ver.rc) compiled via a zig rc windres shim that handles both CMake's positional args and PostgreSQL's -i/-o/--include-dir form. The win CMake toolchain sets CMAKE_RC_COMPILER to that shim.
  • Tcl re-vendored to close the last gap (2026-07-06). vendor/tcl had been trimmed to the native unix/linux build set (no win/, no macosx/), so cross-Tcl could not build from vendored source. The official tcl9.0.4 win/ + macosx/ subdirs are now restored, so all three ports build Tcl from vendor/; the native Linux build is unchanged (still testEngineTcl 13/0). All ports now match.