# Session Recovery — last updated 2026-05-27 Living recovery doc. Update on every meaningful change. If session is lost, read this top-to-bottom + the memory notes referenced inside, then reread the actual diffs in tree to ground assumptions. ## Headline state - **Smoke**: 148/148 green. Demos 9/9 (helloBeep/helloText/helloWindow/ orcaFrame/qdProbe/heavyRelocs/frame/reversi/minicad). - **Active config**: ptr32 (`p:32:16`), full IMG0..IMG15 caller-clobber on JSL, greedy regalloc at -O1+. **Inline-threshold lowered to 50 target-wide** (was LLVM default 225; was 75 earlier this session). - **Branch**: `main`. - **vs Calypsi (2026-05-27)** — Layer 2 + recent peepholes: - **Cycle benches geomean**: **0.62× Calypsi**. 9 of 10 below 1.0×; only `fib` trails at 1.06× (recursive overhead, structural). See cycle bench table below. - **Lua 5.1.5**: default config 1.13× Calypsi; with Layer 2 0.93×. - **CoreMark 1.0**: with Layer 2 0.79× Calypsi (we beat by 21%). - **vs Calypsi static-inst ratio (synthetic bench)**: sumSquares **0.84×** (26 vs 31 — we beat), mul16to32 **0.25×** (1 vs 4 — we beat), evalAt 1.86× (472 vs 254 — structural floor; ABI overhaul rejected). - **New code-gen options (2026-05-25)** — see docs/USAGE.md "Advanced: pointer-deref code generation": - Layer 1 ptr32 deref-fold (always on): LDY offset instead of CLC/ADC carry chain. ~3 instr saved per struct-field access. - `-mllvm -w65816-dbr-safe-ptrs` (Layer 2, opt-in): uses `lda (d,S),Y` for ptr32 derefs assuming bank-byte == DBR. 5 instr → 1 instr per deref. Lua -20.6%. **MISCOMPILES cross-bank pointers — opt in per-TU only when safe.** - Inline-threshold lowered to 50 (was 225). Lua -23% total, CoreMark matrix.o 1.37× → 0.97× Calypsi. Override with `-mllvm -inline-threshold=N`. - **Cycle benches per-call (2026-05-27, Layer 2)** — via `scripts/benchCyclesPrecise.sh` vs `scripts/benchCyclesCalypsi.sh`: ``` Bench Ours Calypsi Ratio dotProduct 1534 5712 0.27× bsearch 682 2387 0.29× sumOfSquares 6820 16368 0.42× bubbleSort 11594 17050 0.68× strLen 767 1023 0.75× djb2Hash 2046 2643 0.77× popcount 1194 1534 0.78× strcpy 1108 1194 0.93× memcmp 682 716 0.95× fib 11594 10912 1.06× ``` Geomean **0.62×**. Older HBL-tick numbers (per-iter, 100 iter loops) from `benchCycles.sh` are still available but lower resolution. - **Recent session wins (2026-05-27)**: - **Y-as-counter for strLen** — structural rewrite: drop STX/INX/INC, use Y as offset AND counter. strLen 1279 → 767 cyc (-40%); 0.75× Calypsi (was 1.25×). - **Stack-rel dead-store elim** — companion to DP version with SP tracking across PHA/PHP/PEA/PEI/PER/PLA/PLP/PLX/PLY/PHX/PHY. strcpy 1194 → 1108 (-7%, 0.93× Calypsi, beats by 7%). Refactored as a static helper called from the recursive-call bail too so fib gets it. fib 12106 → 11594 (-4%, 1.06× Calypsi). - **DP-indirect-Y for iter** (follow-on to X-iter peephole): rewrites `TXA;STA stack-rel S;INX;…;LDA (S,s),Y` to `STX_DP D;INX;…;LDA (D),Y`. Saves 4 cyc/iter. - **Dead INC_HI_IF_CARRY elim** — when the StackRel ptr-hi slot is never read, elide the carry-bookkeeping for Layer 2 ptr32 loops. Wide impact across strLen/strcpy/djb2Hash/memcmp. - **Recent session wins (earlier — 2026-05-20)**: - 8 always-on peepholes + extended phase 4 in W65816StackRelToImg (evalAt 498→472, fib -35%, 35 libc fns shrunk) - __muldi3 32-bit short-circuit (dmul 1605→1033, -36%) - case-(b) ImgCalleeSave bracket hoist enables phase 4 to elide TAY/TYA round-trip in synergy - FP cycle benches added (dadd/dmul/ddiv) with per-bench iter count - Documented LSR-dp cycle mystery as HBL-counter wrap artifact - Game-like benches added: particles (i16 physics), mandelbrot (i32 fp) - **elideStoreForwarding now reached via early-return bail paths**: particles 5005→2253 cyc/iter (-55%). Was being skipped for any function where main IMG promotion bailed (SpAdj invalid, no accesses, or > 16 hot slots). ## Uncommitted, must keep `git status --short` (5 modified, no untracked of consequence): 1. `SESSION_RECOVERY.md` — this doc. 2. `scripts/smokeTest.sh` — added "omfEmit `--stack-size` emits a DP/Stack `~Direct` segment" check. Validates 3-segment layout (ExpressLoad + code + DP/Stack) when `--stack-size` is supplied; parses the third segment header against KIND/LENGTH/RESSPC/ALIGN/ SEGNUM=3/name="~Direct" expectations. 3. `src/link816/omfEmit.cpp` — `emitDpStackSeg(length, segNum)` plus the `--stack-size N` CLI flag. Validation: 256 ≤ N ≤ 65536, page- aligned. **`--stack-size` implicitly enables `--expressload`** — the GS/OS Loader's slow path silently rejects multi-seg OMFs (see §D below for the empirical evidence). 4. `src/llvm/lib/Target/W65816/W65816ISelLowering.cpp` — `LowerShift` now inlines i32 SHL/SRL/SRA by N=1..4 instead of routing to `__lshrsi3`/`__ashlsi3`/`__ashrsi3`. See §E. 5. `src/llvm/lib/Target/W65816/W65816StackSlotCleanup.cpp` — pre-existing uncommitted change from prior turns; verify against git log before re-staging if recovery is fresh. Earlier-mentioned files (snprintf.c, W65816InstrInfo.cpp, W65816SjLjFinalize.cpp) have been checkpoint-committed and are no longer in `git status`. ## Already-committed in this session arc Per `git log --oneline -20` these are the recent checkpoint commits; the diffs they contain are real and load-bearing. The big ones (search by file or grep): - **JSLpseudo Defs += IMG0..IMG15** in `W65816InstrInfo.td`. With the wider Defs, regalloc spills IMG-class vregs around calls instead of treating them as preserved. - **`W65816RegisterInfo.cpp` `eliminateFrameIndex` for `STAfi`**: PHA-bracketed for non-A source (IMG/X/Y). The `lda dp; sta d,s` chain clobbered A; bracket preserves A while shifting offset by +2 between PHA and PLA. Defs=[A] kept on STAfi as safe over-approximation. - **`W65816RegisterInfo.cpp` `eliminateFrameIndex` for `LDAfi`**: if `Dst = IMGn`, append `STA dp` so the IMG slot actually receives the loaded value. Previously only loaded into A; downstream `COPY $x = $imgN` (= `ldx $D?`) read garbage. **This was the smoking gun for `dadd(1.5, 2.5) → 0x4010_0000_3000_3000`.** - **`W65816LowerWide32.cpp`** fixed-point erase loop. Was single-pass; REG_SEQUENCE got skipped if a not-yet-erased COPY consumer kept it alive at the iteration moment. Removed ~40 dead Wide32 vregs from `__adddf3`'s pre-RA MIR. - **`src/llvm/test/CodeGen/W65816/i64-first-arg-img16.ll`** relaxed `stx 0xd / sta 0xd` to `0x{{[cd]}}` (regalloc now picks IMG8..15 too). ## Fixes landed (full list with rationale) Each entry: what / why / where / what regression it would cause if reverted. ### A. Hash-shell DELETE bug → IMG caller-clobber **Symptom**: `dbDelete("age")` returned 0 ("not found") instead of 1. DELETE never ran; `COUNT` stayed at 2. **Root cause**: `dbDelete` did `stx 0xd0` to save k_high, called `hashKey`, then `pei 0xd0` to push k_high to strcmp. `hashKey` used $D0 as scratch in its loop body (`sta 0xd0` storing the iterator's running-ptr-low). $D0 was clobbered by the time `pei 0xd0` ran. JSLpseudo Defs only listed `A, X, Y, DPF0` — IMG slots were not modelled as caller-clobber. **Fix**: `JSLpseudo Defs += [IMG0..IMG15]`. **Cascading fallout** (each required its own fix): #### A1. copyPhysReg vreg fallback `storeRegToStackSlot`'s unpaired-Wide32 default branch hit `unreachable` when called with a vreg source. Basic regalloc's InlineSpiller does this. Fix: short-circuit virtual-reg cases to `TargetOpcode::COPY`. #### A2. LowerWide32 fixed-point erase Single-pass erase left ~40 dead Wide32 vregs in `__adddf3`. Pattern: ``` %X:wide32 = REG_SEQUENCE ... %Y:wide32 = COPY %X ... uses of %Y rewritten by Pass 3 ... ``` Single-pass: REG_SEQUENCE skipped (COPY consumer still alive), then COPY erased (now %X dead but loop already passed it). Fix: iterate until no progress. #### A3. STAfi PHA-bracket Without bracket, regalloc could schedule `$img0 = COPY $a` AFTER a `STAfi`-with-IMG-source whose internal `lda dp` clobbered $a, silently storing X's value where A's was expected. #### A4. LDAfi-IMG-dest STA dp **The big one.** With narrow IMG, regalloc kept Wide16 vregs in IMG slots across calls, never needed `$imgN = LDAfi %stack.X`. With full IMG, every cross-call spill needed it. The expansion only emitted `LDA d,s` (load A) — never wrote to the IMG slot. Downstream `COPY $x = $imgN` (= `ldx $D?`) read stale prior data. Manifested as `dadd(1.5, 2.5) → 0x4010_0000_3000_3000` (mantissa garbage). **Diagnostic that found it**: diff post-RA MIR narrow vs full IMG. Pre-RA MIR was identical. Full had 6 `$imgN = LDAfi` instances; narrow had 0. Narrow used `COPY $imgN = $a` patterns instead — those work correctly. #### A5. FileCheck regex `src/llvm/test/CodeGen/W65816/i64-first-arg-img16.ll` expected `stx 0xd / sta 0xd`. Under full IMG clobber, regalloc picks IMG8..15 ($C0/$C2) for cross-call arg saves. Relaxed to `0x{{[cd]}}`. ### B. C++ try/catch source-level path Two bugs blocking real `clang++ -fsjlj-exceptions` source code: #### B1. W65816SjLjFinalize catchtab ordering `runOnFunction` erased landingpad insts at line ~245, then built the catchtab at line ~290 via `LPadBB->getLandingPadInst()`. By that point, landingpads were nullptr. The build loop's `if (!LP) continue;` skipped every entry. Catchtab ended with just `(0,0)` sentinel. LSDA was 4 bytes of zeros. `findCatch` saw `ctx->lsda == 0`'s entry and bailed. Result: any `throw` aborted. Fix: capture catch-clause typeinfo Constants into a `DenseMap` BEFORE erasing landingpads; the catchtab build loop reads from the saved map. #### B2. copyPhysReg IMG-to-IMG PHA-bracket Comment said "Caller is responsible for ensuring A is dead at this program point (regalloc usually arranges this)." It doesn't, in practice. Regalloc inserted IMG-to-IMG copies between `$a = COPY $img10` and `STAfi $a, slot`. Unbracketed `lda src; sta dst` clobbered A. The subsequent STAfi spilled garbage. Visible as `*p = 42` after `__cxa_allocate_exception` storing 42 to wrong addr (indirect-long setup got hi-half at lo-slot). Fix: PHA-bracket. Cost +7 cyc / +2 bytes per IMG-IMG copy (rare). **Verified end-to-end** via MAME breakpoints: `begin_catch` entered with correct ExcHeader, `end_catch` entered with A=42, doTest returns A=42 from real C++ `try { throw 42; } catch (int x) { return x; }`. ### C. Cleanup wins - `runtime/src/snprintf.c:106` — removed `optnone` on `emitULong`. Smoke green. - `runtime/src/snprintf.c:303` — removed `optnone` on `snprintf`. Smoke green. ### D. `omfEmit --stack-size` — DP/Stack segment for GS/OS Loader Added `emitDpStackSeg` (`src/link816/omfEmit.cpp`). KIND=0x1012 (DP/Stack | PRIVATE), LENGTH=RESSPC=requested-bytes, ALIGN=0x100, BANKSIZE=0, body is a single END opcode. Apps can now request a stack of any page-aligned size from 256B to 64KB (replacing GS/OS Loader's default 4KB allocation). **Loader gotcha** (cost ~1 hour to debug): plain (non-ExpressLoad) multi-segment OMFs do NOT launch under real GS/OS 6.0.2 — the Loader's slow path silently rejects the file and our entry point never runs. ExpressLoad-wrapped multi-segment OMFs DO work. Fix: `--stack-size` now implicitly enables `--expressload` (the Loader's slow path is empirically broken for our 2-seg layout). The DP/Stack seg is appended AFTER the user code seg as SEGNUM=3; the Loader walks all segments by KIND after the ExpressLoad fast-load step finishes. Verified: `runViaFinder.sh /tmp/test_el_dp.omf --check 0x70=0x42 0x71=0x99` passes under real GS/OS 6.0.2 with `--stack-size 4096 --expressload`. Verified failure mode: same payload with `--stack-size` alone (no `--expressload`) → `0x70=0x00` (program never executed). Documented in `feedback_loader_multi_seg_needs_expressload.md`. Smoke updated: 132/132 expects 3 segs (ExpressLoad + code + DP/Stack) when `--stack-size` is supplied. ### E. i32 shift-by-N inlined (was full libcall) — speed win `W65816ISelLowering.cpp` `LowerShift` now inlines i32 SHL/SRL/SRA by N=1..4. Previously every i32 shift went through `__lshrsi3`/ `__ashlsi3`/`__ashrsi3` — ~300+ cyc per call. popcount benchmark: **8320 → 6888 cyc/call, 17% faster**. Implementation extracts `Wide32` halves via `extractWide32Lo/Hi`, applies per-step `lsr; ror`-equivalent SDAG ops with explicit carry propagation (`(Hi & 1) << 15` for SRL/SRA's lo-fill, `Lo >> 15` for SHL's hi-fill), recombines via `buildWide32`. N>4 still routes to libcall — the unrolled cost (~5 i16 ops × N) crosses libcall overhead at N≈5. Documented in `feedback_i32_shift_inline.md`. ## Still-open work areas Each carries a fair-warning note for whoever picks it up. ### 1. qsort/bsearch `optnone` — REMOVED 2026-05-08 Source-restructured `qsort`: split the inner loop into a `__attribute__((noinline))` helper `qsortInner` (4 args: base, cur, size, cmp). Outer `qsort` just iterates `i = 1..nmemb-1` and calls `qsortInner(base, base + i*size, size, cmp)`. This drops outer qsort's i32-vreg simultaneous-live count below the inline-spill OOM threshold; both halves compile cleanly at -O2 + basic regalloc. `bsearch` `optnone` was kept-for-symmetry — once removed, it just worked. The IMG-clobber + LDAfi-IMG-store backend fixes from 2026-05-07 had already resolved its underlying pressure issue. Smoke stays green (now 132/132). ### 2. gmtime_r `optnone` `runtime/src/timeExt.c:69`. NOT a backend bug — IR-level optimization issue (loop rotation + IndVar simplify mis-evaluating `days >= 365L + (__isLeap(...) ? 1 : 0)`). Fixing requires deciding which combine pass is wrong and why. Out of scope for backend work. ### 3. softDouble noinlines `runtime/src/softDouble.c:30` (`dpack`) and `:51` (`dclass`). Removing `dpack` noinline broke dadd this session — register pressure for `__adddf3`/`__muldf3`/`__divdf3`. Architectural for the same reason as qsort. ### 4. Greedy regalloc retry — TRIED, blocked Tested 2026-05-08. Greedy fails immediately on `atoi` in libc.c: ``` LiveRangeEdit.cpp:200: void llvm::LiveRangeEdit::eliminateDeadDef(...): Assertion `MI->allDefsAreDead() && "Def isn't really dead"' failed. ``` Same upstream LLVM bug class as the dadd full-IMG attempt — sub-register pair partial defs that the regalloc treats as fully dead. Greedy is genuinely incompatible with the W65816's split-half subreg-pair patterns until the upstream LLVM issue is patched. Reverted to basic regalloc. Document `feedback_greedy_high_pressure.md` already covers this. ### 5. gmtime_r `optnone` — TRIED, blocked Tested 2026-05-08. Hoisting `yearLen` to a long local (avoiding the double-recompute of `365L + (__isLeap ? 1 : 0)`) didn't help; adding `volatile` to the local also didn't help. IR optimizer is still folding the comparison to compile-time-false. Source-level C restructuring won't dodge it; needs IR-pass-level work to identify which combine pass mis-evaluates and why. optnone stays. ## How to verify recovery ```bash cd /home/scott/claude/llvm816 git status # 5 modified files listed above cd tools/llvm-mos-build && ninja llc clang # rebuild backend (~5 min) cd /home/scott/claude/llvm816 cd src/link816 && make && cd ../.. # rebuild link816 + omfEmit bash runtime/build.sh # build runtime bash scripts/smokeTest.sh # should end "all smoke checks passed" bash scripts/benchCyclesPrecise.sh # popcount should be ~6888 cyc ``` Loader smoke (validates DP/Stack seg under real GS/OS 6.0.2): ```bash # Build a simple test program with --stack-size, run via Finder. tools/omfEmit --input X.bin --map X.map --base 0x1000 --entry __start \ --output /tmp/t.omf --stack-size 4096 --relocs X.relocs bash scripts/runViaFinder.sh /tmp/t.omf --check 0x70=0x42 0x71=0x99 ``` If smoke fails, the likely cause is one of the 5 uncommitted files got reverted; check `git status` and re-apply. If popcount bench regressed past ~7500 cyc, suspect the i32-shift-inline change in `W65816ISelLowering.cpp` was lost. ## Diagnostic tools that worked For posterity — these are the patterns that paid off this session. ### Pre-RA vs post-RA MIR diff ```bash clang -mllvm -stop-before=regallocbasic -S ... # pre-RA clang -mllvm -stop-after=virtregrewriter -S ... # post-RA (post-virtregrewriter) ``` Diff narrow-IMG vs full-IMG post-RA MIR for the failing function. Pre-RA is identical (same IR), so the diff isolates regalloc-decision divergence. Look at every NEW pattern that appears only in the failing build — `$imgN = LDAfi` was the smoking gun for dadd. ### Pass-by-pass IR/MIR dumps ```bash clang -mllvm -print-after=w65816-lower-wide32 -S ... clang -mllvm -print-after-all -S ... 2>dump.txt ``` ### MAME debugger via xvfb-run ```bash xvfb-run -a mame apple2gs ... -debug -debugger qt -oslog -seconds_to_run N ``` With autoboot Lua: load .bin into bank 0 (skip `$C000..$CFFF` I/O), set CPU state, then `cpu.debug:bpset(addr, condition, action)` with actions like `"logerror \"...\\n\",a,x,...; go"`. `logerror` with format args goes to stdout under `-oslog`. Memory reads in expressions: `b@(addr)`, `w@(addr)`. Watchpoints: `cpu.debug:wpset(prog_space, "w", addr, len, condition, action)`. **AVOID**: `add_machine_pause_notifier` + `cpu.debug:go()` in callback — segfaults from reentrancy. `printf` in actions stays in debugger console (not -oslog). `tracelog` also debugger-console only. ### Trace methodology (find divergence point) 1. Set BPs at every `JSLpseudo` callee in the failing function. 2. Capture A/X/Y/DPF0 at each return. 3. Find first divergent return between known-good and failing builds. 4. The instruction sequence between previous-OK and first-divergent return is where the bug lives. This pattern found the dadd bug at `jsl@0x207f → __lshrsi3(0x8001_8000, 3)` in 30 minutes. Recommended. ## Memory notes referenced (Filenames under `/home/scott/.claude/projects/-home-scott-claude-llvm816/memory/`.) - `feedback_strstr9_long_haystack.md` — the hash-shell bug story. - `feedback_cpp_subset.md` — C++ subset, including the SjLj fix. - `feedback_ptr32_frame_limit.md` — was 5 days stale; updated 2026-05-07 to "DONE, 131/131 smoke green". - `feedback_jslpseudo_caller_save.md`, `feedback_libcall_img_clobber.md`, `feedback_img_slot_expansion.md`, `feedback_greedy_high_pressure.md` — related backend topics. - `feedback_loader_multi_seg_needs_expressload.md` — **new 2026-05-08**. Multi-seg OMFs need ExpressLoad to launch under real Loader. - `feedback_i32_shift_inline.md` — **new 2026-05-08**. Inline i32 shift-by-N for N=1..4; first quantified bench-vs-self speed win. - `feedback_speed_over_size.md` — **new 2026-05-07**. Optimization priorities: cycle count over byte count, full stop. ## Next session candidates (ranked) evalAt at 1.86× vs Calypsi is the structural floor for peephole work (see `feedback_evalat_structural_gap.md`). Further gains need: 1. **i64-by-pointer ABI** (rejected this session — diminishing returns). Pass doubles by ptr instead of value: saves ~120 cyc per evalAt call. Requires runtime rewrite, OMF compat checks, every double caller updated. Risk:reward too high for the size of the gain. 2. **__divdf3 / __adddf3 algorithmic improvements**. ddiv 1261 cyc could drop via Newton-Raphson reciprocal multiplication (a*1/b instead of bit-by-bit long division). Major rewrite, but our __muldi3 short-circuit makes the multiplications cheap now. 3. **Higher-resolution cycle timer**. HBL counter is 8-bit and wraps at ~256 ticks; combining scan-line position + frame counter would give per-bench resolution better than ±65 cyc. Would unblock benchmarking sub-loop changes (e.g., the LSR-dp shift form). 4. **More peepholes from the audit**. Phase 4 STA_StackRel extension landed but doesn't fire in current libc (frame sizes too large). If callers shrink frames via better SSM, more functions become eligible.