diff --git a/STATUS.md b/STATUS.md index 3f19959..152039c 100644 --- a/STATUS.md +++ b/STATUS.md @@ -32,7 +32,19 @@ which runs correctly under MAME (apple2gs). - Loops with goto / break / continue, nested loops, state machines. - `` varargs with int / long / unsigned long long mixed args. - Heap: `malloc` / `free` (libc.c first-fit allocator) — linked-list - reverse with `cons` works; free-list coalesce verified. + reverse with `cons` works; free-list coalesce verified. Under GS/OS + and GNO the heap is a locked Memory Manager handle (`__heapInitMM`); + bare metal uses link816's window above BSS, stopped at the $C000 I/O + page in banks $00/$01/$E0/$E1. Running the heap out returns NULL + (smoke: bare-metal exhaustion, GS/OS 26 KB BSS + heap + NewHandle). +- `memcpy` / `memmove` / `memset` (libgcc.s): MVN / MVP block moves at + 7 cycles a byte (the old C byte loops were ~70 / ~100 / ~55), a 16-bit + `[dp],y` loop below 34 bytes, runs split at every 64 KB bank boundary + of source and destination, a per-call stub on the stack (interrupt- and + thread-safe, any code bank), DBR and D preserved. memset is a 16-bit + store loop (5.9 cycles a byte). Smoke: byte-exact model over every + length class / bank crossing / overlap direction, DBR+D probe, and + injected mid-MVN interrupts whose handler moves memory. - Strings: hand-rolled `strlen`, `strcmp`, `strcpy`, `strchr`, atoi/itoa roundtrip. - Soft-float (single): all four ops + comparisons, MAME-verified. @@ -75,7 +87,16 @@ which runs correctly under MAME (apple2gs). cosh/tanh. Accuracy is in the ~1e-6 range — good enough for typical numeric work, far short of glibc-quality. These are slow (each call is dozens to hundreds of soft-double libcalls) - — pre-compute or cache when possible. + — pre-compute or cache when possible. Exact on the bits (C99 + Annex F special values throughout): fmod and remainder (long + division, any quotient), modf / modff, frexp / ldexp (subnormals, + rounded subnormal results, no 16-bit exponent wrap), fpclassify / + isnormal in the argument's own width (subnormals included). hypot + scales instead of overflowing; atan2 has every signed-zero / + infinity case. The soft-float itself flushes subnormal operands + and results to zero, so subnormals survive only these bit-level + functions. Smoke: tests/fpNan/genFpTables.py tables (special, + Annex F, edge) compared bit for bit on MAME. - `setjmp` / `longjmp` from libgcc.s. - Static constructors via crt0's init_array walk. - `` file I/O with two backends: @@ -85,20 +106,22 @@ which runs correctly under MAME (apple2gs). - **GS/OS** — `fopen` falls through to `gsosOpen` for any path not in the mfs table. Routes through the GS/OS class-1 dispatcher via wrappers in `runtime/src/iigsGsos.s` (Open/Read/Write/Close/ - SetMark/GetMark/SetEOF/GetEOF). The full stdio surface + Flush/SetMark/GetMark/SetEOF/GetEOF). stdio keeps no buffer of + its own; `fflush` (and `fflush(NULL)`) issues GS/OS Flush, so + written data and the file's EOF reach the volume without fclose + (smoke: write, fflush, stop MAME, read the file off the image; + without fflush the file is empty). The full stdio surface (`fread/fwrite/fseek/ftell/fclose/fgetc/fputc/fputs/fgets/ungetc/ feof/ferror/clearerr/rewind/fprintf/vfprintf`) dispatches on backend. link816 honors weak symbols so programs that don't use the GS/OS backend don't have to link `iigsGsos.o`. - - **Validation status:** code path compiles, links, and runs under - `runViaFinder.sh --data` injection. `fopen` + `gsosOpen` hangs - when invoked under real GS/OS 6.0.2 (JSL $E100A8 doesn't return); - root cause not yet diagnosed. Stub-dispatcher GS/OS smoke (the - existing one) validates the wrapper contract independently. An - XFAIL'd end-to-end smoke is in `scripts/smokeTest.sh` gated - behind `GSOS_FILE_SMOKE=1` for use after the dispatcher path is - fixed. `runViaFinder.sh --data /PATH=local_file` is the - automated-injection mechanism for runtime-test data files. + - **Validation status:** end to end under real GS/OS 6.0.2 via + `runViaFinder.sh`: fopen / fread of an injected file (smoke "GS/OS + fopen/fread", green since 2026-06-02 -- the old hang was crt0Gsos + re-zeroing BSS over GS/OS state), fwrite + fflush onto the disk + image (smoke "GS/OS fflush"), and a 26 KB-BSS program's file round + trip. `runViaFinder.sh --data /PATH=local_file` injects data files; + `--save-data ` hands back the data disk after the run. - stdin/stdout/stderr route through `putchar` as before. - ``: wcslen / wcscmp / wcsncmp / wcscpy / wcsncpy / wcscat / wcschr / wcsrchr; mbtowc / wctomb / mbstowcs / @@ -455,6 +478,152 @@ Work is now optimization-focused; the toolchain is feature-complete for the common-case C / minimal-C++ workload. Priority is speed (cycle counts), not size. +**Recently landed (2026-10-05, speed pass 2):** + +CoreMark 21800 -> 20583 ticks (-5.6%, CRCs match); Space Taxi's synced +attract segment 699 -> 751 presents (+7.4%, stRenderFrame -21% of its +cycles); UBER 51/51 hashes, input and sample-pixel rows +21..27%, +sprite draw/save +5..7%, no row slower (two readings 0.3-0.4% lower are +window quantization: traced, their per-iteration cycles went down); +corpus .text 765695 -> 741321 bytes (-3.2%). Each item below has a +W65816_DISABLE_* switch. + +- **Global arrays are `lda G+K,x` / `sta G+K,x`** (DBR-relative abs,X; + i32 halves off one X) when the access is based on a sized global: + the index is lowered to 16 bits, K stays in the operand only for an + index known non-negative (GLOBAL_INDEX; global-index-abs.ll; + globalArrFill/Sum 6308/6479 -> 2387 cycles). +- **Multiplies**: a product known to fit 16 bits is a 16-bit multiply + (`sim->spr[idx]`'s u8 x 6-byte record no longer calls __umulhisi3), + and an i32 times a two-signed-digit constant (6, 10, 12, 24, ...) is + shifts and an add (MUL_SHIFT_ADD); NarrowI32Mul does the same for + products it proves narrow across blocks (NARROW_PRODUCT; CoreMark's + bitextract -> __mulhi3, -0.34% cycles). +- **Light calls**: libgcc.s helpers and memcpy/memmove/memset are + JSLlight (clobber A, X, Y, $F0 only), so values stay in IMG0..7 + across them; loop homes take loops that call them, and run in + functions with more than 16 hot slots too (LIGHT_CALLS, + LOOP_HOMES_HOT). +- **isLegalAddressingMode**: no negative or scaled offsets, so LSR + stops building `adc #0xff..` pointer math (ADDR_MODES). +- **i8 constants** load as a 16-bit `lda #imm` (I8_IMM16). +- **IMG8..15 homes for far pointers across calls** (StackRelToImg, + after the loop homes): the entry swaps each slot with a free $C0..$CE + word, every return swaps back, FarPtrStageReuse addresses the + pointer in place. Taken only where it pays on every path (cheapest + path, loops once) or twice over by block frequency and a quarter on + every path (CALLEE_SAVED_HOMES). +- **Miscompiles fixed**: a spill's STAfi kept the opcode's implicit + def of A, so MachineCopyPropagation deleted a live `$a = COPY` before + it (spill-stafi-no-a-def.mir); a longjmp skips the returns that give + callers their IMG8..15 words back -- a returns-twice call now + clobbers IMG8..15, so the setjmp function keeps all eight for its + caller (setjmp-img-clobber.ll, tests/regress/setjmpImgSave.c in smoke). +- Rejected: coalescing heuristics (+5.9K..+16.8K instructions, CoreMark + +2.5..3.2%), region split off for single-register classes (+293 + instructions), SimplifyCFG sink-common off (a new greedy fallback), + a wider addressing mode (sprite restore -1.1%), IMG8..15 homes + unconditionally (12 UBER rows down, audio tick -18%) or by expected + saving alone (spriteMasks, hookPerTickHigh, computeCollisions + +11..102%), per-loop swaps (+3.2K instructions, +1 present). + Left: Y-indexed versioned pointer loops; short branches where BRL + pairs sit today (~0.3% of Space Taxi, ~0.5% of CoreMark; link816 + would have to relax them); CoreMark spends ~14% in multiply helpers. + +**Recently landed (2026-10-04, speed pass):** + +CoreMark 29450 -> 21800 ticks (-26% time, CRCs match); Space Taxi's +synced attract segment 575 -> 697 presents (+21%); every timed JoeyLib +UBER op at or above the old compiler (51/51 hashes, sprite draws +19..25%, +input +7..14%); corpus .text 935424 -> 764879 bytes (-18%). + +- **Far pointer offsets and indexes ride in Y**: an i32 load/store's + second half peels its constant offset into `ldy #` like the first + (constants already lowered to REG_SEQUENCE included); a zero-extended + 16-bit index whose known bits keep it in the bank goes into Y for + `[dp],y` instead of a 32-bit add (LD/ST/STB_PTR_IDX). +- **Frame slots ordered by access density** (orderFrameObjects, loop + depth weighted): hot slots sit near SP, so frames past 200 bytes + reach them `,s` instead of through $F6 (big-frame probe -3.2%). +- **StackSlotCleanup**: a slot that only forwards an IMG value is read + from the IMG (LDAimg); stores to slots nothing reads are erased and + the slots leave the frame (the AGI interpreter's vmStepCore frame + 940 -> 216 bytes, agiVm .text -51%). +- **Spills and reloads**: a virtual Wide32 spills/reloads both halves + by sub-register (it moved the low half only); reloads keep A when it + is read after them (aLiveFrom, which follows successors and reads a + return's uses); the far ($F6) IMG reload stored its value; the wrap + walk counts non-A STAfi as N/Z corrupters. +- **Miscompile fixed: StackRelToImg's `inc dp` fold** took a tied op + after the store (`and #0x1ff` reading the incremented A) for a write + of A, so `head = (head + 1) & 511` masked the byte just copied instead: + JoeyLib's IIgs serial read never caught up with the ring's tail + (verify-iigs-serial-loss hung). Latent; this pass's new code shape + exposed it. Phase 11's STZ peephole had the same scan order + (dp-inc-fold-tied-use.ll). +- **W65816SplitLoopExits** (IR): a critical loop exit into PHIs gets its + own block unless the value leaving is the one the loop carries + (core_list_find -27%). +- **SinkLoopInc** sees through a short single-predecessor chain; + **NarrowI32Mul** sends sext16 x sext16 to the new `__mulhisi3`. +- **libgcc multiplies and 16-bit divides**: 8x8 quarter squares (table + indexed by Y and X), eight shift-right steps when one operand is below + 256, sixteen otherwise (a 16-bit multiplier was ~730 cycles, now ~255); + one 16-bit divide core keeps the remainder in A and starts a dividend + below 256 eight steps in. `tests/regress/genMulDiv.py` (smoke) checks + every helper across operand size classes. +- **Stack releases by cost** (W65816Helpers::emitStackRelease, a cycle + worth two bytes): call cleanups and epilogues pick N/2 PLY, + TSC/CLC/ADC/TCS (A free: void calls, void returns) or TAY ... TYA; the + prologue drops TAY/TYA when no argument comes in A. It pulled up to + seven PLYs (35 cycles) after a call. +- **W65816ReturnPopMerge** (pre-emit): the releases on a path into a + return (last call's cleanup, epilogue, often behind a BRA to a shared + return block) become one, with a return of its own (jlDrawPixel's call + path 28 -> 9 cycles of release). +- **W65816CommuteReload** (pre-emit, last): `sta s ; lda x ; ora s` + (ORA/AND/EOR/ADC) is `sta s ; ora x` -- every far pointer null test. +- Rejected: Sched::Source (greedy allocation fails in 7 corpus TUs, + +982 instructions where it succeeds). Left: DP homes for loops that + call (Space Taxi's render loops call pasteCell per cell; `,s` + accesses are 24% of its cycles), Y-indexed versioned pointer loops. + +**Recently landed (2026-10-04):** + +- **Float/double compares against constants are integer tests** -- + `combineFPSetCCConst` (W65816ISelLowering) rewrites what InstCombine + makes of sign-bit and magnitude tests (`x ==/!= 0.0`, `x 0.0`, + `fabs(x) < c`, `isnan`, `x == x`) into exact integer compares, + NaN / signed-zero / subnormal correct; each used to be a soft-float + libcall (float `== 0` 1347 -> 716 cycles, double `fabs(x) < 2.5` + 1833 -> 835). i64 SETCC/BR_CC/SELECT_CC lower as one four-word + CMP/SBC chain (the old lowering got 63 i64 predicates wrong). +- **W65816SinkLoopInc** (pre-RA): a loop's pointer/counter step moves + below the last use of the old value, so the pointer is bumped in place + and the back-edge copy goes (sumPtr -31%, copy16 -20.5%, strcpy -20%). + Loop-local far-pointer DP homes now also cover loops whose exits are + shared or carry live A / flags (copy-then-call loops -26..-34%). +- **SpillToX bridges merged spill slots** dead after the reload + (two-loop probe -14.8%); X live-out and a PHP inside the window are + now checked. +- **Address of a weak function** takes the program bank ($BE) when the + function is linked and null when not (it was bank 0, so a call + through it crashed under GS/OS). +- **Four older miscompiles fixed**: StackRelToImg promoted incoming + stack arguments of small PHA-frame functions (libc `ungetc`, so scanf + pushback), and its `lda n,s ; pha ; lda n,s` elision deleted callee- + save restores (wmemcpy, rename, fminf, 11 Lua functions returned with + $C0 clobbered); StackSlotCleanup's wrap walk took byte stores as flag + definers (5th miss of that class); LowerWide32's never-reached + LDAptr32S rewrite was removed. +- Runtime: exact `math.c` edge cases, MVN `memcpy`/`memmove`/`memset`, + `fflush` -> GS/OS Flush, bare-metal heap stops at the I/O page (see + "What works"). CoreMark +2.3% over 2026-09-29, .text -1.0% across + the corpus. Left: Y-indexed versioned pointer loops, DP homes for + non-innermost loops and $F6 frames (the largest remaining staging + pools: snprintf, Space Taxi's render and sim), homes across calls + (measured break-even). + **Recently landed (2026-07-03):** - **StackSlotCleanup flag-wrap correctness fix** — a conditional branch diff --git a/demos/rsrcProbe.apl b/demos/rsrcProbe.apl index 119b3c1..1574791 100644 Binary files a/demos/rsrcProbe.apl and b/demos/rsrcProbe.apl differ diff --git a/docs/USAGE.md b/docs/USAGE.md index eb25ed8..a5a01dc 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -246,7 +246,7 @@ Each argument: | `--text-base 0x1000` | Where the code goes in memory. `0x1000` is conventional (first 4 KB of bank 0 is reserved for stack + zero page). | | `runtime/crt0.o` | **Must come first.** The C runtime startup — sets up the stack, calls `main`, halts cleanly on return. | | `runtime/libc.o` | Core C library (`printf`, `malloc`, `strlen`, etc.). | -| `runtime/libgcc.o` | Compiler-provided helpers for things the 65816 can't do natively (16×16 multiply, 32-bit divide, etc.). Required for almost every program. | +| `runtime/libgcc.o` | Compiler-provided helpers for things the 65816 can't do natively (16×16 multiply, 32-bit divide, etc.), plus `memcpy` / `memmove` / `memset` (MVN / MVP block moves; the compiler emits calls to them for struct copies and clears). Required for almost every program. | | `hello.o` | Your code. | `link816` will print something like: @@ -388,7 +388,7 @@ the rest. | `runtime/crt0.o` | **Always.** C runtime startup. | | `runtime/crt0Gsos.o` | Instead of `crt0.o` for programs launched by the GS/OS Loader. | | `runtime/libc.o` | `printf`, `malloc`, `strlen`, the usual. Almost always. | -| `runtime/libgcc.o` | Compiler helpers — multiply, divide, shift. Almost always. | +| `runtime/libgcc.o` | Compiler helpers — multiply, divide, shift — and `memcpy` / `memmove` / `memset`. Almost always. | | `runtime/snprintf.o` | If you use `sprintf` / `snprintf` / `vsnprintf`. | | `runtime/sscanf.o` | If you use `sscanf` / `vsscanf` / `fscanf`. | | `runtime/softDouble.o` | If you use `double`-precision arithmetic anywhere. | diff --git a/runtime/include/c++/cmath b/runtime/include/c++/cmath index 01c0c0b..b8b6b75 100644 --- a/runtime/include/c++/cmath +++ b/runtime/include/c++/cmath @@ -10,7 +10,7 @@ // inherited as-is from . `std::HUGE_VAL_v` is provided as a // constexpr alias since macros can't live inside namespaces. // -// Per the C++ standard, isnan/isinf/isfinite/signbit/fpclassify must be +// Per the C++ standard, isnan/isinf/isfinite/signbit/fpclassify/isnormal must be // functions when is in scope (not C-style type-generic macros). // We #undef the macros and re-declare them as inline functions // in namespace std and at global scope. @@ -32,6 +32,8 @@ #undef islessequal #undef islessgreater #undef isunordered +#undef fpclassify +#undef isnormal inline bool isnan(double x) { return ::__isnan_d(x) != 0; } inline bool isinf(double x) { return ::__isinf_d(x) != 0; } @@ -43,6 +45,11 @@ inline bool isless(double x, double y) { return __builtin_isless(x, y); } inline bool islessequal(double x, double y) { return __builtin_islessequal(x, y); } inline bool islessgreater(double x, double y) { return __builtin_islessgreater(x, y); } inline bool isunordered(double x, double y) { return __builtin_isunordered(x, y); } +// fpclassify(double) is 's extern "C" function; a float keeps its +// own width (see ). +inline int fpclassify(float x) { return ::__fpclassify_f(x); } +inline bool isnormal(float x) { return ::__fpclassify_f(x) == FP_NORMAL; } +inline bool isnormal(double x) { return ::fpclassify(x) == FP_NORMAL; } namespace std { @@ -177,6 +184,7 @@ using ::scalblnf; // ---- Classification (function form) --------------------------------- using ::fpclassify; +using ::isnormal; // ---- Comparison (function form per C++ ) --------------------- using ::isgreater; diff --git a/runtime/include/math.h b/runtime/include/math.h index b996b44..7dbdadb 100644 --- a/runtime/include/math.h +++ b/runtime/include/math.h @@ -160,12 +160,20 @@ double scalbln (double x, long n); float scalblnf(float x, long n); // ---- Classification ------------------------------------------------ +// fpclassify classifies in the argument's own width: a float goes to +// __fpclassify_f, so a float subnormal is FP_SUBNORMAL rather than the +// zero the widening to double would make it (the soft-float flushes +// subnormals). The values must match runtime/src/math.c's copies. #define FP_NAN 0 #define FP_INFINITE 1 #define FP_NORMAL 2 #define FP_SUBNORMAL 3 #define FP_ZERO 4 -int fpclassify(double x); +int (fpclassify)(double x); +int __fpclassify_f(float x); +#define fpclassify(x) (sizeof(x) == sizeof(float) ? \ + __fpclassify_f((float)(x)) : (fpclassify)((double)(x))) +#define isnormal(x) (fpclassify(x) == FP_NORMAL) // ---- Common constants ----------------------------------------------- // (Not in C99 strict, but defined by glibc/BSD math.h and widely used.) diff --git a/runtime/src/iigsGsos.s b/runtime/src/iigsGsos.s index ae49075..fd6b20d 100644 --- a/runtime/src/iigsGsos.s +++ b/runtime/src/iigsGsos.s @@ -50,6 +50,7 @@ .globl gsosRead .globl gsosWrite .globl gsosClose + .globl gsosFlush .globl gsosGetEOF .globl gsosSetEOF .globl gsosSetMark @@ -128,6 +129,11 @@ gsosWrite: gsosClose: gsosDispatch 0x2014, __gsosClosePb +; FlushGS: write the file's buffered blocks and directory entry (EOF) +; to the volume. libc's fflush reaches it through a hook fopen installs. +gsosFlush: + gsosDispatch 0x2015, __gsosFlushPb + gsosGetEOF: gsosDispatch 0x2019, __gsosGetEofPb diff --git a/runtime/src/iigsGsosStub.s b/runtime/src/iigsGsosStub.s index 3737f73..7c5df18 100644 --- a/runtime/src/iigsGsosStub.s +++ b/runtime/src/iigsGsosStub.s @@ -92,9 +92,9 @@ ; lying about success through the universal-success dispatcher. ; ; NOTE: this file does NOT define per-call symbols (gsosOpen, gsosRead, -; gsosWrite, gsosClose, gsosGetEOF, gsosSetEOF, gsosSetMark, gsosGetMark, -; gsosCreate, gsosDestroy, gsosChangePath, gsosGetPrefix, gsosGetFileInfo, -; gsosGetDirEntry). libc.c declares those as +; gsosWrite, gsosClose, gsosFlush, gsosGetEOF, gsosSetEOF, gsosSetMark, +; gsosGetMark, gsosCreate, gsosDestroy, gsosChangePath, gsosGetPrefix, +; gsosGetFileInfo, gsosGetDirEntry). libc.c declares those as ; undefined-weak, so when the stub is linked instead of iigsGsos.o ; they resolve to address 0 -- the libc-side callers all gate through ; __gsosAvailable() and short-circuit before dereferencing them. Any diff --git a/runtime/src/libc.c b/runtime/src/libc.c index 9df4b97..7a56665 100644 --- a/runtime/src/libc.c +++ b/runtime/src/libc.c @@ -1,6 +1,6 @@ // Minimal libc for the W65816 backend. Provides: -// string.h: memcpy, memset, memmove, memcmp, strlen, strcpy, strcmp, -// strncpy, strncmp, strchr, strrchr +// string.h: memcmp, strlen, strcpy, strcmp, strncpy, strncmp, strchr, +// strrchr (memcpy, memmove and memset are in libgcc.s) // ctype.h: isdigit, isalpha, isalnum, isspace, isupper, islower, // toupper, tolower, isxdigit, isprint, ispunct // stdlib.h: abs, labs, atoi @@ -150,6 +150,7 @@ extern u16 gsosOpen (__GsosOpenParm *p) __attribute__((weak, retain, used)) extern u16 gsosRead (__GsosIORecGS *p) __attribute__((weak, retain, used)); extern u16 gsosWrite (__GsosIORecGS *p) __attribute__((weak, retain, used)); extern u16 gsosClose (__GsosRefNumRecGS *p) __attribute__((weak, retain, used)); +extern u16 gsosFlush (__GsosRefNumRecGS *p) __attribute__((weak, retain, used)); extern u16 gsosGetEOF (__GsosEOFRecGS *p) __attribute__((weak, retain, used)); extern u16 gsosSetEOF (__GsosSetPosRecGS *p) __attribute__((weak, retain, used)); extern u16 gsosSetMark(__GsosSetPosRecGS *p) __attribute__((weak, retain, used)); @@ -169,7 +170,7 @@ extern u16 gsosGetDirEntry(__GsosDirEntryParm *p) __attribute__((weak, retain, u // source of truth that distinguishes "real GS/OS available" from // "stub linked" from "no GS/OS surface at all", so newly-added // wrappers can refuse to silently lie about success. -extern int __gsosIsRealImpl __attribute__((weak, retain, used)); +extern int __gsosIsRealImpl __attribute__((weak)); int __gsosAvailable(void) { if (&__gsosIsRealImpl == (int *)0) { @@ -179,31 +180,11 @@ int __gsosAvailable(void) { } // ---- string.h ---- - -void *memcpy(void *dst, const void *src, size_t n) { - char *d = (char *)dst; - const char *s = (const char *)src; - while (n--) *d++ = *s++; - return dst; -} - -void *memmove(void *dst, const void *src, size_t n) { - char *d = (char *)dst; - const char *s = (const char *)src; - if (d < s) { - while (n--) *d++ = *s++; - } else { - d += n; s += n; - while (n--) *--d = *--s; - } - return dst; -} - -void *memset(void *dst, int c, size_t n) { - char *d = (char *)dst; - while (n--) *d++ = (char)c; - return dst; -} +// +// memcpy, memmove and memset are MVN / MVP block moves in libgcc.s. +void *memcpy(void *dst, const void *src, size_t n); +void *memmove(void *dst, const void *src, size_t n); +void *memset(void *dst, int c, size_t n); int memcmp(const void *a, const void *b, size_t n) { const u8 *p = (const u8 *)a; @@ -519,8 +500,8 @@ char *stpncpy(char *dst, const char *src, size_t n) { // Linker-supplied weak symbols; fallback to fixed defaults so a static // link without crt0 still has SOMETHING. -extern char __heap_start[] __attribute__((weak, retain, used)); -extern char __heap_end[] __attribute__((weak, retain, used)); +extern char __heap_start[] __attribute__((weak)); +extern char __heap_end[] __attribute__((weak)); #define HEAP_DEFAULT_START ((char *)0x4000) #define HEAP_DEFAULT_END ((char *)0xBF00) @@ -554,10 +535,34 @@ static void heapSetWindow(char *start, char *end) { } +// Banks $00, $01, $E0 and $E1 have the I/O page at $C000-$CFFF. link816's +// heap window runs from the end of BSS to $FFFE of the entry bank, which is +// plain RAM under GS/OS (where __heapInitMM replaces the window with a +// Memory Manager handle anyway), but a bare-metal crt0.s program lives in +// bank 0, where the window spanned the soft switches: running the heap +// out zeroed them and took the machine down instead of returning NULL. A +// window in such a bank stops at the I/O page. +#define IO_PAGE_START 0xC000UL +#define BANK_MASK 0xFF0000UL +#define OFFSET_MASK 0x00FFFFUL + +static int bankHasIoPage(unsigned long a) { + unsigned long bank = a & BANK_MASK; + return bank == 0x000000UL || bank == 0x010000UL || + bank == 0xE00000UL || bank == 0xE10000UL; +} + + static void mallocInitOnce(void) { if (bumpPtr) return; - heapSetWindow(__heap_start ? __heap_start : HEAP_DEFAULT_START, - __heap_end ? __heap_end : HEAP_DEFAULT_END); + char *start = __heap_start ? __heap_start : HEAP_DEFAULT_START; + char *end = __heap_end ? __heap_end : HEAP_DEFAULT_END; + unsigned long s = (unsigned long)(void *)start; + if (bankHasIoPage(s) && (s & OFFSET_MASK) < IO_PAGE_START && + ((unsigned long)(void *)end & OFFSET_MASK) > IO_PAGE_START) { + end = (char *)(void *)((s & BANK_MASK) | IO_PAGE_START); + } + heapSetWindow(start, end); } // GS/OS and GNO: back the malloc heap with a locked Memory-Manager handle @@ -1178,7 +1183,7 @@ typedef struct __sFILE { // Zero-cost when nothing fails; lets an app report WHY a write/read died // (fwrite itself can only say "short") without a debugger on the target. // __gsosLastOp values: 1=Open 2=Create 3=Read 4=Write 5=SetEOF 6=SetMark -// 7=Close 8=GetEOF 9=GetMark. +// 7=Close 8=GetEOF 9=GetMark 10=Flush. u16 __gsosLastErr; u16 __gsosLastOp; u16 __gsosLastRefNum; @@ -1318,7 +1323,50 @@ int fputs(const char *s, FILE *stream) { return -1; } -int fflush(FILE *stream) { (void)stream; return 0; } +// fflush's way to GS/OS Flush ($2015), installed by fopen's GS/OS path, so +// a program that never opens a GS/OS file does not link the dispatch +// wrappers through fflush (as __autoDeleteFn below does for remove). +// `volatile` for the same reason as __autoDeleteFn. +typedef u16 (*__GsosFlushFn)(__GsosRefNumRecGS *p); +static __GsosFlushFn volatile __gsosFlushFn = (__GsosFlushFn)0; + + +// stdio keeps no buffer of its own -- fputc / fwrite hand every byte to +// GS/OS at once -- but the FST keeps written blocks and the file's EOF in +// memory until Flush or Close, so a crash or a power-off loses whatever +// was written since the open. fflush issues Flush for a GS/OS stream; +// other streams have nothing to push. Returns 0, or EOF (-1) with the +// stream's error flag set and __gsosLastOp = 10 when GS/OS fails. +static int fflushOne(FILE *stream) { + if (stream->kind != FILE_KIND_GSOS || !__gsosFlushFn) { + return 0; + } + __GsosRefNumRecGS r = { 1, stream->refNum }; + u16 rc = __gsosFlushFn(&r); + if (rc != 0) { + stream->err = 1; + __gsosLatch(10, rc, stream->refNum, 0, 0); + return -1; + } + return 0; +} + + +// fflush(NULL) flushes every open stream. +int fflush(FILE *stream) { + if (stream) { + return fflushOne(stream); + } + int rc = 0; + // Kept a loop: unrolled it is 21 calls for a rarely used form. + #pragma clang loop unroll(disable) + for (int i = 3; i < MFS_MAX_FILES; i++) { + if (fflushOne(&__mfs[i]) != 0) { + rc = -1; + } + } + return rc; +} // Indirect hook for tmpfile-auto-delete on fclose. When `tmpfile()` // is called it installs `remove` here; programs that never call @@ -1627,6 +1675,7 @@ FILE *fopen(const char *path, const char *mode) { f->unget = -1; f->path = path; f->refNum = op.refNum; + __gsosFlushFn = gsosFlush; if (truncate) { // "w" / "w+" — truncate to zero length. __GsosSetPosRecGS e = { 3, op.refNum, 0, 0 }; diff --git a/runtime/src/libcGno.c b/runtime/src/libcGno.c index b3a3b6b..4570734 100644 --- a/runtime/src/libcGno.c +++ b/runtime/src/libcGno.c @@ -46,6 +46,7 @@ typedef struct { uint16_t pCount; void *oldPathname; void *newPathname; } GnoCha #define GSOS_READ 0x2012 #define GSOS_WRITE 0x2013 #define GSOS_CLOSE 0x2014 +#define GSOS_FLUSH 0x2015 #define GSOS_SETMARK 0x2016 #define GSOS_GETMARK 0x2017 #define GSOS_SETEOF 0x2018 @@ -75,6 +76,7 @@ KEEP uint16_t gsosOpen(GnoOpenParm *p) { return __gnoGsosCall(p, GSOS_OPEN); KEEP uint16_t gsosRead(GnoIORec *p) { return __gnoGsosCall(p, GSOS_READ); } KEEP uint16_t gsosWrite(GnoIORec *p) { return __gnoGsosCall(p, GSOS_WRITE); } KEEP uint16_t gsosClose(GnoRefNumRec *p) { return __gnoGsosCall(p, GSOS_CLOSE); } +KEEP uint16_t gsosFlush(GnoRefNumRec *p) { return __gnoGsosCall(p, GSOS_FLUSH); } KEEP uint16_t gsosGetEOF(GnoEOFRec *p) { return __gnoGsosCall(p, GSOS_GETEOF); } KEEP uint16_t gsosSetEOF(GnoEOFRec *p) { return __gnoGsosCall(p, GSOS_SETEOF); } KEEP uint16_t gsosSetMark(GnoMarkRec *p) { return __gnoGsosCall(p, GSOS_SETMARK); } diff --git a/runtime/src/libgcc.s b/runtime/src/libgcc.s index 9cacdef..bd69236 100644 --- a/runtime/src/libgcc.s +++ b/runtime/src/libgcc.s @@ -2,7 +2,10 @@ ; Provides the helpers that the LLVM backend lowers integer multiply, ; shift, divide, and modulo operations to. Implementations are ; correct-but-unoptimised; they exist to unblock end-to-end testing, -; not to compete with hand-tuned 65816 math libraries. +; not to compete with hand-tuned 65816 math libraries. The exception is +; memcpy / memmove / memset at the end of this file (MVN / MVP block +; moves, in their own section): the backend lowers struct copies and +; clears to them too, so they live here rather than in libc.c. ; ; Calling convention (matches W65816ISelLowering::LowerCall): ; - Arg 0 in A (16-bit M). @@ -165,113 +168,684 @@ __heapInitMM: rtl ; -------------------------------------------------------------------- -; __mulhi3 — 16-bit multiply. A * (4,S) -> A. -; Signed and unsigned share an implementation: only the low 16 bits of -; the product are returned, which is identical for both. Uses -; shift-and-add over the multiplier bits. +; 16x16 multiplies. One method, inlined in __mulhi3 and __umulhisi3 and +; a JSR subroutine (__umul16_core) for __mulhisi3 and __mulsi3: the +; multiplier in $e0, the multiplicand in $e2, the product's low word +; returned in A and its high word in X. Clobbers Y and $e0..$e5 (never +; the IMG words $c0..$df: W65816StackRelToImg keeps values there across +; these calls). +; Both operands below 256: quarter squares, a*b = T[a+b] - T[|a-b|] +; with T[k] = k*k/4 (__umulhisi3_qsq), ~60 cycles. +; One below 256: eight steps with it as the multiplier, ~150 cycles. +; Otherwise sixteen steps, ~255 cycles. +; A step of the shift-right method: the multiplier's next bit is in C; if +; set, the product's high word gains the multiplicand (the carry out in +; C); then the high word and the multiplier word shift right together, +; the product's low bits filling the multiplier word from the top as its +; own bits leave at the bottom. The loop this replaced shifted the +; multiplicand left in 32 bits and added in 32 bits, ~44 cycles a +; multiplier bit: ~730 cycles for a 16-bit multiplier. ; -------------------------------------------------------------------- + +; __mulhi3 -- 16-bit multiply, A * (4,S) -> A. Signed and unsigned are +; the same low word. .globl __mulhi3 __mulhi3: - sta 0xe0 ; multiplier - lda 0x4, s - sta 0xe2 ; multiplicand - stz 0xe4 ; running product = 0 -.Lmul_loop: - lda 0xe0 - beq .Lmul_done - lsr a sta 0xe0 - bcc .Lmul_skip - lda 0xe4 + ora 0x4, s + and #0xff00 + bne .Lmh_wide + ; Both below 256: a*b = T[a+b] - T[|a-b|], T[k] = k*k/4, the two + ; entries indexed by Y and X. + lda 0xe0 clc - adc 0xe2 - sta 0xe4 -.Lmul_skip: - asl 0xe2 - bra .Lmul_loop -.Lmul_done: - lda 0xe4 - rtl - -; -------------------------------------------------------------------- -; __umulhisi3 — unsigned 16x16 -> 32 multiply. A = multiplier (16-bit), -; (4,s) = multiplicand (16-bit). Returns A:X = 32-bit product (A=lo, -; X=hi). Used by the i32-mul-of-zext-i16 DAG combine in the W65816 -; backend to avoid the full __mulsi3 (32x32 -> 32) call when the user -; writes `(u32)a * b` with a/b as u16 (e.g. `unsigned long s += i*i;`). -; ~16 iterations instead of 16-32 for the equivalent __mulsi3 fast path, -; AND avoids zext/zero-fill overhead in arg setup. -; -; The accumulator and shifted multiplicand both need 32-bit space: -; $e0/$e1 multiplier (shifted right; tested for add) -; $e2..$e5 multiplicand (16-bit input, shifts up into the high -; half over the loop) -; $e6..$e9 32-bit product accumulator -; -------------------------------------------------------------------- - .globl __umulhisi3 -__umulhisi3: - ; Fast path: if both args fit in 8 bits, use a quarter-square - ; table for 8x8 → 16 (result hi half is always 0). Identity: - ; a*b = T[a+b] - T[|a-b|] where T[k] = floor(k²/4) - ; 511-entry, 2-byte table = 1022 bytes ROM. Fast path is ~55 - ; cyc vs ~150 cyc for the shift-and-add fallback at i ~ 50. - ; Hot in `i*i` patterns where i is a loop counter ≤ 255. - sta 0xe0 ; save arg0 - ora 0x4, s ; OR with arg1 - and #0xff00 ; isolate hi bytes - bne .Lumulhisi_slow ; if either hi byte nonzero, fallback - ; Both args fit in 8 bits — quarter-square path. - lda 0xe0 ; arg0 - clc - adc 0x4, s ; A = a + b - asl a ; A = 2*(a+b) — word index - tax - lda __umulhisi3_qsq, x ; T[a+b] - sta 0xe2 ; stash - lda 0xe0 ; arg0 + adc 0x4, s + asl a + tay + lda 0xe0 sec - sbc 0x4, s ; A = a - b - bpl .Lumulhisi_qsq_pos - ; A negative; negate to get |a-b|. + sbc 0x4, s + bcs .Lmh_qpos eor #0xffff inc a -.Lumulhisi_qsq_pos: - asl a ; A = 2*|a-b| +.Lmh_qpos: + asl a tax - lda 0xe2 ; T[a+b] + lda __umulhisi3_qsq, y sec - sbc __umulhisi3_qsq, x ; T[a+b] - T[|a-b|] - ldx #0 ; result hi half = 0 + sbc __umulhisi3_qsq, x rtl -.Lumulhisi_slow: - ; Multiplier already at $e0 from the fast-path check above. +.Lmh_wide: lda 0x4, s - sta 0xe2 ; multiplicand lo in $e2/$e3 - stz 0xe4 ; multiplicand hi (initially 0) in $e4/$e5 - stz 0xe6 ; product lo at $e6/$e7 - stz 0xe8 ; product hi at $e8/$e9 -.Lumulhisi_loop: + sta 0xe2 lda 0xe0 - beq .Lumulhisi_done - lsr a - sta 0xe0 - bcc .Lumulhisi_skip - ; Add 32-bit multiplicand to 32-bit product. + and #0xff00 + beq .Lmh_m8 + lda 0xe2 + and #0xff00 + bne .Lmh_m16 + ; The multiplicand is the short one: swap, so eight steps do. + lda 0xe0 + ldx 0xe2 + stx 0xe0 + sta 0xe2 +.Lmh_m8: + lda #0x0 + lsr 0xe0 + bcc .Lmh_e0 clc - lda 0xe6 adc 0xe2 - sta 0xe6 - lda 0xe8 - adc 0xe4 - sta 0xe8 -.Lumulhisi_skip: - ; Shift 32-bit multiplicand left by 1. - asl 0xe2 - rol 0xe4 - bra .Lumulhisi_loop -.Lumulhisi_done: - ldx 0xe8 - lda 0xe6 +.Lmh_e0: + ror a + ror 0xe0 + bcc .Lmh_e1 + clc + adc 0xe2 +.Lmh_e1: + ror a + ror 0xe0 + bcc .Lmh_e2 + clc + adc 0xe2 +.Lmh_e2: + ror a + ror 0xe0 + bcc .Lmh_e3 + clc + adc 0xe2 +.Lmh_e3: + ror a + ror 0xe0 + bcc .Lmh_e4 + clc + adc 0xe2 +.Lmh_e4: + ror a + ror 0xe0 + bcc .Lmh_e5 + clc + adc 0xe2 +.Lmh_e5: + ror a + ror 0xe0 + bcc .Lmh_e6 + clc + adc 0xe2 +.Lmh_e6: + ror a + ror 0xe0 + bcc .Lmh_e7 + clc + adc 0xe2 +.Lmh_e7: + ror a + ror 0xe0 + ; A = product bits 23..8, $e1 = bits 7..0. + sta 0xe4 + xba + and #0xff00 + sta 0xe4 + lda 0xe0 + xba + and #0xff + ora 0xe4 + rtl +.Lmh_m16: + lda #0x0 + lsr 0xe0 + bcc .Lmh_s0 + clc + adc 0xe2 +.Lmh_s0: + ror a + ror 0xe0 + bcc .Lmh_s1 + clc + adc 0xe2 +.Lmh_s1: + ror a + ror 0xe0 + bcc .Lmh_s2 + clc + adc 0xe2 +.Lmh_s2: + ror a + ror 0xe0 + bcc .Lmh_s3 + clc + adc 0xe2 +.Lmh_s3: + ror a + ror 0xe0 + bcc .Lmh_s4 + clc + adc 0xe2 +.Lmh_s4: + ror a + ror 0xe0 + bcc .Lmh_s5 + clc + adc 0xe2 +.Lmh_s5: + ror a + ror 0xe0 + bcc .Lmh_s6 + clc + adc 0xe2 +.Lmh_s6: + ror a + ror 0xe0 + bcc .Lmh_s7 + clc + adc 0xe2 +.Lmh_s7: + ror a + ror 0xe0 + bcc .Lmh_s8 + clc + adc 0xe2 +.Lmh_s8: + ror a + ror 0xe0 + bcc .Lmh_s9 + clc + adc 0xe2 +.Lmh_s9: + ror a + ror 0xe0 + bcc .Lmh_s10 + clc + adc 0xe2 +.Lmh_s10: + ror a + ror 0xe0 + bcc .Lmh_s11 + clc + adc 0xe2 +.Lmh_s11: + ror a + ror 0xe0 + bcc .Lmh_s12 + clc + adc 0xe2 +.Lmh_s12: + ror a + ror 0xe0 + bcc .Lmh_s13 + clc + adc 0xe2 +.Lmh_s13: + ror a + ror 0xe0 + bcc .Lmh_s14 + clc + adc 0xe2 +.Lmh_s14: + ror a + ror 0xe0 + bcc .Lmh_s15 + clc + adc 0xe2 +.Lmh_s15: + ror a + ror 0xe0 + lda 0xe0 + rtl + +; __umulhisi3 -- unsigned 16x16 -> 32: A * (4,S) -> A (low) : X (high). +; W65816NarrowI32Mul calls it for an i32 multiply of two values whose high +; words are known zero. + .globl __umulhisi3 +__umulhisi3: + sta 0xe0 + ora 0x4, s + and #0xff00 + bne .Lmu_wide + ; Both below 256: a*b = T[a+b] - T[|a-b|], T[k] = k*k/4, the two + ; entries indexed by Y and X. + lda 0xe0 + clc + adc 0x4, s + asl a + tay + lda 0xe0 + sec + sbc 0x4, s + bcs .Lmu_qpos + eor #0xffff + inc a +.Lmu_qpos: + asl a + tax + lda __umulhisi3_qsq, y + sec + sbc __umulhisi3_qsq, x + ldx #0x0 + rtl +.Lmu_wide: + lda 0x4, s + sta 0xe2 + lda 0xe0 + and #0xff00 + beq .Lmu_m8 + lda 0xe2 + and #0xff00 + bne .Lmu_m16 + ; The multiplicand is the short one: swap, so eight steps do. + lda 0xe0 + ldx 0xe2 + stx 0xe0 + sta 0xe2 +.Lmu_m8: + lda #0x0 + lsr 0xe0 + bcc .Lmu_e0 + clc + adc 0xe2 +.Lmu_e0: + ror a + ror 0xe0 + bcc .Lmu_e1 + clc + adc 0xe2 +.Lmu_e1: + ror a + ror 0xe0 + bcc .Lmu_e2 + clc + adc 0xe2 +.Lmu_e2: + ror a + ror 0xe0 + bcc .Lmu_e3 + clc + adc 0xe2 +.Lmu_e3: + ror a + ror 0xe0 + bcc .Lmu_e4 + clc + adc 0xe2 +.Lmu_e4: + ror a + ror 0xe0 + bcc .Lmu_e5 + clc + adc 0xe2 +.Lmu_e5: + ror a + ror 0xe0 + bcc .Lmu_e6 + clc + adc 0xe2 +.Lmu_e6: + ror a + ror 0xe0 + bcc .Lmu_e7 + clc + adc 0xe2 +.Lmu_e7: + ror a + ror 0xe0 + ; A = product bits 23..8, $e1 = bits 7..0. + sta 0xe4 + xba + and #0xff + tax + lda 0xe4 + xba + and #0xff00 + sta 0xe4 + lda 0xe0 + xba + and #0xff + ora 0xe4 + rtl +.Lmu_m16: + lda #0x0 + lsr 0xe0 + bcc .Lmu_s0 + clc + adc 0xe2 +.Lmu_s0: + ror a + ror 0xe0 + bcc .Lmu_s1 + clc + adc 0xe2 +.Lmu_s1: + ror a + ror 0xe0 + bcc .Lmu_s2 + clc + adc 0xe2 +.Lmu_s2: + ror a + ror 0xe0 + bcc .Lmu_s3 + clc + adc 0xe2 +.Lmu_s3: + ror a + ror 0xe0 + bcc .Lmu_s4 + clc + adc 0xe2 +.Lmu_s4: + ror a + ror 0xe0 + bcc .Lmu_s5 + clc + adc 0xe2 +.Lmu_s5: + ror a + ror 0xe0 + bcc .Lmu_s6 + clc + adc 0xe2 +.Lmu_s6: + ror a + ror 0xe0 + bcc .Lmu_s7 + clc + adc 0xe2 +.Lmu_s7: + ror a + ror 0xe0 + bcc .Lmu_s8 + clc + adc 0xe2 +.Lmu_s8: + ror a + ror 0xe0 + bcc .Lmu_s9 + clc + adc 0xe2 +.Lmu_s9: + ror a + ror 0xe0 + bcc .Lmu_s10 + clc + adc 0xe2 +.Lmu_s10: + ror a + ror 0xe0 + bcc .Lmu_s11 + clc + adc 0xe2 +.Lmu_s11: + ror a + ror 0xe0 + bcc .Lmu_s12 + clc + adc 0xe2 +.Lmu_s12: + ror a + ror 0xe0 + bcc .Lmu_s13 + clc + adc 0xe2 +.Lmu_s13: + ror a + ror 0xe0 + bcc .Lmu_s14 + clc + adc 0xe2 +.Lmu_s14: + ror a + ror 0xe0 + bcc .Lmu_s15 + clc + adc 0xe2 +.Lmu_s15: + ror a + ror 0xe0 + tax + lda 0xe0 + rtl + +; __umul16_core -- the same as a JSR subroutine: $e0 * $e2 -> A : X. +__umul16_core: + lda 0xe0 + ora 0xe2 + and #0xff00 + bne .Lmc_wide + ; Both below 256: a*b = T[a+b] - T[|a-b|], T[k] = k*k/4, the two + ; entries indexed by Y and X. + lda 0xe0 + clc + adc 0xe2 + asl a + tay + lda 0xe0 + sec + sbc 0xe2 + bcs .Lmc_qpos + eor #0xffff + inc a +.Lmc_qpos: + asl a + tax + lda __umulhisi3_qsq, y + sec + sbc __umulhisi3_qsq, x + ldx #0x0 + rts +.Lmc_wide: + lda 0xe0 + and #0xff00 + beq .Lmc_m8 + lda 0xe2 + and #0xff00 + bne .Lmc_m16 + ; The multiplicand is the short one: swap, so eight steps do. + lda 0xe0 + ldx 0xe2 + stx 0xe0 + sta 0xe2 +.Lmc_m8: + lda #0x0 + lsr 0xe0 + bcc .Lmc_e0 + clc + adc 0xe2 +.Lmc_e0: + ror a + ror 0xe0 + bcc .Lmc_e1 + clc + adc 0xe2 +.Lmc_e1: + ror a + ror 0xe0 + bcc .Lmc_e2 + clc + adc 0xe2 +.Lmc_e2: + ror a + ror 0xe0 + bcc .Lmc_e3 + clc + adc 0xe2 +.Lmc_e3: + ror a + ror 0xe0 + bcc .Lmc_e4 + clc + adc 0xe2 +.Lmc_e4: + ror a + ror 0xe0 + bcc .Lmc_e5 + clc + adc 0xe2 +.Lmc_e5: + ror a + ror 0xe0 + bcc .Lmc_e6 + clc + adc 0xe2 +.Lmc_e6: + ror a + ror 0xe0 + bcc .Lmc_e7 + clc + adc 0xe2 +.Lmc_e7: + ror a + ror 0xe0 + ; A = product bits 23..8, $e1 = bits 7..0. + sta 0xe4 + xba + and #0xff + tax + lda 0xe4 + xba + and #0xff00 + sta 0xe4 + lda 0xe0 + xba + and #0xff + ora 0xe4 + rts +.Lmc_m16: + lda #0x0 + lsr 0xe0 + bcc .Lmc_s0 + clc + adc 0xe2 +.Lmc_s0: + ror a + ror 0xe0 + bcc .Lmc_s1 + clc + adc 0xe2 +.Lmc_s1: + ror a + ror 0xe0 + bcc .Lmc_s2 + clc + adc 0xe2 +.Lmc_s2: + ror a + ror 0xe0 + bcc .Lmc_s3 + clc + adc 0xe2 +.Lmc_s3: + ror a + ror 0xe0 + bcc .Lmc_s4 + clc + adc 0xe2 +.Lmc_s4: + ror a + ror 0xe0 + bcc .Lmc_s5 + clc + adc 0xe2 +.Lmc_s5: + ror a + ror 0xe0 + bcc .Lmc_s6 + clc + adc 0xe2 +.Lmc_s6: + ror a + ror 0xe0 + bcc .Lmc_s7 + clc + adc 0xe2 +.Lmc_s7: + ror a + ror 0xe0 + bcc .Lmc_s8 + clc + adc 0xe2 +.Lmc_s8: + ror a + ror 0xe0 + bcc .Lmc_s9 + clc + adc 0xe2 +.Lmc_s9: + ror a + ror 0xe0 + bcc .Lmc_s10 + clc + adc 0xe2 +.Lmc_s10: + ror a + ror 0xe0 + bcc .Lmc_s11 + clc + adc 0xe2 +.Lmc_s11: + ror a + ror 0xe0 + bcc .Lmc_s12 + clc + adc 0xe2 +.Lmc_s12: + ror a + ror 0xe0 + bcc .Lmc_s13 + clc + adc 0xe2 +.Lmc_s13: + ror a + ror 0xe0 + bcc .Lmc_s14 + clc + adc 0xe2 +.Lmc_s14: + ror a + ror 0xe0 + bcc .Lmc_s15 + clc + adc 0xe2 +.Lmc_s15: + ror a + ror 0xe0 + tax + lda 0xe0 + rts + +; __mulhisi3 -- signed 16x16 -> 32: A * (4,S) -> A (low) : X (high). +; W65816NarrowI32Mul calls it for an i32 multiply of two sign-extended +; 16-bit values (and __mulsi3 jumps here for them). The magnitudes go +; through the unsigned core, so small negative operands take its short +; paths too; the product is negated when the signs differ. + .globl __mulhisi3 +__mulhisi3: + sta 0xe0 + eor 0x4, s + sta 0xe6 ; bit 15: the product is negative + lda 0xe0 + bpl .Lmhs_apos + eor #0xffff + inc a + sta 0xe0 +.Lmhs_apos: + lda 0x4, s + bpl .Lmhs_bpos + eor #0xffff + inc a +.Lmhs_bpos: + sta 0xe2 + jsr __umul16_core + ldy 0xe6 + bpl .Lmhs_done + eor #0xffff + clc + adc #0x1 + sta 0xe4 + txa + eor #0xffff + adc #0x0 + tax + lda 0xe4 +.Lmhs_done: rtl ; -------------------------------------------------------------------- @@ -334,149 +908,107 @@ __ashrhi3: rtl ; -------------------------------------------------------------------- -; __udivhi3 — A unsigned / (4,S) -> A. -; Restoring shift-subtract division. Common helper; __umodhi3 reuses -; the algorithm and returns the remainder instead. -; Scratch: $e6 = numerator, $e8 = denominator, -; $ea = quotient, $ec = remainder. +; 16-bit divide and modulo. __udivmod16 (JSR): $e0 / $e2 -> quotient in +; $e0, remainder in A. Each step shifts the dividend's top bit into the +; remainder, held in A; a 17th bit carried out of A means the remainder is +; past any divisor, so it subtracts without comparing. The quotient's +; bits enter $e0 from the bottom as the dividend's leave at the top. A +; dividend below 256 starts eight bits in. ~28 cycles a step (the loop +; this replaced shifted three words a step, ~45). +; Signed: magnitudes through the same core; the quotient negated when the +; signs differ, the remainder given the dividend's sign (C99). ; -------------------------------------------------------------------- +__udivmod16: + ldy #0x10 + lda 0xe0 + and #0xff00 + bne .Lud16_go + lda 0xe0 + xba + sta 0xe0 + ldy #0x8 +.Lud16_go: + lda #0x0 +.Lud16_loop: + asl 0xe0 + rol a + bcs .Lud16_sub + cmp 0xe2 + bcc .Lud16_skip +.Lud16_sub: + sbc 0xe2 + inc 0xe0 +.Lud16_skip: + dey + bne .Lud16_loop + rts + .globl __udivhi3 __udivhi3: - ; Public entry: A=dividend, (4,S)=divisor. Set up scratch and - ; call the same JSR-based core used by signed divide. - sta 0xe6 + sta 0xe0 lda 0x4, s - sta 0xe8 - jsr __udivmod_core - lda 0xea + sta 0xe2 + jsr __udivmod16 + lda 0xe0 rtl .globl __umodhi3 __umodhi3: - sta 0xe6 + sta 0xe0 lda 0x4, s - sta 0xe8 - jsr __udivmod_core - lda 0xec + sta 0xe2 + jsr __udivmod16 rtl -; -------------------------------------------------------------------- -; __divhi3 / __modhi3 — signed 16-bit divide and modulo. Strategy: -; - Stash sign of dividend in $ee bit 0 (used by modulo). -; - Stash result sign of quotient (sign(a) XOR sign(b)) in $ee bit 1 -; (used by divide). -; - Take absolute values, run the unsigned core, then negate the -; appropriate result if its sign bit is set. -; C99: quotient truncates toward zero; remainder takes the sign of the -; dividend. -; -------------------------------------------------------------------- .globl __divhi3 __divhi3: - jsr __divmod_setup - jsr __udivmod_core - ; Quotient is in $ea. Negate if bit 1 of $ee is set. - pei 0xea - lda 0xee - and #0x2 - beq .Ldiv_pos - pla + sta 0xe0 + eor 0x4, s + sta 0xe4 ; bit 15: the quotient is negative + lda 0xe0 + bpl .Ldh_apos eor #0xffff - clc - adc #0x1 - rtl -.Ldiv_pos: - pla + inc a + sta 0xe0 +.Ldh_apos: + lda 0x4, s + bpl .Ldh_bpos + eor #0xffff + inc a +.Ldh_bpos: + sta 0xe2 + jsr __udivmod16 + lda 0xe0 + ldy 0xe4 + bpl .Ldh_done + eor #0xffff + inc a +.Ldh_done: rtl .globl __modhi3 __modhi3: - jsr __divmod_setup - jsr __udivmod_core - ; Remainder is in $ec. Negate if bit 0 of $ee is set (dividend - ; was negative). - pei 0xec - lda 0xee - and #0x1 - beq .Lmod_pos - pla + sta 0xe0 + sta 0xe4 ; bit 15: the dividend (and remainder) is negative + lda 0xe0 + bpl .Lmh_apos eor #0xffff - clc - adc #0x1 + inc a + sta 0xe0 +.Lmh_apos: + lda 0x4, s + bpl .Lmh_bpos + eor #0xffff + inc a +.Lmh_bpos: + sta 0xe2 + jsr __udivmod16 + ldy 0xe4 + bpl .Lmh_done + eor #0xffff + inc a +.Lmh_done: rtl -.Lmod_pos: - pla - rtl - -; -------------------------------------------------------------------- -; __divmod_setup — common prologue for __divhi3/__modhi3. Reads -; A=dividend and (4,S)=divisor (the public-entry stack frame is intact -; because we used JSR not JSL, so (4,S) still points to the user's -; pushed arg1 relative to the original JSL). Computes |a| -> $e6, -; |b| -> $e8, and sign tracker -> $ee: -; bit 0 = 1 if dividend was negative (modulo result sign) -; bit 1 = 1 if dividend XOR divisor signs differ (quotient sign) -; Uses JSR/RTS, same bank. -; -------------------------------------------------------------------- -__divmod_setup: - ; Sign tracker. STZ doesn't touch A — preserves the value - ; we still need below. - stz 0xee - ; Dividend sign + abs value. - cmp #0x8000 - bcc .Lset_a_pos - ; Negative: set bits 0 and 1 (dividend sign, result sign so far). - pha - lda 0xee - ora #0x3 - sta 0xee - pla - eor #0xffff - clc - adc #0x1 -.Lset_a_pos: - sta 0xe6 - ; Divisor sign + abs value. After our JSR (pushed 2 bytes of - ; near-return), the user's arg1 has shifted up by 2 from (4,S) - ; to (6,S). - lda 0x6, s - cmp #0x8000 - bcc .Lset_b_pos - ; Negative: flip bit 1 of $ee (XOR with sign of dividend). - pha - lda 0xee - eor #0x2 - sta 0xee - pla - eor #0xffff - clc - adc #0x1 -.Lset_b_pos: - sta 0xe8 - rts - -; -------------------------------------------------------------------- -; __udivmod_core — internal restoring divide. Inputs at $e6/$e8, -; outputs quotient at $ea, remainder at $ec. JSR/RTS local helper. -; -------------------------------------------------------------------- -__udivmod_core: - stz 0xea - stz 0xec - ldx #0x10 -.Lcore_loop: - asl 0xe6 - rol 0xec - asl 0xea - lda 0xec - cmp 0xe8 - bcc .Lcore_skip - sec - sbc 0xe8 - sta 0xec - inc 0xea -.Lcore_skip: - dex - bne .Lcore_loop - rts ; ==================================================================== ; 32-bit (long / si) helpers. @@ -499,166 +1031,78 @@ __udivmod_core: ; ==================================================================== ; -------------------------------------------------------------------- -; __mulsi3 — 32-bit multiply. Shift-and-add over 32 bits of the -; multiplier. Result = (a * b) mod 2^32. +; __mulsi3 -- 32-bit multiply, (a * b) mod 2^32. ; ; ABI: A = a_lo, X = a_hi (the i32-first-arg in A:X convention), ; (4,s) = b_lo, (6,s) = b_hi. Result returned in A:X (lo:hi). +; +; Both high words zero: one 16x16 multiply. Both operands sign-extended +; 16-bit values (`(long)s16 * s16`, and the small negatives the old +; 32-step loop paid ~1100 cycles for): __mulhisi3, whose (4,s) is our +; b_lo. Otherwise a_lo*b_lo + ((a_lo*b_hi + a_hi*b_lo) << 16), three +; trips through __umul16_core. ; -------------------------------------------------------------------- .globl __mulsi3 __mulsi3: - ; Stash a (multiplier) into $e0/$e2. - sta 0xe0 - stx 0xe2 - ; Stash b (multiplicand) into $e4/$e6. + sta 0xe8 ; a_lo + stx 0xea ; a_hi lda 0x4, s - sta 0xe4 + sta 0xec ; b_lo lda 0x6, s - sta 0xe6 - ; Clear running product at $e8/$ea. - stz 0xe8 - stz 0xea - ; Asymmetric-size swap: the loop iterates based on the multiplier's - ; bit width. If a is large (a_hi != 0) but b is small (b_hi == 0), - ; swap so the small one becomes the multiplier — the early-exit - ; check (lda 0xe0; beq done) then fires sooner. Saves a lot for - ; djb2-style `h * 33` patterns where h grows but the constant - ; multiplier is small. Cost: ~12 cyc unconditional check; benefit: - ; ~10 cyc per saved iter × ~10 iters = 100 cyc. - lda 0xe2 - beq .Lmulsi_no_swap ; a_hi == 0, no need to swap (or already covered by u16 path below) - lda 0xe6 - bne .Lmulsi_no_swap ; b_hi != 0, both large, no benefit - ; Swap (a_lo, a_hi) ↔ (b_lo, b_hi). Both are at DP; XOR-swap or - ; load/store; we use load/store with A scratch since the values are - ; 16 bits each. - lda 0xe0 - ldx 0xe4 - stx 0xe0 - sta 0xe4 - lda 0xe2 - ldx 0xe6 - stx 0xe2 - sta 0xe6 -.Lmulsi_no_swap: - ; Fast path: if multiplier's high half ($e2) is 0, we only - ; need 16 loop iterations AND we can drop the multiplier-hi - ; shift step entirely (lsr $e2 + bcc + ora #$8000) — that step - ; only ever fires when hi has bits to shift out, which it - ; doesn't here. Saves ~8 cyc/iter × 16 iters = ~128 cyc/call - ; vs the generic 16-iter path. Common in C code where both - ; source operands are zext'd from i16 (e.g. `i*i` with i a - ; `unsigned short`) — sumOfSquares benchmark hits this on every - ; iteration. - lda 0xe2 - bne .Lmulsi_full - ; Quarter-square fast path: when BOTH args fit in 8 bits (e2==0 - ; already verified, e6==0, e0<256, e4<256), use the same T[k]=k^2/4 - ; table that __umulhisi3 uses to compute a*b = T[a+b] - T[|a-b|] - ; in ~50 cyc vs the u16 loop's ~150-250 cyc for small multipliers. - ; Common when dotProduct etc. sign-extend small i16 values to i32. - lda 0xe6 - bne .Lmulsi_u16_entry - lda 0xe0 - ora 0xe4 - and #0xff00 - bne .Lmulsi_u16_entry - ; T[a+b] -> $e8 (product lo). - lda 0xe0 - clc - adc 0xe4 + sta 0xee ; b_hi + ora 0xea + bne .Lmsi_wide + lda 0xe8 + sta 0xe0 + lda 0xec + sta 0xe2 + jsr __umul16_core + rtl +.Lmsi_wide: + ; a_hi == sign(a_lo) and b_hi == sign(b_lo)? `asl ; lda #0 ; adc + ; #$ffff ; eor #$ffff` makes $ffff from a set bit 15, 0 from a clear one. + lda 0xe8 asl a - tax - lda __umulhisi3_qsq, x - sta 0xe8 - ; |a-b| as word index. - lda 0xe0 - sec - sbc 0xe4 - bpl .Lmulsi_qsq_pos + lda #0x0 + adc #0xffff eor #0xffff - inc a -.Lmulsi_qsq_pos: + cmp 0xea + bne .Lmsi_full + lda 0xec asl a + lda #0x0 + adc #0xffff + eor #0xffff + cmp 0xee + bne .Lmsi_full + lda 0xe8 + brl __mulhisi3 +.Lmsi_full: + lda 0xe8 + sta 0xe0 + lda 0xee + sta 0xe2 + jsr __umul16_core ; a_lo * b_hi, low word + sta 0xe6 + lda 0xea + sta 0xe0 + lda 0xec + sta 0xe2 + jsr __umul16_core ; a_hi * b_lo, low word + clc + adc 0xe6 + sta 0xe6 + lda 0xe8 + sta 0xe0 + lda 0xec + sta 0xe2 + jsr __umul16_core ; a_lo * b_lo + sta 0xe4 + txa + clc + adc 0xe6 tax - lda 0xe8 - sec - sbc __umulhisi3_qsq, x - ldx #0 - rtl -.Lmulsi_u16_entry: - ldy #0x10 -.Lmulsi_u16_loop: - ; Shift multiplier right; bit-out tested for add. Bottom of loop - ; checks multiplier==0 BEFORE the multiplicand shift, so on the - ; iter that clears the multiplier we save 14 cyc of unused - ; asl/rol on the multiplicand. Combined with the early-exit - ; saves ~30 cyc/call on small multipliers (1-50 range typical - ; for sumOfSquares). - lda 0xe0 - lsr a - sta 0xe0 - bcc .Lmulsi_u16_noadd - clc - lda 0xe8 - adc 0xe4 - sta 0xe8 - lda 0xea - adc 0xe6 - sta 0xea -.Lmulsi_u16_noadd: - lda 0xe0 - beq .Lmulsi_done - asl 0xe4 - rol 0xe6 - dey - bne .Lmulsi_u16_loop -.Lmulsi_done: - ldx 0xea - lda 0xe8 - rtl -.Lmulsi_full: - ldy #0x20 -.Lmulsi_loop: - ; Test bit 0 of multiplier (lo word). - lda 0xe0 - lsr a - sta 0xe0 - bcc .Lmulsi_noadd - ; Add multiplicand to product (32-bit). - clc - lda 0xe8 - adc 0xe4 - sta 0xe8 - lda 0xea - adc 0xe6 - sta 0xea -.Lmulsi_noadd: - ; Stream multiplier hi's LSB into lo's MSB so subsequent iters - ; test bits 16..31 via the same lo-bit test. - lsr 0xe2 - bcc .Lmulsi_no_borrow - lda 0xe0 - ora #0x8000 - sta 0xe0 -.Lmulsi_no_borrow: - ; Early exit: if BOTH halves of multiplier are 0, no more bits - ; remain. Saves the multiplicand shift on the terminating iter - ; AND the rest of the loop on small multipliers. - lda 0xe0 - bne .Lmulsi_shift_mc - lda 0xe2 - beq .Lmulsi_full_done -.Lmulsi_shift_mc: - ; Shift multiplicand left for the next iter's potential add. - asl 0xe4 - rol 0xe6 - dey - bne .Lmulsi_loop -.Lmulsi_full_done: - ; Result is in $e8 (lo) / $ea (hi). - ldx 0xea - lda 0xe8 + lda 0xe4 rtl ; -------------------------------------------------------------------- @@ -734,56 +1178,93 @@ __ashrsi3: rtl ; -------------------------------------------------------------------- -; __udivmodsi_core — internal 32-bit unsigned divide. Inputs in -; $e0/$e2 (numerator) and $e4/$e6 (denominator); outputs quotient in -; $e8/$ea and remainder in $ec/$ee. 32-iteration restoring divide. -; JSR/RTS local helper. +; __udivmodsi_core -- internal 32-bit unsigned divide (JSR / RTS). +; Dividend in $e0/$e2, divisor in $e4/$e6; quotient out in $e0/$e2, +; remainder in $e8/$ea. +; +; A divisor below 2^16 keeps the remainder in A, as __udivmod16 does: +; the dividend's high word first (skipped, quotient high word 0, when it +; is already below the divisor), then its low word, 16 steps each. A +; divisor of 2^16 or more makes a quotient below 2^16: the remainder +; starts as the dividend's high word and 16 steps shift the low word +; through a 32-bit remainder (a bit carried out of it means the remainder +; is past the divisor). ~450..900 cycles for the first form and ~1000 +; for the second, against ~2400 for the 32 three-word steps before. ; -------------------------------------------------------------------- __udivmodsi_core: - stz 0xe8 - stz 0xea - stz 0xec - stz 0xee ; was `sta 0xee` — A held b_hi at entry, - ; so for divisors > 0xFFFF (b_hi != 0) - ; the remainder started contaminated and - ; produced wrong quotients. Bug masked - ; for b_hi==0 (e.g. /60, /1000) because - ; sta-of-zero == stz. Caught by /86400. - ldy #0x20 -.Lcoresi_loop: - ; Shift numerator left through remainder. - asl 0xe0 - rol 0xe2 - rol 0xec - rol 0xee - ; Shift quotient left. - asl 0xe8 - rol 0xea - ; Compare remainder to denominator (32-bit). - lda 0xee - cmp 0xe6 - bcc .Lcoresi_skip - bne .Lcoresi_take - lda 0xec + lda 0xe6 + bne .Lud32_wide + lda 0xe2 cmp 0xe4 - bcc .Lcoresi_skip -.Lcoresi_take: - ; Remainder >= denominator: subtract and set quotient bit 0. - sec - lda 0xec + bcs .Lud32_hi + stz 0xe2 + bra .Lud32_lo +.Lud32_hi: + lda #0x0 + ldy #0x10 +.Lud32_hloop: + asl 0xe2 + rol a + bcs .Lud32_hsub + cmp 0xe4 + bcc .Lud32_hskip +.Lud32_hsub: sbc 0xe4 - sta 0xec - lda 0xee - sbc 0xe6 - sta 0xee - inc 0xe8 -.Lcoresi_skip: + inc 0xe2 +.Lud32_hskip: dey - bne .Lcoresi_loop + bne .Lud32_hloop +.Lud32_lo: + ldy #0x10 +.Lud32_lloop: + asl 0xe0 + rol a + bcs .Lud32_lsub + cmp 0xe4 + bcc .Lud32_lskip +.Lud32_lsub: + sbc 0xe4 + inc 0xe0 +.Lud32_lskip: + dey + bne .Lud32_lloop + sta 0xe8 + stz 0xea + rts +.Lud32_wide: + lda 0xe2 + sta 0xe8 + stz 0xea + stz 0xe2 + ldy #0x10 +.Lud32_wloop: + asl 0xe0 + rol 0xe8 + rol 0xea + bcs .Lud32_wsub + lda 0xea + cmp 0xe6 + bcc .Lud32_wskip + bne .Lud32_wsub + lda 0xe8 + cmp 0xe4 + bcc .Lud32_wskip +.Lud32_wsub: + lda 0xe8 + sec + sbc 0xe4 + sta 0xe8 + lda 0xea + sbc 0xe6 + sta 0xea + inc 0xe0 +.Lud32_wskip: + dey + bne .Lud32_wloop rts ; -------------------------------------------------------------------- -; __udivsi3 — unsigned 32/32 -> 32 divide. +; __udivsi3 -- unsigned 32/32 -> 32 divide. ; -------------------------------------------------------------------- .globl __udivsi3 __udivsi3: @@ -795,12 +1276,12 @@ __udivsi3: lda 0x6, s sta 0xe6 jsr __udivmodsi_core - ldx 0xea - lda 0xe8 + ldx 0xe2 + lda 0xe0 rtl ; -------------------------------------------------------------------- -; __umodsi3 — unsigned 32/32 -> 32 modulo. +; __umodsi3 -- unsigned 32/32 -> 32 modulo. ; -------------------------------------------------------------------- .globl __umodsi3 __umodsi3: @@ -811,12 +1292,12 @@ __umodsi3: lda 0x6, s sta 0xe6 jsr __udivmodsi_core - ldx 0xee - lda 0xec + ldx 0xea + lda 0xe8 rtl ; -------------------------------------------------------------------- -; __divsi3 / __modsi3 — signed 32-bit divide / modulo. Strategy mirrors +; __divsi3 / __modsi3 -- signed 32-bit divide / modulo. Strategy mirrors ; the i16 helpers: stash signs, take abs, run unsigned core, negate ; result(s) as needed. Sign tracker bits in $f0: ; bit 0 = dividend was negative (modulo result sign) @@ -826,11 +1307,33 @@ __umodsi3: __divsi3: jsr __divmodsi_setup jsr __udivmodsi_core - ; Quotient at $e8/$ea. Negate if bit 1 of $f0 is set. + ; Quotient at $e0/$e2. Negate if bit 1 of $f0 is set. lda 0xf0 and #0x2 beq .Ldivsi_pos - ; 32-bit two's complement of quotient. + lda 0xe0 + eor #0xffff + clc + adc #0x1 + sta 0xe0 + lda 0xe2 + eor #0xffff + adc #0x0 + sta 0xe2 +.Ldivsi_pos: + ldx 0xe2 + lda 0xe0 + rtl + + .globl __modsi3 +__modsi3: + jsr __divmodsi_setup + jsr __udivmodsi_core + ; Remainder at $e8/$ea. Negate if bit 0 of $f0 set (dividend + ; was negative -- C99 remainder takes dividend's sign). + lda 0xf0 + and #0x1 + beq .Lmodsi_pos lda 0xe8 eor #0xffff clc @@ -840,34 +1343,11 @@ __divsi3: eor #0xffff adc #0x0 sta 0xea -.Ldivsi_pos: +.Lmodsi_pos: ldx 0xea lda 0xe8 rtl - .globl __modsi3 -__modsi3: - jsr __divmodsi_setup - jsr __udivmodsi_core - ; Remainder at $ec/$ee. Negate if bit 0 of $f0 set (dividend - ; was negative — C99 remainder takes dividend's sign). - lda 0xf0 - and #0x1 - beq .Lmodsi_pos - lda 0xec - eor #0xffff - clc - adc #0x1 - sta 0xec - lda 0xee - eor #0xffff - adc #0x0 - sta 0xee -.Lmodsi_pos: - ldx 0xee - lda 0xec - rtl - ; -------------------------------------------------------------------- ; __divmodsi_setup — common prologue for __divsi3 / __modsi3. ; Reads A=a_lo, X=a_hi (i32-first-arg ABI), (4,s)=b_lo, (6,s)=b_hi. @@ -1567,3 +2047,510 @@ __umulhisi3_qsq: .short 0xf040, 0xf138, 0xf231, 0xf32a, 0xf424, 0xf51e, 0xf619, 0xf714 .short 0xf810, 0xf90c, 0xfa09, 0xfb06, 0xfc04, 0xfd02, 0xfe01 + +; -------------------------------------------------------------------- +; memcpy / memmove / memset -- block moves with MVN / MVP. +; +; C signatures (size_t is 32 bits): +; void *memcpy (void *dst, const void *src, size_t n); +; void *memmove(void *dst, const void *src, size_t n); +; void *memset (void *dst, int c, size_t n); +; dst arrives in A (offset) : X (bank, pad byte 0); the other arguments +; are on the stack after the 3-byte return address: src at 4,s and n at +; 8,s, or c at 4,s and n at 6,s. All three return dst in A:X. +; +; Counts below MEM_SHORT copy through [dp],y with 16-bit loads and stores +; (10.5 cycles a byte): MVN's setup costs as much as that many bytes of +; the loop. A long-indirect-indexed address is a full 24-bit sum, so +; these loops cross bank boundaries on their own. +; +; Longer copies run MVN (forward) or MVP (backward), 7 cycles a byte. +; Both wrap X and Y inside their banks, so a copy is cut into runs that +; stop at every 64 KB boundary of the source and of the destination (most +; copies are one run). The bank operands are immediates, so each call +; builds its own 4-byte stub on the stack -- MVN/MVP, dst bank, src bank, +; RTL -- and RTLs into it; the stack is always bank 0, so this works from +; any code bank. Nothing shared is modified: an interrupt that lands +; mid-move (MVN/MVP resume after RTI) may itself call these routines, as +; may another GNO thread, without disturbing the interrupted move. MVN / +; MVP set DBR to the destination bank; the caller's DBR is saved on entry +; and restored on exit. D is never changed; the direct-page bytes used +; are the $E0-$EB libcall scratch, so an interrupt handler that runs C +; must switch D (or save that scratch) like it must for any C code. +; +; memset needs no block move: a 16-bit store loop fills at 5.9 cycles a +; byte, faster than MVN, without its setup. It fills in passes of at +; most MEM_PASS bytes so the 16-bit index never wraps. +; -------------------------------------------------------------------- + +MEM_SRC = 0xe0 ; DP: source (memset: fill) far pointer +MEM_DST = 0xe4 ; DP: destination far pointer, the result +MEM_REM = 0xe8 ; DP: memset bytes still to fill, 32 bits +MEM_SHORT = 34 ; from this count on, copies use MVN / MVP +MEM_PASS = 0x8000 ; memset pass length for fills of 64 KB up +OP_MVN = 0x54 +OP_MVP = 0x44 +OP_RTL = 0x6b +OP_FWD = 0x10 ; the opcode bit that tells MVN from MVP + +; The multi-run frame, as stack offsets from its base S0 (the stack +; pointer at the top of every run). +F_CNT = 1 ; bytes - 1 in this run (MVN/MVP's A) +F_REM = 3 ; bytes still to move, 32 bits +F_STUB = 7 ; stub: opcode, dst bank, src bank, RTL +F_DBR = 11 ; the caller's DBR + + .section .text.memcpy,"ax",@progbits + .globl memcpy + .globl memmove + .globl memset + +memcpy: + rep #0x30 + sta MEM_DST + stx MEM_DST+2 + lda 4, s + sta MEM_SRC + lda 6, s + sta MEM_SRC+2 + lda 10, s ; n.hi + bne .LmemFwdFar + lda 8, s ; n.lo + cmp #MEM_SHORT + bcs .LmemFwdLong + tay + beq .LmemRet +; Copy [0, Y) top down in words, an odd last byte first: right for any +; copy but an overlapping one with dst below src. Y < $8000. +.LmemDown: + tya + lsr a ; C = odd count + bcc .LmemDownPair + dey + sep #0x20 + .a8 + lda [MEM_SRC], y + sta [MEM_DST], y + rep #0x20 + .a16 +.LmemDownPair: + dey + dey ; the last whole word, or $FFFE: none left + bmi .LmemRet +.LmemDownWords: + lda [MEM_SRC], y + sta [MEM_DST], y + dey + dey + bpl .LmemDownWords +.LmemRet: + lda MEM_DST + ldx MEM_DST+2 + rtl + +; One MVN when neither block crosses a bank boundary. A = n < 64 KB. +.LmemFwdLong: + dec a + tax ; n - 1 + clc + adc MEM_SRC + bcs .LmemFwdFar + txa + clc + adc MEM_DST + bcs .LmemFwdFar + ldx MEM_SRC + ldy MEM_DST + lda #OP_MVN +; One MVN / MVP (A = its opcode) of n bytes from X to Y, in the source's +; and the destination's banks. The stub's RTL returns to .LmemOneBack. +.LmemOne: + phb + sep #0x20 + .a8 + xba ; keep the opcode in B + lda #OP_RTL + pha + lda MEM_SRC+2 + pha + lda MEM_DST+2 + pha + xba + pha ; the stub is 4,s..7,s from here + phk + per .LmemOneBack-1 + lda #0 + pha ; the stack's bank + rep #0x20 + .a16 + tsc + clc + adc #4 ; the stub's address - 1 + pha + lda 19, s ; n.lo (n was 8,s on entry, 11 bytes ago) + dec a + rtl +.LmemOneBack: + pla ; drop the stub + pla + plb + lda MEM_DST + ldx MEM_DST+2 + rtl +.LmemFwdFar: + brl .LmemFwdRuns + +memmove: + rep #0x30 + sta MEM_DST + stx MEM_DST+2 + lda 4, s + sta MEM_SRC + lda 6, s + sta MEM_SRC+2 + ; dst below src moves forward, dst above src backward; the pad + ; bytes above the banks take no part in the compare. + sep #0x20 + .a8 + lda MEM_DST+2 + cmp MEM_SRC+2 + rep #0x20 + .a16 + bne .LmemOrdered + lda MEM_DST + cmp MEM_SRC + beq .LmemMoveRet +.LmemOrdered: + bcc .LmemMoveUp + lda 10, s ; n.hi + bne .LmemBwdRuns + lda 8, s ; n.lo + cmp #MEM_SHORT + bcs .LmemBwdLong + tay + beq .LmemMoveRet + brl .LmemDown +.LmemMoveUp: + lda 10, s ; n.hi + bne .LmemMoveFwdRuns + lda 8, s ; n.lo + cmp #MEM_SHORT + bcs .LmemMoveFwdLong + tay + beq .LmemMoveRet +; Copy [0, n) bottom up in words, an odd last byte last: right for an +; overlapping copy with dst below src. A = n; C survives the loop. + ldy #0 + lsr a ; C = odd count, A = whole words + tax + beq .LmemUpOdd +.LmemUpWords: + lda [MEM_SRC], y + sta [MEM_DST], y + iny + iny + dex + bne .LmemUpWords +.LmemUpOdd: + bcc .LmemMoveRet + sep #0x20 + .a8 + lda [MEM_SRC], y + sta [MEM_DST], y + rep #0x20 + .a16 +.LmemMoveRet: + lda MEM_DST + ldx MEM_DST+2 + rtl +.LmemMoveFwdLong: + brl .LmemFwdLong +.LmemMoveFwdRuns: + brl .LmemFwdRuns + +; One MVP from the last bytes when neither block crosses a bank +; boundary. A = n < 64 KB. +.LmemBwdLong: + dec a + tay ; n - 1 + clc + adc MEM_SRC + bcs .LmemBwdRuns + tax ; the source's last byte + tya + clc + adc MEM_DST + bcs .LmemBwdRuns + tay ; the destination's last byte + lda #OP_MVP + brl .LmemOne + +; Several runs, backward: X / Y = the last byte's offset in its bank, the +; stub's bank bytes = that byte's bank. +.LmemBwdRuns: + phb + pea OP_RTL << 8 + pea OP_MVP + ; S0 + 6 here: the stub is 1,s..4,s and n 13,s..16,s. + lda MEM_SRC + clc + adc 13, s + tax + sep #0x20 + .a8 + lda MEM_SRC+2 + adc 15, s + sta 3, s + rep #0x20 + .a16 + txa + bne .LmemBwdSrc + sep #0x20 + .a8 + lda 3, s + dec a + sta 3, s + rep #0x20 + .a16 +.LmemBwdSrc: + dex + lda MEM_DST + clc + adc 13, s + tay + sep #0x20 + .a8 + lda MEM_DST+2 + adc 15, s + sta 2, s + rep #0x20 + .a16 + tya + bne .LmemBwdDst + sep #0x20 + .a8 + lda 2, s + dec a + sta 2, s + rep #0x20 + .a16 +.LmemBwdDst: + dey + bra .LmemRem + +; Several runs, forward: X / Y = the first byte's offset. +.LmemFwdRuns: + phb + pea OP_RTL << 8 + pea OP_MVN + sep #0x20 + .a8 + lda MEM_SRC+2 + sta 3, s + lda MEM_DST+2 + sta 2, s + rep #0x20 + .a16 + ldx MEM_SRC + ldy MEM_DST +.LmemRem: + lda 15, s ; n.hi + pha + lda 15, s ; n.lo + pha + pha ; F_CNT +; One run: as many bytes as fit before the source or the destination +; leaves its bank, or the rest of the copy if that is less. +.LmemRun: + lda F_STUB, s + and #OP_FWD + beq .LmemRunDown + txa + eor #0xffff ; bytes - 1 up to the source bank's end + sta F_CNT, s + tya + eor #0xffff ; and to the destination bank's end + bra .LmemRunMin +.LmemRunDown: + txa ; bytes - 1 down to the source bank's start + sta F_CNT, s + tya ; and to the destination bank's start +.LmemRunMin: + cmp F_CNT, s + bcs .LmemRunRem + sta F_CNT, s +.LmemRunRem: + lda F_REM+2, s + bne .LmemRunGo ; 64 KB or more to go: a bank edge is first + lda F_REM, s + dec a + cmp F_CNT, s + bcs .LmemRunGo + sta F_CNT, s +.LmemRunGo: + lda F_CNT, s + eor #0xffff ; -(bytes in this run) + clc + adc F_REM, s + sta F_REM, s + lda F_REM+2, s + adc #0xffff + sta F_REM+2, s + ; RTL into the stub (bank 0), whose RTL comes back here. + phk + per .LmemRunBack-1 + sep #0x20 + .a8 + lda #0 + pha + rep #0x20 + .a16 + tsc + clc + adc #F_STUB+3 ; S0 - 4 now: the stub's address - 1 + pha + lda F_CNT+6, s + rtl +.LmemRunBack: + lda F_REM, s + ora F_REM+2, s + beq .LmemRunsDone + ; X / Y wrapped to 0 (forward) or $FFFF (backward) where the source / + ; destination left its bank: step that bank byte. + lda F_STUB, s + and #OP_FWD + sep #0x20 + .a8 + beq .LmemStepDown + cpx #0 + bne .LmemStepUpDst + lda F_STUB+2, s + inc a + sta F_STUB+2, s +.LmemStepUpDst: + cpy #0 + bne .LmemStepDone + lda F_STUB+1, s + inc a + sta F_STUB+1, s + bra .LmemStepDone +.LmemStepDown: + cpx #0xffff + bne .LmemStepDownDst + lda F_STUB+2, s + dec a + sta F_STUB+2, s +.LmemStepDownDst: + cpy #0xffff + bne .LmemStepDone + lda F_STUB+1, s + dec a + sta F_STUB+1, s +.LmemStepDone: + rep #0x20 + .a16 + bra .LmemRun +.LmemRunsDone: + tsc + clc + adc #F_DBR-1 ; drop the run count, the rest and the stub + tcs + plb + lda MEM_DST + ldx MEM_DST+2 + rtl + +memset: + rep #0x30 + sta MEM_DST + stx MEM_DST+2 + sta MEM_SRC ; the fill pointer moves; MEM_DST is the result + stx MEM_SRC+2 + lda 4, s ; c: only its low byte counts + sep #0x20 + .a8 + xba + lda 4, s + rep #0x20 + .a16 + tax ; X = the fill byte, twice + lda 8, s ; n.hi + sta MEM_REM+2 + bne .LmemSetBig + lda 6, s ; n.lo + tay + beq .LmemSetRet +; Fill [0, Y) top down: an odd last byte, words down to a multiple of 8 +; bytes, then 8 bytes a loop. +.LmemSetFill: + tya + lsr a ; C = odd count + bcc .LmemSetPeel + dey + txa + sep #0x20 + .a8 + sta [MEM_SRC], y + rep #0x20 + .a16 +.LmemSetPeel: + tya + bit #0x0006 + beq .LmemSetOcts + txa + dey + dey + sta [MEM_SRC], y + bra .LmemSetPeel +.LmemSetOcts: + txa + cpy #0 + beq .LmemSetDone +.LmemSetOct: + dey + dey + sta [MEM_SRC], y + dey + dey + sta [MEM_SRC], y + dey + dey + sta [MEM_SRC], y + dey + dey + sta [MEM_SRC], y + bne .LmemSetOct +.LmemSetDone: + lda MEM_REM+2 + bne .LmemSetStep +.LmemSetRet: + lda MEM_DST + ldx MEM_DST+2 + rtl +; 64 KB or more: MEM_PASS bytes a pass while the rest's high word is +; nonzero (so it is nonzero after every such pass's fill), then the rest. +.LmemSetBig: + lda 6, s ; n.lo + sta MEM_REM +.LmemSetPass: + ldy #MEM_PASS + bra .LmemSetFill +.LmemSetStep: + lda MEM_SRC + clc + adc #MEM_PASS + sta MEM_SRC + lda MEM_SRC+2 + adc #0 + sta MEM_SRC+2 + lda MEM_REM + sec + sbc #MEM_PASS + sta MEM_REM + lda MEM_REM+2 + sbc #0 + sta MEM_REM+2 + bne .LmemSetPass + ldy MEM_REM + bne .LmemSetFill + bra .LmemSetRet diff --git a/runtime/src/math.c b/runtime/src/math.c index 1e3e24c..906095a 100644 --- a/runtime/src/math.c +++ b/runtime/src/math.c @@ -49,6 +49,28 @@ static float fFromBits(uint32_t b) { #define D_SIGN 0x8000000000000000ULL #define D_INF 0x7FF0000000000000ULL #define D_ONE 0x3FF0000000000000ULL +#define D_MANT 0x000FFFFFFFFFFFFFULL // the stored mantissa bits +#define D_LEAD 0x0010000000000000ULL // the implicit leading 1 +#define D_HUGE2 0x7FE0000000000000ULL // 2^1023: doubling it overflows +#define D_TINY2 0x0020000000000000ULL // 2^-1021: halving it loses a bit +#define D_BIAS 1023 +#define D_EMAX 1023 // the largest finite exponent +#define D_EMIN (-1022) // the smallest normal exponent +#define D_MBITS 52 // stored mantissa bits +#define F_SIGN 0x80000000UL +#define F_MANT 0x007FFFFFUL +#define F_EXP 0xFFu // a float's biased exponent field +#define F_BIAS 127 +#define F_MBITS 23 + +// fpclassify's answers. MUST stay in lock-step with math.h, which this +// file cannot include (the smoke test builds it without the runtime's +// include path, where would be the host's). +#define FP_NAN 0 +#define FP_INFINITE 1 +#define FP_NORMAL 2 +#define FP_SUBNORMAL 3 +#define FP_ZERO 4 // Used before their definitions (this file does not include math.h). @@ -101,6 +123,56 @@ static int isZeroBits(uint64_t b) { } +// The mantissa of the finite, nonzero magnitude a with its leading 1 at +// bit 52, and its unbiased exponent: a subnormal is shifted up until it +// has one (the soft-float flushes subnormals, so this works on the bits). +static uint64_t dUnpack(uint64_t a, int *eOut) { + int e = (int)(a >> D_MBITS); + uint64_t m = a & D_MANT; + if (e == 0) { + e = 1; + while ((m & D_LEAD) == 0) { + m <<= 1; + e--; + } + } else { + m |= D_LEAD; + } + *eOut = e - D_BIAS; + return m; +} + + +// sign | m * 2^(e - 52), m's leading 1 at bit 52. Past the largest +// exponent it is an infinity; below the smallest normal one it is rounded +// to nearest, ties to even, into a subnormal or zero (a carry into bit 52 +// lands on the smallest normal, as it should). e is a long so that huge +// ldexp scales cannot wrap a 16-bit int. +static double dBuild(uint64_t sign, uint64_t m, long e) { + unsigned shift; + uint64_t q; + uint64_t rest; + uint64_t half; + if (e > D_EMAX) { + return dFromBits(sign | D_INF); + } + if (e >= D_EMIN) { + return dFromBits(sign | ((uint64_t)(e + D_BIAS) << D_MBITS) | (m & D_MANT)); + } + if (e < D_EMIN - D_MBITS - 1) { + return dFromBits(sign); // under half the smallest subnormal + } + shift = (unsigned)(D_EMIN - e); + q = m >> shift; + rest = m & (((uint64_t)1 << shift) - 1); + half = (uint64_t)1 << (shift - 1); + if (rest > half || (rest == half && (q & 1))) { + q++; + } + return dFromBits(sign | q); +} + + // sqrt and log of an infinity or a NaN: +inf maps to itself, a NaN to // itself quieted (x + x does both), -inf is a domain error like any other // negative argument. Without this the series below turned +inf into a @@ -182,10 +254,9 @@ NO_SHRINK(ceil) float ceilf(float x) { } -// fmod(x, y) = x - trunc(x / y) * y, which takes the sign of x. The -// special cases follow C99 Annex F: a NaN operand, an infinite x or a zero -// y give NaN; an infinite y or a zero x give x back. They are screened on -// the bits, which also replaced a `y == 0.0` libcall compare. +// fmod's special cases follow C99 Annex F: a NaN operand, an infinite x or +// a zero y give NaN; an infinite y gives x back. They are screened on the +// bits, which also replaced a `y == 0.0` libcall compare. static COLD double fmodSpecial(double x, double y) { uint64_t xb = dToBits(x); uint64_t yb = dToBits(y); @@ -195,31 +266,59 @@ static COLD double fmodSpecial(double x, double y) { if (isNonFinite(xb) || isZeroBits(yb)) { return __builtin_nan(""); // domain error } - return x; // infinite y, or a zero x + return x; // infinite y } +// fmod(x, y) is always exactly representable, so it is computed exactly: +// long division of the mantissas, one quotient bit a step, keeping only +// the remainder (as musl and fdlibm do). It takes x's sign, a zero one +// included. x - trunc(x / y) * y, used before, was off by whole multiples +// of y once x / y rounded (fmod(1e20, 0.1), fmod(-1e300, 7)). The loop +// runs once per bit of exponent difference, so fmod(DBL_MAX, 1e-300) is +// slow but exact. double fmod(double x, double y) { uint64_t xb = dToBits(x); uint64_t yb = dToBits(y); - if (isNonFinite(xb) || isNonFinite(yb) || isZeroBits(yb) || isZeroBits(xb)) { + uint64_t sign; + uint64_t xa; + uint64_t ya; + uint64_t mx; + uint64_t my; + int ex; + int ey; + if (isNonFinite(xb) || isNonFinite(yb) || isZeroBits(yb)) { return fmodSpecial(x, y); } - double q = x / y; - // trunc: floor for positives, ceil for negatives. The sign test reads - // the bits, which is cheaper than a `<` libcall compare. - double t; - if (dToBits(q) >> 63) { - t = -floor(-q); - } else { - t = floor(q); + sign = xb & D_SIGN; + xa = xb & ~D_SIGN; + ya = yb & ~D_SIGN; + if (xa <= ya) { + // |x| < |y| is its own remainder (a zero x too); |x| == |y| leaves 0. + return (xa == ya) ? dFromBits(sign) : x; } - double r = x - t * y; - // An exact multiple leaves x + -x, which is +0; the result takes x's sign. - if (isZeroBits(dToBits(r))) { - return (xb >> 63) ? -0.0 : 0.0; + mx = dUnpack(xa, &ex); + my = dUnpack(ya, &ey); + for (; ex > ey; ex--) { + if (mx >= my) { + mx -= my; + if (mx == 0) { + return dFromBits(sign); + } + } + mx <<= 1; } - return r; + if (mx >= my) { + mx -= my; + if (mx == 0) { + return dFromBits(sign); + } + } + while ((mx & D_LEAD) == 0) { + mx <<= 1; + ex--; + } + return dBuild(sign, mx, ex); } @@ -399,6 +498,8 @@ float powf(float x, float y) { #define M_PI 3.14159265358979323846 #define M_2PI 6.28318530717958647692 #define M_HALF_PI 1.57079632679489661923 +#define M_QUART_PI 0.78539816339744830962 +#define M_3QUAR_PI 2.35619449019234492885 #define M_LN2 0.69314718055994530942 @@ -523,6 +624,18 @@ float expf(float x) { } +// 1/(2n+1) for the log and atan series: a table of the correctly rounded +// constants (the run-time division gave the same values at ~60000 cycles +// each). +#define LOG_TERMS 8 +#define ATAN_TERMS 18 // ~1e-15 at tan(pi/8) +static const double kInvOdd[ATAN_TERMS + 1] = { + 1.0, 1.0 / 3.0, 1.0 / 5.0, 1.0 / 7.0, 1.0 / 9.0, 1.0 / 11.0, 1.0 / 13.0, + 1.0 / 15.0, 1.0 / 17.0, 1.0 / 19.0, 1.0 / 21.0, 1.0 / 23.0, 1.0 / 25.0, + 1.0 / 27.0, 1.0 / 29.0, 1.0 / 31.0, 1.0 / 33.0, 1.0 / 35.0, 1.0 / 37.0 +}; + + // log: x = 2^k * m where m in [1, 2). log(x) = k*ln2 + log(m). // log(m) for m in [1,2) computed via the substitution u = (m-1)/(m+1) // giving log(m) = 2 * (u + u^3/3 + u^5/5 + ...) — converges fast for @@ -546,17 +659,11 @@ double log(double x) { b = (b & ~((uint64_t)0x7FF << 52)) | ((uint64_t)1023 << 52); double m; __builtin_memcpy(&m, &b, sizeof(m)); - // 1/(2n+1) for the series: a table of the correctly rounded constants - // (the run-time division gave the same values at ~60000 cycles each). - static const double kInvOdd[9] = { - 1.0, 1.0 / 3.0, 1.0 / 5.0, 1.0 / 7.0, 1.0 / 9.0, 1.0 / 11.0, - 1.0 / 13.0, 1.0 / 15.0, 1.0 / 17.0 - }; double u = (m - 1.0) / (m + 1.0); double u2 = u * u; double term = u; double s = term; - for (int n = 1; n <= 8; n++) { + for (int n = 1; n <= LOG_TERMS; n++) { term = term * u2; s = s + term * kInvOdd[n]; } @@ -569,38 +676,51 @@ float logf(float x) { } -// atan: range-reduce |x| > 1 via atan(x) = pi/2 - atan(1/x), then -// further reduce via atan(x) = 2*atan(x / (1 + sqrt(1 + x*x))). -// For the reduced argument, Taylor: x - x^3/3 + x^5/5 - ... +// atan: |x| >= 1 goes through atan(x) = pi/2 - atan(1/x), and from +// tan(pi/8) on, atan(t) = pi/4 + atan((t - 1) / (t + 1)), which leaves +// |t| <= tan(pi/8) ~ 0.414 for the series t - t^3/3 + t^5/5 - ..., whose +// terms then shrink 0.17x apiece. Without the pi/4 step (which the old +// comment promised but the code never did) the series ran at |t| up to 1, +// where 25 terms are only ~1% accurate: atan(1) was 0.795, not 0.785. +#define ATAN_SHIFT_TOP 0x3FDAu // top word of 0.40625, just under tan(pi/8) double atan(double x) { - uint64_t b; - __builtin_memcpy(&b, &x, sizeof(b)); - int sign = (int)(b >> 63) & 1; - if (sign) { - b ^= ((uint64_t)1 << 63); - __builtin_memcpy(&x, &b, sizeof(x)); + uint64_t b = dToBits(x); + int sign = (int)(b >> 63) & 1; + int invert = 0; + int shift = 0; + double x2; + double term; + double s; + if (isNanBits(b)) { + return x + x; } - int invert = 0; - // Test |x| > 1 via bit pattern: exponent > 1023 means value >= 1. - int e = (int)((b >> 52) & 0x7FF) - 1023; - if (e >= 0) { + if (sign) { + b ^= D_SIGN; + x = dFromBits(b); + } + // |x| >= 1 from the bits: the exponent is 1023 or more. + if (((topWord(b) >> 4) & 0x7FFu) >= (unsigned)D_BIAS) { x = 1.0 / x; invert = 1; } - // Taylor for |x| <= 1: x - x^3/3 + x^5/5 - x^7/7 + x^9/9 - ... - // Need many terms when |x| close to 1; cap at 30 for ~1e-6 accuracy. - double x2 = x * x; - double term = x; - double s = term; - for (int n = 1; n <= 25; n++) { + if (topWord(dToBits(x)) >= ATAN_SHIFT_TOP) { + x = (x - 1.0) / (x + 1.0); + shift = 1; + } + x2 = x * x; + term = x; + s = term; + for (int n = 1; n <= ATAN_TERMS; n++) { term = term * x2; - double denom = (double)(2 * n + 1); if (n & 1) { - s = s - term / denom; + s = s - term * kInvOdd[n]; } else { - s = s + term / denom; + s = s + term * kInvOdd[n]; } } + if (shift) { + s = M_QUART_PI + s; + } if (invert) { s = M_HALF_PI - s; } @@ -616,24 +736,47 @@ float atanf(float x) { } -double atan2(double y, double x) { - // Quadrant adjustment. - uint64_t xb, yb; - __builtin_memcpy(&xb, &x, sizeof(xb)); - __builtin_memcpy(&yb, &y, sizeof(yb)); - int xNeg = (int)(xb >> 63) & 1; - int yNeg = (int)(yb >> 63) & 1; - // Strip sign of x for the quotient; restore quadrant later. - uint64_t xa = xb & ~((uint64_t)1 << 63); - if (xa == 0) { - // x == 0 (handles +0 and -0): return ±pi/2 - return yNeg ? -M_HALF_PI : M_HALF_PI; +// atan2 with a zero, an infinity or a NaN, per C99 F.9.1.4: a NaN operand +// gives NaN; otherwise the result takes y's sign and its magnitude is +// y = inf: pi/4 (x = +inf), 3pi/4 (x = -inf), pi/2 (x finite); +// x = inf: 0 (x = +inf) or pi (x = -inf) for a finite y; +// y = 0: 0 for x > 0 or x = +0, pi for x < 0 or x = -0; +// x = 0: pi/2 for a finite, nonzero y. +// Every x = +-0 case used to give +-pi/2 (atan2(0, 0) too), and the +// infinite pairs went through inf / inf to NaN. +static COLD double atan2Special(double y, double x, uint64_t yb, uint64_t xb) { + uint64_t ya = yb & ~D_SIGN; + uint64_t xa = xb & ~D_SIGN; + int xNeg = (int)(xb >> 63); + double r; + if (isNanBits(xb) || isNanBits(yb)) { + return x + y; } - double a = atan(y / x); - if (!xNeg) { + if (ya == D_INF) { + r = (xa == D_INF) ? (xNeg ? M_3QUAR_PI : M_QUART_PI) : M_HALF_PI; + } else if (xa == D_INF || ya == 0) { + r = xNeg ? M_PI : 0.0; + } else { + r = M_HALF_PI; + } + return (yb >> 63) ? -r : r; +} + + +// atan2: atan(y / x) moved to x's half-plane. A y / x that overflows or +// underflows still lands right (atan(+-inf) is +-pi/2, atan(+-0) +-0). +double atan2(double y, double x) { + uint64_t xb = dToBits(x); + uint64_t yb = dToBits(y); + double a; + if (isNonFinite(xb) || isNonFinite(yb) || isZeroBits(xb) || isZeroBits(yb)) { + return atan2Special(y, x, yb, xb); + } + a = atan(y / x); + if ((xb >> 63) == 0) { return a; } - return yNeg ? a - M_PI : a + M_PI; + return (yb >> 63) ? a - M_PI : a + M_PI; } @@ -822,39 +965,21 @@ float fdimf(float x, float y) { return (float)fdim((double)x, (double)y); } // ---- FP decomposition ---------------------------------------------- -// ldexp(x, n) = x * 2^n. Implemented by adjusting the exponent field -// in place. Handles subnormals and overflow only crudely (overflow → -// infinity; underflow → zero). +// ldexp(x, n) = x * 2^n on the bits: exact unless the result leaves the +// double range, where it is +-inf or a correctly rounded subnormal / zero. +// It used to multiply a zero or subnormal x by 2 n times (ldexp(0, 30000) +// ran 30000 soft multiplies, and the soft-float flushed the subnormal to +// 0 on the first), and its 16-bit exponent sum wrapped: ldexp(1, 32767) +// was 0, not inf. double ldexp(double x, int n) { uint64_t b = dToBits(x); - int e = (int)((b >> 52) & 0x7FF); - if (e == 0 || e == 0x7FF) { - // 0 or denorm or inf/nan — return as-is for inf/nan, else - // fall back to multiplication (handles subnormals correctly - // enough for typical use). - if (e == 0x7FF) return x; - // For zero / subnormal, multiply by 2^n via repeated *2. - if (n > 0) { - while (n--) x *= 2.0; - return x; - } - if (n < 0) { - while (n++) x *= 0.5; - return x; - } + uint64_t m; + int e; + if (isNonFinite(b) || isZeroBits(b)) { return x; } - int newE = e + n; - if (newE >= 0x7FF) { - // Overflow → ±infinity. - return (b >> 63) ? -1e308 * 1e308 : 1e308 * 1e308; - } - if (newE <= 0) { - // Underflow → ±0 (skip subnormal handling). - return (b >> 63) ? -0.0 : 0.0; - } - b = (b & 0x800FFFFFFFFFFFFFULL) | ((uint64_t)newE << 52); - return dFromBits(b); + m = dUnpack(b & ~D_SIGN, &e); + return dBuild(b & D_SIGN, m, (long)e + n); } @@ -865,15 +990,19 @@ float ldexpf(float x, int n) { // frexp(x, *e): split x into normalized fraction in [0.5, 1) (or 0) // and integer exponent. x = frac * 2^e. +// A subnormal x is normalized first (it used to come back unchanged with +// *eOut = 0); zeros, infinities and NaNs come back as they are, *eOut 0. double frexp(double x, int *eOut) { uint64_t b = dToBits(x); - int e = (int)((b >> 52) & 0x7FF); - if (e == 0) { *eOut = 0; return x; } // zero or subnormal - if (e == 0x7FF) { *eOut = 0; return x; } // inf or nan - *eOut = e - 1022; // bias adjustment for [0.5,1) - // Force exponent to 1022 (so result is in [0.5, 1)). - b = (b & 0x800FFFFFFFFFFFFFULL) | ((uint64_t)1022 << 52); - return dFromBits(b); + uint64_t m; + int e; + if (isNonFinite(b) || isZeroBits(b)) { + *eOut = 0; + return x; + } + m = dUnpack(b & ~D_SIGN, &e); + *eOut = e + 1; + return dFromBits((b & D_SIGN) | ((uint64_t)(D_BIAS - 1) << D_MBITS) | (m & D_MANT)); } @@ -884,19 +1013,54 @@ float frexpf(float x, int *eOut) { // modf(x, *iptr): split into integer and fractional parts, both with -// the same sign as x. +// the same sign as x, on the bits. x - trunc(x) made modf(+-inf) a NaN +// (inf - inf) and gave a +0 fraction for -2.0 and -0.0. C99 F.9.3.12: +// modf(+-inf) is +-0 with *iptr = +-inf; a NaN is both parts. double modf(double x, double *iptr) { - double ip = trunc(x); - *iptr = ip; - return x - ip; + uint64_t b = dToBits(x); + uint64_t sign = b & D_SIGN; + int e = (int)((topWord(b) >> 4) & 0x7FFu) - D_BIAS; + uint64_t frac; + if (e >= D_MBITS) { + *iptr = x; // integral, infinite or NaN + return isNanBits(b) ? x : dFromBits(sign); + } + if (e < 0) { + *iptr = dFromBits(sign); // |x| < 1, subnormals included + return x; + } + frac = D_MANT >> e; // the bits below the unit + if ((b & frac) == 0) { + *iptr = x; + return dFromBits(sign); + } + *iptr = dFromBits(b & ~frac); + return x - *iptr; // exact } +// modff on the float's own bits: through double, a subnormal float +// flushed to zero on the widening. float modff(float x, float *iptr) { - double ipd; - double frac = modf((double)x, &ipd); - *iptr = (float)ipd; - return (float)frac; + uint32_t b = fToBits(x); + uint32_t sign = b & F_SIGN; + int e = (int)((b >> F_MBITS) & F_EXP) - F_BIAS; + uint32_t frac; + if (e >= F_MBITS) { + *iptr = x; + return ((b & ~F_SIGN) > ((uint32_t)F_EXP << F_MBITS)) ? x : fFromBits(sign); + } + if (e < 0) { + *iptr = fFromBits(sign); + return x; + } + frac = F_MANT >> e; + if ((b & frac) == 0) { + *iptr = x; + return fFromBits(sign); + } + *iptr = fFromBits(b & ~frac); + return x - *iptr; } @@ -948,22 +1112,51 @@ float expm1f(float x) { return (float)expm1((double)x); } // ---- hypot --------------------------------------------------------- // -// hypot(x, y) = sqrt(x*x + y*y). This implementation does NOT scale -// to avoid overflow — for |x|, |y| < ~1e150 the naive form is fine, -// past that you'd want the standard scale-by-max trick. +// hypot(x, y) = sqrt(x*x + y*y), with both operands first scaled (on the +// bits, exactly) by the power of two that brings the larger into [1, 2), +// so neither square can overflow or underflow: hypot(1e200, 1e200) was +// +inf and hypot(1e-200, 1e-200) was 0. The result is scaled back with +// dBuild, which rounds into the subnormals. A smaller operand more than +// HYPOT_SPAN binary orders below the larger cannot change the rounded +// result. C99 F.9.4.3: an infinite operand gives +inf even when the +// other is a NaN. +#define HYPOT_SPAN 60 double hypot(double x, double y) { - uint64_t xb = dToBits(x); - uint64_t yb = dToBits(y); - // C99 F.9.4.3: an infinite operand gives +inf even when the other is a - // NaN; otherwise a NaN operand gives NaN (through the arithmetic). - if ((xb & ~D_SIGN) == D_INF || (yb & ~D_SIGN) == D_INF) { + uint64_t xa = dToBits(x) & ~D_SIGN; + uint64_t ya = dToBits(y) & ~D_SIGN; + uint64_t t; + uint64_t mx; + uint64_t my; + uint64_t mr; + int ex; + int ey; + int er; + double sx; + double sy; + if (xa == D_INF || ya == D_INF) { return __builtin_inf(); } - double xx = x * x; - double yy = y * y; - double s = xx + yy; - return sqrt(s); + if (xa > D_INF || ya > D_INF) { + return x + y; // a NaN, quieted + } + if (xa < ya) { + t = xa; + xa = ya; + ya = t; + } + if (ya == 0) { + return dFromBits(xa); + } + mx = dUnpack(xa, &ex); + my = dUnpack(ya, &ey); + if (ex - ey > HYPOT_SPAN) { + return dFromBits(xa); + } + sx = dFromBits(D_ONE | (mx & D_MANT)); + sy = dFromBits(((uint64_t)(D_BIAS + ey - ex) << D_MBITS) | (my & D_MANT)); + mr = dUnpack(dToBits(sqrt(sx * sx + sy * sy)), &er); + return dBuild(0, mr, (long)er + ex); } @@ -1012,22 +1205,50 @@ float cbrtf(float x) { // have. Callers who depend on extra precision must use Kahan summation // or similar. +// From 2^28 up, x * x + 1 (or - 1) is x * x and log(x + sqrt(x * x)) is +// log(2x) = log(x) + ln 2; computing it that way also keeps x * x from +// overflowing (asinh(1e200) and acosh(1e200) were +inf). +#define HYPER_BIG 0x41B0000000000000ULL + +// asinh is odd, so it runs on |x| and takes x's sign: log(x + sqrt(x * x +// + 1)) for a negative x cancels to log(0) = -inf (asinh(-1e10) was -inf, +// asinh(-inf) NaN, asinh(-0) +0). double asinh(double x) { - // asinh(x) = log(x + sqrt(x*x + 1)) — stable for all finite x. - return log(x + sqrt(x * x + 1.0)); + uint64_t b = dToBits(x); + uint64_t a = b & ~D_SIGN; + double ax = dFromBits(a); + double r; + if (isZeroBits(b)) { + return x; + } + if (isNonFinite(b)) { + return x + x; + } + if (a >= HYPER_BIG) { + r = log(ax) + M_LN2; + } else { + r = log(ax + sqrt(ax * ax + 1.0)); + } + return (b >> 63) ? -r : r; } +// acosh(x) = log(x + sqrt(x*x - 1)) for x >= 1, NaN below 1 (and for a +// NaN: the big-x branch takes it, as its bits compare above HYPER_BIG). double acosh(double x) { - // acosh(x) = log(x + sqrt(x*x - 1)) for x >= 1. Returns NaN for x < 1. if (x < 1.0) return __builtin_nanf(""); + if (dToBits(x) >= HYPER_BIG) { + return log(x) + M_LN2; + } return log(x + sqrt(x * x - 1.0)); } double atanh(double x) { // atanh(x) = 0.5 * log((1+x)/(1-x)) for |x| < 1. Returns +/-inf - // at x = +/-1 and NaN outside [-1, 1]. + // at x = +/-1 and NaN outside [-1, 1]; atanh(+-0) is +-0 (the formula + // gives +0 for both). + if (isZeroBits(dToBits(x))) return x; if (x >= 1.0) return (x == 1.0) ? __builtin_inff() : __builtin_nan(""); if (x <= -1.0) return (x == -1.0) ? -__builtin_inff() : __builtin_nan(""); return 0.5 * log((1.0 + x) / (1.0 - x)); @@ -1059,18 +1280,48 @@ double nan(const char *tagp) { float nanf(const char *tagp) { (void)tagp; return __builtin_nanf(""); } +// IEEE 754 remainder: x - n*y where n = x/y rounded to nearest, ties to +// even. Exact, like fmod, which it starts from (fdlibm's method): r = +// fmod(x, 2|y|) is x - n*y for the even n just below, then one or two +// subtractions of |y| (exact: r is within a factor of two of |y|) land on +// the nearest, ties going to the even n. A |y| so small that |y| / 2 is +// subnormal is scaled up first (ldexp is exact), because the soft-float +// flushes subnormal operands to zero. x - rint(x / y) * y, used before, +// rounded x / y (remainder(1e20, 0.1) was off by units of 0.1) and gave +// remainder(-4, 2) = +0, not -0: a zero result takes x's sign. C99 +// F.9.7.2: a NaN operand, an infinite x or a zero y give NaN; +// remainder(x, +-inf) is x for a finite x. +#define REM_SCALE 64 + double remainder(double x, double y) { - // IEEE 754 remainder: x - n*y where n = round-to-nearest-even(x/y), - // which is rint (round() rounded ties away: remainder(5, 2) was -1). - // remainder(x, +-inf) is x for a finite x (C99 F.9.7.2); x / y would - // make it 0 * inf = NaN. - if (y == 0.0) return __builtin_nanf(""); - uint64_t yb = dToBits(y); - if ((yb & ~D_SIGN) == D_INF && !isNonFinite(dToBits(x))) { - return x; + uint64_t xb = dToBits(x); + uint64_t yb = dToBits(y); + uint64_t sign = xb & D_SIGN; + uint64_t ya = yb & ~D_SIGN; + int scaled = ya < D_TINY2; + double ay = dFromBits(ya); + double r; + double half; + if (isNonFinite(xb) || isNonFinite(yb) || isZeroBits(yb)) { + return fmodSpecial(x, y); } - double quotient = rint(x / y); - return x - quotient * y; + r = (ya < D_HUGE2) ? fmod(x, ldexp(ay, 1)) : x; + r = fabs(r); + if (scaled) { + r = ldexp(r, REM_SCALE); + ay = ldexp(ay, REM_SCALE); + } + half = ay * 0.5; + if (r > half) { + r = r - ay; + if (r >= half) { + r = r - ay; + } + } + if (scaled) { + r = ldexp(r, -REM_SCALE); + } + return dFromBits(dToBits(r) ^ sign); } @@ -1138,11 +1389,31 @@ double scalbln(double x, long n) { return ldexp(x, (int)n); } float scalblnf(float x, long n) { return ldexpf(x, (int)n); } +// fpclassify on the bits, in the argument's own width (math.h's macro +// sends a float to __fpclassify_f). Subnormals are FP_SUBNORMAL: a double +// one used to be FP_NORMAL (nothing told them apart), and a float one +// FP_ZERO, because the widening to double flushed it to zero first. int fpclassify(double x) { - if (__isnan_d(x)) return 0; // FP___builtin_nanf("") - if (__isinf_d(x)) return 1; // FP_INFINITE - if (x == 0.0) return 4; // FP_ZERO - return 2; // FP_NORMAL - // FP_SUBNORMAL (= 3) not distinguished from normal in this minimal - // implementation — subnormals are valid but classified as normal. + uint64_t b = dToBits(x); + unsigned e = (topWord(b) >> 4) & 0x7FFu; + if (e == 0x7FFu) { + return (b & D_MANT) ? FP_NAN : FP_INFINITE; + } + if (e != 0u) { + return FP_NORMAL; + } + return isZeroBits(b) ? FP_ZERO : FP_SUBNORMAL; +} + + +int __fpclassify_f(float x) { + uint32_t b = fToBits(x); + uint32_t e = (b >> F_MBITS) & F_EXP; + if (e == F_EXP) { + return (b & F_MANT) ? FP_NAN : FP_INFINITE; + } + if (e != 0u) { + return FP_NORMAL; + } + return (b & ~F_SIGN) ? FP_SUBNORMAL : FP_ZERO; } diff --git a/scripts/runInMame.sh b/scripts/runInMame.sh index 2974c7d..7c6d3f2 100755 --- a/scripts/runInMame.sh +++ b/scripts/runInMame.sh @@ -18,6 +18,14 @@ # 2+ for safe data writes (bank 0 zero page is scribbled by IIgs ROM # during execution). # +# Optional environment: +# MAME_CHECK_FRAME frame of the checks (default 300; the load is at 30). +# MAME_SECS emulated seconds before MAME quits (default 6). +# MAME_LUA_EXTRA a Lua file appended to the autoboot script, for a +# check that needs its own hooks (write taps, injected +# interrupts). It runs once at startup, like the rest; +# its MAME-* prints reach the output. +# # Exit 0 if all reads match, 1 otherwise. set -euo pipefail @@ -118,6 +126,9 @@ $LUA_CHECKS end end) EOF +if [ -n "${MAME_LUA_EXTRA:-}" ]; then + cat "$MAME_LUA_EXTRA" >> "$LUA_PATH" +fi # MAME and MAME_ROMPATH let a client project (joeylib) supply its own # patched MAME and ROM set instead of the ones installMame.sh sets up. diff --git a/scripts/runViaFinder.sh b/scripts/runViaFinder.sh index 490968a..ef75ad6 100755 --- a/scripts/runViaFinder.sh +++ b/scripts/runViaFinder.sh @@ -4,6 +4,7 @@ # specific frames to verify the program executed. # # Usage: runViaFinder.sh [--data /VOL/PATH/NAME=local_file]... +# [--save-data ] # --check =... # The OMF file is injected as /DATA/HELLO on a separate 800K data # disk; Lua drives Finder to open the Data volume and launch HELLO. @@ -19,6 +20,10 @@ # readable file via gsosOpen. Pass multiple `--data` options to inject # more than one file. # +# --save-data copies the data disk to once MAME has stopped (MAME +# writes the floppy image back when it exits), so a caller can read what +# the program left on it -- e.g. with `cadius EXTRACTFILE`. +# # Memory checks happen at frame 5400 (~90s emulated, well after the # launch path completes) and exit 0 / 1 depending on whether each # requested address holds the requested value. @@ -34,14 +39,20 @@ OMF="$1" shift [ -f "$OMF" ] || { echo "missing: $OMF" >&2; exit 2; } -# Collect optional --data injections before --check. +# Collect optional --data injections and --save-data before --check. +USAGE="usage: $0 [--data /VOL/PATH/NAME=path]... [--save-data ] --check =..." DATA_INJECTS=() -while [ $# -gt 0 ] && [ "$1" = "--data" ]; do - [ $# -ge 2 ] || { echo "usage: $0 [--data /VOL/PATH/NAME=path]... --check =..." >&2; exit 2; } - DATA_INJECTS+=("$2") +SAVE_DATA="" +while [ $# -gt 0 ] && { [ "$1" = "--data" ] || [ "$1" = "--save-data" ]; }; do + [ $# -ge 2 ] || { echo "$USAGE" >&2; exit 2; } + if [ "$1" = "--data" ]; then + DATA_INJECTS+=("$2") + else + SAVE_DATA="$2" + fi shift 2 done -[ "${1:-}" = "--check" ] || { echo "usage: $0 [--data /VOL/PATH/NAME=path]... --check =..." >&2; exit 2; } +[ "${1:-}" = "--check" ] || { echo "$USAGE" >&2; exit 2; } shift PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" @@ -162,6 +173,9 @@ OUT=$(SDL_VIDEODRIVER=dummy SDL_AUDIODRIVER=dummy timeout 150 mame apple2gs -rom -video none -nothrottle -sound none \ -seconds_to_run 130 -flop3 "$WORK/disk.po" -flop4 "$WORK/data.po" \ -autoboot_script "$WORK/finder.lua" &1) +if [ -n "$SAVE_DATA" ]; then + cp "$WORK/data.po" "$SAVE_DATA" +fi # Verify each expected value. fail=0 diff --git a/scripts/smokeTest.sh b/scripts/smokeTest.sh index 3c55fc4..204e0d5 100755 --- a/scripts/smokeTest.sh +++ b/scripts/smokeTest.sh @@ -1797,6 +1797,38 @@ EOF fi rm -f "$cFpCmpFile" "$oFpCmpFile" "$binFpCmpFile" + # Float and double compares against constants that the operand's bit + # pattern decides are integer compares (W65816 combineFPSetCCConst, + # 2026-10): x against a zero, an infinity or (== / !=) any non-NaN + # constant, fabs(x) against a constant >= 0, x against itself, and the + # ordered / unordered tests. Every C operator, its negation, + # isunordered and islessgreater, against ten constants and on fabs(x), + # plus the classification builtins, as a branch and as a value, on 34 + # values (signed zeros, subnormals, normals one ulp apart, the largest + # finites, infinities, quiet / signaling / all-ones NaNs). Compares + # left to the libcalls run too. kExp is host-computed by + # tests/fpNan/genFpCmpConst.py; the double table runs in two halves. + log "check: MAME float/double compares against constants (integer bit tests)" + cFpcFile="$(mktemp --suffix=.c)" + oFpcFile="$(mktemp --suffix=.o)" + binFpcFile="$(mktemp --suffix=.bin)" + for fpcPart in "f 1 1" "d 1 2" "d 2 2"; do + # shellcheck disable=SC2086 + python3 "$PROJECT_ROOT/tests/fpNan/genFpCmpConst.py" $fpcPart > "$cFpcFile" + "$CLANG" --target=w65816 -O2 -ffunction-sections \ + -I"$PROJECT_ROOT/runtime/include" -c "$cFpcFile" -o "$oFpcFile" + "$PROJECT_ROOT/tools/link816" -o "$binFpcFile" --text-base 0x1000 \ + "$oCrt0File" "$oFpcFile" "$PROJECT_ROOT/runtime/softFloat.o" \ + "$PROJECT_ROOT/runtime/softDouble.o" "$oLibgccFile" >/dev/null 2>&1 \ + || die "MAME float/double compares against constants ($fpcPart): link failed" + if ! MAME_CHECK_FRAME=900 MAME_SECS=20 \ + bash "$PROJECT_ROOT/scripts/runInMame.sh" "$binFpcFile" --check \ + 0x025000=0000 0x025002=ffff 0x025004=c0de >/dev/null 2>&1; then + die "MAME: float/double compare against a constant wrong ($fpcPart; mailbox 0x025000 = mismatches, 0x025002 = first case)" + fi + done + rm -f "$cFpcFile" "$oFpcFile" "$binFpcFile" + # NaN and infinity through the soft-float arithmetic, conversions, # classification and math.c (C99 Annex F special values, plus ordinary # values for every math function touched): a NaN operand comes back @@ -2567,6 +2599,443 @@ EOF fi rm -f "$cFpNanFile" "$oFpNanFile" "$binFpNanFile" + # math.c's exact and bit-level functions, bit for bit against Python's + # C99 answers (tests/fpNan/genFpTables.py edge): fmod and remainder + # exact for any quotient (fmod(1e20, 0.1), remainder(-1e300, 7)), modf / + # modff signs and infinities (modf(-2) is -0, modf(inf) +0 with *iptr + # inf), atan2's signed zeros and infinities, fpclassify / isnormal on + # subnormals in both widths, hypot without overflow or underflow + # (1e200, 1e200), frexp / ldexp on subnormals and exponents past 16 + # bits, asinh / acosh / atanh signs and overflow. 57 of these 164 + # cases failed before 2026-09-30. fmod(DBL_MAX, 1e-300) alone runs its + # long division ~2000 steps; the table finishes by about frame 1100. + log "check: MAME math.c exact / edge cases: fmod, remainder, modf, atan2, fpclassify, hypot, frexp, ldexp" + cMathEdge="$(mktemp --suffix=.c)" + oMathEdge="$(mktemp --suffix=.o)" + binMathEdge="$(mktemp --suffix=.bin)" + cat > "$cMathEdge" <<'EOF' +// math.c's exact and bit-level functions against host-computed bit +// patterns (tests/fpNan/genFpTables.py edge): exact fmod / remainder for +// any quotient, modf / modff signs and infinities, atan2's signed zeros +// and infinities, fpclassify / isnormal on subnormals in both widths, +// hypot without overflow or underflow, frexp / ldexp on subnormals and +// huge exponents, asinh / acosh / atanh signs and overflow. A NaN result +// counts as the default NaN. Mailbox: 0x025000 = mismatches, 0x025002 = +// first failing case, 0x025004 = 0xC0DE done; the first failure's result +// at 0x025010 (8 bytes). +#include +typedef unsigned char u8; +typedef unsigned short u16; +typedef unsigned long u32; +typedef unsigned long long u64; +#define NOI __attribute__((noinline)) static +typedef struct { + u16 op; + u64 a; + u64 b; + u64 want; + u16 approx; +} CaseT; +enum { + E_MODFFRAC, E_MODFINT, E_MODFFFRAC, E_MODFFINT, E_ATAN2, E_CLSD, E_CLSF, + E_HYPOT, E_FMOD, E_REMAINDER, E_FREXPM, E_FREXPE, E_LDEXP, E_ASINH, + E_ACOSH, E_ATANH +}; +#define NE 164 +static const CaseT kEdge[NE] = { + { E_MODFFRAC, 0xC000000000000000ULL, 0x0000000000000000ULL, 0x8000000000000000ULL, 0 }, + { E_MODFINT, 0xC000000000000000ULL, 0x0000000000000000ULL, 0xC000000000000000ULL, 0 }, + { E_MODFFRAC, 0x8000000000000000ULL, 0x0000000000000000ULL, 0x8000000000000000ULL, 0 }, + { E_MODFINT, 0x8000000000000000ULL, 0x0000000000000000ULL, 0x8000000000000000ULL, 0 }, + { E_MODFFRAC, 0x7FF0000000000000ULL, 0x0000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_MODFINT, 0x7FF0000000000000ULL, 0x0000000000000000ULL, 0x7FF0000000000000ULL, 0 }, + { E_MODFFRAC, 0xFFF0000000000000ULL, 0x0000000000000000ULL, 0x8000000000000000ULL, 0 }, + { E_MODFINT, 0xFFF0000000000000ULL, 0x0000000000000000ULL, 0xFFF0000000000000ULL, 0 }, + { E_MODFFRAC, 0x7FF8000000000000ULL, 0x0000000000000000ULL, 0x7FF8000000000000ULL, 0 }, + { E_MODFINT, 0x7FF8000000000000ULL, 0x0000000000000000ULL, 0x7FF8000000000000ULL, 0 }, + { E_MODFFRAC, 0xC00E000000000000ULL, 0x0000000000000000ULL, 0xBFE8000000000000ULL, 0 }, + { E_MODFINT, 0xC00E000000000000ULL, 0x0000000000000000ULL, 0xC008000000000000ULL, 0 }, + { E_MODFFRAC, 0x3FE0000000000000ULL, 0x0000000000000000ULL, 0x3FE0000000000000ULL, 0 }, + { E_MODFINT, 0x3FE0000000000000ULL, 0x0000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_MODFFRAC, 0x7E37E43C8800759CULL, 0x0000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_MODFINT, 0x4330000000000002ULL, 0x0000000000000000ULL, 0x4330000000000002ULL, 0 }, + { E_MODFFRAC, 0x4330000000000002ULL, 0x0000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_MODFFRAC, 0x0000000000000001ULL, 0x0000000000000000ULL, 0x0000000000000001ULL, 0 }, + { E_MODFINT, 0x8000000000000001ULL, 0x0000000000000000ULL, 0x8000000000000000ULL, 0 }, + { E_MODFFFRAC, 0x00000000C0000000ULL, 0x0000000000000000ULL, 0x0000000080000000ULL, 0 }, + { E_MODFFINT, 0x00000000C0000000ULL, 0x0000000000000000ULL, 0x00000000C0000000ULL, 0 }, + { E_MODFFFRAC, 0x0000000080000000ULL, 0x0000000000000000ULL, 0x0000000080000000ULL, 0 }, + { E_MODFFFRAC, 0x000000007F800000ULL, 0x0000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_MODFFINT, 0x00000000FF800000ULL, 0x0000000000000000ULL, 0x00000000FF800000ULL, 0 }, + { E_MODFFFRAC, 0x000000007FC00000ULL, 0x0000000000000000ULL, 0x000000007FC00000ULL, 0 }, + { E_MODFFINT, 0x000000007FC00000ULL, 0x0000000000000000ULL, 0x000000007FC00000ULL, 0 }, + { E_MODFFFRAC, 0x00000000C0700000ULL, 0x0000000000000000ULL, 0x00000000BF400000ULL, 0 }, + { E_MODFFINT, 0x00000000C0700000ULL, 0x0000000000000000ULL, 0x00000000C0400000ULL, 0 }, + { E_MODFFFRAC, 0x000000004B000002ULL, 0x0000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_MODFFFRAC, 0x0000000000000001ULL, 0x0000000000000000ULL, 0x0000000000000001ULL, 0 }, + { E_ATAN2, 0x0000000000000000ULL, 0x0000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_ATAN2, 0x0000000000000000ULL, 0x8000000000000000ULL, 0x400921FB54442D18ULL, 0 }, + { E_ATAN2, 0x8000000000000000ULL, 0x0000000000000000ULL, 0x8000000000000000ULL, 0 }, + { E_ATAN2, 0x8000000000000000ULL, 0x8000000000000000ULL, 0xC00921FB54442D18ULL, 0 }, + { E_ATAN2, 0x0000000000000000ULL, 0xBFF0000000000000ULL, 0x400921FB54442D18ULL, 0 }, + { E_ATAN2, 0x8000000000000000ULL, 0xBFF0000000000000ULL, 0xC00921FB54442D18ULL, 0 }, + { E_ATAN2, 0x0000000000000000ULL, 0x3FF0000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_ATAN2, 0x8000000000000000ULL, 0x3FF0000000000000ULL, 0x8000000000000000ULL, 0 }, + { E_ATAN2, 0x3FF0000000000000ULL, 0x0000000000000000ULL, 0x3FF921FB54442D18ULL, 0 }, + { E_ATAN2, 0xBFF0000000000000ULL, 0x8000000000000000ULL, 0xBFF921FB54442D18ULL, 0 }, + { E_ATAN2, 0x3FF0000000000000ULL, 0xFFF0000000000000ULL, 0x400921FB54442D18ULL, 0 }, + { E_ATAN2, 0xBFF0000000000000ULL, 0xFFF0000000000000ULL, 0xC00921FB54442D18ULL, 0 }, + { E_ATAN2, 0x3FF0000000000000ULL, 0x7FF0000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_ATAN2, 0xBFF0000000000000ULL, 0x7FF0000000000000ULL, 0x8000000000000000ULL, 0 }, + { E_ATAN2, 0x7FF0000000000000ULL, 0x3FF0000000000000ULL, 0x3FF921FB54442D18ULL, 0 }, + { E_ATAN2, 0xFFF0000000000000ULL, 0xBFF0000000000000ULL, 0xBFF921FB54442D18ULL, 0 }, + { E_ATAN2, 0x7FF0000000000000ULL, 0xFFF0000000000000ULL, 0x4002D97C7F3321D2ULL, 0 }, + { E_ATAN2, 0xFFF0000000000000ULL, 0xFFF0000000000000ULL, 0xC002D97C7F3321D2ULL, 0 }, + { E_ATAN2, 0x7FF0000000000000ULL, 0x7FF0000000000000ULL, 0x3FE921FB54442D18ULL, 0 }, + { E_ATAN2, 0xFFF0000000000000ULL, 0x7FF0000000000000ULL, 0xBFE921FB54442D18ULL, 0 }, + { E_ATAN2, 0x7FF8000000000000ULL, 0x3FF0000000000000ULL, 0x7FF8000000000000ULL, 0 }, + { E_ATAN2, 0x3FF0000000000000ULL, 0x7FF8000000000000ULL, 0x7FF8000000000000ULL, 0 }, + { E_ATAN2, 0x3FF0000000000000ULL, 0x3FF0000000000000ULL, 0x3FE921FB54442D18ULL, 1 }, + { E_ATAN2, 0xBFF0000000000000ULL, 0xC000000000000000ULL, 0xC0056C6E7397F5AEULL, 1 }, + { E_ATAN2, 0x4000000000000000ULL, 0xBFF0000000000000ULL, 0x4000468A8ACE4DF6ULL, 1 }, + { E_ATAN2, 0x7E37E43C8800759CULL, 0x01A56E1FC2F8F359ULL, 0x3FF921FB54442D18ULL, 0 }, + { E_ATAN2, 0x01A56E1FC2F8F359ULL, 0xFE37E43C8800759CULL, 0x400921FB54442D18ULL, 0 }, + { E_CLSD, 0x3FF0000000000000ULL, 0x0000000000000000ULL, 0x0000000000000012ULL, 0 }, + { E_CLSD, 0x8000000000000000ULL, 0x0000000000000000ULL, 0x0000000000000004ULL, 0 }, + { E_CLSD, 0x0000000000000001ULL, 0x0000000000000000ULL, 0x0000000000000003ULL, 0 }, + { E_CLSD, 0x800FFFFFFFFFFFFFULL, 0x0000000000000000ULL, 0x0000000000000003ULL, 0 }, + { E_CLSD, 0x0010000000000000ULL, 0x0000000000000000ULL, 0x0000000000000012ULL, 0 }, + { E_CLSD, 0x7FF0000000000000ULL, 0x0000000000000000ULL, 0x0000000000000001ULL, 0 }, + { E_CLSD, 0x7FF8000000000000ULL, 0x0000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_CLSD, 0xFE37E43C8800759CULL, 0x0000000000000000ULL, 0x0000000000000012ULL, 0 }, + { E_CLSF, 0x000000003F800000ULL, 0x0000000000000000ULL, 0x0000000000000012ULL, 0 }, + { E_CLSF, 0x0000000080000000ULL, 0x0000000000000000ULL, 0x0000000000000004ULL, 0 }, + { E_CLSF, 0x0000000000000001ULL, 0x0000000000000000ULL, 0x0000000000000003ULL, 0 }, + { E_CLSF, 0x00000000807FFFFFULL, 0x0000000000000000ULL, 0x0000000000000003ULL, 0 }, + { E_CLSF, 0x0000000000800000ULL, 0x0000000000000000ULL, 0x0000000000000012ULL, 0 }, + { E_CLSF, 0x000000007F800000ULL, 0x0000000000000000ULL, 0x0000000000000001ULL, 0 }, + { E_CLSF, 0x000000007FC00000ULL, 0x0000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_CLSF, 0x00000000FF61B1E6ULL, 0x0000000000000000ULL, 0x0000000000000012ULL, 0 }, + { E_HYPOT, 0x6974E718D7D7625AULL, 0x6974E718D7D7625AULL, 0x697D8F9811335B57ULL, 1 }, + { E_HYPOT, 0x698F5AA543C31387ULL, 0xE994E718D7D7625AULL, 0x699A20DF0DCD3AF0ULL, 1 }, + { E_HYPOT, 0x16687E92154EF7ACULL, 0x16687E92154EF7ACULL, 0x167151F68876F410ULL, 1 }, + { E_HYPOT, 0x81C01297D23AB683ULL, 0x01C56E1FC2F8F359ULL, 0x01CAC9A7B3B7302FULL, 1 }, + { E_HYPOT, 0x7FE1CCF385EBC8A0ULL, 0x7FE1CCF385EBC8A0ULL, 0x7FE92C80954C51F5ULL, 0 }, + { E_HYPOT, 0x7E37E43C8800759CULL, 0x3FF0000000000000ULL, 0x7E37E43C8800759CULL, 0 }, + { E_HYPOT, 0x0000000000000001ULL, 0x0000000000000001ULL, 0x0000000000000001ULL, 0 }, + { E_HYPOT, 0x8000000000000000ULL, 0x8000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_HYPOT, 0x8000000000000000ULL, 0xC014000000000000ULL, 0x4014000000000000ULL, 0 }, + { E_HYPOT, 0xFFF0000000000000ULL, 0x7FF8000000000000ULL, 0x7FF0000000000000ULL, 0 }, + { E_HYPOT, 0x7FF8000000000000ULL, 0x4000000000000000ULL, 0x7FF8000000000000ULL, 0 }, + { E_HYPOT, 0x4008000000000000ULL, 0x4010000000000000ULL, 0x4014000000000000ULL, 1 }, + { E_FMOD, 0x7E37E43C8800759CULL, 0x4008000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_FMOD, 0xFE37E43C8800759CULL, 0x401C000000000000ULL, 0xBFF0000000000000ULL, 0 }, + { E_FMOD, 0x4415AF1D78B58C40ULL, 0x3FB999999999999AULL, 0x3FB5BA7DA6F389F0ULL, 0 }, + { E_FMOD, 0x4016000000000000ULL, 0xC000000000000000ULL, 0x3FF8000000000000ULL, 0 }, + { E_FMOD, 0xC016000000000000ULL, 0xC000000000000000ULL, 0xBFF8000000000000ULL, 0 }, + { E_FMOD, 0xC018000000000000ULL, 0x4008000000000000ULL, 0x8000000000000000ULL, 0 }, + { E_FMOD, 0x4018000000000000ULL, 0xC008000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_FMOD, 0x3FF0000000000000ULL, 0x4008000000000000ULL, 0x3FF0000000000000ULL, 0 }, + { E_FMOD, 0x0000000000000003ULL, 0x0000000000000002ULL, 0x0000000000000001ULL, 0 }, + { E_FMOD, 0x3FF8000000000000ULL, 0x0000000000000001ULL, 0x0000000000000000ULL, 0 }, + { E_FMOD, 0x7FEFFFFFFFFFFFFFULL, 0x01A56E1FC2F8F359ULL, 0x0151C210546B28C0ULL, 0 }, + { E_FMOD, 0x43B0000000000001ULL, 0x3D70000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_FMOD, 0x401E000000000000ULL, 0x4000000000000000ULL, 0x3FF8000000000000ULL, 0 }, + { E_FMOD, 0x8000000000000000ULL, 0x4008000000000000ULL, 0x8000000000000000ULL, 0 }, + { E_FMOD, 0x3FF0000000000000ULL, 0xFFF0000000000000ULL, 0x3FF0000000000000ULL, 0 }, + { E_FMOD, 0x7FF0000000000000ULL, 0x3FF0000000000000ULL, 0x7FF8000000000000ULL, 0 }, + { E_FMOD, 0x7FF8000000000000ULL, 0x3FF0000000000000ULL, 0x7FF8000000000000ULL, 0 }, + { E_FMOD, 0x3FF0000000000000ULL, 0x0000000000000000ULL, 0x7FF8000000000000ULL, 0 }, + { E_REMAINDER, 0x7E37E43C8800759CULL, 0x4008000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_REMAINDER, 0xFE37E43C8800759CULL, 0x401C000000000000ULL, 0xBFF0000000000000ULL, 0 }, + { E_REMAINDER, 0x4415AF1D78B58C40ULL, 0x3FB999999999999AULL, 0xBF8EF8DF95307D50ULL, 0 }, + { E_REMAINDER, 0xC010000000000000ULL, 0x4000000000000000ULL, 0x8000000000000000ULL, 0 }, + { E_REMAINDER, 0x4010000000000000ULL, 0xC000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_REMAINDER, 0x4008000000000000ULL, 0x4000000000000000ULL, 0xBFF0000000000000ULL, 0 }, + { E_REMAINDER, 0xC008000000000000ULL, 0x4000000000000000ULL, 0x3FF0000000000000ULL, 0 }, + { E_REMAINDER, 0x4014000000000000ULL, 0xC000000000000000ULL, 0x3FF0000000000000ULL, 0 }, + { E_REMAINDER, 0x3FF8000000000000ULL, 0x01A56E1FC2F8F359ULL, 0x018D87654EA9F100ULL, 0 }, + { E_REMAINDER, 0x0000000000000003ULL, 0x0000000000000002ULL, 0x8000000000000001ULL, 0 }, + { E_REMAINDER, 0x7FEFFFFFFFFFFFFFULL, 0xFFEE42D130773B76ULL, 0x7FABD2ECF88C4890ULL, 0 }, + { E_REMAINDER, 0x3FF0000000000000ULL, 0x7FF0000000000000ULL, 0x3FF0000000000000ULL, 0 }, + { E_REMAINDER, 0x7FF0000000000000ULL, 0x3FF0000000000000ULL, 0x7FF8000000000000ULL, 0 }, + { E_REMAINDER, 0x3FF0000000000000ULL, 0x8000000000000000ULL, 0x7FF8000000000000ULL, 0 }, + { E_FREXPM, 0x0000000000000001ULL, 0x0000000000000000ULL, 0x3FE0000000000000ULL, 0 }, + { E_FREXPE, 0x0000000000000001ULL, 0x0000000000000000ULL, 0x000000000000FBCFULL, 0 }, + { E_FREXPM, 0x800FFFFFFFFFFFFFULL, 0x0000000000000000ULL, 0xBFEFFFFFFFFFFFFEULL, 0 }, + { E_FREXPE, 0x800FFFFFFFFFFFFFULL, 0x0000000000000000ULL, 0x000000000000FC02ULL, 0 }, + { E_FREXPM, 0x7E37E43C8800759CULL, 0x0000000000000000ULL, 0x3FE7E43C8800759CULL, 0 }, + { E_FREXPE, 0x7E37E43C8800759CULL, 0x0000000000000000ULL, 0x00000000000003E5ULL, 0 }, + { E_FREXPM, 0x8000000000000000ULL, 0x0000000000000000ULL, 0x8000000000000000ULL, 0 }, + { E_FREXPE, 0x8000000000000000ULL, 0x0000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_FREXPM, 0xFFF0000000000000ULL, 0x0000000000000000ULL, 0xFFF0000000000000ULL, 0 }, + { E_FREXPM, 0x3FE8000000000000ULL, 0x0000000000000000ULL, 0x3FE8000000000000ULL, 0 }, + { E_FREXPE, 0x3FE8000000000000ULL, 0x0000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_LDEXP, 0x3FF0000000000000ULL, 0x00000000000003FFULL, 0x7FE0000000000000ULL, 0 }, + { E_LDEXP, 0x3FF0000000000000ULL, 0x0000000000000400ULL, 0x7FF0000000000000ULL, 0 }, + { E_LDEXP, 0xBFF0000000000000ULL, 0x0000000000007FFFULL, 0xFFF0000000000000ULL, 0 }, + { E_LDEXP, 0x3FF0000000000000ULL, 0x0000000000008000ULL, 0x0000000000000000ULL, 0 }, + { E_LDEXP, 0x0000000000000000ULL, 0x0000000000007530ULL, 0x0000000000000000ULL, 0 }, + { E_LDEXP, 0x8000000000000000ULL, 0x0000000000008AD0ULL, 0x8000000000000000ULL, 0 }, + { E_LDEXP, 0x3FF0000000000000ULL, 0x000000000000FBCEULL, 0x0000000000000001ULL, 0 }, + { E_LDEXP, 0x3FF8000000000000ULL, 0x000000000000FBCEULL, 0x0000000000000002ULL, 0 }, + { E_LDEXP, 0x3FF0000000000000ULL, 0x000000000000FBCDULL, 0x0000000000000000ULL, 0 }, + { E_LDEXP, 0x4008000000000000ULL, 0x000000000000FBCDULL, 0x0000000000000002ULL, 0 }, + { E_LDEXP, 0x0000000000000001ULL, 0x0000000000000432ULL, 0x3FF0000000000000ULL, 0 }, + { E_LDEXP, 0x800FFFFFFFFFFFFFULL, 0x0000000000000001ULL, 0x801FFFFFFFFFFFFEULL, 0 }, + { E_LDEXP, 0x3FE8000000000000ULL, 0x000000000000FC02ULL, 0x000C000000000000ULL, 0 }, + { E_LDEXP, 0x7FF0000000000000ULL, 0x000000000000FFFBULL, 0x7FF0000000000000ULL, 0 }, + { E_LDEXP, 0x7FF8000000000000ULL, 0x0000000000000005ULL, 0x7FF8000000000000ULL, 0 }, + { E_ASINH, 0x8000000000000000ULL, 0x0000000000000000ULL, 0x8000000000000000ULL, 0 }, + { E_ASINH, 0x0000000000000000ULL, 0x0000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_ASINH, 0xFFF0000000000000ULL, 0x0000000000000000ULL, 0xFFF0000000000000ULL, 0 }, + { E_ASINH, 0x7FF0000000000000ULL, 0x0000000000000000ULL, 0x7FF0000000000000ULL, 0 }, + { E_ASINH, 0x7FF8000000000000ULL, 0x0000000000000000ULL, 0x7FF8000000000000ULL, 0 }, + { E_ASINH, 0xC202A05F20000000ULL, 0x0000000000000000ULL, 0xC037B810429A7C2AULL, 1 }, + { E_ASINH, 0x6974E718D7D7625AULL, 0x0000000000000000ULL, 0x407CD35CD6CAD20FULL, 1 }, + { E_ASINH, 0xFE37E43C8800759CULL, 0x0000000000000000ULL, 0xC0859BBFD8B83E43ULL, 1 }, + { E_ASINH, 0x3FE0000000000000ULL, 0x0000000000000000ULL, 0x3FDECC2CAEC5160AULL, 1 }, + { E_ASINH, 0xC000000000000000ULL, 0x0000000000000000ULL, 0xBFF719218313D087ULL, 1 }, + { E_ACOSH, 0x6974E718D7D7625AULL, 0x0000000000000000ULL, 0x407CD35CD6CAD20FULL, 1 }, + { E_ACOSH, 0x7FF0000000000000ULL, 0x0000000000000000ULL, 0x7FF0000000000000ULL, 0 }, + { E_ACOSH, 0x3FF0000000000000ULL, 0x0000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_ACOSH, 0x3FE0000000000000ULL, 0x0000000000000000ULL, 0x7FF8000000000000ULL, 0 }, + { E_ACOSH, 0xFFF0000000000000ULL, 0x0000000000000000ULL, 0x7FF8000000000000ULL, 0 }, + { E_ACOSH, 0x7FF8000000000000ULL, 0x0000000000000000ULL, 0x7FF8000000000000ULL, 0 }, + { E_ACOSH, 0x4000000000000000ULL, 0x0000000000000000ULL, 0x3FF5124271980434ULL, 1 }, + { E_ATANH, 0x8000000000000000ULL, 0x0000000000000000ULL, 0x8000000000000000ULL, 0 }, + { E_ATANH, 0x0000000000000000ULL, 0x0000000000000000ULL, 0x0000000000000000ULL, 0 }, + { E_ATANH, 0xBFE0000000000000ULL, 0x0000000000000000ULL, 0xBFE193EA7AAD030AULL, 1 }, + { E_ATANH, 0x3FF0000000000000ULL, 0x0000000000000000ULL, 0x7FF0000000000000ULL, 0 }, +}; +static float fFrom(u64 b) { + union { u32 u; float f; } c; + c.u = (u32)b; + return c.f; +} +static double dFrom(u64 b) { + union { u64 u; double d; } c; + c.u = b; + return c.d; +} +static u64 dBits(double d) { + union { u64 u; double d; } c; + c.d = d; + return isnan(d) ? 0x7FF8000000000000ULL : c.u; +} +static u64 fBits(float f) { + union { u32 u; float f; } c; + c.f = f; + return isnan(f) ? 0x7FC00000ULL : c.u; +} +// 1 when got is within 1e-6 relative of want (both finite). +static int near(u64 got, u64 want) { + double g = dFrom(got); + double w = dFrom(want); + double d = g - w; + if (!isfinite(g) || !isfinite(w)) { + return 0; + } + if (d < 0.0) { + d = -d; + } + if (w < 0.0) { + w = -w; + } + return d <= w * 1e-6; +} +NOI u64 run(u16 op, u64 a, u64 b) { + double d; + float f; + int e; + switch (op) { + case E_MODFFRAC: return dBits(modf(dFrom(a), &d)); + case E_MODFINT: (void)modf(dFrom(a), &d); return dBits(d); + case E_MODFFFRAC: return fBits(modff(fFrom(a), &f)); + case E_MODFFINT: (void)modff(fFrom(a), &f); return fBits(f); + case E_ATAN2: return dBits(atan2(dFrom(a), dFrom(b))); + case E_CLSD: return (u64)(fpclassify(dFrom(a)) | ((isnormal(dFrom(a)) != 0) << 4)); + case E_CLSF: return (u64)(fpclassify(fFrom(a)) | ((isnormal(fFrom(a)) != 0) << 4)); + case E_HYPOT: return dBits(hypot(dFrom(a), dFrom(b))); + case E_FMOD: return dBits(fmod(dFrom(a), dFrom(b))); + case E_REMAINDER: return dBits(remainder(dFrom(a), dFrom(b))); + case E_FREXPM: return dBits(frexp(dFrom(a), &e)); + case E_FREXPE: (void)frexp(dFrom(a), &e); return (u16)e; + case E_LDEXP: return dBits(ldexp(dFrom(a), (int)(short)b)); + case E_ASINH: return dBits(asinh(dFrom(a))); + case E_ACOSH: return dBits(acosh(dFrom(a))); + case E_ATANH: return dBits(atanh(dFrom(a))); + } + return 0xDEADULL; +} +int main(void) { + u16 bad = 0u; + u16 first = 0xFFFFu; + u16 i; + for (i = 0u; i < NE; i++) { + u64 got = run(kEdge[i].op, kEdge[i].a, kEdge[i].b); + if (got != kEdge[i].want && + !(kEdge[i].approx && near(got, kEdge[i].want))) { + if (bad == 0u) { + first = i; + *(volatile u64 *)0x025010UL = got; + } + bad++; + } + } + *(volatile u16 *)0x025000UL = bad; + *(volatile u16 *)0x025002UL = first; + *(volatile u16 *)0x025004UL = 0xC0DEu; + for (;;) { + } + return 0; +} +EOF + "$CLANG" --target=w65816 -O2 -ffunction-sections \ + -I"$PROJECT_ROOT/runtime/include" -c "$cMathEdge" -o "$oMathEdge" + "$PROJECT_ROOT/tools/link816" -o "$binMathEdge" --text-base 0x1000 \ + "$oCrt0File" "$oMathEdge" "$PROJECT_ROOT/runtime/math.o" \ + "$PROJECT_ROOT/runtime/softFloat.o" \ + "$PROJECT_ROOT/runtime/softDouble.o" "$oLibgccFile" >/dev/null 2>&1 + if ! MAME_CHECK_FRAME=2500 MAME_SECS=45 \ + bash "$PROJECT_ROOT/scripts/runInMame.sh" "$binMathEdge" --check \ + 0x025000=0000 0x025002=ffff 0x025004=c0de >/dev/null 2>&1; then + die "MAME: math.c edge cases wrong (mailbox 0x025000 = mismatches, 0x025002 = first case, 0x025010 = its result)" + fi + rm -f "$cMathEdge" "$oMathEdge" "$binMathEdge" + + # memcpy / memmove / memset (libgcc.s): MVN / MVP block moves with a + # 16-bit [dp],y loop for short counts. tests/memBlock/memBlockTest.c + # runs them against a byte-exact model -- every length class (0, 1, + # short, both sides of the MVN threshold, over 64 KB), odd and even + # addresses, source and destination across a bank boundary, overlapping + # memmove both ways, memset including c > 255, the returned pointer and + # guard bytes -- and memBlockRegs.s checks DBR and D come back intact on + # every path (DBR = $05, D = $0100 at the call). memBlockIrq.c then + # gets interrupts injected (memBlockIrq.lua) while an outer move's stub + # is mid-MVN; the handler moves memory in other banks, and both must + # come out right: the bank operands live in a per-call stack stub, not + # in shared code. + log "check: MAME memcpy/memmove/memset: lengths, bank crossings, overlap, DBR and D" + memDir="$PROJECT_ROOT/tests/memBlock" + oMemT="$(mktemp --suffix=.o)" + oMemR="$(mktemp --suffix=.o)" + binMemT="$(mktemp --suffix=.bin)" + "$CLANG" --target=w65816 -O2 -ffunction-sections -c \ + "$memDir/memBlockTest.c" -o "$oMemT" + "$LLVM_MC" -arch=w65816 -filetype=obj "$memDir/memBlockRegs.s" -o "$oMemR" + "$PROJECT_ROOT/tools/link816" -o "$binMemT" --text-base 0x1000 \ + "$oCrt0File" "$oMemT" "$oMemR" "$oLibgccFile" >/dev/null 2>&1 + if ! MAME_CHECK_FRAME=6500 MAME_SECS=115 \ + bash "$PROJECT_ROOT/scripts/runInMame.sh" "$binMemT" --check \ + 0x025000=0000 0x025002=ffff 0x025004=c0de 0x025008=0fff >/dev/null 2>&1; then + die "MAME: memcpy/memmove/memset wrong (0x025000 = mismatches, 0x025002 = first case, 0x025008 = DBR/D probe bits)" + fi + log "check: MAME memcpy/memmove survive interrupts mid-MVN whose handler moves memory" + oMemI="$(mktemp --suffix=.o)" + oMemE="$(mktemp --suffix=.o)" + binMemI="$(mktemp --suffix=.bin)" + "$CLANG" --target=w65816 -O2 -ffunction-sections -c \ + "$memDir/memBlockIrq.c" -o "$oMemI" + "$LLVM_MC" -arch=w65816 -filetype=obj "$memDir/memBlockIrqEntry.s" -o "$oMemE" + "$PROJECT_ROOT/tools/link816" -o "$binMemI" --text-base 0x1000 \ + "$oCrt0File" "$oMemI" "$oMemE" "$oLibgccFile" >/dev/null 2>&1 + if ! MAME_LUA_EXTRA="$memDir/memBlockIrq.lua" MAME_CHECK_FRAME=3500 MAME_SECS=65 \ + bash "$PROJECT_ROOT/scripts/runInMame.sh" "$binMemI" --check \ + 0x025000=0000 0x025002=0000 0x025004=c0de 0x025006=0001 >/dev/null 2>&1; then + die "MAME: block move broken by a mid-MVN interrupt (0x025000 = bad outer, 0x025002 = bad nested, 0x025006 = 1 when >= 8 interrupts landed)" + fi + rm -f "$oMemT" "$oMemR" "$binMemT" "$oMemI" "$oMemE" "$binMemI" + + # malloc / calloc run the heap out to NULL. Bare metal (crt0.s) has no + # Memory Manager, so malloc uses link816's window, [end of BSS, $FFFE) + # of the entry bank -- bank 0 here, where that spans the $C000-$CFFF + # I/O page. calloc(1, 720) in a loop used to march into the soft + # switches after 45 blocks and crash MAME; libc now stops such a window + # at $C000 (48 blocks fit below it, then NULL). The + # check also wants every oversized request (over 0x7FF0, or a product + # over 64K) and the request after exhaustion to be NULL (LLVM816 ask + # #4: calloc once returned 1 there). + log "check: MAME calloc runs the bare-metal heap out to NULL, short of the I/O page" + cHeapX="$(mktemp --suffix=.c)" + oHeapX="$(mktemp --suffix=.o)" + binHeapX="$(mktemp --suffix=.bin)" + cat > "$cHeapX" <<'EOF' +// calloc past the end of the heap returns NULL, never a bogus pointer, and +// never a block on the bank-0 I/O page. Mailbox: 0x025000 = allocations +// that succeeded, 0x025002 = bad pointers (outside the heap window, on +// the I/O page or with a bad payload), 0x025004 = 0xC0DE, 0x025006 = 1 +// when every oversized request and the one after exhaustion were NULL. +#include +typedef unsigned long u32; +extern char __heap_start[]; +extern char __heap_end[]; +#define CHUNK 720u +#define MAXP 64u +static char *gP[MAXP]; +static void *volatile gOver[5]; +int main(void) { + unsigned short n = 0u; + unsigned short bad = 0u; + unsigned short i; + u32 lo = (u32)__heap_start; + u32 hi = (u32)__heap_end; + for (;;) { + char *p = (char *)calloc(1u, CHUNK); + if (!p) { + break; + } + if ((u32)p < lo || (u32)p + CHUNK > hi || (u32)p + CHUNK > 0xC000UL || n >= MAXP) { + bad++; + break; + } + for (i = 0u; i < CHUNK; i++) { + if (p[i] != 0) { + bad++; + break; + } + p[i] = (char)n; + } + gP[n++] = p; + } + for (i = 0u; i < n; i++) { + unsigned short k; + for (k = 0u; k < CHUNK; k++) { + if (gP[i][k] != (char)i) { + bad++; + break; + } + } + } + *(volatile unsigned short *)0x025000UL = n; + *(volatile unsigned short *)0x025002UL = bad; + // Each result is stored before it is tested: LLVM may fold `malloc(n) + // == 0` to false for an allocation nothing else uses. + gOver[0] = calloc(1u, 0x7FF1u); + gOver[1] = calloc(2u, 0x8000u); + gOver[2] = malloc(0xFFFFu); + gOver[3] = malloc(0x10000UL); + gOver[4] = calloc(1u, CHUNK); + *(volatile unsigned short *)0x025006UL = (unsigned short)(gOver[0] == 0 && gOver[1] == 0 && + gOver[2] == 0 && gOver[3] == 0 && gOver[4] == 0); + *(volatile unsigned short *)0x025004UL = 0xC0DEu; + for (;;) { + } + return 0; +} +EOF + "$CLANG" --target=w65816 -O2 -ffunction-sections -I"$PROJECT_ROOT/runtime/include" \ + -c "$cHeapX" -o "$oHeapX" + "$PROJECT_ROOT/tools/link816" -o "$binHeapX" --text-base 0x1000 \ + "$oCrt0File" "$oHeapX" "$PROJECT_ROOT/runtime/libc.o" \ + "$PROJECT_ROOT/runtime/extras.o" "$oLibgccFile" >/dev/null 2>&1 + if ! MAME_CHECK_FRAME=600 MAME_SECS=12 \ + bash "$PROJECT_ROOT/scripts/runInMame.sh" "$binHeapX" --check \ + 0x025002=0000 0x025004=c0de 0x025006=0001 >/dev/null 2>&1; then + die "MAME: heap exhaustion wrong (0x025002 = bad blocks, 0x025006 = 1 when the oversized / exhausted requests were NULL)" + fi + rm -f "$cHeapX" "$oHeapX" "$binHeapX" + # Regression: W65816SepRepCleanup's PHI-copy hoist must not move a # `STA slot` above an `AND/ORA/EOR/ADC/SBC/CMP_StackRel slot` read in # the gap. Before the fix (gap check only guarded LDA_StackRel), @@ -2607,7 +3076,28 @@ EOF 0x025000=0025 >/dev/null 2>&1; then die "MAME: byteStoreO0 != 0x0025 (-O0 byte-store wrap regression)" fi - rm -f "$oBso" "$binBso" "$oCrt0File" + rm -f "$oBso" "$binBso" + + # Regression: a longjmp skips the returns that give each caller its + # IMG8..IMG15 ($C0..$CE) words back. walk() keeps two pointers in + # $C2..$C8 homes and longjmps out; outer() keeps its own there across + # the call to the setjmp function f0(). Fix = a returns-twice call + # clobbers IMG8..IMG15, so f0() saves and restores all eight. + # outer() = 0x10, bufC[1] = 5, bufE[2] = 9 (broken: 9, 5, 8). + log "check: MAME runs setjmpImgSave regression (longjmp keeps IMG8..15)" + oSji="$(mktemp --suffix=.o)" + binSji="$(mktemp --suffix=.bin)" + "$CLANG" --target=w65816 -O2 -ffunction-sections \ + -I "$PROJECT_ROOT/runtime/include" -c \ + "$PROJECT_ROOT/tests/regress/setjmpImgSave.c" -o "$oSji" + "$PROJECT_ROOT/tools/link816" -o "$binSji" --text-base 0x1000 \ + "$oCrt0File" "$oSji" "$oLibgccFile" 2>/dev/null + if ! bash "$PROJECT_ROOT/scripts/runInMame.sh" "$binSji" --check \ + 0x025000=c0de 0x025002=0010 0x025004=0005 \ + 0x025006=0009 >/dev/null 2>&1; then + die "MAME: setjmpImgSave wrong (longjmp left IMG8..15 clobbered)" + fi + rm -f "$oSji" "$binSji" "$oCrt0File" fi # Fuzzer: generate 20 small random C programs and verify all compile. @@ -4020,6 +4510,55 @@ EOF fi rm -f "$cI32CmpFile" "$oI32CmpFile" "$binI32CmpFile" + # Every i64 predicate on values at each 16-bit word boundary and both + # sides of the sign bit, as a branch, a value and a select against + # registers and as a branch and a value against ten constants, then + # loops whose i64 counters cross 32-bit boundaries. Pins the + # four-word CMP/SBC chain (lowerI32Cmp for SETCC / BR_CC / SELECT_CC + # on i64, 2026-10). The generic expansion it replaced answered 63 of + # the value-form compares against constants wrongly, through a + # StackRelToImg argument-slot bug. kExp from tests/regress/genI64Cmp.py. + log "check: MAME i64 compares: every predicate on word-boundary values, loops across 4G" + cI64CmpFile="$(mktemp --suffix=.c)" + oI64CmpFile="$(mktemp --suffix=.o)" + binI64CmpFile="$(mktemp --suffix=.bin)" + python3 "$PROJECT_ROOT/tests/regress/genI64Cmp.py" > "$cI64CmpFile" + "$CLANG" --target=w65816 -O2 -ffunction-sections -c \ + "$cI64CmpFile" -o "$oI64CmpFile" + "$PROJECT_ROOT/tools/link816" -o "$binI64CmpFile" --text-base 0x1000 \ + "$oCrt0F" "$oLibgccFile" "$oI64CmpFile" \ + >/dev/null 2>&1 + if ! MAME_CHECK_FRAME=900 MAME_SECS=20 \ + bash "$PROJECT_ROOT/scripts/runInMame.sh" "$binI64CmpFile" --check \ + 0x025000=0000 0x025002=ffff 0x025004=c0de >/dev/null 2>&1; then + die "MAME: i64 compare predicates/loops wrong (mailbox 0x025000 = mismatches, 0x025002 = first case)" + fi + rm -f "$cI64CmpFile" "$oI64CmpFile" "$binI64CmpFile" + + # Every multiply / divide helper in libgcc.s (__mulhi3, + # __umulhisi3, __mulhisi3, __mulsi3, the 16- and 32-bit divides and + # modulos, signed and unsigned) against host C on the word-boundary + # values and an xorshift32 stream of size-classed operands, so each + # fast path and its boundaries run: one checksum per operation. The + # helpers were rewritten 2026-10 (shift-right multiply, remainder-in- + # A divide). kExp from tests/regress/genMulDiv.py. + log "check: MAME mul/div helpers: 12 operations x 1200 operand pairs against host C" + cMulDivFile="$(mktemp --suffix=.c)" + oMulDivFile="$(mktemp --suffix=.o)" + binMulDivFile="$(mktemp --suffix=.bin)" + python3 "$PROJECT_ROOT/tests/regress/genMulDiv.py" 300 > "$cMulDivFile" + "$CLANG" --target=w65816 -O2 -ffunction-sections -c \ + "$cMulDivFile" -o "$oMulDivFile" + "$PROJECT_ROOT/tools/link816" -o "$binMulDivFile" --text-base 0x1000 \ + "$oCrt0F" "$oLibgccFile" "$oMulDivFile" \ + >/dev/null 2>&1 + if ! MAME_CHECK_FRAME=3000 MAME_SECS=60 \ + bash "$PROJECT_ROOT/scripts/runInMame.sh" "$binMulDivFile" --check \ + 0x025000=0000 0x025002=c0de >/dev/null 2>&1; then + die "MAME: mul/div helpers wrong (mailbox 0x025000 = bitmask of failing operations)" + fi + rm -f "$cMulDivFile" "$oMulDivFile" "$binMulDivFile" + # The 2026-09-29 compare / loop lowering: sign tests on i8 / i16 / # i32 (bmi / bpl, `and #$80`, the `cmp #$8000` carry form under a PHI # copy), i32 == / != split into per-half branches (W65816CmpPrepare, @@ -4546,6 +5085,266 @@ EOF fi rm -f "$cCpFile" "$oCpFile" "$binCpFile" + # Loop shapes the 2026-10 backend work changed, against host-computed + # values: a pointer-inequality sum whose pointer is bumped in place + # (W65816SinkLoopInc), also across $02:FFFF -> $03:0000; `while + # (n--)` copy / sum loops followed by a call (loop-local DP homes at + # an exit the zero-trip guard shares); two loops whose counters share + # a spill slot (W65816SpillToX); an i16 counter passed after its + # increment. + log "check: MAME loop shapes: in-place pointer bumps, loop-local homes, shared spill slots" + cLsFile="$(mktemp --suffix=.c)" + oLsFile="$(mktemp --suffix=.o)" + binLsFile="$(mktemp --suffix=.bin)" + cat > "$cLsFile" <<'EOF' +// Loop shapes the 2026-10 backend work changed, each against a +// host-computed value. Mailbox: 0x025000 = mismatches, 0x025002 = first +// failing check, 0x025004 = 0xC0DE when done. +typedef unsigned char u8; +typedef unsigned short u16; +typedef unsigned long u32; +#define NOI __attribute__((noinline)) +static u8 gBuf[2048]; +static u16 gW1[100]; +static u16 gW2[100]; +static volatile u16 gSink; +static volatile u16 gAcc; +static u16 gVa = 3; +static u16 gWa = 5; +// Pointer-inequality sum: the pointer is bumped in place (SinkLoopInc). +NOI static u16 sumPtr(const u8 *p, const u8 *end) { + u16 s = 0; + while (p != end) { + s += *p++; + } + return s; +} +NOI static void sink(u16 v) { gSink = v; } +// `while (n--)` loops followed by a call: loop-local DP homes at an exit +// the zero-trip guard shares. +NOI static void copyWords(u16 *dst, const u16 *src, u16 n) { + while (n--) { + *dst++ = (u16)(*src++ ^ 0x5A5Au); + } + sink(1); +} +NOI static u16 sumWords(const u16 *p, u16 n) { + u16 s = 0; + while (n--) { + s += *p++; + } + sink(s); + return s; +} +// Two loops whose counters share a spill slot (SpillToX). +NOI static void twoLoops(u16 n) { + u16 i; + for (i = 0; i < n; i++) { + gAcc = (u16)(gAcc + gVa); + } + for (i = 0; i < n; i++) { + gAcc = (u16)(gAcc + gWa); + } +} +NOI static u16 f2(u16 a, u16 b) { return (u16)(a * b + a); } +// An i16 counter whose old value is passed after the increment. +NOI static u16 counterArg(u16 n) { + u16 s = 0; + u16 i; + for (i = 0; i < n; i++) { + s = (u16)(s + f2(i, (u16)(i + 1))); + } + return s; +} +static u16 gBad; +static u16 gFirst = 0xFFFFu; +static void check(u16 got, u16 want, u16 id) { + if (got != want) { + if (gBad == 0u) { + gFirst = id; + } + gBad++; + } +} +int main(void) { + u16 i; + u16 cw = 0; + volatile u8 *far = (volatile u8 *)0x02FFF0UL; + for (i = 0; i < 2048u; i++) { + gBuf[i] = (u8)(i * 7u + 3u); + } + for (i = 0; i < 32u; i++) { + far[i] = (u8)(0xA0u + i); + } + for (i = 0; i < 100u; i++) { + gW2[i] = (u16)(i * 0x0101u + 0x1234u); + } + check(sumPtr(gBuf, gBuf + 2047), 0xFB04u, 1u); + check(sumPtr(gBuf + 1, gBuf + 2048), 0xFBFDu, 2u); + // Crosses $02:FFFF -> $03:0000: the bump carries into the bank byte. + check(sumPtr((const u8 *)0x02FFF0UL, (const u8 *)0x030010UL), 0x15F0u, 3u); + copyWords(gW1, gW2, 100); + for (i = 0; i < 100u; i++) { + cw = (u16)(cw * 3u + gW1[i]); + } + check(cw, 0x88E2u, 4u); + check(sumWords(gW2, 100), 0x85A6u, 5u); + gAcc = 0; + twoLoops(100); + check(gAcc, 0x0320u, 6u); + check(counterArg(50), 0xA77Bu, 7u); + *(volatile u16 *)0x025000UL = gBad; + *(volatile u16 *)0x025002UL = gFirst; + *(volatile u16 *)0x025004UL = 0xC0DEu; + for (;;) { + } + return 0; +} +EOF + "$CLANG" --target=w65816 -O2 -ffunction-sections -c \ + "$cLsFile" -o "$oLsFile" + "$PROJECT_ROOT/tools/link816" -o "$binLsFile" --text-base 0x1000 \ + "$oCrt0F" "$oLibgccFile" "$oLsFile" \ + >/dev/null 2>&1 + if ! bash "$PROJECT_ROOT/scripts/runInMame.sh" "$binLsFile" --check \ + 0x025000=0000 0x025002=ffff 0x025004=c0de >/dev/null 2>&1; then + die "MAME: loop shape wrong (mailbox 0x025000 = mismatches, 0x025002 = first check)" + fi + rm -f "$cLsFile" "$oLsFile" "$binLsFile" + + # Callee-saved IMG8..IMG15 ($C0..$CE) come back unchanged from runtime + # calls, a PHA-frame function reads its stack arguments, and a + # compare keeps its flags across byte stores. Before 2026-10 + # StackRelToImg deleted the load of ImgCalleeSave's restore (`lda + # 3,s ; pha ; lda 3,s` read as one word), so wmemcpy, wmemmove, + # fminf (and rename, eleven Lua functions) handed back $C0 holding + # their return value; it promoted ungetc's FILE pointer argument to + # a DP word nothing loaded; and the PHP/PLP wrap walk took a byte + # store for the compare, so demos/randProbe.c's `r == 0x41C6` read + # the byte split's AND. The runtime objects are the ones compiled + # above with the compiler under test. + log "check: MAME runtime calls keep IMG8..15; stack args read; compares survive byte stores" + cRtcFile="$(mktemp --suffix=.c)" + oRtcFile="$(mktemp --suffix=.o)" + binRtcFile="$(mktemp --suffix=.bin)" + cat > "$cRtcFile" <<'EOF' +// Callee-saved IMG8..IMG15 ($C0..$CE) survive runtime calls, a small +// PHA-frame function reads its stack arguments, and a compare keeps its +// flags across byte stores. Mailbox: 0x025000 = mismatches, 0x025002 = +// first failing check (1 wmemcpy, 2 wmemmove, 3 fminf, 4..6 ungetLike, +// 7..8 splitTest), 0x025004 = 0xC0DE when done. +#include +#include +typedef unsigned short u16; +#define NOI __attribute__((noinline)) +#define IMG ((volatile u16 *)0x0000C0UL) +static wchar_t gA[8] = {1, 2, 3, 4, 5, 6, 7, 8}; +static wchar_t gB[8]; +volatile float gF1 = 2.5f; +volatile float gF2 = -1.0f; +// libc's ungetc, on a stand-in for its FILE: a 4-byte frame and a pointer +// argument read four times. W65816StackRelToImg promoted that argument +// to a DP word nothing loaded, and the stores went through garbage. +typedef struct { + char pad[2]; + char eof; + char pad2[19]; + int unget; +} FakeFileT; +NOI int ungetLike(int c, FakeFileT *s); +NOI int ungetLike(int c, FakeFileT *s) { + if (!s || c < 0) { + return -1; + } + if (s->unget >= 0) { + return -1; + } + s->unget = c & 0xFF; + s->eof = 0; + return c & 0xFF; +} +// demos/randProbe.c: the compare's BEQ lost to the byte split's AND when +// the wrap walk took the byte store for the test. +static volatile unsigned char gOut[3]; +NOI void splitTest(u16 r); +NOI void splitTest(u16 r) { + gOut[0] = (unsigned char)(r & 0xFFu); + gOut[1] = (unsigned char)((r >> 8) & 0xFFu); + gOut[2] = (r == 0x41C6u) ? 0x00u : 0x99u; +} +static FakeFileT gFile; +static FakeFileT *volatile gFileP = &gFile; +static u16 gBad; +static u16 gFirst = 0xFFFFu; +static void fail(u16 id) { + if (gBad == 0u) { + gFirst = id; + } + gBad++; +} +NOI static void fill(void) { + u16 i; + for (i = 0; i < 8u; i++) { + IMG[i] = (u16)(0x5A00u + i); + } +} +NOI static void checkImg(u16 id) { + u16 i; + for (i = 0; i < 8u; i++) { + if (IMG[i] != (u16)(0x5A00u + i)) { + fail(id); + return; + } + } +} +int main(void) { + fill(); + wmemcpy(gB, gA, 8); + checkImg(1u); + fill(); + wmemmove(gB + 1, gB, 6); + checkImg(2u); + fill(); + gF1 = fminf(gF1, gF2); + checkImg(3u); + gFile.unget = -1; + gFile.eof = 1; + if (ungetLike('Q', gFileP) != 'Q') { + fail(4u); + } + if (gFile.unget != 'Q' || gFile.eof != 0) { + fail(5u); + } + if (ungetLike('R', gFileP) != -1) { + fail(6u); + } + splitTest(0x41C6u); + if (gOut[0] != 0xC6u || gOut[1] != 0x41u || gOut[2] != 0x00u) { + fail(7u); + } + splitTest(0x00C6u); + if (gOut[0] != 0xC6u || gOut[1] != 0x00u || gOut[2] != 0x99u) { + fail(8u); + } + *(volatile u16 *)0x025000UL = gBad; + *(volatile u16 *)0x025002UL = gFirst; + *(volatile u16 *)0x025004UL = 0xC0DEu; + for (;;) { + } + return 0; +} +EOF + "$CLANG" --target=w65816 -O2 -ffunction-sections \ + -I"$PROJECT_ROOT/runtime/include" -c "$cRtcFile" -o "$oRtcFile" + "$PROJECT_ROOT/tools/link816" -o "$binRtcFile" --text-base 0x1000 \ + "$oCrt0F" "$oRtcFile" "$oExtrasF" "$oMathF" "$oSfF" "$oSdF" \ + "$oLibcF" "$oLibgccFile" >/dev/null 2>&1 + if ! bash "$PROJECT_ROOT/scripts/runInMame.sh" "$binRtcFile" --check \ + 0x025000=0000 0x025002=ffff 0x025004=c0de >/dev/null 2>&1; then + die "MAME: runtime call clobbered IMG8..15 or a stack argument was lost (mailbox 0x025000 = mismatches, 0x025002 = first check)" + fi + rm -f "$cRtcFile" "$oRtcFile" "$binRtcFile" + # Regression: free() coalescing must remove blocks absorbed # into a lower-address neighbour from the free list. Old code # extended the lower block but left the absorbed entry in @@ -8141,7 +8940,7 @@ print('OK') omfElr="$(mktemp --suffix=.omf)" cat > "$cElrFile" <<'EOF' __attribute__((noinline)) static int helper(int x) { return x + 1; } -void main(void) { *(volatile unsigned char *)0x70 = (unsigned char)helper(0x40); } +int main(void) { *(volatile unsigned char *)0x70 = (unsigned char)helper(0x40); return 0; } EOF "$CLANG" --target=w65816 -O2 -c "$cElrFile" -o "$oElrFile" "$PROJECT_ROOT/tools/link816" -o "$binElr" --text-base 0x1000 \ @@ -8206,8 +9005,9 @@ print(f'OK: ExpressLoad header points at {nRel} cRELOC sites') omfR1="$(mktemp --suffix=.omf)" cat > "$cR1" <<'EOF' __attribute__((noinline)) static int helper(int x) { return x + 1; } -void main(void) { +int main(void) { *(volatile unsigned char *)0x00007F = (unsigned char)helper(0x40); + return 0; } EOF "$CLANG" --target=w65816 -O2 -c "$cR1" -o "$oR1" @@ -8341,6 +9141,215 @@ EOF "$omfGsf" "$testFileGsf" fi +# fflush reaches the disk. stdio keeps no buffer, but the GS/OS FST keeps +# written blocks and the file's EOF in memory until Flush ($2015) or +# Close, so a program that crashes after logging loses the log. fflush +# used to be `return 0`. The program writes three lines, calls fflush, +# and spins without fclose or exit; runViaFinder then stops MAME, which +# writes the floppy image back as the guest left it (a power-off), and +# cadius reads the file back. The control build skips fflush and must +# find the file still empty, or the check could not tell. Skips under +# SMOKE_SKIP_GSOSFLUSH=1. +if [ "${SMOKE_SKIP_GSOSFLUSH:-0}" != "1" ] \ + && [ -x "$CLANG" ] && [ -x "$CADIUS" ] && [ -f "$SYSDISK" ] \ + && command -v mame >/dev/null 2>&1; then + log "check: GS/OS fflush puts written lines on the disk with the file still open" + flushDir="$(mktemp -d)" + cat > "$flushDir/flush.c" <<'EOF' +#include +#define MB ((volatile unsigned char *)0x70) +int main(void) { + FILE *f = fopen("/DATA/FLUSHLOG", "w"); + MB[0] = 0u; + MB[1] = 0u; + if (f) { + MB[0] = 0x11u; + fputs("fflush reaches the disk\n", f); + fprintf(f, "line %d of %d\n", 2, 3); + fputs("third line, then the power goes\n", f); +#ifdef DO_FLUSH + if (fflush(f) == 0) { + MB[1] = 0x22u; + } +#else + MB[1] = 0x22u; +#endif + } + for (;;) { + } + return 0; +} +EOF + printf 'fflush reaches the disk\nline 2 of 3\nthird line, then the power goes\n' \ + > "$flushDir/expected.txt" + for flushMode in flush control; do + flushDef=() + [ "$flushMode" = flush ] && flushDef=(-DDO_FLUSH) + "$CLANG" --target=w65816 -I"$PROJECT_ROOT/runtime/include" -O2 -ffunction-sections \ + "${flushDef[@]}" -c "$flushDir/flush.c" -o "$flushDir/$flushMode.o" + "$PROJECT_ROOT/tools/link816" -o "$flushDir/$flushMode.bin" --text-base 0x1000 \ + --map "$flushDir/$flushMode.map" --reloc-out "$flushDir/$flushMode.reloc" \ + "$PROJECT_ROOT/runtime/crt0Gsos.o" "$flushDir/$flushMode.o" \ + "$PROJECT_ROOT/runtime/libc.o" \ + "$PROJECT_ROOT/runtime/snprintf.o" \ + "$PROJECT_ROOT/runtime/extras.o" \ + "$PROJECT_ROOT/runtime/softFloat.o" \ + "$PROJECT_ROOT/runtime/softDouble.o" \ + "$PROJECT_ROOT/runtime/iigsGsos.o" \ + "$PROJECT_ROOT/runtime/iigsToolbox.o" "$PROJECT_ROOT/runtime/iigsToolboxExtra.o" \ + "$PROJECT_ROOT/runtime/libgcc.o" >/dev/null \ + || die "GS/OS fflush smoke: link failed" + "$PROJECT_ROOT/tools/omfEmit" --input "$flushDir/$flushMode.bin" \ + --map "$flushDir/$flushMode.map" --base 0x1000 --entry __start \ + --output "$flushDir/$flushMode.omf" --name HELLO --expressload \ + --relocs "$flushDir/$flushMode.reloc" >/dev/null 2>&1 + [ -s "$flushDir/$flushMode.omf" ] || die "GS/OS fflush smoke: omfEmit produced no OMF" + if ! bash "$PROJECT_ROOT/scripts/runViaFinder.sh" "$flushDir/$flushMode.omf" \ + --save-data "$flushDir/$flushMode.po" \ + --check 0x70=0x11 0x71=0x22 >/dev/null 2>&1; then + die "GS/OS fflush smoke ($flushMode): fopen or fflush failed (\$70 = 0x11 opened, \$71 = 0x22 flushed)" + fi + mkdir -p "$flushDir/out-$flushMode" + "$CADIUS" EXTRACTFILE "$flushDir/$flushMode.po" /DATA/FLUSHLOG \ + "$flushDir/out-$flushMode/" >/dev/null 2>&1 || true + flushGot="$(ls "$flushDir/out-$flushMode"/FLUSHLOG* 2>/dev/null | head -1)" + if [ -n "$flushGot" ] && cmp -s "$flushGot" "$flushDir/expected.txt"; then + flushSame=1 + else + flushSame=0 + fi + if [ "$flushMode" = flush ] && [ "$flushSame" != 1 ]; then + die "GS/OS fflush smoke: the flushed lines are not on the disk image" + fi + if [ "$flushMode" = control ] && [ "$flushSame" = 1 ]; then + die "GS/OS fflush smoke: the lines reached the disk without fflush too; the check cannot tell" + fi + done + rm -rf "$flushDir" +fi + +# A GS/OS program with 26 KB of static BSS keeps it: the Loader reserves +# the whole segment, BSS included, so malloc's Memory-Manager heap +# (__heapInitMM), a 32 KB NewHandle and GS/OS's own file buffers all land +# elsewhere; the heap fills and then says NULL; fopen / fwrite / fread +# round-trip a file. (LLVM816 asks #4 and #5, fixed 2026-07-06 and +# re-confirmed 2026-09-30.) --full-bank: under GS/OS the entry bank has +# no I/O page to skip. Skips under SMOKE_SKIP_GSOSBIGBSS=1. +if [ "${SMOKE_SKIP_GSOSBIGBSS:-0}" != "1" ] \ + && [ -x "$CLANG" ] && [ -x "$CADIUS" ] && [ -f "$SYSDISK" ] \ + && command -v mame >/dev/null 2>&1; then + log "check: GS/OS 26 KB BSS stays clear of the heap, NewHandle and file buffers" + bbDir="$(mktemp -d)" + cat > "$bbDir/bigBss.c" <<'EOF' +// A GS/OS program with 26 KB of static BSS. The Loader must +// reserve the whole segment (BSS included), so no Memory Manager block -- +// malloc's heap handle (__heapInitMM), GS/OS's own file buffers, a 32 KB +// NewHandle -- may land on it, and fopen must work. $70 = 0x11 the BSS +// pattern survived everything, $71 = 0x22 the heap filled up with no +// block on the BSS and then said NULL, $72 = 0x33 the NewHandle block is +// clear of the BSS, $73 = 0x44 a file written and read back matched. +#include +#include +#include +#include +typedef unsigned char u8; +typedef unsigned short u16; +typedef unsigned long u32; +#define MB ((volatile u8 *)0x70) +#define BIG 26624u +#define CHUNK 512u +#define MAX_CHUNK 200u +#define FILE_LEN 4096u +#define NH_LEN 0x8000UL +#define NH_LOCKED 0x8000u +static u8 gBig[BIG]; + + +static u8 pat(u16 i) { + return (u8)(i * 7u + (i >> 8)); +} + + +static int overlaps(u32 a, u32 n, u32 lo, u32 hi) { + return a < hi && a + n > lo; +} + + +int main(void) { + u32 lo = (u32)gBig; + u32 hi = lo + BIG; + u16 i; + u16 n = 0u; + u16 bad = 0u; + u8 *rd; + void *p; + void **h; + FILE *f; + for (i = 0u; i < BIG; i++) { + gBig[i] = pat(i); + } + rd = (u8 *)malloc(FILE_LEN); + if (!rd || overlaps((u32)rd, FILE_LEN, lo, hi)) { + bad++; + } + while (n < MAX_CHUNK && (p = malloc(CHUNK)) != 0) { + if (overlaps((u32)p, CHUNK, lo, hi)) { + bad++; + } + memset(p, 0xEE, CHUNK); + n++; + } + if (n > 0u && n < MAX_CHUNK && bad == 0u) { + MB[1] = 0x22u; + } + h = (void **)NewHandle(NH_LEN, MMStartUp(), NH_LOCKED, 0); + if (h && *h && !overlaps((u32)*h, NH_LEN, lo, hi)) { + MB[2] = 0x33u; + } + f = fopen("/DATA/BIGBSS", "w"); + if (f && rd && fwrite(gBig, 1u, FILE_LEN, f) == FILE_LEN && fclose(f) == 0) { + f = fopen("/DATA/BIGBSS", "r"); + if (f && fread(rd, 1u, FILE_LEN, f) == FILE_LEN && fclose(f) == 0 && + memcmp(rd, gBig, FILE_LEN) == 0) { + MB[3] = 0x44u; + } + } + for (i = 0u; i < BIG; i++) { + if (gBig[i] != pat(i)) { + break; + } + } + if (i == BIG) { + MB[0] = 0x11u; + } + return 0; +} +EOF + "$CLANG" --target=w65816 -I"$PROJECT_ROOT/runtime/include" -O2 -ffunction-sections \ + -c "$bbDir/bigBss.c" -o "$bbDir/bigBss.o" + "$PROJECT_ROOT/tools/link816" -o "$bbDir/bigBss.bin" --text-base 0x1000 --full-bank \ + --map "$bbDir/bigBss.map" --reloc-out "$bbDir/bigBss.reloc" \ + "$PROJECT_ROOT/runtime/crt0Gsos.o" "$bbDir/bigBss.o" \ + "$PROJECT_ROOT/runtime/libc.o" \ + "$PROJECT_ROOT/runtime/snprintf.o" \ + "$PROJECT_ROOT/runtime/extras.o" \ + "$PROJECT_ROOT/runtime/softFloat.o" \ + "$PROJECT_ROOT/runtime/softDouble.o" \ + "$PROJECT_ROOT/runtime/iigsGsos.o" \ + "$PROJECT_ROOT/runtime/iigsToolbox.o" "$PROJECT_ROOT/runtime/iigsToolboxExtra.o" \ + "$PROJECT_ROOT/runtime/libgcc.o" >/dev/null \ + || die "GS/OS big-BSS smoke: link failed" + "$PROJECT_ROOT/tools/omfEmit" --input "$bbDir/bigBss.bin" --map "$bbDir/bigBss.map" \ + --base 0x1000 --entry __start --output "$bbDir/bigBss.omf" \ + --name HELLO --expressload --relocs "$bbDir/bigBss.reloc" >/dev/null 2>&1 + [ -s "$bbDir/bigBss.omf" ] || die "GS/OS big-BSS smoke: omfEmit produced no OMF" + if ! bash "$PROJECT_ROOT/scripts/runViaFinder.sh" "$bbDir/bigBss.omf" \ + --check 0x70=0x11 0x71=0x22 0x72=0x33 0x73=0x44 >/dev/null 2>&1; then + die "GS/OS big-BSS smoke FAILED (\$70 BSS intact, \$71 heap, \$72 NewHandle, \$73 file round trip)" + fi + rm -rf "$bbDir" +fi + # GS/OS program exit: returning from main is exit(main's value), which must # run the atexit handlers (last registered first), then the C++ dtors, then # QUIT back to the Finder. crt0Gsos used to issue QUIT with the pointer @@ -8403,6 +9412,77 @@ EOF rm -f "$cGsx" "$oGsx" "$binGsx" "$mapGsx" "$relGsx" "$omfGsx" fi +# A weak function's address under GS/OS: the program bank when the function +# is linked (a call through the pointer reaches it), a null pointer when it +# is not. The bank half used to be a literal 0 for every extern-weak +# symbol, so a call through `&weakFn` jumped into bank 0 (the runtime's +# gsosFlush hook crashed); now it is `lda #weakFn ; beq ; lda $BE`. $70 = +# 0x5A from hookFn, $71 = 0xCC (missingFn's address tested null), $72 = +# 0xA5 at the end of main. Skips under SMOKE_SKIP_GSOSEXIT=1 with the exit +# check above. +if [ "${SMOKE_SKIP_GSOSEXIT:-0}" != "1" ] \ + && [ -x "$CLANG" ] && [ -x "$CADIUS" ] && [ -f "$SYSDISK" ] \ + && command -v mame >/dev/null 2>&1; then + log "check: GS/OS weak function pointer reaches its definition, undefined weak is null" + cGsw="$(mktemp --suffix=.c)" + cGswHook="$(mktemp --suffix=.c)" + oGsw="$(mktemp --suffix=.o)" + oGswHook="$(mktemp --suffix=.o)" + binGsw="$(mktemp --suffix=.bin)" + mapGsw="$(mktemp --suffix=.map)" + relGsw="$(mktemp --suffix=.reloc)" + omfGsw="$(mktemp --suffix=.omf)" + cat > "$cGsw" <<'EOF' +#define MB ((volatile unsigned char *)0x70) +typedef void (*Fn)(void); +extern void hookFn(void) __attribute__((weak)); +extern void missingFn(void) __attribute__((weak)); +volatile Fn gHook; +volatile Fn gMissing; +int main(void) { + MB[0] = 0u; + gHook = hookFn; + gMissing = missingFn; + if (gHook) { + gHook(); + } + MB[1] = gMissing ? 0xBBu : 0xCCu; + MB[2] = 0xA5u; + return 0; +} +EOF + cat > "$cGswHook" <<'EOF' +void hookFn(void) { *(volatile unsigned char *)0x70 = 0x5Au; } +EOF + "$CLANG" --target=w65816 -I"$PROJECT_ROOT/runtime/include" -O2 -ffunction-sections -c \ + "$cGsw" -o "$oGsw" + "$CLANG" --target=w65816 -I"$PROJECT_ROOT/runtime/include" -O2 -ffunction-sections -c \ + "$cGswHook" -o "$oGswHook" + "$PROJECT_ROOT/tools/link816" -o "$binGsw" --text-base 0x1000 \ + --map "$mapGsw" --reloc-out "$relGsw" \ + "$PROJECT_ROOT/runtime/crt0Gsos.o" "$oGsw" "$oGswHook" \ + "$PROJECT_ROOT/runtime/libc.o" \ + "$PROJECT_ROOT/runtime/snprintf.o" \ + "$PROJECT_ROOT/runtime/extras.o" \ + "$PROJECT_ROOT/runtime/softFloat.o" \ + "$PROJECT_ROOT/runtime/softDouble.o" \ + "$PROJECT_ROOT/runtime/iigsGsos.o" \ + "$PROJECT_ROOT/runtime/iigsToolbox.o" "$PROJECT_ROOT/runtime/iigsToolboxExtra.o" \ + "$PROJECT_ROOT/runtime/libgcc.o" >/dev/null \ + || die "GS/OS weak pointer smoke: link failed" + "$PROJECT_ROOT/tools/omfEmit" --input "$binGsw" --map "$mapGsw" \ + --base 0x1000 --entry __start --output "$omfGsw" \ + --name HELLO --expressload --relocs "$relGsw" >/dev/null 2>&1 + [ -s "$omfGsw" ] || die "GS/OS weak pointer smoke: omfEmit produced empty/missing OMF" + if ! bash "$PROJECT_ROOT/scripts/runViaFinder.sh" "$omfGsw" \ + --check 0x70=0x5a --check 0x71=0xcc --check 0x72=0xa5 >/dev/null 2>&1; then + bash "$PROJECT_ROOT/scripts/runViaFinder.sh" "$omfGsw" \ + --check 0x70=0x5a --check 0x71=0xcc --check 0x72=0xa5 2>&1 | tail -5 >&2 + die "GS/OS weak pointer smoke FAILED (call through &weakFn / null undefined weak)" + fi + rm -f "$cGsw" "$cGswHook" "$oGsw" "$oGswHook" "$binGsw" "$mapGsw" "$relGsw" "$omfGsw" +fi + # Phase C.2: cINTERSEG cross-segment JSL under real GS/OS Loader. # Links a 2-user-seg program (main in seg 1, pinnedFn in seg 2 via # __attribute__((section(".text.seg2")))), uses --multi-seg-expressload diff --git a/src/link816/link816.cpp b/src/link816/link816.cpp index 518a9de..6e78c6f 100644 --- a/src/link816/link816.cpp +++ b/src/link816/link816.cpp @@ -1767,8 +1767,11 @@ struct Linker { // __heap_start / __heap_end: everything from bss_end to the top // of the entry segment's BANK. // - // There is no IO window and no language card to dodge here. Those - // exist only in banks $00/$01/$E0/$E1; GS/OS places bank-aligned + // Under GS/OS there is no IO window and no language card to dodge + // here. (A bare-metal link -- crt0.s, loaded at $00:1000 -- DOES put + // this window in bank 0, across the $C000-$CFFF I/O page; libc's + // heap init stops such a window at $C000 in banks $00/$01/$E0/$E1.) + // Those exist only in banks $00/$01/$E0/$E1; GS/OS places bank-aligned // OMF segments in free banks (measured under MAME: joeylib's // examples run in $08-$0B, and bank $00 shows up only in ROM and // GS/OS glue), and crt0Gsos points DBR at the entry segment's diff --git a/src/llvm/lib/Target/W65816/CMakeLists.txt b/src/llvm/lib/Target/W65816/CMakeLists.txt index cbf2acf..c21d075 100644 --- a/src/llvm/lib/Target/W65816/CMakeLists.txt +++ b/src/llvm/lib/Target/W65816/CMakeLists.txt @@ -36,10 +36,14 @@ add_llvm_target(W65816CodeGen W65816SjLjFinalize.cpp W65816LowerWide32.cpp W65816I32IncFold.cpp + W65816ReturnPopMerge.cpp + W65816CommuteReload.cpp W65816ImgCalleeSave.cpp W65816NarrowI32Mul.cpp W65816CmpPrepare.cpp + W65816SplitLoopExits.cpp W65816CmpSink.cpp + W65816SinkLoopInc.cpp W65816PromoteFiToImg.cpp W65816StackRelToImg.cpp W65816StackSlotMerge.cpp diff --git a/src/llvm/lib/Target/W65816/W65816.h b/src/llvm/lib/Target/W65816/W65816.h index 206fbbe..ff99ede 100644 --- a/src/llvm/lib/Target/W65816/W65816.h +++ b/src/llvm/lib/Target/W65816/W65816.h @@ -102,7 +102,7 @@ FunctionPass *createW65816WidenAcc16(); // Targets fast-regalloc's habit of spilling A unnecessarily; the // 3x speedup is the biggest single per-iteration win we can get // without switching to a smarter allocator. See W65816SpillToX.cpp. -FunctionPass *createW65816SpillToX(); +FunctionPass *createW65816SpillToX(bool SharedSlots = false); // Pre-emit peephole: rewrite `LDY #neg ; (LDA|STA) (sr,S),Y` to // pre-add the offset to the pointer with Y=0. The 65816 spec for @@ -131,6 +131,11 @@ FunctionPass *createW65816NarrowI32Mul(); // one at a time. See W65816CmpPrepare.cpp. FunctionPass *createW65816CmpPrepare(); +// IR pass before ISel: split each critical edge out of a loop into a block +// with PHIs, so PHI elimination's copies run once per exit instead of on +// every trip. See W65816SplitLoopExits.cpp. +FunctionPass *createW65816SplitLoopExits(); + // Pre-RA pass, inserted right after the register coalescer: moves the // instruction that sets a conditional branch's flags down past the back- // edge COPYs PHI elimination left between them, so the copies (and the @@ -176,11 +181,14 @@ FunctionPass *createW65816StackRelToImg(); // vregs. Without this, greedy/basic regalloc can't fit the pair- // pressure of i64-via-2-i32-via-Wide32 traffic in i64-heavy // functions (RegAllocBase crashes during allocatePhysRegs). After -// this pass, only i16 vregs reach regalloc, and the pair structure -// lives only in the LDAptr32S / STAptr32S / STBptr32S pseudos which -// take 2 i16 ptr operands directly. +// this pass, only i16 vregs reach regalloc. FunctionPass *createW65816LowerWide32(); +// Pre-RA, SSA: move a loop-carried increment below the last use of the +// value it increments so the coalescer can merge the two (no back-edge +// copy of a bumped pointer). See W65816SinkLoopInc.cpp. +FunctionPass *createW65816SinkLoopInc(); + // Pre-emit peephole: detect the post-PEI 6-instruction `i32 += 1` // pattern (LDA-ADCi16imm-STA-LDA-ADCEi16imm-STA on consecutive i16 // stack-rel halves) and rewrite to LDA-INA-STA + INC_HI_IF_CARRY. @@ -188,6 +196,17 @@ FunctionPass *createW65816LowerWide32(); // See W65816I32IncFold.cpp. FunctionPass *createW65816I32IncFold(); +// Pre-emit: where a path to a return is only stack releases (the last +// call's argument cleanup, then the epilogue, often behind a branch to a +// shared return block), give the block one release for the total and a +// return of its own. See W65816ReturnPopMerge.cpp. +FunctionPass *createW65816ReturnPopMerge(); + +// Pre-emit, last: `sta slot ; lda x ; op slot` with a commutative op +// (ORA / AND / EOR / ADC) becomes `sta slot ; op x`. See +// W65816CommuteReload.cpp. +FunctionPass *createW65816CommuteReload(); + // Post-RA, pre-PEI pass that emits prologue save + epilogue restore for // IMG8..IMG15 if the function uses them. Makes IMG8..IMG15 behave as // callee-saved at the asm level without going through LLVM's CSR @@ -228,7 +247,9 @@ void initializeW65816LowerWide32Pass(PassRegistry &); void initializeW65816ImgCalleeSavePass(PassRegistry &); void initializeW65816NarrowI32MulPass(PassRegistry &); void initializeW65816CmpPreparePass(PassRegistry &); +void initializeW65816SplitLoopExitsPass(PassRegistry &); void initializeW65816CmpSinkPass(PassRegistry &); +void initializeW65816SinkLoopIncPass(PassRegistry &); void initializeW65816PromoteFiToImgPass(PassRegistry &); void initializeW65816StackSlotMergePass(PassRegistry &); void initializeW65816StackRelToImgPass(PassRegistry &); diff --git a/src/llvm/lib/Target/W65816/W65816AsmPrinter.cpp b/src/llvm/lib/Target/W65816/W65816AsmPrinter.cpp index 0a596e4..9181f6a 100644 --- a/src/llvm/lib/Target/W65816/W65816AsmPrinter.cpp +++ b/src/llvm/lib/Target/W65816/W65816AsmPrinter.cpp @@ -532,17 +532,23 @@ void W65816AsmPrinter::emitInstruction(const MachineInstr *MI) { // no help; INTERSEG is for cross-segment refs. Reading PBR at // runtime via $BE is the only working mechanism on real hardware. // - // EXCEPTION — external-weak symbols: a `&weakfn` whose definition may + // EXCEPTION -- external-weak symbols: a `&weakfn` whose definition may // be absent at link time must be representable as NULL. If we emit // `lda $BE` for its bank, the runtime PBR (non-zero when GS/OS loads // the segment in a non-zero bank) makes the pointer always non-null, // so `if (weakfn) weakfn()` is always taken and jumps to the - // unresolved (=0) target — a crash. For an external-weak global the - // bank must be a literal 0 instead, so an undefined-weak ref is a - // genuine null and a strong link-time def still works (its low16 is - // non-zero so the null test passes and the direct call uses the - // symbol's own reloc, not this bank byte). Defined/normal globals - // and compiler external symbols (libgcc helpers) keep `lda $BE`. + // unresolved (=0) target -- a crash. A literal 0 bank fixed that but + // broke the other case: when a definition IS linked, `p = weakfn; + // p()` called bank 0 (the runtime's `&gsosFlush` hook crashed under + // GS/OS). So the bank depends on the symbol's low 16 bits, which the + // linker resolves to 0 for an undefined weak: + // lda #weakfn ; beq L ; lda $BE ; L: + // giving 0 (a null pointer) or the program bank. A defined weak at + // offset 0 of its bank would read as null; nothing is placed there + // (the entry segment starts with crt0, and link816 sends other + // segments' function pointers through entry-segment trampolines). + // Defined/normal globals and compiler external symbols (libgcc + // helpers) keep the plain `lda $BE`. // // CODE symbols in a multi-segment OMF app are NOT an exception, even // though every overflow segment lives in its own bank: link816 @@ -554,10 +560,16 @@ void W65816AsmPrinter::emitInstruction(const MachineInstr *MI) { // on 2026-09-22 and broke every such pointer (bank of the callee + // offset of the trampoline). Do not resurrect it. const MachineOperand &Sym = MI->getOperand(1); - bool WeakUndef = Sym.isGlobal() && - Sym.getGlobal()->hasExternalWeakLinkage(); - if (WeakUndef) { - emitOpImm(W65816::LDA_Imm16, 0); + if (Sym.isGlobal() && Sym.getGlobal()->hasExternalWeakLinkage()) { + MCSymbol *NullSym = OutContext.createTempSymbol(); + EmitToStreamer(*OutStreamer, + MCInstBuilder(W65816::LDA_Imm16) + .addOperand(lowerOperand(Sym, MCInstLowering))); + EmitToStreamer(*OutStreamer, + MCInstBuilder(W65816::BEQ).addExpr( + MCSymbolRefExpr::create(NullSym, OutContext))); + emitOpImm(W65816::LDA_DP, kRuntimePbrStashDP); + OutStreamer->emitLabel(NullSym); } else { emitOpImm(W65816::LDA_DP, kRuntimePbrStashDP); } @@ -648,8 +660,11 @@ void W65816AsmPrinter::emitInstruction(const MachineInstr *MI) { // STA runs in our LDAi8imm-set M=8 mode. Saves 4B / 6cyc per // hit. We mark the next-SEP-to-skip via a per-AsmPrinter flag // so the SEP visit drops it. - emitSepM(); - emitOpImm(W65816::LDA_Imm8, MI->getOperand(1).getImm() & 0xFF); + // + // Otherwise the byte is a 16-bit `lda #$00nn` with no mode switch at + // all: an i8 value's high byte is don't-care, and nothing reads the + // flags of a constant load (LDAi8imm models no Defs=[P]). 3 bytes / + // 3 cycles against SEP / LDA / REP's 6 / 8. bool SkipRep = false; // Walk past mode-neutral MIs (X-flag-only ops, branches, transfers // that don't touch A) to find the next SEP/REP — same idea as the @@ -701,6 +716,13 @@ void W65816AsmPrinter::emitInstruction(const MachineInstr *MI) { SkipRep = true; SkipNextSta8Wrap = true; } + static const bool Wide16Imm = !getenv("W65816_DISABLE_I8_IMM16"); + if (!SkipRep && Wide16Imm) { + emitOpImm(W65816::LDA_Imm16, MI->getOperand(1).getImm() & 0xFF); + return; + } + emitSepM(); + emitOpImm(W65816::LDA_Imm8, MI->getOperand(1).getImm() & 0xFF); if (!SkipRep) { emitRepM(); } @@ -1042,7 +1064,9 @@ void W65816AsmPrinter::emitInstruction(const MachineInstr *MI) { return; } case W65816::JSLpseudo: - case W65816::JSLpseudo32: { + case W65816::JSLpseudo32: + case W65816::JSLlight: + case W65816::JSLlight32: { EmitToStreamer(*OutStreamer, MCInstBuilder(W65816::JSL_Long) .addOperand(lowerOperand(MI->getOperand(0), MCInstLowering))); diff --git a/src/llvm/lib/Target/W65816/W65816CommuteReload.cpp b/src/llvm/lib/Target/W65816/W65816CommuteReload.cpp new file mode 100644 index 0000000..80cac47 --- /dev/null +++ b/src/llvm/lib/Target/W65816/W65816CommuteReload.cpp @@ -0,0 +1,167 @@ +//===-- W65816CommuteReload.cpp - Fold a reload into a commutative op ----===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===---------------------------------------------------------------------===// +// +// Pre-emit peephole, last: A is stored, something else is loaded, and the +// stored word is the other operand of a commutative op: +// +// sta 3,s sta 3,s +// lda 0xd0 => ora 0xd0 +// ora 3,s +// +// After the store A equals the slot, so the op can take the loaded operand +// instead and the load goes (4 or 5 cycles, 2 or 3 bytes). The op reads +// memory at the same point the load did, so even an operand that overlaps +// the slot reads the same bytes, and N / Z come out the same. ORA, AND, +// EOR and ADC (a CLC may sit between the load and an ADC; LDA leaves C +// alone either way). The two allocator-assigned operands of a far +// pointer's null test (`stx dp ; sta d,s ; lda dp ; ora d,s`) are the +// common case. The slot is a stack or direct-page word only, never an +// absolute address that could be an I/O register reading back something +// else. +// +// Kill switch: W65816_DISABLE_COMMUTE_RELOAD=1. +// +//===---------------------------------------------------------------------===// + +#include "W65816.h" +#include "W65816InstrInfo.h" +#include "W65816Subtarget.h" +#include "llvm/CodeGen/MachineFunctionPass.h" +#include "llvm/CodeGen/MachineInstrBuilder.h" + +using namespace llvm; + +#define DEBUG_TYPE "w65816-commute-reload" + +namespace { + +class W65816CommuteReload : public MachineFunctionPass { +public: + static char ID; + W65816CommuteReload() : MachineFunctionPass(ID) {} + bool runOnMachineFunction(MachineFunction &MF) override; + StringRef getPassName() const override { + return "W65816 fold a reload into a commutative op"; + } +}; + +char W65816CommuteReload::ID = 0; + +// One addressing mode of LDA and the same mode of ORA / AND / EOR / ADC. +struct ModeRow { + unsigned Lda; + unsigned Ora; + unsigned And; + unsigned Eor; + unsigned Adc; +}; + +const ModeRow Modes[] = { + {W65816::LDA_Abs, W65816::ORA_Abs, W65816::AND_Abs, W65816::EOR_Abs, + W65816::ADC_Abs}, + {W65816::LDA_AbsX, W65816::ORA_AbsX, W65816::AND_AbsX, W65816::EOR_AbsX, + W65816::ADC_AbsX}, + {W65816::LDA_AbsY, W65816::ORA_AbsY, W65816::AND_AbsY, W65816::EOR_AbsY, + W65816::ADC_AbsY}, + {W65816::LDA_DP, W65816::ORA_DP, W65816::AND_DP, W65816::EOR_DP, + W65816::ADC_DP}, + {W65816::LDA_DPInd, W65816::ORA_DPInd, W65816::AND_DPInd, + W65816::EOR_DPInd, W65816::ADC_DPInd}, + {W65816::LDA_DPIndLong, W65816::ORA_DPIndLong, W65816::AND_DPIndLong, + W65816::EOR_DPIndLong, W65816::ADC_DPIndLong}, + {W65816::LDA_DPIndLongY, W65816::ORA_DPIndLongY, W65816::AND_DPIndLongY, + W65816::EOR_DPIndLongY, W65816::ADC_DPIndLongY}, + {W65816::LDA_DPIndX, W65816::ORA_DPIndX, W65816::AND_DPIndX, + W65816::EOR_DPIndX, W65816::ADC_DPIndX}, + {W65816::LDA_DPIndY, W65816::ORA_DPIndY, W65816::AND_DPIndY, + W65816::EOR_DPIndY, W65816::ADC_DPIndY}, + {W65816::LDA_DPX, W65816::ORA_DPX, W65816::AND_DPX, W65816::EOR_DPX, + W65816::ADC_DPX}, + {W65816::LDA_Imm16, W65816::ORA_Imm16, W65816::AND_Imm16, + W65816::EOR_Imm16, W65816::ADC_Imm16}, + {W65816::LDA_Imm8, W65816::ORA_Imm8, W65816::AND_Imm8, W65816::EOR_Imm8, + W65816::ADC_Imm8}, + {W65816::LDA_Long, W65816::ORA_Long, W65816::AND_Long, W65816::EOR_Long, + W65816::ADC_Long}, + {W65816::LDA_LongX, W65816::ORA_LongX, W65816::AND_LongX, + W65816::EOR_LongX, W65816::ADC_LongX}, + {W65816::LDA_StackRel, W65816::ORA_StackRel, W65816::AND_StackRel, + W65816::EOR_StackRel, W65816::ADC_StackRel}, + {W65816::LDA_StackRelIndY, W65816::ORA_StackRelIndY, + W65816::AND_StackRelIndY, W65816::EOR_StackRelIndY, + W65816::ADC_StackRelIndY}, +}; + +const ModeRow *rowOfLoad(unsigned Opc) { + for (const ModeRow &R : Modes) + if (R.Lda == Opc) + return &R; + return nullptr; +} + +// The op of Row with the kind of Opc (an ORA / AND / EOR / ADC in Ref's +// mode), or 0. +unsigned sameKind(unsigned Opc, const ModeRow &Ref, const ModeRow &Row) { + if (Opc == Ref.Ora) + return Row.Ora; + if (Opc == Ref.And) + return Row.And; + if (Opc == Ref.Eor) + return Row.Eor; + if (Opc == Ref.Adc) + return Row.Adc; + return 0; +} + +} // namespace + +bool W65816CommuteReload::runOnMachineFunction(MachineFunction &MF) { + if (getenv("W65816_DISABLE_COMMUTE_RELOAD")) + return false; + const TargetInstrInfo *TII = MF.getSubtarget().getInstrInfo(); + const ModeRow *DpRow = rowOfLoad(W65816::LDA_DP); + const ModeRow *SrRow = rowOfLoad(W65816::LDA_StackRel); + bool Changed = false; + for (MachineBasicBlock &MBB : MF) { + for (auto It = MBB.begin(); It != MBB.end(); ++It) { + unsigned StOpc = It->getOpcode(); + if ((StOpc != W65816::STA_DP && StOpc != W65816::STA_StackRel) || + !It->getOperand(0).isImm()) + continue; + const ModeRow *SlotRow = StOpc == W65816::STA_DP ? DpRow : SrRow; + auto Ld = std::next(It); + if (Ld == MBB.end()) + continue; + const ModeRow *LdRow = rowOfLoad(Ld->getOpcode()); + if (!LdRow) + continue; + auto Op = std::next(Ld); + if (Op != MBB.end() && Op->getOpcode() == W65816::CLC) + ++Op; + if (Op == MBB.end() || Op->getNumOperands() == 0 || + !Op->getOperand(0).isImm() || + Op->getOperand(0).getImm() != It->getOperand(0).getImm()) + continue; + unsigned NewOpc = sameKind(Op->getOpcode(), *SlotRow, *LdRow); + if (!NewOpc) + continue; + MachineInstrBuilder New = + BuildMI(MBB, Op, Op->getDebugLoc(), TII->get(NewOpc)); + for (const MachineOperand &MO : Ld->explicit_operands()) + New.add(MO); + Op->eraseFromParent(); + Ld->eraseFromParent(); + Changed = true; + } + } + return Changed; +} + +FunctionPass *llvm::createW65816CommuteReload() { + return new W65816CommuteReload(); +} diff --git a/src/llvm/lib/Target/W65816/W65816FrameLowering.cpp b/src/llvm/lib/Target/W65816/W65816FrameLowering.cpp index b62b6e6..642be87 100644 --- a/src/llvm/lib/Target/W65816/W65816FrameLowering.cpp +++ b/src/llvm/lib/Target/W65816/W65816FrameLowering.cpp @@ -16,9 +16,13 @@ #include "W65816InstrInfo.h" #include "W65816MachineFunctionInfo.h" #include "W65816Subtarget.h" +#include "llvm/ADT/DenseMap.h" +#include "llvm/ADT/DenseSet.h" +#include "llvm/CodeGen/MachineDominators.h" #include "llvm/CodeGen/MachineFrameInfo.h" #include "llvm/CodeGen/MachineFunction.h" #include "llvm/CodeGen/MachineInstrBuilder.h" +#include "llvm/CodeGen/MachineLoopInfo.h" #include "llvm/CodeGen/MachineRegisterInfo.h" #include "llvm/IR/Function.h" #include "llvm/Support/CommandLine.h" @@ -101,6 +105,82 @@ bool W65816FrameLowering::hasReservedCallFrame(const MachineFunction &MF) const return false; } +// Lay out a large frame with its most-used objects nearest S. A slot more +// than 0xFF bytes above S is out of `off,s` reach and every access to it +// goes through the $F6 frame pointer (`sty $fa ; ldy #off ; lda [$f6],y ; +// ldy $fa`, 18 cycles against 5). PEI places the objects in this list +// from the top of the frame down, so the last ones end up nearest S; the +// default order (creation order) puts the IR's allocas -- arrays among +// them -- below nothing but the spill slots, and which scalars landed out +// of reach was an accident of numbering (Lua's luaV_execute did 400+ far +// accesses in its dispatch loop). Sort ascending by access density: +// uses weighted by loop depth, divided by size, so big arrays (reached +// through a pointer built once) go to the top and hot scalars to the +// bottom. Small frames keep the default order: everything is in reach. +void W65816FrameLowering::orderFrameObjects( + const MachineFunction &MF, SmallVectorImpl &ObjectsToAllocate) const { + const MachineFrameInfo &MFI = MF.getFrameInfo(); + uint64_t Total = 0; + for (int FI : ObjectsToAllocate) + Total += MFI.getObjectSize(FI); + if (ObjectsToAllocate.size() < 2 || Total <= 0xC0 || + getenv("W65816_DISABLE_FRAME_ORDER")) + return; + MachineFunction &MutMF = const_cast(MF); + MachineDominatorTree MDT(MutMF); + MachineLoopInfo MLI(MDT); + // Only the accesses that address the object `off,s` (or `[$f6],y`) + // count: an ADDframe builds its address with `adc #off` from anywhere. + auto isSlotAccess = [](unsigned Opc) { + switch (Opc) { + case W65816::LDAfi: case W65816::STAfi: case W65816::STA8fi: + case W65816::ADCfi: case W65816::ADCEfi: case W65816::SBCfi: + case W65816::SBCEfi: case W65816::ANDfi: case W65816::ORAfi: + case W65816::EORfi: case W65816::CMPfi: case W65816::LDAfi_indY: + case W65816::STAfi_indY: case W65816::STA8fi_indY: + return true; + default: + return false; + } + }; + DenseMap Uses; + // Objects some access starts past the end of: that is the PHA-bracket + // idiom (W65816ImgCalleeSave's `pha ; lda dp ; sta slot+2,s ; pla`), + // whose +2 is right only `off,s`, never through $F6, which a PHA does + // not move. They stay nearest S, where creation order (they are made + // last) always put them. + DenseSet PinnedNear; + for (const MachineBasicBlock &MBB : MF) { + // 8x per loop level, capped so a deep nest cannot overflow the sum. + uint64_t W = uint64_t(1) << (3 * std::min(MLI.getLoopDepth(&MBB), 6u)); + for (const MachineInstr &MI : MBB) { + if (!isSlotAccess(MI.getOpcode())) + continue; + for (unsigned I = 0, E = MI.getNumOperands(); I != E; ++I) { + const MachineOperand &MO = MI.getOperand(I); + if (!MO.isFI() || MO.getIndex() < 0) + continue; + Uses[MO.getIndex()] += W; + if (I + 1 < E && MI.getOperand(I + 1).isImm() && + (MI.getOperand(I + 1).getImm() < 0 || + MI.getOperand(I + 1).getImm() >= + (int64_t)MFI.getObjectSize(MO.getIndex()))) + PinnedNear.insert(MO.getIndex()); + } + } + } + llvm::stable_sort(ObjectsToAllocate, [&](int A, int B) { + bool PA = PinnedNear.count(A); + bool PB = PinnedNear.count(B); + if (PA != PB) + return PB; + uint64_t SA = std::max(MFI.getObjectSize(A), 1); + uint64_t SB = std::max(MFI.getObjectSize(B), 1); + // Uses[A] / SA < Uses[B] / SB, without the division. + return Uses.lookup(A) * SB < Uses.lookup(B) * SA; + }); +} + void W65816FrameLowering::determineCalleeSaves(MachineFunction &MF, BitVector &SavedRegs, RegScavenger *RS) const { @@ -200,20 +280,25 @@ void W65816FrameLowering::emitPrologue(MachineFunction &MF, } if (StackSize > 0) { - // Cycle math: each PHA is 4 cyc; the TSC-sequence (TAY+TSC+SEC+ - // SBC+TCS+TYA) is 13 cyc total. N PHAs win on cycles when 4*N <= 13, - // i.e. up to 3 PHAs (6-byte frame). At N=4 (8 bytes): 16 cyc PHAs vs - // 13 cyc TSC-seq → TSC wins. Threshold at 6 bytes for speed. - // (Bytes: N PHAs cost N bytes; TSC-seq costs 8 bytes. We're - // optimizing for speed per the project directive.) - if (StackSize <= 6 && (StackSize % 2) == 0) { + // Cycle math: each PHA is 4 cyc and 1 byte; TSC+SEC+SBC+TCS is 9 cyc + // and 6 bytes, 13 and 8 with the TAY/TYA that keeps an argument in A. + // Under the shared cost model (W65816Helpers::cycleByteCost) PHAs win + // up to 2 of them (4-byte frame) when A is free and up to 3 (6 bytes) + // when it carries an argument in. + bool KeepA = MBB.isLiveIn(W65816::A); + uint64_t Pushes = StackSize / 2; + uint64_t PushCost = W65816Helpers::cycleByteCost(4 * Pushes, Pushes); + uint64_t SubCost = KeepA ? W65816Helpers::cycleByteCost(13, 8) + : W65816Helpers::cycleByteCost(9, 6); + if ((StackSize % 2) == 0 && PushCost <= SubCost) { // Small frame: N/2 PHAs. Preserves A. - for (uint64_t i = 0; i < StackSize / 2; ++i) + for (uint64_t i = 0; i < Pushes; ++i) BuildMI(MBB, MBBI, DL, TII.get(W65816::PHA)); } else { - // Larger frame: TAY/TSC/.../TYA bracket. Preserves A via Y as a - // temp. - BuildMI(MBB, MBBI, DL, TII.get(W65816::TAY)); + // Larger frame: TSC/.../TCS, bracketed by TAY/TYA (Y as a temp) + // when A carries an argument in. + if (KeepA) + BuildMI(MBB, MBBI, DL, TII.get(W65816::TAY)); BuildMI(MBB, MBBI, DL, TII.get(W65816::TSC)); BuildMI(MBB, MBBI, DL, TII.get(W65816::SEC)); BuildMI(MBB, MBBI, DL, TII.get(W65816::SBC_Imm16)) @@ -243,7 +328,8 @@ void W65816FrameLowering::emitPrologue(MachineFunction &MF, // probe writes before invoking it. BuildMI(MBB, MBBI, DL, TII.get(W65816::STZ_DP)).addImm(0xF8); } - BuildMI(MBB, MBBI, DL, TII.get(W65816::TYA)); + if (KeepA) + BuildMI(MBB, MBBI, DL, TII.get(W65816::TYA)); } } // VLA function with no static frame (or PHA-only): still need FP. @@ -311,11 +397,8 @@ void W65816FrameLowering::emitEpilogue(MachineFunction &MF, bool YLive = false; bool XLive = false; if (MBBI != MBB.end() && MBBI->isReturn()) { - for (const MachineOperand &MO : MBBI->operands()) { - if (!MO.isReg() || !MO.isImplicit() || !MO.isUse()) continue; - if (MO.getReg() == W65816::Y) YLive = true; - else if (MO.getReg() == W65816::X) XLive = true; - } + YLive = MBBI->readsRegister(W65816::Y, STI.getRegisterInfo()); + XLive = MBBI->readsRegister(W65816::X, STI.getRegisterInfo()); } // VLA cleanup: restore entry SP from DP $F4 (saved in prologue). @@ -376,38 +459,16 @@ void W65816FrameLowering::emitEpilogue(MachineFunction &MF, // Prologue is always 16-bit M now (see emitPrologue). No 8-bit // epilogue branch needed. - // 16-bit M epilogue. Mirror the prologue: A holds the return value - // at this point and MUST be preserved. Small frames release via - // N/2 PLY (pop into Y, discard); larger frames use - // TAY/TSC/CLC/ADC #N/TCS/TYA. - // Mirror the prologue threshold (see comment there). - if (StackSize <= 6 && (StackSize % 2) == 0 && !YLive) { - // PLY clobbers Y, which is fine when Y isn't a return reg. - for (uint64_t i = 0; i < StackSize / 2; ++i) - BuildMI(MBB, MBBI, DL, TII.get(W65816::PLY)); - return; - } - if (YLive) { - // Y is a return register (i64 / double). Save A via DP $E0 - // instead of TAY so Y survives. 4 cyc slower than TAY/TYA but - // correct. X is allowed to be live too — none of these touch X. - BuildMI(MBB, MBBI, DL, TII.get(W65816::STA_DP)).addImm(0xE0); - BuildMI(MBB, MBBI, DL, TII.get(W65816::TSC)); - BuildMI(MBB, MBBI, DL, TII.get(W65816::CLC)); - BuildMI(MBB, MBBI, DL, TII.get(W65816::ADC_Imm16)) - .addImm(StackSize); - BuildMI(MBB, MBBI, DL, TII.get(W65816::TCS)); - BuildMI(MBB, MBBI, DL, TII.get(W65816::LDA_DP)).addImm(0xE0); - (void)XLive; - return; - } - BuildMI(MBB, MBBI, DL, TII.get(W65816::TAY)); - BuildMI(MBB, MBBI, DL, TII.get(W65816::TSC)); - BuildMI(MBB, MBBI, DL, TII.get(W65816::CLC)); - BuildMI(MBB, MBBI, DL, TII.get(W65816::ADC_Imm16)) - .addImm(StackSize); - BuildMI(MBB, MBBI, DL, TII.get(W65816::TCS)); - BuildMI(MBB, MBBI, DL, TII.get(W65816::TYA)); + // 16-bit M epilogue. The return value in A (and X / Y for wider + // ones) must survive; a void function's A is free, and then a bare + // TSC/CLC/ADC/TCS (9 cyc) beats both N/2 PLY (5 cyc each) past two of + // them and the TAY/TYA-bracketed form (13 cyc), which in turn beats + // PLYs past three. An i64 / double return keeps Y too, so A goes + // through DP $E0 there. + bool KeepA = MBBI == MBB.end() || !MBBI->isReturn() || + MBBI->readsRegister(W65816::A, STI.getRegisterInfo()); + W65816Helpers::emitStackRelease(MBB, MBBI, DL, TII, StackSize, KeepA, + XLive, YLive); } MachineBasicBlock::iterator W65816FrameLowering::eliminateCallFramePseudoInstr( @@ -520,9 +581,16 @@ MachineBasicBlock::iterator W65816FrameLowering::eliminateCallFramePseudoInstr( reestablishFP(); BuildMI(MBB, I, DL, TII.get(W65816::LDA_DP)).addImm(0xE0); (void)XLive; - } else if (N > 0 && N <= 14 && (N % 2) == 0 && !usesDpFP) { - for (int i = 0; i < N / 2; ++i) - BuildMI(MBB, I, DL, TII.get(W65816::PLY)); + } else if (!usesDpFP) { + // Y is free (no i64 return). When the callee's A result is not + // read (a void call, or one whose value is unused), A is free too + // and TSC/CLC/ADC/TCS releases any size in 9 cycles; otherwise + // N/2 PLY wins up to three of them over TAY/.../TYA. X can carry + // an i32 result's high half, so it is kept. + W65816Helpers::emitStackRelease( + MBB, I, DL, TII, N, + W65816Helpers::aLiveFrom(MBB, std::next(I)), + /*KeepX=*/true, /*KeepY=*/false); } else { BuildMI(MBB, I, DL, TII.get(W65816::TAY)); BuildMI(MBB, I, DL, TII.get(W65816::TSC)); diff --git a/src/llvm/lib/Target/W65816/W65816FrameLowering.h b/src/llvm/lib/Target/W65816/W65816FrameLowering.h index 4326a0e..f5691e4 100644 --- a/src/llvm/lib/Target/W65816/W65816FrameLowering.h +++ b/src/llvm/lib/Target/W65816/W65816FrameLowering.h @@ -35,6 +35,9 @@ public: RegScavenger *RS) const override; bool hasReservedCallFrame(const MachineFunction &MF) const override; + + void orderFrameObjects(const MachineFunction &MF, + SmallVectorImpl &ObjectsToAllocate) const override; }; } // namespace llvm diff --git a/src/llvm/lib/Target/W65816/W65816ISelLowering.cpp b/src/llvm/lib/Target/W65816/W65816ISelLowering.cpp index 5711dd6..af05fc6 100644 --- a/src/llvm/lib/Target/W65816/W65816ISelLowering.cpp +++ b/src/llvm/lib/Target/W65816/W65816ISelLowering.cpp @@ -26,6 +26,9 @@ #include "llvm/CodeGen/MachineRegisterInfo.h" #include "llvm/CodeGen/SelectionDAG.h" #include "llvm/CodeGen/TargetLoweringObjectFileImpl.h" +#include "llvm/ADT/StringSwitch.h" +#include "llvm/Analysis/ValueTracking.h" +#include "llvm/IR/GlobalVariable.h" #include "llvm/Support/KnownBits.h" #include "llvm/IR/DiagnosticInfo.h" #include "llvm/IR/Function.h" @@ -443,6 +446,13 @@ W65816TargetLowering::W65816TargetLowering(const TargetMachine &TM, setOperationAction(ISD::SELECT_CC, MVT::i32, Custom); setOperationAction(ISD::SELECT, MVT::i32, Custom); setOperationAction(ISD::Constant, MVT::i32, Custom); + // An i64 compare is one CMP/SBC chain over its four words (lowerI32Cmp) + // instead of the generic expansion's nested per-half selects. The type + // legalizer looks these up by the compare operands' type; a SELECT_CC + // with an i64 result also asks ReplaceNodeResults, which declines. + setOperationAction(ISD::SETCC, MVT::i64, Custom); + setOperationAction(ISD::BR_CC, MVT::i64, Custom); + setOperationAction(ISD::SELECT_CC, MVT::i64, Custom); } // Disable jump tables. Generating them costs us BRIND (indirect @@ -476,6 +486,9 @@ W65816TargetLowering::W65816TargetLowering(const TargetMachine &TM, // See PerformDAGCombine. setTargetDAGCombine(ISD::STORE); setTargetDAGCombine(ISD::LOAD); + // Float / double compares the operand's bits decide become integer + // compares instead of soft-float libcalls (see combineFPSetCCConst). + setTargetDAGCombine(ISD::SETCC); } // Map an LLVM SETCC condition to a W65816 branch. Returns the condition @@ -680,6 +693,20 @@ static SDValue lookThroughRegSeq(SDValue X, unsigned WantSub) { } return SDValue(); } +// The i16 low half of an i32 extension LowerExtend has not lowered yet +// (its operand, widened from i8 / i1 the way LowerExtend widens it), or +// null for anything else. +static SDValue extendSource(SelectionDAG &DAG, const SDLoc &DL, SDValue X) { + unsigned Opc = X.getOpcode(); + if (X.getValueType() != MVT::i32 || + (Opc != ISD::ZERO_EXTEND && Opc != ISD::SIGN_EXTEND && + Opc != ISD::ANY_EXTEND)) + return SDValue(); + SDValue In = X.getOperand(0); + if (In.getValueType() == MVT::i8 || In.getValueType() == MVT::i1) + return DAG.getNode(Opc, DL, MVT::i16, In); + return In.getValueType() == MVT::i16 ? In : SDValue(); +} static SDValue extractWide32Lo(SelectionDAG &DAG, const SDLoc &DL, SDValue X) { // For constants, materialise the lo half as an i16 constant directly // — getTargetExtractSubreg on a Constant SDNode produces a malformed @@ -721,6 +748,18 @@ static SDValue extractWide32Hi(SelectionDAG &DAG, const SDLoc &DL, SDValue X) { } if (SDValue Half = lookThroughRegSeq(X, llvm::sub_hi)) return Half; + // The high half LowerExtend gives an extension it has not reached yet: + // a known zero (an i32 shift of a zero-extended index computed it with + // a constant-zero operand the DAG could no longer fold), the sign fill, + // or undef. + if (SDValue In = extendSource(DAG, DL, X)) { + if (X.getOpcode() == ISD::ZERO_EXTEND) + return DAG.getConstant(0, DL, MVT::i16); + if (X.getOpcode() == ISD::SIGN_EXTEND) + return DAG.getNode(ISD::SRA, DL, MVT::i16, In, + DAG.getConstant(15, DL, MVT::i16)); + return DAG.getUNDEF(MVT::i16); + } return DAG.getTargetExtractSubreg(llvm::sub_hi, DL, MVT::i16, X); } @@ -732,8 +771,8 @@ static SDValue extractWide32Hi(SelectionDAG &DAG, const SDLoc &DL, SDValue X) { // The bank-byte carry-in is intentionally dropped: the `[dp],Y` deref // adds Y to the 24-bit pointer without propagating beyond 16 bits. // Caller's responsibility that the target object doesn't span a bank. -static bool peelPtr32Offset(SelectionDAG &DAG, SDLoc DL, SDValue Ptr, - SDValue &OutBase, uint16_t &OutOff) { +static bool peelPtr32OffsetOnce(SelectionDAG &DAG, SDLoc DL, SDValue Ptr, + SDValue &OutBase, uint16_t &OutOff) { if (Ptr.getValueType() != MVT::i32) return false; // Pre-LowerI32Bin shape: `ISD::ADD(BaseWide32, i32 const)`. LowerLoad // runs before LowerI32Bin in legalization order, so the ADD is still @@ -741,13 +780,29 @@ static bool peelPtr32Offset(SelectionDAG &DAG, SDLoc DL, SDValue Ptr, if (Ptr.getOpcode() == ISD::ADD) { SDValue L = Ptr.getOperand(0); SDValue R = Ptr.getOperand(1); - auto *KC = dyn_cast(R); - if (!KC) { - KC = dyn_cast(L); - if (!KC) return false; + // The constant may already be split into a REG_SEQUENCE of its + // halves (LowerI32Constant runs on it first when the ADD's other + // operand was lowered earlier). + auto constOf = [](SDValue V, uint64_t &Out) { + if (auto *C = dyn_cast(V)) { + Out = C->getZExtValue(); + return true; + } + auto *CL = dyn_cast_or_null( + lookThroughRegSeq(V, llvm::sub_lo).getNode()); + auto *CH = dyn_cast_or_null( + lookThroughRegSeq(V, llvm::sub_hi).getNode()); + if (!CL || !CH) + return false; + Out = (CL->getZExtValue() & 0xFFFFu) | + ((CH->getZExtValue() & 0xFFFFu) << 16); + return true; + }; + uint64_t K = 0; + if (!constOf(R, K)) { + if (!constOf(L, K)) return false; L = R; } - uint64_t K = KC->getZExtValue(); if (K == 0 || K > 0xFFFFu) return false; OutOff = (uint16_t)K; OutBase = L; @@ -774,6 +829,143 @@ static bool peelPtr32Offset(SelectionDAG &DAG, SDLoc DL, SDValue Ptr, return true; } +// Peel every constant offset off Ptr, not just the outermost: an i32 +// load or store at `p + K` is split into halves at `p + K` and +// `(p + K) + 2`, and the inner add has two users, so the combiner does +// not reassociate the second. Peeling one level left the high half +// addressed through the computed `p + K` -- an i32 add and a staging of +// its result into $E0 / $E2 per access -- instead of `[p],Y` with +// Y = K + 2. Offsets add up while they fit Y. +static bool peelPtr32Offset(SelectionDAG &DAG, SDLoc DL, SDValue Ptr, + SDValue &OutBase, uint16_t &OutOff) { + SDValue Base; + uint16_t Off = 0; + if (!peelPtr32OffsetOnce(DAG, DL, Ptr, Base, Off)) + return false; + uint32_t Total = Off; + SDValue Inner; + uint16_t InnerOff = 0; + while (peelPtr32OffsetOnce(DAG, DL, Base, Inner, InnerOff) && + Total + InnerOff <= 0xFFFFu) { + Total += InnerOff; + Base = Inner; + } + OutBase = Base; + OutOff = (uint16_t)Total; + return true; +} + +// `P = Base + zext(V)` for a 16-bit V, in each shape it reaches LowerLoad +// in: (add Base, (zext V)) -- or sext of a value whose sign bit is known +// clear --, (add Base, REG_SEQUENCE(V, 0)) once LowerExtend has run, +// (add Base, X) for any X whose high word is known zero, and LowerI32Bin's +// REG_SEQUENCE(ADDC(BaseLo, V), ADDE(BaseHi, 0)). +static bool matchZextIndexAdd(SelectionDAG &DAG, const SDLoc &DL, SDValue P, + SDValue &Base, SDValue &V) { + if (P.getValueType() != MVT::i32) + return false; + auto zextOf = [&](SDValue X, SDValue &Out) { + if (X.getOpcode() == ISD::ZERO_EXTEND || + (X.getOpcode() == ISD::SIGN_EXTEND && + DAG.SignBitIsZero(X.getOperand(0)))) { + SDValue In = X.getOperand(0); + if (In.getValueType() == MVT::i16) { + Out = In; + return true; + } + if (In.getValueType() == MVT::i8 || In.getValueType() == MVT::i1) { + Out = DAG.getNode(ISD::ZERO_EXTEND, DL, MVT::i16, In); + return true; + } + return false; + } + if (X.isMachineOpcode() && + X.getMachineOpcode() == TargetOpcode::REG_SEQUENCE) { + SDValue Lo = lookThroughRegSeq(X, llvm::sub_lo); + SDValue Hi = lookThroughRegSeq(X, llvm::sub_hi); + auto *HiC = dyn_cast_or_null(Hi.getNode()); + if (Lo && HiC && HiC->isZero() && !isa(Lo)) { + Out = Lo; + return true; + } + return false; + } + // Anything else whose high word is known zero: a zero-extending load + // (an argument's `zeroext` byte read off the stack), a mask, ... + if (!isa(X) && + DAG.computeKnownBits(X).countMinLeadingZeros() >= 16) { + Out = DAG.getNode(ISD::TRUNCATE, DL, MVT::i16, X); + return true; + } + return false; + }; + if (P.getOpcode() == ISD::ADD) { + for (unsigned I = 0; I < 2; ++I) { + if (zextOf(P.getOperand(1 - I), V)) { + Base = P.getOperand(I); + return true; + } + } + return false; + } + if (!P.isMachineOpcode() || + P.getMachineOpcode() != TargetOpcode::REG_SEQUENCE) + return false; + SDValue Lo = lookThroughRegSeq(P, llvm::sub_lo); + SDValue Hi = lookThroughRegSeq(P, llvm::sub_hi); + if (!Lo || !Hi || Lo.getOpcode() != ISD::ADDC || + Hi.getOpcode() != ISD::ADDE || Hi.getOperand(2) != Lo.getValue(1)) + return false; + for (unsigned I = 0; I < 2; ++I) { + auto *HiK = dyn_cast(Hi.getOperand(1 - I)); + if (HiK && HiK->isZero() && + !isa(Lo.getOperand(1 - I))) { + V = Lo.getOperand(1 - I); + Base = buildWide32(DAG, DL, Lo.getOperand(I), Hi.getOperand(I)); + return true; + } + } + return false; +} + +// `Ptr = Base + zext(Idx) + Off` with Idx + Off provably below 0x10000: +// the deref puts Idx + Off in Y (`[dp],Y` adds Y to the whole 24-bit +// pointer, as the i32 add carried into the bank byte), so the address +// costs no i32 add and no staging of a computed pointer. `p[i]` with an +// unsigned 8-bit index, or any 16-bit one at offset 0, takes this form. +static bool peelPtr32Index(SelectionDAG &DAG, const SDLoc &DL, SDValue Ptr, + SDValue &OutBase, SDValue &OutIdx, + uint16_t &OutOff) { + uint32_t K = 0; + SDValue P = Ptr; + SDValue B; + uint16_t O = 0; + if (peelPtr32Offset(DAG, DL, P, B, O)) { + P = B; + K = O; + } + SDValue Base; + SDValue V; + // An address several accesses share (`row[0] .. row[15]` off one + // `row = base + (i << 5)`) is better built once: each Idx access pays + // `lda i ; clc ; adc #k ; tay` against one i32 add and an `ldy #k`. + // Two uses are the halves of one i32 access. + if (P.getNode()->use_size() > 2 || + !matchZextIndexAdd(DAG, DL, P, Base, V)) + return false; + if (peelPtr32Offset(DAG, DL, Base, B, O) && K + O <= 0xFFFFu) { + Base = B; + K += O; + } + KnownBits KB = DAG.computeKnownBits(V); + if (KB.getMaxValue().getZExtValue() + K > 0xFFFFu) + return false; + OutBase = Base; + OutIdx = V; + OutOff = (uint16_t)K; + return true; +} + // The integer predicates lowerI32Cmp handles (every one C can produce). static bool isI32CmpCC(ISD::CondCode CC) { switch (CC) { @@ -786,10 +978,24 @@ static bool isI32CmpCC(ISD::CondCode CC) { } } -// An i32 compare reduced to something one flag test can consume. Either an -// equivalent i16 compare (L16 CC16 R16; Glue is null), which the caller feeds -// back through the i16 BR_CC / SETCC / SELECT_CC paths, or the flags of a -// CMP/SBC chain (Glue) that the caller tests with TCC. +// An i64 == / != and a sign test (x < 0, x > -1) are left to the type +// legalizer, whose expansions -- (lo ^ lo') | (hi ^ hi') == 0, or the high +// half alone -- work on the i32 halves and fold constant halves there. The +// other relational predicates are lowered here: the generic expansion nests +// per-half compares in selects. +static bool isI64EqNe(SDValue LHS, SDValue RHS, ISD::CondCode CC) { + if (LHS.getValueType() != MVT::i64) + return false; + if (CC == ISD::SETEQ || CC == ISD::SETNE) + return true; + return (CC == ISD::SETLT && isNullConstant(RHS)) || + (CC == ISD::SETGT && isAllOnesConstant(RHS)); +} + +// An i32 or i64 compare reduced to something one flag test can consume. +// Either an equivalent i16 compare (L16 CC16 R16; Glue is null), which the +// caller feeds back through the i16 BR_CC / SETCC / SELECT_CC paths, or the +// flags of a CMP/SBC chain (Glue) that the caller tests with TCC. struct I32CmpLowering { SDValue L16; SDValue R16; @@ -798,28 +1004,88 @@ struct I32CmpLowering { W65816CC::CondCode TCC = W65816CC::COND_INVALID; }; -// Lower an i32 compare without materialising per-half booleans. The old -// lowering built (hiLT | (hiEQ & loLT)) out of three i16 setcc diamonds plus -// an AND and an OR, then branched on the result: ~40 instructions and three -// stack temporaries for one `i < n`, the dominant cost of every 32-bit loop. +// The four i16 words of the i64 value V, least significant first, appended +// to Words. An i64 compare is lowered while the type legalizer is still +// splitting i64 values into i32 halves: EXTRACT_ELEMENT names a half, which +// the legalizer resolves once V is expanded, and the sub-register extracts +// of it are machine nodes nothing folds. So constants, AND / OR / XOR with +// a constant and zero extensions are taken apart here instead, where the +// words they make constant still fold: a mask test such as softDouble's +// `(x & ~sign) > inf` then compares x's words, not materialised all-ones +// and zero words. +static void i64Words(SelectionDAG &DAG, const SDLoc &DL, SDValue V, + SmallVectorImpl &Words, unsigned Depth = 0) { + auto word = [&](uint64_t K, unsigned I) { + return DAG.getConstant((K >> (16 * I)) & 0xFFFFu, DL, MVT::i16); + }; + if (auto *C = dyn_cast(V)) { + for (unsigned I = 0; I != 4; ++I) + Words.push_back(word(C->getZExtValue(), I)); + return; + } + unsigned Opc = V.getOpcode(); + if (Depth < 4 && (Opc == ISD::AND || Opc == ISD::OR || Opc == ISD::XOR) && + isa(V.getOperand(1))) { + SmallVector X; + i64Words(DAG, DL, V.getOperand(0), X, Depth + 1); + uint64_t K = V.getConstantOperandVal(1); + for (unsigned I = 0; I != 4; ++I) + Words.push_back(DAG.getNode(Opc, DL, MVT::i16, X[I], word(K, I))); + return; + } + if (Opc == ISD::ZERO_EXTEND && V.getOperand(0).getValueType() == MVT::i32) { + SDValue X = V.getOperand(0); + SDValue Zero = DAG.getConstant(0, DL, MVT::i16); + Words.append({extractWide32Lo(DAG, DL, X), extractWide32Hi(DAG, DL, X), + Zero, Zero}); + return; + } + for (unsigned H = 0; H != 2; ++H) { + SDValue Half = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i32, V, + DAG.getIntPtrConstant(H, DL)); + Words.push_back(extractWide32Lo(DAG, DL, Half)); + Words.push_back(extractWide32Hi(DAG, DL, Half)); + } +} + +// The i16 words of an i32 or i64 compare operand, least significant first. +static SmallVector cmpWords(SelectionDAG &DAG, const SDLoc &DL, + SDValue V) { + SmallVector Words; + if (V.getValueType() == MVT::i64) { + i64Words(DAG, DL, V, Words); + } else { + Words.push_back(extractWide32Lo(DAG, DL, V)); + Words.push_back(extractWide32Hi(DAG, DL, V)); + } + return Words; +} + +// Lower an i32 or i64 compare without materialising per-word booleans. The +// old i32 lowering built (hiLT | (hiEQ & loLT)) out of three i16 setcc +// diamonds plus an AND and an OR, then branched on the result: ~40 +// instructions and three stack temporaries for one `i < n`, the dominant +// cost of every 32-bit loop. The generic i64 expansion nested that twice. // // Ordered predicates use the classic 65xx multi-precision compare: -// lda lhs.lo ; cmp rhs.lo ; lda lhs.hi ; sbc rhs.hi -// The CMP leaves C = no-borrow of the low halves, the SBC folds that borrow -// into the high halves, so C ends up as the unsigned 32-bit (lhs >= rhs). -// No branch, no temporary, one BCC/BCS. The SBC's result is dead; only its -// C matters, and neither LDA nor any other N/Z-only op between the two (the -// reload of lhs.hi) touches C. Signed predicates bias both high halves by -// 0x8000 first, which turns signed order into unsigned order (the same trick -// normalizeCC uses for i16); against a constant the bias folds away. +// lda lhs.w0 ; cmp rhs.w0 ; lda lhs.w1 ; sbc rhs.w1 [; ... w2, w3] +// The CMP leaves C = no-borrow of the low words, each SBC folds the borrow +// into the next words, so C ends up as the unsigned (lhs >= rhs). No +// branch, no temporary, one BCC/BCS. The SBCs' results are dead; only C +// matters, and neither LDA nor any other N/Z-only op between them (the +// reload of the next word) touches C. Signed predicates bias both top +// words by 0x8000 first, which turns signed order into unsigned order (the +// same trick normalizeCC uses for i16); against a constant the bias folds +// away. // -// EQ/NE reduce to one i16 test: (lhs.lo ^ rhs.lo) | (lhs.hi ^ rhs.hi) is -// zero exactly when the halves match, and an XOR with a constant 0 half -// folds, so `x == 5` is just `lda lo ; eor #5 ; ora hi ; beq`. +// EQ/NE reduce to one i16 test: the OR of every (lhs.wN ^ rhs.wN) is zero +// exactly when the words match, and an XOR with a constant 0 word folds, so +// `x == 5` is just `lda lo ; eor #5 ; ora hi ; beq`. static I32CmpLowering lowerI32Cmp(SelectionDAG &DAG, const SDLoc &DL, SDValue LHS, SDValue RHS, ISD::CondCode CC) { I32CmpLowering R; + EVT VT = LHS.getValueType(); // CMP / SBC want the comparand (constant or memory) on the right. if (isa(LHS) && !isa(RHS)) { std::swap(LHS, RHS); @@ -827,21 +1093,20 @@ static I32CmpLowering lowerI32Cmp(SelectionDAG &DAG, const SDLoc &DL, } if (CC == ISD::SETEQ || CC == ISD::SETNE) { - SDValue Zero = DAG.getConstant(0, DL, MVT::i16); - SDValue LL = extractWide32Lo(DAG, DL, LHS); - SDValue LH = extractWide32Hi(DAG, DL, LHS); + SmallVector L = cmpWords(DAG, DL, LHS); // x == 0 / x != 0 needs no XOR at all: (lo | hi) == 0. Hot in // `while (x)` and in every count-down loop. - if (isNullConstant(RHS)) { - R.L16 = DAG.getNode(ISD::OR, DL, MVT::i16, LL, LH); - } else { - SDValue RL = extractWide32Lo(DAG, DL, RHS); - SDValue RH = extractWide32Hi(DAG, DL, RHS); - SDValue XLo = DAG.getNode(ISD::XOR, DL, MVT::i16, LL, RL); - SDValue XHi = DAG.getNode(ISD::XOR, DL, MVT::i16, LH, RH); - R.L16 = DAG.getNode(ISD::OR, DL, MVT::i16, XLo, XHi); + bool Zero = isNullConstant(RHS); + SmallVector Rw; + if (!Zero) + Rw = cmpWords(DAG, DL, RHS); + SDValue Acc; + for (unsigned K = 0; K != L.size(); ++K) { + SDValue W = Zero ? L[K] : DAG.getNode(ISD::XOR, DL, MVT::i16, L[K], Rw[K]); + Acc = Acc ? DAG.getNode(ISD::OR, DL, MVT::i16, Acc, W) : W; } - R.R16 = Zero; + R.L16 = Acc; + R.R16 = DAG.getConstant(0, DL, MVT::i16); R.CC16 = CC; return R; } @@ -850,9 +1115,9 @@ static I32CmpLowering lowerI32Cmp(SelectionDAG &DAG, const SDLoc &DL, // (x <= K is x < K+1), unless K+1 would wrap. Then only the four // predicates the CMP/SBC chain answers directly remain. if (auto *KC = dyn_cast(RHS)) { - uint32_t K = (uint32_t)KC->getZExtValue(); - bool UMax = K == 0xFFFFFFFFu; - bool SMax = K == 0x7FFFFFFFu; + const APInt &K = KC->getAPIntValue(); + bool UMax = K.isMaxValue(); + bool SMax = K.isMaxSignedValue(); ISD::CondCode Strict = ISD::SETCC_INVALID; if (CC == ISD::SETULE && !UMax) Strict = ISD::SETULT; @@ -864,7 +1129,7 @@ static I32CmpLowering lowerI32Cmp(SelectionDAG &DAG, const SDLoc &DL, Strict = ISD::SETGE; if (Strict != ISD::SETCC_INVALID) { CC = Strict; - RHS = DAG.getConstant((uint64_t)(K + 1u), DL, MVT::i32); + RHS = DAG.getConstant(K + 1, DL, VT); } } // x <= y is y >= x, x > y is y < x: swap rather than add a second branch. @@ -877,17 +1142,18 @@ static I32CmpLowering lowerI32Cmp(SelectionDAG &DAG, const SDLoc &DL, break; } - SDValue LL = extractWide32Lo(DAG, DL, LHS); - SDValue LH = extractWide32Hi(DAG, DL, LHS); - SDValue RL = extractWide32Lo(DAG, DL, RHS); - SDValue RH = extractWide32Hi(DAG, DL, RHS); - // A constant whose low half is 0 makes the low CMP always leave C = 1, - // so the SBC is just a CMP of the high halves: the whole compare is the - // i16 compare of the high halves with the same predicate (x < 0x30000 is - // hi < 3, and signed x < 0 is hi < 0). - if (isNullConstant(RL)) { - R.L16 = LH; - R.R16 = RH; + SmallVector L = cmpWords(DAG, DL, LHS); + SmallVector Rw = cmpWords(DAG, DL, RHS); + // A constant low word of 0 makes its CMP always leave C = 1, so the next + // SBC is just a CMP: drop such words. When only the top word is left the + // whole compare is the i16 compare of the top words with the same + // predicate (x < 0x30000 is hi < 3, and signed x < 0 is hi < 0). + unsigned First = 0; + while (First + 1 != L.size() && isNullConstant(Rw[First])) + ++First; + if (First + 1 == L.size()) { + R.L16 = L.back(); + R.R16 = Rw.back(); R.CC16 = CC; return R; } @@ -895,14 +1161,15 @@ static I32CmpLowering lowerI32Cmp(SelectionDAG &DAG, const SDLoc &DL, bool Signed = (CC == ISD::SETLT || CC == ISD::SETGE); if (Signed) { SDValue Bias = DAG.getConstant(0x8000, DL, MVT::i16); - LH = DAG.getNode(ISD::XOR, DL, MVT::i16, LH, Bias); - RH = DAG.getNode(ISD::XOR, DL, MVT::i16, RH, Bias); + L.back() = DAG.getNode(ISD::XOR, DL, MVT::i16, L.back(), Bias); + Rw.back() = DAG.getNode(ISD::XOR, DL, MVT::i16, Rw.back(), Bias); } - SDValue LoFlags = DAG.getNode(W65816ISD::CMP, DL, MVT::Glue, LL, RL); - SDValue HiSub = DAG.getNode(ISD::SUBE, DL, - DAG.getVTList(MVT::i16, MVT::Glue), LH, RH, - LoFlags); - R.Glue = HiSub.getValue(1); + SDValue Flags = DAG.getNode(W65816ISD::CMP, DL, MVT::Glue, L[First], + Rw[First]); + for (unsigned K = First + 1; K != L.size(); ++K) + Flags = DAG.getNode(ISD::SUBE, DL, DAG.getVTList(MVT::i16, MVT::Glue), + L[K], Rw[K], Flags).getValue(1); + R.Glue = Flags; R.TCC = (CC == ISD::SETULT || CC == ISD::SETLT) ? W65816CC::COND_LO : W65816CC::COND_HS; return R; @@ -916,9 +1183,11 @@ SDValue W65816TargetLowering::LowerBR_CC(SDValue Op, SelectionDAG &DAG) const { SDValue Dest = Op.getOperand(4); SDLoc DL(Op); EVT VT = LHS.getValueType(); - // i32 BR_CC: a CMP/SBC chain feeding one Bxx, or an equivalent i16 + // i32 / i64 BR_CC: a CMP/SBC chain feeding one Bxx, or an equivalent i16 // compare that re-enters this function (see lowerI32Cmp). - if (VT == MVT::i32) { + if (isI64EqNe(LHS, RHS, CC)) + return SDValue(); + if (VT == MVT::i32 || VT == MVT::i64) { if (!isI32CmpCC(CC)) { reportUnsupported(DAG, DL, "unsupported i32 branch condition code"); return Chain; // safe placeholder; the diagnostic fails compilation @@ -1026,10 +1295,14 @@ SDValue W65816TargetLowering::LowerSETCC(SDValue Op, SelectionDAG &DAG) const { ISD::CondCode CC = cast(Op.getOperand(2))->get(); SDLoc DL(Op); EVT VT = Op.getValueType(); - // i32 SETCC: one select on the CMP/SBC chain's flags, or the equivalent - // i16 SETCC (see lowerI32Cmp). Result type is i16 (the legalizer keeps - // the boolean result type narrow regardless of LHS width). - if (LHS.getValueType() == MVT::i32) { + // i32 / i64 SETCC: one select on the CMP/SBC chain's flags, or the + // equivalent i16 SETCC (see lowerI32Cmp). Result type is i16 (the + // legalizer keeps the boolean result type narrow regardless of LHS + // width). i64 arrives here from the type legalizer (SETCC is Custom for + // i64), before its operands are split. + if (isI64EqNe(LHS, RHS, CC)) + return SDValue(); + if (LHS.getValueType() == MVT::i32 || LHS.getValueType() == MVT::i64) { if (!isI32CmpCC(CC)) { reportUnsupported(DAG, DL, "unsupported i32 comparison condition code"); return DAG.getUNDEF(VT); // safe placeholder @@ -1063,12 +1336,15 @@ SDValue W65816TargetLowering::LowerSELECT_CC(SDValue Op, ISD::CondCode CC = cast(Op.getOperand(4))->get(); SDLoc DL(Op); - // i32 SELECT_CC. An i8/i16 result selects straight on the CMP/SBC + // i32 / i64 SELECT_CC. An i8/i16 result selects straight on the CMP/SBC // chain's flags (or on the equivalent i16 compare). An i32 result needs // two per-half selects and a glue value has only one consumer, so it - // materialises the i16 boolean (LowerSETCC's i32 path) and selects both + // materialises the i16 boolean (LowerSETCC's wide path) and selects both // halves on that. Never creates an i32 W65816::CMP (no pattern for it). - if (LHS.getValueType() == MVT::i32) { + // An i64 result was split into i32 ones by the type legalizer first. + if (isI64EqNe(LHS, RHS, CC)) + return SDValue(); + if (LHS.getValueType() == MVT::i32 || LHS.getValueType() == MVT::i64) { if (Op.getValueType() != MVT::i32 && isI32CmpCC(CC)) { I32CmpLowering C = lowerI32Cmp(DAG, DL, LHS, RHS, CC); if (!C.Glue) @@ -1196,6 +1472,247 @@ static SDValue foldGlobalAddr(SelectionDAG &DAG, const SDLoc &DL, SDValue Ptr, return SDValue(); } +// The global (with its constant offset) a pointer operand is, in the +// shapes LowerLoad / LowerStore see it: a raw GlobalAddress, or the +// Wrapper / WrapperBank pair LowerGlobalAddress builds. +static const GlobalAddressSDNode *globalBaseOf(SDValue P) { + if (auto *GA = dyn_cast(P)) + return GA; + SDValue Lo = lookThroughRegSeq(P, llvm::sub_lo); + SDValue Hi = lookThroughRegSeq(P, llvm::sub_hi); + if (!Lo || !Hi || Lo.getOpcode() != W65816ISD::Wrapper || + Hi.getOpcode() != W65816ISD::WrapperBank) + return nullptr; + auto *GLo = dyn_cast(Lo.getOperand(0)); + auto *GHi = dyn_cast(Hi.getOperand(0)); + if (!GLo || !GHi || GLo->getGlobal() != GHi->getGlobal() || + GLo->getOffset() != GHi->getOffset()) + return nullptr; + return GLo; +} + +// X * K (VT i16 or i32) as shifts and adds when K's signed-digit (non- +// adjacent) form has at most MaxDigits digits -- `x * 200` is (x << 8) - +// (x << 6) + (x << 3), a few dozen cycles against a multiply libcall. +// Each shift continues from the previous term. Null when K takes more. +static SDValue mulConstShiftAdd(SelectionDAG &DAG, const SDLoc &DL, EVT VT, + SDValue X, uint64_t K, unsigned MaxDigits) { + unsigned Bits = VT.getSizeInBits(); + K &= Bits == 16 ? 0xFFFFull : 0xFFFFFFFFull; + if (K == 0) + return DAG.getConstant(0, DL, VT); + SmallVector, 8> Digits; + uint64_t R = K; + for (unsigned B = 0; R != 0 && B < Bits; ++B, R >>= 1) { + if ((R & 1) == 0) + continue; + int D = (R & 3) == 3 ? -1 : 1; + Digits.push_back({B, D}); + R = D > 0 ? R - 1 : R + 1; + } + if (Digits.size() > MaxDigits) + return SDValue(); + SDValue Term = X; + unsigned Prev = 0; + SDValue Sum; + SmallVector Neg; + for (auto [B, D] : Digits) { + if (B != Prev) + Term = DAG.getNode(ISD::SHL, DL, VT, Term, + DAG.getConstant(B - Prev, DL, MVT::i16)); + Prev = B; + if (D < 0) + Neg.push_back(Term); + else + Sum = Sum ? DAG.getNode(ISD::ADD, DL, VT, Sum, Term) : Term; + } + if (!Sum) + Sum = DAG.getConstant(0, DL, VT); + for (SDValue T : Neg) + Sum = DAG.getNode(ISD::SUB, DL, VT, Sum, T); + return Sum; +} + +// X * K in 16 bits: shifts and adds for up to three signed digits, else +// a plain MUL (the __mulhi3 call). +static SDValue mulConst16(SelectionDAG &DAG, const SDLoc &DL, SDValue X, + uint64_t K) { + if (SDValue R = mulConstShiftAdd(DAG, DL, MVT::i16, X, K, 3)) + return R; + return DAG.getNode(ISD::MUL, DL, MVT::i16, X, + DAG.getConstant(K & 0xFFFFu, DL, MVT::i16)); +} + +// The low word of an i32 value, computed in 16 bits where the low word +// depends on nothing above it: extensions, shifts left, adds, +// multiplies and logic distribute over the truncation. Anything else +// is a TRUNCATE of the i32 value. +static SDValue lowWord(SelectionDAG &DAG, const SDLoc &DL, SDValue X, + unsigned Depth = 0) { + if (X.getValueType() == MVT::i16) + return X; + if (auto *C = dyn_cast(X)) + return DAG.getConstant(C->getZExtValue() & 0xFFFFu, DL, MVT::i16); + unsigned Opc = X.getOpcode(); + if (Opc == ISD::ZERO_EXTEND || Opc == ISD::SIGN_EXTEND || + Opc == ISD::ANY_EXTEND) { + SDValue In = X.getOperand(0); + if (In.getValueType() == MVT::i16) + return In; + if (In.getValueType() == MVT::i8 || In.getValueType() == MVT::i1) + return DAG.getNode(Opc, DL, MVT::i16, In); + } + if (Depth < 4 && X.getValueType() == MVT::i32) { + // A right shift's low word needs the high word only when that is + // not known zero. + if (Opc == ISD::SHL || Opc == ISD::SRL || Opc == ISD::SRA) + if (auto *C = dyn_cast(X.getOperand(1))) + if (C->getZExtValue() < 16 && + (Opc == ISD::SHL || DAG.computeKnownBits(X.getOperand(0)) + .countMinLeadingZeros() >= 16)) + return DAG.getNode(Opc == ISD::SHL ? ISD::SHL : ISD::SRL, DL, + MVT::i16, + lowWord(DAG, DL, X.getOperand(0), Depth + 1), + DAG.getConstant(C->getZExtValue(), DL, MVT::i16)); + if (Opc == ISD::MUL) + for (unsigned I = 0; I < 2; ++I) + if (auto *C = dyn_cast(X.getOperand(I))) + return mulConst16(DAG, DL, + lowWord(DAG, DL, X.getOperand(1 - I), Depth + 1), + C->getZExtValue()); + if (Opc == ISD::ADD || Opc == ISD::SUB || Opc == ISD::MUL || + Opc == ISD::AND || Opc == ISD::OR || Opc == ISD::XOR) + return DAG.getNode(Opc, DL, MVT::i16, + lowWord(DAG, DL, X.getOperand(0), Depth + 1), + lowWord(DAG, DL, X.getOperand(1), Depth + 1)); + } + return DAG.getNode(ISD::TRUNCATE, DL, MVT::i16, X); +} + +// `Ptr = G + K + X` for a global variable G that the DBR-relative abs-16 +// forms reach (every global but a `.far` one lives in the bank crt0 pins +// to DBR, the bank `lda $BE` supplies for &G). The access becomes +// `lda G+K,x` with X = the low word of X: the CPU adds X to the 24-bit +// DBR:(G+K), the same sum the i32 pointer add makes, provided that X is +// exactly G's offset less K -- i.e. below 0x10000 and not negative. +// +// The IR supplies the bound: the access is based on G (its underlying +// object), so a valid program stays inside G's S bytes, S <= 0x10000. +// X then equals (address - G) - K: in [0, S) when K is 0, and in +// [0, S - K) when X is also known not negative. Otherwise K moves into +// the index (`lda G,x` with X + K, which is address - G). Sym is the +// i16 TargetGlobalAddress, Idx the i16 index. Replaces the far-pointer +// build (`lda #G ; ... lda $BE ; ... sta $E0/$E2 ; lda [$E0],y`, ~30 +// cycles) every `table[i]` used to pay. +// +// P = Base + X comes as an ISD::ADD, or once LowerI32Bin has run as +// REG_SEQUENCE(ADDC(BaseLo, XLo), ADDE(BaseHi, XHi, carry)). +static bool matchGlobalIndex(SelectionDAG &DAG, const SDLoc &DL, SDValue Ptr, + const MachineMemOperand *MMO, SDValue &Sym, + SDValue &Idx) { + static const bool Disabled = getenv("W65816_DISABLE_GLOBAL_INDEX"); + if (Disabled || Ptr.getValueType() != MVT::i32 || !MMO) + return false; + const Value *IRPtr = MMO->getValue(); + if (!IRPtr) + return false; + SDValue P = Ptr; + SDValue B; + uint16_t Off = 0; + int64_t K = 0; + if (peelPtr32Offset(DAG, DL, P, B, Off)) { + P = B; + K = Off; + } + // Candidate splits: {base, i32 index or null, index low word, high word}. + struct Split { + SDValue Base, X, XLo, XHi; + }; + SmallVector Splits; + if (P.getOpcode() == ISD::ADD) { + for (unsigned I = 0; I < 2; ++I) + Splits.push_back({P.getOperand(I), P.getOperand(1 - I), SDValue(), + SDValue()}); + } else { + SDValue Lo = lookThroughRegSeq(P, llvm::sub_lo); + SDValue Hi = lookThroughRegSeq(P, llvm::sub_hi); + if (!Lo || !Hi || Lo.getOpcode() != ISD::ADDC || + Hi.getOpcode() != ISD::ADDE || Hi.getOperand(2) != Lo.getValue(1)) + return false; + for (unsigned I = 0; I < 2; ++I) + Splits.push_back({buildWide32(DAG, DL, Lo.getOperand(I), + Hi.getOperand(I)), + SDValue(), Lo.getOperand(1 - I), + Hi.getOperand(1 - I)}); + } + for (const Split &Sp : Splits) { + SDValue BaseSide = Sp.Base; + int64_t KB = K; + if (peelPtr32Offset(DAG, DL, BaseSide, B, Off)) { + BaseSide = B; + KB += Off; + } + const GlobalAddressSDNode *GA = globalBaseOf(BaseSide); + if (!GA) + continue; + auto *GV = dyn_cast(GA->getGlobal()); + if (!GV || GV->isThreadLocal() || GV->getAddressSpace() != 0 || + (GV->hasSection() && GV->getSection().contains(".far")) || + !GV->getValueType()->isSized()) + return false; + uint64_t Size = DAG.getDataLayout().getTypeAllocSize(GV->getValueType()); + int64_t K0 = KB + GA->getOffset(); + if (Size == 0 || Size > 0x10000u || K0 < 0 || uint64_t(K0) >= Size) + return false; + if (getUnderlyingObject(IRPtr) != GV) + return false; + SDValue XLo = Sp.X ? lowWord(DAG, DL, Sp.X) : Sp.XLo; + // An extract of a pair built since is that pair's low half. + if (XLo.isMachineOpcode() && + XLo.getMachineOpcode() == TargetOpcode::EXTRACT_SUBREG && + XLo.getConstantOperandVal(1) == llvm::sub_lo) + if (SDValue Half = lookThroughRegSeq(XLo.getOperand(0), llvm::sub_lo)) + XLo = Half; + bool NonNeg = Sp.X ? DAG.computeKnownBits(Sp.X).isNonNegative() + : DAG.computeKnownBits(Sp.XHi).isNonNegative(); + if (K0 != 0 && !NonNeg) { + XLo = DAG.getNode(ISD::ADD, DL, MVT::i16, XLo, + DAG.getConstant(K0, DL, MVT::i16)); + K0 = 0; + } + Sym = DAG.getTargetGlobalAddress(GV, DL, MVT::i16, K0); + Idx = XLo; + return true; + } + return false; +} + +// `lda Sym,x` (Opc LDA_AbsX, or LDA8absX for a volatile byte) with +// X = Idx: the i16 value read out of A, its chain as value 1. +static SDValue emitGlobalIndexLoad(SelectionDAG &DAG, const SDLoc &DL, + SDValue Chain, SDValue Sym, SDValue Idx, + unsigned Opc) { + SDValue ToX = DAG.getCopyToReg(Chain, DL, W65816::X, Idx, SDValue()); + SDNode *Lda = DAG.getMachineNode(Opc, DL, + DAG.getVTList(MVT::Other, MVT::Glue), + {Sym, ToX, ToX.getValue(1)}); + return DAG.getCopyFromReg(SDValue(Lda, 0), DL, W65816::A, MVT::i16, + SDValue(Lda, 1)); +} + +// `sta Sym,x` (Opc STA_AbsX, or STA8absX for a byte) of the i16 Val with +// X = Idx; returns the chain. +static SDValue emitGlobalIndexStore(SelectionDAG &DAG, const SDLoc &DL, + SDValue Chain, SDValue Val, SDValue Sym, + SDValue Idx, unsigned Opc) { + SDValue ToX = DAG.getCopyToReg(Chain, DL, W65816::X, Idx, SDValue()); + SDValue ToA = DAG.getCopyToReg(ToX, DL, W65816::A, Val, ToX.getValue(1)); + SDNode *Sta = DAG.getMachineNode(Opc, DL, + DAG.getVTList(MVT::Other, MVT::Glue), + {Sym, ToA, ToA.getValue(1)}); + return SDValue(Sta, 0); +} + // ptr32 foundation hook. In ptr16 mode (PointerWidth=16, current // default) addresses are i16 and we return SDValue() so the legalizer // keeps the load and the existing LDAptr / STAptr selection patterns @@ -1250,6 +1767,23 @@ SDValue W65816TargetLowering::LowerLoad(SDValue Op, SDValue Wide = buildWide32(DAG, DL, Lo, Hi); return DAG.getMergeValues({Wide, ByteLd.getValue(1)}, DL); } + // An element of a global array: `lda G+K,x` and `lda G+K+2,x`, decided + // here while the address is still one ADD (the halves are lowered + // after it). The four bytes lie inside G, so G+K+2 does too. + SDValue GSym, GIdx; + if (MemVT == MVT::i32 && + matchGlobalIndex(DAG, DL, Ptr, Ld->getMemOperand(), GSym, GIdx)) { + auto *GA = cast(GSym); + SDValue GSymHi = DAG.getTargetGlobalAddress(GA->getGlobal(), DL, + MVT::i16, + GA->getOffset() + 2); + SDValue Lo = emitGlobalIndexLoad(DAG, DL, Chain, GSym, GIdx, + W65816::LDA_AbsX); + SDValue Hi = emitGlobalIndexLoad(DAG, DL, Lo.getValue(1), GSymHi, GIdx, + W65816::LDA_AbsX); + return DAG.getMergeValues({buildWide32(DAG, DL, Lo, Hi), + Hi.getValue(1)}, DL); + } EVT PtrVT = Ptr.getValueType(); SDValue Two = DAG.getConstant(2, DL, PtrVT); SDValue Ptr2 = DAG.getNode(ISD::ADD, DL, PtrVT, Ptr, Two); @@ -1308,6 +1842,38 @@ SDValue W65816TargetLowering::LowerLoad(SDValue Op, MemVT, Ld->getMemOperand()); } + // A global indexed by a variable: `lda G+K,x` (see matchGlobalIndex). + // A byte reads the whole word and masks, as the [dp],Y path does; only + // a volatile byte keeps its 8-bit access (LDA8absX). + SDValue GSym, GIdx; + if (matchGlobalIndex(DAG, DL, Ptr, Ld->getMemOperand(), GSym, GIdx)) { + EVT MemVT = Ld->getMemoryVT(); + bool Byte = MemVT == MVT::i8 || MemVT == MVT::i1; + SDValue Val = emitGlobalIndexLoad( + DAG, DL, Chain, GSym, GIdx, + Byte && Ld->isVolatile() ? W65816::LDA8absX : W65816::LDA_AbsX); + SDValue OutChain = Val.getValue(1); + if (Byte) { + ISD::LoadExtType ExtType = Ld->getExtensionType(); + if (MemVT == MVT::i1 && ExtType == ISD::SEXTLOAD) { + SDValue Bit = DAG.getNode(ISD::AND, DL, MVT::i16, Val, + DAG.getConstant(1, DL, MVT::i16)); + Val = DAG.getNode(ISD::SUB, DL, MVT::i16, + DAG.getConstant(0, DL, MVT::i16), Bit); + } else if (MemVT == MVT::i1 || ExtType == ISD::ZEXTLOAD) { + Val = DAG.getNode(ISD::AND, DL, MVT::i16, Val, + DAG.getConstant(MemVT == MVT::i1 ? 1 : 0xFF, DL, + MVT::i16)); + } else if (ExtType == ISD::SEXTLOAD) { + Val = DAG.getNode(ISD::SIGN_EXTEND_INREG, DL, MVT::i16, Val, + DAG.getValueType(MVT::i8)); + } + } + if (VT == MVT::i8) + Val = DAG.getNode(ISD::TRUNCATE, DL, MVT::i8, Val); + return DAG.getMergeValues({Val, OutChain}, DL); + } + // ptr16 mode: address is i16, let the default selection handle it. if (Ptr.getValueType() != MVT::i32) return SDValue(); @@ -1325,7 +1891,13 @@ SDValue W65816TargetLowering::LowerLoad(SDValue Op, // See feedback_ptr32_deref_fold_layer1_mi.md. SDValue LdNode; SDValue Base; uint16_t Off = 0; - if (peelPtr32Offset(DAG, DL, Ptr, Base, Off)) { + SDValue Idx; + if (peelPtr32Index(DAG, DL, Ptr, Base, Idx, Off)) { + SDValue OffN = DAG.getTargetConstant(Off, DL, MVT::i16); + SDValue OpsIdx[] = { Chain, Base, Idx, OffN }; + LdNode = DAG.getMemIntrinsicNode(W65816ISD::LD_PTR_IDX, DL, VTs, OpsIdx, + MemVT, Ld->getMemOperand()); + } else if (peelPtr32Offset(DAG, DL, Ptr, Base, Off)) { SDValue OffN = DAG.getTargetConstant(Off, DL, MVT::i16); SDValue OpsOff[] = { Chain, Base, OffN }; LdNode = DAG.getMemIntrinsicNode(W65816ISD::LD_PTR_OFF, DL, VTs, OpsOff, @@ -1488,6 +2060,17 @@ SDValue W65816TargetLowering::LowerI32Bin(SDValue Op, SDValue LH = extractWide32Hi(DAG, DL, L); SDValue RL = extractWide32Lo(DAG, DL, R); SDValue RH = extractWide32Hi(DAG, DL, R); + // A high half known to be zero is the constant: `adc #0`, not an ADC of + // a register (or a stack slot) holding a materialized zero. A + // zero-extending load, lowered after this add, otherwise reaches here + // as a node whose high half is only an extract_subreg. + auto zeroHigh = [&](SDValue V, SDValue &H) { + if (!isa(H) && + DAG.computeKnownBits(V).countMinLeadingZeros() >= 16) + H = DAG.getConstant(0, DL, MVT::i16); + }; + zeroHigh(L, LH); + zeroHigh(R, RH); SDValue Lo, Hi; switch (Op.getOpcode()) { case ISD::AND: @@ -1605,20 +2188,56 @@ SDValue W65816TargetLowering::LowerStore(SDValue Op, St->getPointerInfo().getWithOffset(2), commonAlignment(St->getAlign(), 2), Fl); } + // An element of a global array: two `sta G+K,x` / `sta G+K+2,x`. + // The four bytes lie inside G (matchGlobalIndex), so G+K+2 does too. + SDValue GSym, GIdx; + if (matchGlobalIndex(DAG, DL, Ptr, St->getMemOperand(), GSym, GIdx)) { + auto *GA = cast(GSym); + SDValue GSymHi = DAG.getTargetGlobalAddress(GA->getGlobal(), DL, + MVT::i16, + GA->getOffset() + 2); + SDValue StLo = emitGlobalIndexStore(DAG, DL, Chain, Lo, GSym, GIdx, + W65816::STA_AbsX); + return emitGlobalIndexStore(DAG, DL, StLo, Hi, GSymHi, GIdx, + W65816::STA_AbsX); + } SDValue Two = DAG.getConstant(2, DL, PtrVT); SDValue Ptr2 = DAG.getNode(ISD::ADD, DL, PtrVT, Ptr, Two); if (PtrVT == MVT::i32) { // ptr32 path — emit two W65816ISD::ST_PTR target nodes, sequentially // chained. The combiner cannot merge target-opaque MemIntrinsic - // stores. + // stores. Both halves address the peeled base with the offset in + // Y (ST_PTR_OFF), as an i16 store does: the high half's `p + 2` + // (or `p + K + 2`) otherwise costs an i32 add and a staging of its + // result. SDVTList VTs = DAG.getVTList(MVT::Other); + SDValue Base = Ptr; + uint16_t Off = 0; + if (!peelPtr32Offset(DAG, DL, Ptr, Base, Off)) + Base = Ptr; + MachineMemOperand *MMOHi = DAG.getMachineFunction().getMachineMemOperand( + St->getMemOperand(), 2, 2); + auto storeHalf = [&](SDValue Ch, SDValue V, uint32_t K, + MachineMemOperand *MMO) { + if (K == 0) { + SDValue Ops[] = { Ch, V, Base }; + return DAG.getMemIntrinsicNode(W65816ISD::ST_PTR, DL, VTs, Ops, + MVT::i16, MMO); + } + SDValue Ops[] = { Ch, V, Base, + DAG.getTargetConstant(K, DL, MVT::i16) }; + return DAG.getMemIntrinsicNode(W65816ISD::ST_PTR_OFF, DL, VTs, Ops, + MVT::i16, MMO); + }; + if (uint32_t(Off) + 2u <= 0xFFFFu) { + SDValue StLo = storeHalf(Chain, Lo, Off, St->getMemOperand()); + return storeHalf(StLo, Hi, uint32_t(Off) + 2u, MMOHi); + } SDValue OpsLo[] = { Chain, Lo, Ptr }; SDValue StLo = DAG.getMemIntrinsicNode( W65816ISD::ST_PTR, DL, VTs, OpsLo, MVT::i16, St->getMemOperand()); SDValue OpsHi[] = { StLo, Hi, Ptr2 }; - MachineMemOperand *MMOHi = DAG.getMachineFunction().getMachineMemOperand( - St->getMemOperand(), 2, 2); SDValue StHi = DAG.getMemIntrinsicNode( W65816ISD::ST_PTR, DL, VTs, OpsHi, MVT::i16, MMOHi); return StHi; @@ -1661,6 +2280,17 @@ SDValue W65816TargetLowering::LowerStore(SDValue Op, St->getMemOperand()); } + // A global indexed by a variable: `sta G+K,x` (see matchGlobalIndex). + SDValue GSym, GIdx; + if ((MemVT == MVT::i8 || MemVT == MVT::i16) && + matchGlobalIndex(DAG, DL, Ptr, St->getMemOperand(), GSym, GIdx)) { + if (Val.getValueType() == MVT::i8) + Val = DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i16, Val); + return emitGlobalIndexStore(DAG, DL, Chain, Val, GSym, GIdx, + MemVT == MVT::i8 ? W65816::STA8absX + : W65816::STA_AbsX); + } + // No i32 ptr → nothing for us to do; let the default ISD::STORE // path handle it. (Also avoids accidentally wrapping an i16 ptr // store into ST_PTR below, whose ptr operand must be i32.) @@ -1676,6 +2306,15 @@ SDValue W65816TargetLowering::LowerStore(SDValue Op, SDVTList VTs = DAG.getVTList(MVT::Other); SDValue Base; uint16_t Off = 0; + SDValue Idx; + if (peelPtr32Index(DAG, DL, Ptr, Base, Idx, Off)) { + unsigned IdxOpc = (MemVT == MVT::i8) ? unsigned(W65816ISD::STB_PTR_IDX) + : unsigned(W65816ISD::ST_PTR_IDX); + SDValue OffN = DAG.getTargetConstant(Off, DL, MVT::i16); + SDValue OpsIdx[] = { Chain, Val, Base, Idx, OffN }; + return DAG.getMemIntrinsicNode(IdxOpc, DL, VTs, OpsIdx, MemVT, + St->getMemOperand()); + } if (peelPtr32Offset(DAG, DL, Ptr, Base, Off)) { unsigned OffOpc = (MemVT == MVT::i8) ? unsigned(W65816ISD::STB_PTR_OFF) : unsigned(W65816ISD::ST_PTR_OFF); @@ -1755,6 +2394,38 @@ static SDValue LowerVASTART(SDValue Op, SelectionDAG &DAG, return DAG.getStore(Chain, DL, VAFI, VAListPtr, MachinePointerInfo(SV)); } +// A call that can return twice (setjmp) clobbers IMG8..IMG15 as well. +// The longjmp that returns through it skips the returns of every function +// it unwinds, and with them the restores that keep those words for each +// caller (W65816ImgCalleeSave, the swaps of W65816StackRelToImg's IMG8..15 +// homes): they come back holding whatever the deepest of them left. So no +// value of this function lives in them across the call, and +// W65816ImgCalleeSave, seeing the defs, keeps all eight for its caller. +void W65816TargetLowering::AdjustInstrPostInstrSelection(MachineInstr &MI, + SDNode *Node) const { + if (!MI.isCall() || MI.getNumOperands() == 0) + return; + const MachineOperand &Callee = MI.getOperand(0); + bool Twice = false; + if (Callee.isGlobal()) { + if (const auto *F = dyn_cast(Callee.getGlobal())) + Twice = F->hasFnAttribute(Attribute::ReturnsTwice) || + F->getName() == "setjmp" || F->getName() == "_setjmp" || + F->getName() == "sigsetjmp"; + } else if (Callee.isSymbol()) { + StringRef Name = Callee.getSymbolName(); + Twice = Name == "setjmp" || Name == "_setjmp" || Name == "sigsetjmp"; + } + if (!Twice) + return; + for (MCPhysReg R : {W65816::IMG8, W65816::IMG9, W65816::IMG10, + W65816::IMG11, W65816::IMG12, W65816::IMG13, + W65816::IMG14, W65816::IMG15}) + MI.addOperand(MachineOperand::CreateReg(R, /*isDef=*/true, + /*isImp=*/true, /*isKill=*/false, + /*isDead=*/true)); +} + SDValue W65816TargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const { switch (Op.getOpcode()) { @@ -2287,6 +2958,65 @@ SDValue W65816TargetLowering::LowerFormalArguments( return Chain; } +// A callee in libgcc.s known to touch only A, X, Y, the stack, the $E0..$EF +// scratch (and $F0 for the signed 32-bit divides), restoring DBR: its call +// is a JSLlight, which leaves IMG0..IMG7 out of the clobbers (see the +// JSLlight definition). A compiler libcall reaches here as an external +// symbol; a call the program wrote (memcpy) as a declared function -- a +// definition in this module is not the runtime's helper. Anything added +// here must be checked against the helper's code: one that starts using +// an IMG word (or calls compiled C) has to come off the list. +static bool isLightCallee(SDValue Callee) { + static const bool Disabled = getenv("W65816_DISABLE_LIGHT_CALLS"); + if (Disabled) + return false; + StringRef Name; + if (auto *ES = dyn_cast(Callee)) { + Name = ES->getSymbol(); + } else if (auto *GA = dyn_cast(Callee)) { + auto *F = dyn_cast(GA->getGlobal()); + if (!F || !F->isDeclaration()) + return false; + Name = F->getName(); + } else { + return false; + } + return StringSwitch(Name) + .Cases({"__mulhi3", "__mulhisi3", "__umulhisi3", "__mulsi3"}, true) + .Cases({"__udivhi3", "__umodhi3", "__divhi3", "__modhi3"}, true) + .Cases({"__udivsi3", "__umodsi3", "__divsi3", "__modsi3"}, true) + .Cases({"__ashlhi3", "__lshrhi3", "__ashrhi3"}, true) + .Cases({"__ashlsi3", "__lshrsi3", "__ashrsi3"}, true) + .Cases({"memcpy", "memmove", "memset"}, true) + .Default(false); +} + +// What a load or store can address without arithmetic of its own: a far +// pointer plus a constant in Y (`ldy #K ; lda [dp],y`), a global plus a +// constant, or a global plus an index in X (`lda G+K,x`). Y and X are +// unsigned 16-bit, so a NEGATIVE offset is not a mode at all: the access +// pays an i32 add of the pointer and a staging of the result. The default +// (any signed 16-bit offset) let loop strength reduction run a loop over +// an array of records on a pointer to a later field, every other field at +// a negative offset -- Space Taxi's sprite-cache scan did four such i32 +// adds per entry. A register base plus an index and a constant is left +// out too: the index would need its own add. +bool W65816TargetLowering::isLegalAddressingMode(const DataLayout &DL, + const AddrMode &AM, Type *Ty, + unsigned AS, + Instruction *I) const { + static const bool Disabled = getenv("W65816_DISABLE_ADDR_MODES"); + if (Disabled) + return TargetLowering::isLegalAddressingMode(DL, AM, Ty, AS, I); + if (AM.BaseOffs < 0 || AM.BaseOffs > 0xFFFF) + return false; + if (AM.Scale != 0 && AM.Scale != 1) + return false; + if (AM.Scale && AM.HasBaseReg && (AM.BaseGV || AM.BaseOffs)) + return false; + return true; +} + SDValue W65816TargetLowering::LowerCall(TargetLowering::CallLoweringInfo &CLI, SmallVectorImpl &InVals) const { @@ -2476,8 +3206,9 @@ W65816TargetLowering::LowerCall(TargetLowering::CallLoweringInfo &CLI, if (Glue.getNode()) CallOps.push_back(Glue); - Chain = DAG.getNode(W65816ISD::CALL, DL, - DAG.getVTList(MVT::Other, MVT::Glue), CallOps); + Chain = DAG.getNode(isLightCallee(Callee) ? W65816ISD::CALL_LIGHT + : W65816ISD::CALL, + DL, DAG.getVTList(MVT::Other, MVT::Glue), CallOps); Glue = Chain.getValue(1); Chain = DAG.getCALLSEQ_END(Chain, StackBytes, 0, Glue, DL); @@ -2661,9 +3392,151 @@ SDValue W65816TargetLowering::LowerReturn( return DAG.getNode(W65816ISD::RET_GLUE, DL, MVT::Other, RetOps); } +// A float / double compare that the operand's bit pattern alone decides, +// done as integer compares instead of a soft-float libcall (__eqsf2, +// __unorddf2, __gtdf2, ...). InstCombine makes most of these out of +// integer code: `(bits & ~sign) == 0` becomes `x == 0.0`, a magnitude test +// against the exponent mask becomes `fabs(x) == inf` or `x ord 0.0`. +// +// With B the operand's bits, A = B with the sign cleared and Inf the bits +// of +infinity, every rule below is exact for every input -- NaNs, signed +// zeros and subnormals included -- under IEEE semantics, which the +// runtime's helpers follow (compiler-rt's contract): +// * a NaN is exactly A > Inf, so `ord` is A <= Inf; +// * x == C for a non-NaN C is B == bits(C), or A == 0 when C is a zero; +// * the non-negative floats order as their bit patterns do and every NaN +// lies above Inf, so fabs(x) against C >= 0 is an unsigned compare of +// A, bounded by Inf where a NaN must answer false; +// * against a zero: x > 0 is B in [1, Inf], x < 0 is B in +// [Sign + 1, Sign | Inf], x != 0 ordered is A in [1, Inf], and >= / <= +// add the other zero's pattern. +// The unordered predicates are the negations of the ordered ones. Anything +// else (a NaN constant, relational compares against other constants, two +// variables) stays a libcall. +static SDValue combineFPSetCCConst(SDNode *N, SelectionDAG &DAG) { + SDValue X = N->getOperand(0); + SDValue Y = N->getOperand(1); + EVT FVT = X.getValueType(); + if (FVT != MVT::f32 && FVT != MVT::f64) + return SDValue(); + ISD::CondCode CC = cast(N->getOperand(2))->get(); + if (isa(X) && !isa(Y)) { + std::swap(X, Y); + CC = ISD::getSetCCSwappedOperands(CC); + } + // The NaN-agnostic codes (fast-math) may give either answer for a NaN; + // take the ordered one's. Then reduce the unordered codes to their + // ordered inverses. + switch (CC) { + case ISD::SETEQ: CC = ISD::SETOEQ; break; + case ISD::SETNE: CC = ISD::SETUNE; break; + case ISD::SETLT: CC = ISD::SETOLT; break; + case ISD::SETLE: CC = ISD::SETOLE; break; + case ISD::SETGT: CC = ISD::SETOGT; break; + case ISD::SETGE: CC = ISD::SETOGE; break; + default: break; + } + bool Negate = false; + switch (CC) { + case ISD::SETUEQ: case ISD::SETUNE: case ISD::SETUO: + case ISD::SETULT: case ISD::SETULE: case ISD::SETUGT: case ISD::SETUGE: + CC = ISD::getSetCCInverse(CC, FVT); + Negate = true; + break; + default: + break; + } + + SDLoc DL(N); + EVT VT = N->getValueType(0); + unsigned W = FVT.getSizeInBits(); + EVT IVT = EVT::getIntegerVT(*DAG.getContext(), W); + APInt Sign = APInt::getSignMask(W); + APInt Inf = APFloat::getInf(FVT.getFltSemantics()).bitcastToAPInt(); + APInt Zero = APInt::getZero(W); + auto bitsOf = [&](SDValue V) { return DAG.getBitcast(IVT, V); }; + auto magOf = [&](SDValue V) { + return DAG.getNode(ISD::AND, DL, IVT, bitsOf(V), + DAG.getConstant(~Sign, DL, IVT)); + }; + auto cmp = [&](SDValue L, const APInt &R, ISD::CondCode IntCC) { + return DAG.getSetCC(DL, VT, L, DAG.getConstant(R, DL, IVT), IntCC); + }; + // Lo <= V <= Hi, unsigned: one subtract and one compare. + auto inRange = [&](SDValue V, const APInt &Lo, const APInt &Hi) { + SDValue Off = DAG.getNode(ISD::SUB, DL, IVT, V, + DAG.getConstant(Lo, DL, IVT)); + return cmp(Off, Hi - Lo, ISD::SETULE); + }; + auto either = [&](SDValue L, SDValue R) { + return DAG.getNode(ISD::OR, DL, VT, L, R); + }; + auto done = [&](SDValue R) { + return Negate ? DAG.getLogicalNOT(DL, R, VT) : R; + }; + + // x ord x, and x == x / >= / <= x, which are true unless x is a NaN. + if (X == Y) { + if (CC == ISD::SETO || CC == ISD::SETOEQ || CC == ISD::SETOGE || + CC == ISD::SETOLE) + return done(cmp(magOf(X), Inf, ISD::SETULE)); + return SDValue(); + } + auto *C = dyn_cast(Y); + if (!C || C->isNaN()) + return SDValue(); + if (CC == ISD::SETO) + return done(cmp(magOf(X), Inf, ISD::SETULE)); + APInt CB = C->getValueAPF().bitcastToAPInt(); + + if (X.getOpcode() == ISD::FABS && (!C->isNegative() || C->isZero())) { + SDValue A = magOf(X.getOperand(0)); + APInt CM = CB & ~Sign; + switch (CC) { + case ISD::SETOEQ: return done(cmp(A, CM, ISD::SETEQ)); + case ISD::SETOLT: return done(cmp(A, CM, ISD::SETULT)); + case ISD::SETOLE: return done(cmp(A, CM, ISD::SETULE)); + case ISD::SETOGE: return done(inRange(A, CM, Inf)); + case ISD::SETOGT: + if (CM == Inf) + return SDValue(); + return done(inRange(A, CM + 1, Inf)); + case ISD::SETONE: + if (CM == Inf) + return done(cmp(A, Inf, ISD::SETULT)); + return SDValue(); + default: + return SDValue(); + } + } + + if (C->isZero()) { + SDValue B = bitsOf(X); + switch (CC) { + case ISD::SETOEQ: return done(cmp(magOf(X), Zero, ISD::SETEQ)); + case ISD::SETONE: return done(inRange(magOf(X), APInt(W, 1), Inf)); + case ISD::SETOGT: return done(inRange(B, APInt(W, 1), Inf)); + case ISD::SETOLT: return done(inRange(B, Sign + 1, Sign | Inf)); + case ISD::SETOGE: + return done(either(cmp(B, Inf, ISD::SETULE), cmp(B, Sign, ISD::SETEQ))); + case ISD::SETOLE: + return done(either(inRange(B, Sign, Sign | Inf), + cmp(B, Zero, ISD::SETEQ))); + default: + return SDValue(); + } + } + + if (CC == ISD::SETOEQ) + return done(cmp(bitsOf(X), CB, ISD::SETEQ)); + return SDValue(); +} + SDValue W65816TargetLowering::PerformDAGCombine(SDNode *N, DAGCombinerInfo &DCI) const { + if (N->getOpcode() == ISD::SETCC) + return combineFPSetCCConst(N, DCI.DAG); // (shl i32 X, K) -> chain of K (add x, x) for small K. After type // legalisation the i32 add splits via ADDC/ADDE pseudos which expand // to native ASL/ROL + carry-chain — much cheaper than the type- @@ -2981,34 +3854,37 @@ SDValue W65816TargetLowering::LowerMUL_I32(SDValue Op, return SDValue(); }; - // Mul-by-constant strength reduction: (X * K) where K-1 or K+1 is - // a small power of 2 (shift count 1..5, matching our inlined i32 - // SHL range) expands to (X << N) +/- X — saves a __mulsi3 libcall - // (~250 cyc) for ~70 cyc of inlined shift+add. Catches djb2Hash's - // `h * 33` = (h << 5) + h. - // - // Patterns covered: - // K = 2^N + 1 in {3,5,9,17,33} → (X << N) + X - // K = 2^N - 1 in {7,15,31} → (X << N) - X - // Larger N hits the i32 SHL libcall path (no longer profitable). - if (auto *CN = dyn_cast(Rhs)) { - int64_t K = CN->getSExtValue(); - for (unsigned N = 1; N <= 5; N++) { - int64_t Pow = int64_t{1} << N; - SDValue ShAmt = DAG.getConstant(N, DL, MVT::i16); - if (K == Pow + 1) { - SDValue Shl = DAG.getNode(ISD::SHL, DL, MVT::i32, Lhs, ShAmt); - return DAG.getNode(ISD::ADD, DL, MVT::i32, Shl, Lhs); - } - if (K == Pow - 1) { - SDValue Shl = DAG.getNode(ISD::SHL, DL, MVT::i32, Lhs, ShAmt); - return DAG.getNode(ISD::SUB, DL, MVT::i32, Shl, Lhs); - } - } + // W65816_DISABLE_MUL_SHIFT_ADD turns off both reductions below. + static const bool Reduce = !getenv("W65816_DISABLE_MUL_SHIFT_ADD"); + + // A product known to fit 16 bits is its low word: the multiply runs in + // 16 bits (shifts and adds for a constant, else the __mulhi3 call) and + // is zero-extended. `sim->spr[idx]`'s byte offset, a u8 index times a + // 6-byte record, called __umulhisi3 on every access. + if (Reduce && DAG.computeKnownBits(Op).countMinLeadingZeros() >= 16) { + SDValue LoL = lowWord(DAG, DL, Lhs); + SDValue Lo; + if (auto *CN = dyn_cast(Rhs)) + Lo = mulConst16(DAG, DL, LoL, CN->getZExtValue()); + else + Lo = DAG.getNode(ISD::MUL, DL, MVT::i16, LoL, lowWord(DAG, DL, Rhs)); + return buildWide32(DAG, DL, Lo, DAG.getConstant(0, DL, MVT::i16)); } SDValue A = narrowToI16(Lhs); SDValue B = narrowToI16(Rhs); + + // Mul-by-constant strength reduction: a constant of at most two signed + // digits, 2^N +- 2^M (3, 5, 6, 7, 9, 10, 12, 15, 24, 33, ...), is one + // i32 add or subtract of two inlined shifts -- against a __mulsi3 call. + // djb2Hash's `h * 33` = (h << 5) + h. Two 16-bit operands keep + // __umulhisi3, whose table lookup for small values is about as quick. + if (auto *CN = dyn_cast(Rhs)) + if (Reduce && !(A && B)) + if (SDValue R = mulConstShiftAdd(DAG, DL, MVT::i32, Lhs, + CN->getZExtValue(), 2)) + return R; + if (A && B) { TargetLowering::ArgListTy Args; Args.push_back({A, Type::getInt16Ty(*DAG.getContext())}); @@ -3184,76 +4060,6 @@ W65816TargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI, return emitRROp(MI, BB, W65816::STAfi, W65816::EORfi, /*HasOut=*/true); case W65816::CMP_RR: return emitRROp(MI, BB, W65816::STAfi, W65816::CMPfi, /*HasOut=*/false); - case W65816::LDAptr32S: - case W65816::STAptr32S: - case W65816::STBptr32S: { - // Split-pair variant: ptr is 2 i16 operands (lo + hi) instead of - // 1 Wide32 reg pair. Used by the W65816LowerWide32 pre-RA pass - // to dodge pair-allocation pressure. Otherwise identical to - // the LDAptr32 inserter below. - MachineFunction *MF = BB->getParent(); - const W65816Subtarget &STI = MF->getSubtarget(); - const W65816InstrInfo &TII = *STI.getInstrInfo(); - DebugLoc DL = MI.getDebugLoc(); - bool IsLoad = MI.getOpcode() == W65816::LDAptr32S; - bool IsByteStore = MI.getOpcode() == W65816::STBptr32S; - Register PtrLo = MI.getOperand(IsLoad ? 1 : 1).getReg(); - Register PtrHi = MI.getOperand(IsLoad ? 2 : 2).getReg(); - - int FILo = MF->getFrameInfo().CreateStackObject(2, Align(2), - /*isSpillSlot=*/false); - int FIHi = MF->getFrameInfo().CreateStackObject(2, Align(2), - /*isSpillSlot=*/false); - BuildMI(*BB, MI.getIterator(), DL, TII.get(W65816::STAfi)) - .addReg(PtrLo).addFrameIndex(FILo).addImm(0); - BuildMI(*BB, MI.getIterator(), DL, TII.get(W65816::STAfi)) - .addReg(PtrHi).addFrameIndex(FIHi).addImm(0); - - // STA_DP's tablegen def has no implicit A Use, so without an - // explicit kill marker between adjacent LDAfi-STA_DP-LDAfi-STA_DP - // pairs the fast regalloc collapses two A-loads into one (the - // first's value is overwritten before STA_DP can store it). Add - // implicit Use of A on the STA_DP to encode the dependency. This - // also helps post-RA passes track A liveness correctly. - BuildMI(*BB, MI.getIterator(), DL, TII.get(W65816::LDAfi), - W65816::A).addFrameIndex(FILo).addImm(0); - BuildMI(*BB, MI.getIterator(), DL, - TII.get(W65816::STA_DP)).addImm(0xE0) - .addReg(W65816::A, RegState::Implicit); - BuildMI(*BB, MI.getIterator(), DL, TII.get(W65816::LDAfi), - W65816::A).addFrameIndex(FIHi).addImm(0); - BuildMI(*BB, MI.getIterator(), DL, - TII.get(W65816::STA_DP)).addImm(0xE2) - .addReg(W65816::A, RegState::Implicit); - - if (IsLoad) { - Register Dst = MI.getOperand(0).getReg(); - BuildMI(*BB, MI.getIterator(), DL, - TII.get(W65816::LDY_Imm16)).addImm(0); - BuildMI(*BB, MI.getIterator(), DL, - TII.get(ptrLoadOpcode(MI))).addImm(0xE0); - BuildMI(*BB, MI.getIterator(), DL, - TII.get(TargetOpcode::COPY), Dst).addReg(W65816::A); - } else { - Register Val = MI.getOperand(0).getReg(); - BuildMI(*BB, MI.getIterator(), DL, - TII.get(TargetOpcode::COPY), W65816::A).addReg(Val); - BuildMI(*BB, MI.getIterator(), DL, - TII.get(W65816::LDY_Imm16)).addImm(0); - // Byte store: one atomic STB_DPIndLongY pseudo (AsmPrinter expands - // to SEP #$20 ; sta [dp],y ; REP #$20). See STB_DPIndLongY in the - // .td: emitting SEP/STA/REP as three MIs let the -O0 fast RA wedge - // a 16-bit reload into the M=8 window, truncating a live value. - if (IsByteStore) - BuildMI(*BB, MI.getIterator(), DL, - TII.get(W65816::STB_DPIndLongY)).addImm(0xE0); - else - BuildMI(*BB, MI.getIterator(), DL, - TII.get(W65816::STA_DPIndLongY)).addImm(0xE0); - } - MI.eraseFromParent(); - return BB; - } case W65816::LDAptr32: case W65816::STAptr32: case W65816::STBptr32: { @@ -3757,7 +4563,10 @@ W65816TargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI, } case W65816::LDAptr32Off: case W65816::STAptr32Off: - case W65816::STBptr32Off: { + case W65816::STBptr32Off: + case W65816::LDAptr32Idx: + case W65816::STAptr32Idx: + case W65816::STBptr32Idx: { // ptr32 deref with constant offset. The 65816's `[dp],Y` adds Y // to the 24-bit pointer at `dp..dp+2` to form the effective // address — so we can stage the RAW pointer at $E0..$E2 and put @@ -3781,10 +4590,33 @@ W65816TargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI, const W65816InstrInfo &TII = *STI.getInstrInfo(); MachineRegisterInfo &MRI = MF->getRegInfo(); DebugLoc DL = MI.getDebugLoc(); - bool IsLoad = MI.getOpcode() == W65816::LDAptr32Off; - bool IsByteStore = MI.getOpcode() == W65816::STBptr32Off; + bool IsLoad = MI.getOpcode() == W65816::LDAptr32Off || + MI.getOpcode() == W65816::LDAptr32Idx; + bool IsByteStore = MI.getOpcode() == W65816::STBptr32Off || + MI.getOpcode() == W65816::STBptr32Idx; + // The Idx forms carry a 16-bit index before the offset: Y = idx + off, + // set before the staging below, which leaves Y alone. + bool HasIdx = MI.getOpcode() == W65816::LDAptr32Idx || + MI.getOpcode() == W65816::STAptr32Idx || + MI.getOpcode() == W65816::STBptr32Idx; Register Ptr = MI.getOperand(1).getReg(); - int64_t Off = MI.getOperand(2).getImm(); + int64_t Off = MI.getOperand(HasIdx ? 3 : 2).getImm(); + if (HasIdx) { + Register YSrc = MI.getOperand(2).getReg(); + if (Off != 0) { + Register T0 = MRI.createVirtualRegister(&W65816::Acc16RegClass); + Register T1 = MRI.createVirtualRegister(&W65816::Acc16RegClass); + BuildMI(*BB, MI.getIterator(), DL, TII.get(TargetOpcode::COPY), T0) + .addReg(YSrc); + BuildMI(*BB, MI.getIterator(), DL, TII.get(W65816::ADCi16imm), T1) + .addReg(T0) + .addImm(Off); + YSrc = T1; + } + BuildMI(*BB, MI.getIterator(), DL, TII.get(TargetOpcode::COPY), + W65816::Y) + .addReg(YSrc); + } // See LDAptr32 inserter above: vreg sub-regs need COPY-with-subreg // (TRI.getSubReg is physreg-only at custom-inserter time). Register PtrLo = MRI.createVirtualRegister(&W65816::Wide16RegClass); @@ -3817,8 +4649,9 @@ W65816TargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI, if (IsLoad) { Register Dst = MI.getOperand(0).getReg(); - BuildMI(*BB, MI.getIterator(), DL, - TII.get(W65816::LDY_Imm16)).addImm(Off); + if (!HasIdx) + BuildMI(*BB, MI.getIterator(), DL, + TII.get(W65816::LDY_Imm16)).addImm(Off); BuildMI(*BB, MI.getIterator(), DL, TII.get(ptrLoadOpcode(MI))).addImm(0xE0); BuildMI(*BB, MI.getIterator(), DL, @@ -3827,8 +4660,9 @@ W65816TargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI, Register Val = MI.getOperand(0).getReg(); BuildMI(*BB, MI.getIterator(), DL, TII.get(TargetOpcode::COPY), W65816::A).addReg(Val); - BuildMI(*BB, MI.getIterator(), DL, - TII.get(W65816::LDY_Imm16)).addImm(Off); + if (!HasIdx) + BuildMI(*BB, MI.getIterator(), DL, + TII.get(W65816::LDY_Imm16)).addImm(Off); // Byte store: one atomic STB_DPIndLongY pseudo (AsmPrinter expands // to SEP #$20 ; sta [dp],y ; REP #$20). See STB_DPIndLongY in the // .td: emitting SEP/STA/REP as three MIs let the -O0 fast RA wedge diff --git a/src/llvm/lib/Target/W65816/W65816ISelLowering.h b/src/llvm/lib/Target/W65816/W65816ISelLowering.h index 5cde32f..195a080 100644 --- a/src/llvm/lib/Target/W65816/W65816ISelLowering.h +++ b/src/llvm/lib/Target/W65816/W65816ISelLowering.h @@ -56,6 +56,16 @@ public: SDValue LowerOperation(SDValue Op, SelectionDAG &DAG) const override; + void AdjustInstrPostInstrSelection(MachineInstr &MI, + SDNode *Node) const override; + + // SELECT_CC is Custom for i64 so that the type legalizer hands its i64 + // compare operands to LowerSELECT_CC. A SELECT_CC with an i64 result + // asks here first; leaving Results empty keeps the default expansion, + // which splits it into i32 selects that come back through LowerOperation. + void ReplaceNodeResults(SDNode *N, SmallVectorImpl &Results, + SelectionDAG &DAG) const override {} + // Lock i16 shift amounts to i16 (not i32) even when i32 is a legal // type. Without this, the DAG combiner promotes i16 shift amounts // to i32 once i32 is registered as legal, leaving (sra i16, i32:K) @@ -177,6 +187,10 @@ public: // a libcall is ~12 instructions, while `(mul x, 3)` -> `(add x, (shl // x, 1))` is 5. i32 stays libcall — the per-half shift+add+chain // expansion comes out larger than the __mulsi3 call. + bool isLegalAddressingMode(const DataLayout &DL, const AddrMode &AM, + Type *Ty, unsigned AS, + Instruction *I = nullptr) const override; + bool decomposeMulByConstant(LLVMContext &Context, EVT VT, SDValue C) const override { return VT == MVT::i16; diff --git a/src/llvm/lib/Target/W65816/W65816InstrInfo.cpp b/src/llvm/lib/Target/W65816/W65816InstrInfo.cpp index 0390b76..9c9b554 100644 --- a/src/llvm/lib/Target/W65816/W65816InstrInfo.cpp +++ b/src/llvm/lib/Target/W65816/W65816InstrInfo.cpp @@ -14,6 +14,7 @@ #include "W65816InstrInfo.h" #include "W65816.h" #include "W65816Subtarget.h" +#include "llvm/ADT/SmallPtrSet.h" #include "llvm/CodeGen/MachineFrameInfo.h" #include "llvm/CodeGen/MachineInstrBuilder.h" #include "llvm/Support/ErrorHandling.h" @@ -36,6 +37,12 @@ void W65816InstrInfo::anchor() {} // REP coalescer and the AsmPrinter's LDAi8imm mode-tracking behave exactly // as they did for the old three-instruction form. bool W65816InstrInfo::expandPostRAPseudo(MachineInstr &MI) const { + if (MI.getOpcode() == W65816::LDAimg) { + BuildMI(*MI.getParent(), MI, MI.getDebugLoc(), get(W65816::LDA_DP)) + .addImm(W65816Helpers::imgDPAddr(MI.getOperand(1).getReg())); + MI.eraseFromParent(); + return true; + } unsigned Access; switch (MI.getOpcode()) { case W65816::STB_DPIndLongY: @@ -151,6 +158,195 @@ bool hasTiedAcc16Src(const MachineInstr &MI) { return false; } + +// One block of aLiveFrom: 1 when A is read from From on, 0 when it is +// overwritten first, -1 when the scan reaches the end of the block. +// Conservative -- anything not known to leave A alone or to write all of +// it without reading it counts as a read, as does a SEP / REP (an 8-bit +// load writes half of A). A non-A LDAfi / STAfi further on counts as +// leaving A alone: it saves A itself when A is read after it, and when A +// is not, A is dead here too. +static int scanALive(const MachineBasicBlock &MBB, + MachineBasicBlock::const_iterator From) { + const TargetRegisterInfo *TRI = + MBB.getParent()->getSubtarget().getRegisterInfo(); + for (auto It = From; It != MBB.end(); ++It) { + const MachineInstr &N = *It; + if (N.isDebugInstr()) + continue; + switch (N.getOpcode()) { + case W65816::LDAfi: + if (N.getOperand(0).getReg() == W65816::A) + return 0; + continue; + case W65816::STAfi: + case W65816::STA8fi: + if (N.getOperand(0).getReg() == W65816::A) + return 1; + continue; + // Write all of A (in 16-bit M) and read nothing of it. + case W65816::LDA_StackRel: case W65816::LDA_DP: case W65816::LDA_Abs: + case W65816::LDA_Long: case W65816::LDA_Imm16: case W65816::LDAi16imm: + case W65816::LDAi16imm_bank: case W65816::LDAabs: + case W65816::LDA_DPIndLongY: case W65816::LDAimg: + case W65816::TXA: case W65816::TYA: case W65816::TSC: case W65816::TDC: + case W65816::PLA: + return 0; + // Leave A alone. + case W65816::LDX_Imm16: case W65816::LDX_DP: case W65816::LDX_Abs: + case W65816::LDY_Imm16: case W65816::LDY_DP: case W65816::LDY_Abs: + case W65816::LDXi16imm: + case W65816::STX_DP: case W65816::STX_Abs: + case W65816::STY_DP: case W65816::STY_Abs: + case W65816::STZ_DP: case W65816::STZ_Abs: + case W65816::INX: case W65816::DEX: case W65816::INY: case W65816::DEY: + case W65816::PHX: case W65816::PHY: case W65816::PLX: case W65816::PLY: + case W65816::CLC: case W65816::SEC: case W65816::PHP: case W65816::PLP: + continue; + default: + break; + } + if (N.readsRegister(W65816::A, TRI)) + return 1; + // A call that clobbers A (by def or register mask) ends its life; one + // that preserves it, like a direct branch, leaves A alone. + if (N.isCall()) { + if (N.modifiesRegister(W65816::A, TRI)) + return 0; + continue; + } + if (N.isReturn()) + return 0; + if (N.isBranch() && !N.isIndirectBranch()) + continue; + if (N.isCopy()) { + Register Dst = N.getOperand(0).getReg(); + Register Src = N.getOperand(1).getReg(); + if (Src == W65816::A) + return 1; + if (Dst == W65816::A) + return 0; + // IMG <-> IMG / X / Y copies leave A alone (IMG to IMG goes + // through A inside PHA / PLA); X <-> Y goes through A. + if ((Dst == W65816::X || Dst == W65816::Y) && + (Src == W65816::X || Src == W65816::Y)) + return 1; + continue; + } + return 1; + } + return -1; +} + + +bool aLiveFrom(const MachineBasicBlock &MBB, + MachineBasicBlock::const_iterator From) { + int R = scanALive(MBB, From); + if (R >= 0) + return R == 1; + // Off the end: live when any successor reads A before writing it. A + // block re-entered from the top was already scanned from there, so a + // loop back adds nothing; past the budget, assume a read. + SmallVector Work; + SmallPtrSet Seen; + const MachineBasicBlock *B = &MBB; + unsigned Budget = 8; + for (;;) { + if (B->succ_empty()) + return true; + for (const MachineBasicBlock *S : B->successors()) + if (Seen.insert(S).second) + Work.push_back(S); + do { + if (Work.empty()) + return false; + if (Budget-- == 0) + return true; + B = Work.pop_back_val(); + R = scanALive(*B, B->begin()); + if (R == 1) + return true; + } while (R == 0); + } +} + + +namespace { +// The ways to release stack, in the order a tie in cost prefers them. +enum ReleaseForm { RelNone, RelPly, RelPlx, RelAdc, RelAdcY, RelAdcX, RelAdcDp }; +} // namespace + +// Cycle and byte counts in 16-bit M and X: PLY / PLX 5 and 1 for two +// bytes; TSC / CLC / ADC # / TCS 9 and 6; TAY / TYA or TAX / TXA around +// that 13 and 8; STA $E0 / LDA $E0 around it 17 and 10. +static ReleaseForm pickRelease(int64_t Bytes, bool KeepA, bool KeepX, + bool KeepY, unsigned &Cost) { + Cost = 0; + if (Bytes <= 0) + return RelNone; + ReleaseForm Best = RelAdcDp; + Cost = cycleByteCost(17, 10); + auto Try = [&](ReleaseForm F, unsigned Cycles, unsigned Size) { + unsigned C = cycleByteCost(Cycles, Size); + if (C < Cost) { + Cost = C; + Best = F; + } + }; + unsigned Pulls = Bytes / 2; + if (Bytes % 2 == 0 && Bytes <= 32) { + if (!KeepY) + Try(RelPly, 5 * Pulls, Pulls); + else if (!KeepX) + Try(RelPlx, 5 * Pulls, Pulls); + } + if (!KeepA) + Try(RelAdc, 9, 6); + else if (!KeepY) + Try(RelAdcY, 13, 8); + else if (!KeepX) + Try(RelAdcX, 13, 8); + return Best; +} + + +unsigned stackReleaseCost(int64_t Bytes, bool KeepA, bool KeepX, bool KeepY) { + unsigned Cost; + pickRelease(Bytes, KeepA, KeepX, KeepY, Cost); + return Cost; +} + + +void emitStackRelease(MachineBasicBlock &MBB, MachineBasicBlock::iterator I, + const DebugLoc &DL, const TargetInstrInfo &TII, + int64_t Bytes, bool KeepA, bool KeepX, bool KeepY) { + unsigned Cost; + ReleaseForm F = pickRelease(Bytes, KeepA, KeepX, KeepY, Cost); + if (F == RelNone) + return; + if (F == RelPly || F == RelPlx) { + for (int64_t i = 0; i < Bytes / 2; ++i) + BuildMI(MBB, I, DL, TII.get(F == RelPly ? W65816::PLY : W65816::PLX)); + return; + } + if (F == RelAdcY) + BuildMI(MBB, I, DL, TII.get(W65816::TAY)); + else if (F == RelAdcX) + BuildMI(MBB, I, DL, TII.get(W65816::TAX)); + else if (F == RelAdcDp) + BuildMI(MBB, I, DL, TII.get(W65816::STA_DP)).addImm(0xE0); + BuildMI(MBB, I, DL, TII.get(W65816::TSC)); + BuildMI(MBB, I, DL, TII.get(W65816::CLC)); + BuildMI(MBB, I, DL, TII.get(W65816::ADC_Imm16)).addImm(Bytes); + BuildMI(MBB, I, DL, TII.get(W65816::TCS)); + if (F == RelAdcY) + BuildMI(MBB, I, DL, TII.get(W65816::TYA)); + else if (F == RelAdcX) + BuildMI(MBB, I, DL, TII.get(W65816::TXA)); + else if (F == RelAdcDp) + BuildMI(MBB, I, DL, TII.get(W65816::LDA_DP)).addImm(0xE0); +} + } // namespace W65816Helpers } // namespace llvm @@ -349,68 +545,77 @@ void W65816InstrInfo::copyPhysReg(MachineBasicBlock &MBB, llvm_unreachable("W65816: cross-class copyPhysReg not yet implemented"); } +// A STAfi for a spill. The opcode carries Defs = [A] as an over- +// approximation (see its definition), but STAfi never changes A: +// eliminateFrameIndex moves a non-A source through A inside PHA / PLA +// whenever A is still read. The allocator places spills without checking +// A's interference, so one can land after `$a = COPY $img0` with A live +// into the next block; with the implicit def in place, post-RA +// MachineCopyPropagation took the copy as overwritten and deleted it (a +// loop header then read whatever A last held). A spill names no def of A. +MachineInstrBuilder W65816InstrInfo::spillSTAfi(MachineBasicBlock &MBB, + MachineBasicBlock::iterator MI, + const DebugLoc &DL) const { + MachineInstrBuilder B = BuildMI(MBB, MI, DL, get(W65816::STAfi)); + MachineInstr *NewMI = B.getInstr(); + for (unsigned I = NewMI->getNumOperands(); I-- > 0;) { + const MachineOperand &MO = NewMI->getOperand(I); + if (MO.isReg() && MO.isImplicit() && MO.isDef() && + MO.getReg() == W65816::A) + NewMI->removeOperand(I); + } + return B; +} + void W65816InstrInfo::storeRegToStackSlot( MachineBasicBlock &MBB, MachineBasicBlock::iterator MI, Register SrcReg, bool isKill, int FrameIdx, const TargetRegisterClass *RC, Register VReg, MachineInstr::MIFlag Flags) const { // STAfi gets eliminated by W65816RegisterInfo::eliminateFrameIndex into - // a real STA d,S. Source is implicit A; emit the pseudo with the FI - // and zero offset. When regalloc hands us a spill from X or Y, bridge - // through A (TXA / TYA) — same rationale as loadRegFromStackSlot. + // a real STA d,S, moving a source other than A through A. DebugLoc DL = MI != MBB.end() ? MI->getDebugLoc() : DebugLoc(); - // Wide32 spill: split into 2 i16 stores at offsets 0 and 2 of the - // 4-byte spill slot. Bridge each half through A using copyPhysReg. + // Wide32 spill: two i16 stores, offsets 0 and 2 of the 4-byte slot, + // each STAfi straight from its half (eliminateFrameIndex moves a non-A + // half through A, keeping A when it is still read). The allocator + // spills a VIRTUAL Wide32 register through here too (the vregs + // W65816LowerWide32 cannot split -- the pointer PHIs of a loop nest); + // the old code took that for the unpaired-physreg case and stored the + // low half only, so the reload brought back a pointer with whatever + // bank its register pair last held: CoreMark's core_list_mergesort, + // once its loop exits got blocks of their own, walked off the list. if (RC == &W65816::Wide32RegClass || RC == &W65816::Acc32RegClass || RC == &W65816::AnyWide32RegClass) { - Register Lo, Hi; - switch (SrcReg) { - case W65816::AX32: Lo = W65816::A; Hi = W65816::X; break; - case W65816::IMG01: Lo = W65816::IMG0; Hi = W65816::IMG1; break; - case W65816::IMG23: Lo = W65816::IMG2; Hi = W65816::IMG3; break; - case W65816::IMG45: Lo = W65816::IMG4; Hi = W65816::IMG5; break; - case W65816::IMG67: Lo = W65816::IMG6; Hi = W65816::IMG7; break; - case W65816::IMG89: Lo = W65816::IMG8; Hi = W65816::IMG9; break; - case W65816::IMG1011: Lo = W65816::IMG10; Hi = W65816::IMG11; break; - case W65816::IMG1213: Lo = W65816::IMG12; Hi = W65816::IMG13; break; - case W65816::IMG1415: Lo = W65816::IMG14; Hi = W65816::IMG15; break; - default: - // Regalloc occasionally hands us an UNPAIRED single i16 physreg - // (Acc16 / Img16 / Idx16) for a Wide32-class spill — happens when - // only one sub-reg is live at the spill point and the regalloc - // decides to spill it through the Wide32 path anyway. Treat as - // a single i16 store of the lone half at offset 0; the matching - // reload mirrors this (only the lo half is read back). The hi - // half slot at offset 2 is left unwritten — the reload's hi load - // reads zero-init stack memory which is fine because nothing - // genuinely needed the hi value (otherwise the regalloc would - // have allocated a real pair). - if (SrcReg != W65816::A) { - copyPhysReg(MBB, MI, DL, W65816::A, SrcReg, false); - } - BuildMI(MBB, MI, DL, get(W65816::STAfi)) - .addReg(W65816::A).addFrameIndex(FrameIdx).addImm(0); + if (SrcReg.isVirtual()) { + spillSTAfi(MBB, MI, DL) + .addReg(SrcReg, {}, llvm::sub_lo) + .addFrameIndex(FrameIdx).addImm(0); + spillSTAfi(MBB, MI, DL) + .addReg(SrcReg, getKillRegState(isKill), llvm::sub_hi) + .addFrameIndex(FrameIdx).addImm(2); return; } - // Bridge lo through A, store at offset 0; bridge hi through A, - // store at offset 2. This is brittle in the face of regalloc - // expectations — Wide32 spills are best avoided by keeping the - // pair in registers if at all possible. - if (Lo != W65816::A) { - copyPhysReg(MBB, MI, DL, W65816::A, Lo, false); + Register Lo = getRegisterInfo().getSubReg(SrcReg, llvm::sub_lo); + Register Hi = getRegisterInfo().getSubReg(SrcReg, llvm::sub_hi); + if (!Lo || !Hi) { + // Regalloc occasionally hands us an UNPAIRED single i16 physreg for + // a Wide32-class spill (only one half live at the spill point). + // Store the lone half at offset 0; the reload mirrors this. + spillSTAfi(MBB, MI, DL) + .addReg(SrcReg, getKillRegState(isKill)) + .addFrameIndex(FrameIdx).addImm(0); + return; } - BuildMI(MBB, MI, DL, get(W65816::STAfi)) - .addReg(W65816::A).addFrameIndex(FrameIdx).addImm(0); - copyPhysReg(MBB, MI, DL, W65816::A, Hi, false); - BuildMI(MBB, MI, DL, get(W65816::STAfi)) - .addReg(W65816::A).addFrameIndex(FrameIdx).addImm(2); + spillSTAfi(MBB, MI, DL) + .addReg(Lo, getKillRegState(isKill)) + .addFrameIndex(FrameIdx).addImm(0); + spillSTAfi(MBB, MI, DL) + .addReg(Hi, getKillRegState(isKill)) + .addFrameIndex(FrameIdx).addImm(2); return; } - if (SrcReg == W65816::X || SrcReg == W65816::Y) { - unsigned XferOp = (SrcReg == W65816::X) ? W65816::TXA : W65816::TYA; - BuildMI(MBB, MI, DL, get(XferOp)); - SrcReg = W65816::A; - } - BuildMI(MBB, MI, DL, get(W65816::STAfi)) + // X / Y sources included: eliminateFrameIndex transfers them through A + // (a TXA / TYA here clobbered A behind the allocator's back). + spillSTAfi(MBB, MI, DL) .addReg(SrcReg, getKillRegState(isKill)) .addFrameIndex(FrameIdx) .addImm(0); @@ -422,66 +627,44 @@ void W65816InstrInfo::loadRegFromStackSlot(MachineBasicBlock &MBB, const TargetRegisterClass *RC, Register VReg, unsigned SubReg, MachineInstr::MIFlag Flags) const { - // LDAfi only knows how to put the value in A. If regalloc asks for - // a spill into X or Y, we have to bridge through A: LDA d,S then - // TAX / TAY. Without this, the MIR has `$x = LDAfi` but the asm - // printer emits just `LDA d,S` (which writes A, not X) — a silent - // miscompile that surfaced as i64 subtract chains using stale X - // values for the second word (caught by udivmod's `a - q*b` mod - // computation). + // LDAfi loads through A; eliminateFrameIndex adds the TAX / TAY / STA dp + // for an X, Y or IMG destination. (Before that existed, `$x = LDAfi` + // printed as a bare `LDA d,S`: i64 subtract chains used stale X words.) DebugLoc DL = MI != MBB.end() ? MI->getDebugLoc() : DebugLoc(); - // Wide32 reload: 2 i16 loads at offsets 0 and 2 of the 4-byte slot. + // Wide32 reload: two i16 loads, the high half first so an AX32 pair's + // X load (through A) comes before A takes the low half. A virtual + // Wide32 register is reloaded by sub-register (see storeRegToStackSlot); + // a physical half other than A is loaded through A by + // eliminateFrameIndex, which keeps A when it is still read (an `lda d,s + // ; tax` / `lda d,s ; sta dp` here clobbered it behind the allocator's + // back). if (RC == &W65816::Wide32RegClass || RC == &W65816::Acc32RegClass || RC == &W65816::AnyWide32RegClass) { - Register Lo, Hi; - switch (DestReg) { - case W65816::AX32: Lo = W65816::A; Hi = W65816::X; break; - case W65816::IMG01: Lo = W65816::IMG0; Hi = W65816::IMG1; break; - case W65816::IMG23: Lo = W65816::IMG2; Hi = W65816::IMG3; break; - case W65816::IMG45: Lo = W65816::IMG4; Hi = W65816::IMG5; break; - case W65816::IMG67: Lo = W65816::IMG6; Hi = W65816::IMG7; break; - case W65816::IMG89: Lo = W65816::IMG8; Hi = W65816::IMG9; break; - case W65816::IMG1011: Lo = W65816::IMG10; Hi = W65816::IMG11; break; - case W65816::IMG1213: Lo = W65816::IMG12; Hi = W65816::IMG13; break; - case W65816::IMG1415: Lo = W65816::IMG14; Hi = W65816::IMG15; break; - default: - // Mirror of the unpaired-spill case in storeRegToStackSlot: - // regalloc handed us a single physreg for a Wide32 reload. - // Just load the lo half from offset 0 into the dest. - BuildMI(MBB, MI, DL, get(W65816::LDAfi), W65816::A) + if (DestReg.isVirtual()) { + BuildMI(MBB, MI, DL, get(W65816::LDAfi)) + .addReg(DestReg, RegState::Define | RegState::Undef, llvm::sub_hi) + .addFrameIndex(FrameIdx).addImm(2); + BuildMI(MBB, MI, DL, get(W65816::LDAfi)) + .addReg(DestReg, RegState::Define, llvm::sub_lo) .addFrameIndex(FrameIdx).addImm(0); - if (DestReg != W65816::A) - copyPhysReg(MBB, MI, DL, DestReg, W65816::A, false); return; } - // Lo half: LDA from offset 0, transfer to Lo if needed. - BuildMI(MBB, MI, DL, get(W65816::LDAfi), W65816::A) - .addFrameIndex(FrameIdx).addImm(0); - if (Lo != W65816::A) - copyPhysReg(MBB, MI, DL, Lo, W65816::A, false); - // Hi half: LDA from offset 2, transfer to Hi. - BuildMI(MBB, MI, DL, get(W65816::LDAfi), W65816::A) + Register Lo = getRegisterInfo().getSubReg(DestReg, llvm::sub_lo); + Register Hi = getRegisterInfo().getSubReg(DestReg, llvm::sub_hi); + if (!Lo || !Hi) { + // Mirror of the unpaired-spill case in storeRegToStackSlot. + BuildMI(MBB, MI, DL, get(W65816::LDAfi), DestReg) + .addFrameIndex(FrameIdx).addImm(0); + return; + } + BuildMI(MBB, MI, DL, get(W65816::LDAfi), Hi) .addFrameIndex(FrameIdx).addImm(2); - if (Hi != W65816::A) - copyPhysReg(MBB, MI, DL, Hi, W65816::A, false); + BuildMI(MBB, MI, DL, get(W65816::LDAfi), Lo) + .addFrameIndex(FrameIdx).addImm(0); return; } - if (DestReg == W65816::A) { - BuildMI(MBB, MI, DL, get(W65816::LDAfi), DestReg) - .addFrameIndex(FrameIdx) - .addImm(0); - return; - } - if (DestReg == W65816::X || DestReg == W65816::Y) { - // Load via A, then transfer. A is implicitly clobbered. - BuildMI(MBB, MI, DL, get(W65816::LDAfi), W65816::A) - .addFrameIndex(FrameIdx) - .addImm(0); - unsigned XferOp = (DestReg == W65816::X) ? W65816::TAX : W65816::TAY; - BuildMI(MBB, MI, DL, get(XferOp)); - return; - } - // Fallback: assume A path (covers Acc16 / Wide16 vregs by class). + // Any destination class: eliminateFrameIndex loads A, X, Y or an IMG + // word. BuildMI(MBB, MI, DL, get(W65816::LDAfi), DestReg) .addFrameIndex(FrameIdx) .addImm(0); @@ -761,7 +944,16 @@ unsigned W65816InstrInfo::getInstSizeInBytes(const MachineInstr &MI) const { // JSLpseudo: jsl is 4 bytes. case W65816::JSLpseudo: case W65816::JSLpseudo32: + case W65816::JSLlight: + case W65816::JSLlight32: return 4; + // LDAi16imm_bank: `lda $BE` (2), or for an external-weak symbol + // `lda #sym ; beq L ; lda $BE` (3 + 2 + 2). + case W65816::LDAi16imm_bank: + return MI.getOperand(1).isGlobal() && + MI.getOperand(1).getGlobal()->hasExternalWeakLinkage() + ? 7 + : 2; default: break; } diff --git a/src/llvm/lib/Target/W65816/W65816InstrInfo.h b/src/llvm/lib/Target/W65816/W65816InstrInfo.h index ae5a719..380ad29 100644 --- a/src/llvm/lib/Target/W65816/W65816InstrInfo.h +++ b/src/llvm/lib/Target/W65816/W65816InstrInfo.h @@ -14,6 +14,7 @@ #define LLVM_LIB_TARGET_W65816_W65816INSTRINFO_H #include "W65816RegisterInfo.h" +#include "llvm/CodeGen/MachineInstrBuilder.h" #include "llvm/CodeGen/TargetInstrInfo.h" #define GET_INSTRINFO_HEADER @@ -59,6 +60,29 @@ bool isTiedAcc16Consumer(unsigned Opc); // per-MI operand check the bridge passes perform on every candidate. bool hasTiedAcc16Src(const MachineInstr &MI); +// Whether A's value may still be read from From on, before something +// overwrites it. Follows the CFG through a few successor blocks and reads +// a return's implicit uses; anything it does not know counts as a read. +bool aLiveFrom(const MachineBasicBlock &MBB, + MachineBasicBlock::const_iterator From); + +// Release Bytes of stack (SP += Bytes) before I in 16-bit M and X, keeping +// A, X and Y as asked, in the cheapest form for the 65816: N/2 PLY or PLX, +// TSC/CLC/ADC/TCS, that bracketed by TAY/TYA or TAX/TXA, or through DP $E0. +// Clobbers the flags. +void emitStackRelease(MachineBasicBlock &MBB, MachineBasicBlock::iterator I, + const DebugLoc &DL, const TargetInstrInfo &TII, + int64_t Bytes, bool KeepA, bool KeepX, bool KeepY); + +// The cost model the stack-release choices share: a cycle is worth two +// bytes of code. +inline unsigned cycleByteCost(unsigned Cycles, unsigned Bytes) { + return 2 * Cycles + Bytes; +} + +// Cost of what emitStackRelease emits for the same request. +unsigned stackReleaseCost(int64_t Bytes, bool KeepA, bool KeepX, bool KeepY); + } // namespace W65816Helpers class W65816InstrInfo : public W65816GenInstrInfo { @@ -79,6 +103,10 @@ public: MachineBasicBlock &MBB, MachineBasicBlock::iterator MI, Register SrcReg, bool isKill, int FrameIndex, const TargetRegisterClass *RC, Register VReg, MachineInstr::MIFlag Flags = MachineInstr::NoFlags) const override; + // A spill's STAfi, without the opcode's Defs = [A] (see the .cpp). + MachineInstrBuilder spillSTAfi(MachineBasicBlock &MBB, + MachineBasicBlock::iterator MI, + const DebugLoc &DL) const; void loadRegFromStackSlot( MachineBasicBlock &MBB, MachineBasicBlock::iterator MI, Register DestReg, int FrameIdx, const TargetRegisterClass *RC, Register VReg, diff --git a/src/llvm/lib/Target/W65816/W65816InstrInfo.td b/src/llvm/lib/Target/W65816/W65816InstrInfo.td index 7e7b67c..73f3f3a 100644 --- a/src/llvm/lib/Target/W65816/W65816InstrInfo.td +++ b/src/llvm/lib/Target/W65816/W65816InstrInfo.td @@ -45,6 +45,11 @@ def W65816retglue : SDNode<"W65816ISD::RET_GLUE", SDTNone, def W65816call : SDNode<"W65816ISD::CALL", SDT_W65816Call, [SDNPHasChain, SDNPOutGlue, SDNPOptInGlue, SDNPVariadic]>; +// A call to a runtime helper known to touch only A/X/Y, its arguments, +// the $E0..$F0 scratch and the stack (see JSLlight). +def W65816callLight : SDNode<"W65816ISD::CALL_LIGHT", SDT_W65816Call, + [SDNPHasChain, SDNPOutGlue, SDNPOptInGlue, + SDNPVariadic]>; def W65816callseq_start : SDNode<"ISD::CALLSEQ_START", SDT_W65816CallSeqStart, @@ -141,6 +146,22 @@ def W65816stPtrOff : SDNode<"W65816ISD::ST_PTR_OFF", SDT_W65816StPtrOff, def W65816stbPtrOff : SDNode<"W65816ISD::STB_PTR_OFF", SDT_W65816StPtrOff, [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>; +// `Idx` siblings: the deref of `ptr + zext(idx) + off`, with idx + off in +// Y. LowerLoad / LowerStore form them only when idx + off provably fits +// 16 bits, so Y never wraps where the i32 add would have carried. +def SDT_W65816LdPtrIdx + : SDTypeProfile<1, 3, [SDTCisVT<0, i16>, SDTCisVT<1, i32>, + SDTCisVT<2, i16>, SDTCisVT<3, i16>]>; +def SDT_W65816StPtrIdx + : SDTypeProfile<0, 4, [SDTCisVT<0, i16>, SDTCisVT<1, i32>, + SDTCisVT<2, i16>, SDTCisVT<3, i16>]>; +def W65816ldPtrIdx : SDNode<"W65816ISD::LD_PTR_IDX", SDT_W65816LdPtrIdx, + [SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>; +def W65816stPtrIdx : SDNode<"W65816ISD::ST_PTR_IDX", SDT_W65816StPtrIdx, + [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>; +def W65816stbPtrIdx : SDNode<"W65816ISD::STB_PTR_IDX", SDT_W65816StPtrIdx, + [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>; + // Trap: produced by LowerTRAP for ISD::TRAP (__builtin_trap, // -fsanitize-trap=undefined). Pure side-effect node, takes the // chain in and threads it out — selected from the (W65816trap) @@ -861,6 +882,17 @@ let mayLoad = 1, hasSideEffects = 0, mayStore = 0, def LDAfi : W65816Pseudo<(outs Acc16:$dst), (ins memfi:$addr), "# LDAfi $dst, $addr", []>; } +// A = an IMG register's DP word, as a real instruction instead of a COPY: +// W65816StackSlotCleanup forwards a temp slot's IMG word to its reload +// with it. The reload's A def is often dead in name only (the STA_DP $E0 +// that consumes it names no register), and a COPY whose defs are dead is +// deleted at expansion; the explicit IMG use keeps that register's def +// in place for the post-RA passes. expandPostRAPseudo lowers it to +// LDA_DP. +let mayLoad = 1, hasSideEffects = 1, mayStore = 0 in { +def LDAimg : W65816Pseudo<(outs Acc16:$dst), (ins Img16:$src), + "# LDAimg $dst, $src", []>; +} // STAfi accepts Wide16 src so greedy can park the value in IMGn instead // of A. When src is in IMGn (or X/Y after class coalescing), eliminate- // FrameIndex emits a PHA-bracketed sequence (`pha; lda dp; sta d+2,s; pla`) @@ -872,6 +904,12 @@ def LDAfi : W65816Pseudo<(outs Acc16:$dst), (ins memfi:$addr), // A's was expected — observed as `dadd(1.5,2.5) → 0x4010_0000_3000_3000` // under full IMG-clobber. // +// The over-approximation must stay off the STAfis the register allocator +// itself inserts (storeRegToStackSlot drops it): the spiller places them +// without checking A's interference, so a spill can land after `$a = COPY +// $img0` with A still live into the next block, and MachineCopyPropagation +// then deleted that copy as overwritten. +// // Note: Defs = [A] triggers a greedy-regalloc assertion failure // (LiveRangeEdit::eliminateDeadDef on a KILL pseudo with non-dead // implicit-def $a) on functions with many cross-call Acc16 vregs @@ -1164,7 +1202,7 @@ def STAfi_indY : W65816Pseudo<(outs), (ins Acc16:$src, memfi:$addr), "# STAfi_indY $src, $addr", []>; } -// Atomic i8 (byte) variant of STAfi_indY. Emitted by the STBptr32S +// Atomic i8 (byte) variant of STAfi_indY. Emitted by the STBptr32 // inserter for Layer-2 (d,S),Y byte stores; eliminateFrameIndex expands // it with a SEP #$20 / REP #$20 wrap so only one byte is written. Kept a // single pseudo through register allocation (same reason as STB_DPIndLongY @@ -1290,6 +1328,23 @@ def STBptr32Off : W65816Pseudo<(outs), (ins Acc16:$val, AnyWide32:$ptr, i16imm:$off), "# STBptr32Off $val, $ptr, $off", []>; } +let usesCustomInserter = 1, hasSideEffects = 1, mayLoad = 1, + Defs = [Y, P] in { +def LDAptr32Idx : W65816Pseudo<(outs Acc16:$dst), + (ins AnyWide32:$ptr, Wide16:$idx, i16imm:$off), + "# LDAptr32Idx $dst, $ptr, $idx, $off", []>; +} +let usesCustomInserter = 1, hasSideEffects = 1, mayStore = 1, + Defs = [Y, P] in { +def STAptr32Idx : W65816Pseudo<(outs), + (ins Acc16:$val, AnyWide32:$ptr, Wide16:$idx, + i16imm:$off), + "# STAptr32Idx $val, $ptr, $idx, $off", []>; +def STBptr32Idx : W65816Pseudo<(outs), + (ins Acc16:$val, AnyWide32:$ptr, Wide16:$idx, + i16imm:$off), + "# STBptr32Idx $val, $ptr, $idx, $off", []>; +} // Direct ptr32 load/store patterns over generic ISD::LOAD / ISD::STORE // when the address is an i32 (AnyWide32) reg. These are unreachable @@ -1326,29 +1381,16 @@ def : Pat<(W65816stPtrOff Acc16:$val, AnyWide32:$ptr, (i16 timm:$off)), (STAptr32Off Acc16:$val, AnyWide32:$ptr, imm:$off)>; def : Pat<(W65816stbPtrOff Acc16:$val, AnyWide32:$ptr, (i16 timm:$off)), (STBptr32Off Acc16:$val, AnyWide32:$ptr, imm:$off)>; - -// Split-pair variants: same semantics as LDAptr32/STAptr32/STBptr32 but -// the ptr is two separate i16 register operands (lo + hi) instead of -// one Wide32 register pair. Used by the W65816LowerWide32 pre-RA pass -// to relieve register-pair allocation pressure: it walks REG_SEQUENCE -// + LDAptr32 chains, decomposes the Wide32 vregs into pairs of i16 -// vregs, and rewrites the LDAptr32-family to take the two halves -// directly. -let usesCustomInserter = 1, hasSideEffects = 1, mayLoad = 1, - Defs = [Y, P] in { -def LDAptr32S : W65816Pseudo<(outs Acc16:$dst), - (ins Wide16:$ptrLo, Wide16:$ptrHi), - "# LDAptr32S $dst, $ptrLo, $ptrHi", []>; -} -let usesCustomInserter = 1, hasSideEffects = 1, mayStore = 1, - Defs = [Y, P] in { -def STAptr32S : W65816Pseudo<(outs), - (ins Acc16:$val, Wide16:$ptrLo, Wide16:$ptrHi), - "# STAptr32S $val, $ptrLo, $ptrHi", []>; -def STBptr32S : W65816Pseudo<(outs), - (ins Acc16:$val, Wide16:$ptrLo, Wide16:$ptrHi), - "# STBptr32S $val, $ptrLo, $ptrHi", []>; -} +// Idx variants: a zero-extended 16-bit index folded into Y with the +// offset (see LowerLoad's peelPtr32Index). +def : Pat<(W65816ldPtrIdx AnyWide32:$ptr, Wide16:$idx, (i16 timm:$off)), + (LDAptr32Idx AnyWide32:$ptr, Wide16:$idx, imm:$off)>; +def : Pat<(W65816stPtrIdx Acc16:$val, AnyWide32:$ptr, Wide16:$idx, + (i16 timm:$off)), + (STAptr32Idx Acc16:$val, AnyWide32:$ptr, Wide16:$idx, imm:$off)>; +def : Pat<(W65816stbPtrIdx Acc16:$val, AnyWide32:$ptr, Wide16:$idx, + (i16 timm:$off)), + (STBptr32Idx Acc16:$val, AnyWide32:$ptr, Wide16:$idx, imm:$off)>; // i8 load via Acc16 pointer producing a true i8 (Acc8) result. Reuses // the existing zextloadi8 16-bit-LDA-and-mask path: load 2 bytes, mask @@ -1977,7 +2019,11 @@ def : Pat<(store // regalloc-side patch (taught to never spill between sub-reg defs of // the same parent vreg) or a backend-side restructure of i64-arg // passing to use stack slots directly instead of register pairs. +// +// A call that can return twice (setjmp) gets IMG8..IMG15 as implicit defs +// on top (AdjustInstrPostInstrSelection): see W65816ISelLowering.cpp. let isCall = 1, hasSideEffects = 0, mayLoad = 0, mayStore = 0, + hasPostISelHook = 1, Defs = [A, X, Y, DPF0, IMG0, IMG1, IMG2, IMG3, IMG4, IMG5, IMG6, IMG7] in { def JSLpseudo : W65816Pseudo<(outs), (ins i16imm:$dst), @@ -1995,3 +2041,23 @@ def : Pat<(W65816call (i16 texternalsym:$dst)), (JSLpseudo texternalsym:$dst)>; // cRELOC opcode rewrites the offset and bank at load time. def : Pat<(W65816call (i32 tglobaladdr:$dst)), (JSLpseudo32 tglobaladdr:$dst)>; def : Pat<(W65816call (i32 texternalsym:$dst)), (JSLpseudo32 texternalsym:$dst)>; + +// A call to one of libgcc.s's hand-written helpers (16/32-bit multiply, +// divide and shift, memcpy / memmove / memset): each touches only A, X, +// Y, the stack, DBR (saved and restored) and the $E0..$EF scratch, plus +// $F0 for the signed 32-bit divides -- never IMG0..IMG7. Leaving them +// out of Defs lets a value stay in an IMG register across the call +// instead of being spilled (every multiply in a loop used to force the +// loop's live values onto the stack). W65816ISelLowering::isLightCallee +// holds the list; a helper that starts using an IMG word must leave it. +let isCall = 1, hasSideEffects = 0, mayLoad = 0, mayStore = 0, + Defs = [A, X, Y, DPF0] in { +def JSLlight : W65816Pseudo<(outs), (ins i16imm:$dst), + "# JSLlight $dst", []>; +def JSLlight32 : W65816Pseudo<(outs), (ins i32imm:$dst), + "# JSLlight32 $dst", []>; +} +def : Pat<(W65816callLight (i16 tglobaladdr:$dst)), (JSLlight tglobaladdr:$dst)>; +def : Pat<(W65816callLight (i16 texternalsym:$dst)), (JSLlight texternalsym:$dst)>; +def : Pat<(W65816callLight (i32 tglobaladdr:$dst)), (JSLlight32 tglobaladdr:$dst)>; +def : Pat<(W65816callLight (i32 texternalsym:$dst)), (JSLlight32 texternalsym:$dst)>; diff --git a/src/llvm/lib/Target/W65816/W65816LowerWide32.cpp b/src/llvm/lib/Target/W65816/W65816LowerWide32.cpp index afc2d9d..9e716e8 100644 --- a/src/llvm/lib/Target/W65816/W65816LowerWide32.cpp +++ b/src/llvm/lib/Target/W65816/W65816LowerWide32.cpp @@ -11,9 +11,7 @@ // functions (`RegAllocBase` crashes during `allocatePhysRegs`) because // the i64-via-2-i32-via-Wide32 chain produces too many simultaneously // live register-pair vregs. After this pass, only i16 vregs remain at -// the regalloc input — Wide32 lives only inside this pass and the new -// LDAptr32S / STAptr32S / STBptr32S pseudos that take 2 i16 ptr operands -// directly. +// the regalloc input. // // Walks the MIR and: // 1. Finds REG_SEQUENCE producing Wide32 / Acc32 / AnyWide32; records @@ -23,9 +21,11 @@ // mapped Wide32 vreg; chains the (lo, hi) mapping forward. // 3. Rewrites EXTRACT_SUBREG of mapped Wide32 vregs by replacing the // destination vreg with the appropriate half (sub_lo or sub_hi). -// 4. Rewrites LDAptr32 / STAptr32 / STBptr32 with a mapped Wide32 ptr -// to the corresponding LDAptr32S / STAptr32S / STBptr32S pseudo -// with two separate i16 operands. +// +// The far-pointer pseudos (LDAptr32 / STAptr32 / STBptr32) never reach +// this pass: they have custom inserters, which finalize-isel runs right +// after instruction selection, and those take their Wide32 pointer apart +// with sub-register COPYs (rewritten in step 3's partial-COPY case). // // Bail / safety: any Wide32 vreg whose def we can't decompose is left // in place — regalloc may still struggle but no miscompile. @@ -343,53 +343,19 @@ bool W65816LowerWide32::runOnMachineFunction(MachineFunction &MF) { auto [Lo, Hi] = wideMap[SrcOp.getReg()]; Register Half = (SubIdx == llvm::sub_lo) ? Lo : Hi; forwardHalf(MI, DstOp.getReg(), Half); - continue; - } - } - - // LDAptr32 / STAptr32 / STBptr32 with a mapped Wide32 ptr: - // rewrite to LDAptr32S / STAptr32S / STBptr32S. - unsigned Opc = MI->getOpcode(); - bool isPtrOp = (Opc == W65816::LDAptr32 || Opc == W65816::STAptr32 || - Opc == W65816::STBptr32); - if (isPtrOp) { - Register Ptr = MI->getOperand(1).getReg(); - if (Ptr.isVirtual() && wideMap.count(Ptr)) { - Register Lo = current(wideMap[Ptr].first); - Register Hi = current(wideMap[Ptr].second); - unsigned NewOpc = (Opc == W65816::LDAptr32) ? W65816::LDAptr32S - : (Opc == W65816::STAptr32) ? W65816::STAptr32S - : W65816::STBptr32S; - DebugLoc DL = MI->getDebugLoc(); - MachineBasicBlock *ParentMBB = MI->getParent(); - if (Opc == W65816::LDAptr32) { - Register Dst = MI->getOperand(0).getReg(); - BuildMI(*ParentMBB, MI->getIterator(), DL, TII.get(NewOpc), Dst) - .addReg(Lo) - .addReg(Hi); - } else { - Register Val = MI->getOperand(0).getReg(); - BuildMI(*ParentMBB, MI->getIterator(), DL, TII.get(NewOpc)) - .addReg(Val) - .addReg(Lo) - .addReg(Hi); - } - useToErase.push_back(MI); - continue; } } } } - // Erase use-side instructions (EXTRACT_SUBREG, LDAptr32-family) first + // Erase use-side instructions (EXTRACT_SUBREG, partial COPYs) first // so the Wide32 vreg becomes dead. // // DBG_VALUE preservation: MRI.replaceRegWith above already redirected // any debug operands to the new half-vreg (replaceRegWith walks // reg_operands which includes debug ops). No further work needed for // the EXTRACT_SUBREG / partial-COPY paths since the user-visible value - // is preserved in the half vreg. For LDAptr32 rewrites we don't move - // a user-named value so DBG_VALUE attribution is unaffected. + // is preserved in the half vreg. for (auto *MI : useToErase) MI->eraseFromParent(); diff --git a/src/llvm/lib/Target/W65816/W65816NarrowI32Mul.cpp b/src/llvm/lib/Target/W65816/W65816NarrowI32Mul.cpp index 490a4bc..d85059a 100644 --- a/src/llvm/lib/Target/W65816/W65816NarrowI32Mul.cpp +++ b/src/llvm/lib/Target/W65816/W65816NarrowI32Mul.cpp @@ -9,7 +9,10 @@ // IR-level peephole. Detects `mul i32 X, Y` where both X and Y have // their top 16 bits provably zero (via LLVM's IR-level computeKnownBits) // and rewrites to a call to `__umulhisi3` — a 16x16 -> 32 unsigned -// multiply (~30% faster than __mulsi3 for `(u32)i * i` patterns). +// multiply (~30% faster than __mulsi3 for `(u32)i * i` patterns) — or +// where both are sign-extended 16-bit values, to `__mulhisi3`, its +// signed sibling. A product that fits 16 bits itself becomes a 16-bit +// multiply, zero-extended. // // Why an IR pass instead of a Custom SDAG lowering: LLVM's IndVarSimplify // loop pass widens narrow induction variables (e.g. an i16 loop counter @@ -53,7 +56,7 @@ public: W65816NarrowI32Mul() : FunctionPass(ID) {} StringRef getPassName() const override { - return "W65816 narrow i32 multiplies to __umulhisi3"; + return "W65816 narrow i32 multiplies to __umulhisi3 / __mulhisi3"; } void getAnalysisUsage(AnalysisUsage &AU) const override { @@ -77,13 +80,13 @@ INITIALIZE_PASS_END(W65816NarrowI32Mul, DEBUG_TYPE, "W65816 narrow i32 multiplies", false, false) -// Get-or-declare `__umulhisi3(i16, i16) -> i32` in the module. -static FunctionCallee getUmulhisi3(Module &M) { +// Get-or-declare `Name(i16, i16) -> i32` (__umulhisi3 / __mulhisi3). +static FunctionCallee getMul16x16(Module &M, StringRef Name) { LLVMContext &Ctx = M.getContext(); Type *I16 = Type::getInt16Ty(Ctx); Type *I32 = Type::getInt32Ty(Ctx); FunctionType *FT = FunctionType::get(I32, {I16, I16}, false); - return M.getOrInsertFunction("__umulhisi3", FT); + return M.getOrInsertFunction(Name, FT); } @@ -105,13 +108,62 @@ static bool top16Zero(Value *V, const DataLayout &DL, ScalarEvolution &SE) { } +// True iff the product of two operands whose high words are zero fits +// 16 bits too: by the product's known bits, or by the operands' largest +// values (known bits and ScalarEvolution's unsigned ranges). +static bool productFitsU16(BinaryOperator *BO, const DataLayout &DL, + ScalarEvolution &SE) { + if (computeKnownBits(BO, DL).countMinLeadingZeros() >= 16) { + return true; + } + auto maxOf = [&](Value *V) { + APInt Max = computeKnownBits(V, DL).getMaxValue(); + if (SE.isSCEVable(V->getType())) { + Max = APIntOps::umin(Max, SE.getUnsignedRangeMax(SE.getSCEV(V))); + } + return Max; + }; + bool Overflow = false; + APInt Product = maxOf(BO->getOperand(0)).umul_ov(maxOf(BO->getOperand(1)), + Overflow); + return !Overflow && Product.getActiveBits() <= 16; +} + + +// True iff V is the sign extension of its low 16 bits: by known sign +// bits, or by ScalarEvolution's signed range. +static bool fitsS16(Value *V, const DataLayout &DL, ScalarEvolution &SE) { + if (ComputeNumSignBits(V, DL) >= 17) { + return true; + } + if (!SE.isSCEVable(V->getType())) { + return false; + } + ConstantRange R = SE.getSignedRange(SE.getSCEV(V)); + return R.getMinSignedBits() <= 16; +} + + bool W65816NarrowI32Mul::runOnFunction(Function &F) { Module *M = F.getParent(); const DataLayout &DL = M->getDataLayout(); Type *I16 = Type::getInt16Ty(F.getContext()); ScalarEvolution &SE = getAnalysis().getSE(); - SmallVector Worklist; + // Each multiply with the helper that computes it: __umulhisi3 when both + // operands' high words are zero, else __mulhisi3 when both are + // sign-extended 16-bit values (`(long)a * b` of two shorts -- CoreMark's + // matrix_mul_matrix, the dot products of fixed-point code), whose + // __mulsi3 call took the 32-step loop for every negative operand. + // + // A product that fits 16 bits itself is a 16-bit multiply, zero-extended: + // shifts and adds for a constant (`(u32)byte * 6`, a 6-byte record's + // offset), else __mulhi3, and a high word the rest of the code knows is + // zero. + static const bool NarrowProducts = + !getenv("W65816_DISABLE_NARROW_PRODUCT"); + enum class Kind { Unsigned, Signed, Product16 }; + SmallVector, 8> Worklist; for (Instruction &I : instructions(F)) { auto *BO = dyn_cast(&I); if (!BO || BO->getOpcode() != Instruction::Mul) { @@ -120,13 +172,16 @@ bool W65816NarrowI32Mul::runOnFunction(Function &F) { if (!BO->getType()->isIntegerTy(32)) { continue; } - if (!top16Zero(BO->getOperand(0), DL, SE)) { + if (top16Zero(BO->getOperand(0), DL, SE) && + top16Zero(BO->getOperand(1), DL, SE)) { + bool Fits = NarrowProducts && productFitsU16(BO, DL, SE); + Worklist.push_back({BO, Fits ? Kind::Product16 : Kind::Unsigned}); continue; } - if (!top16Zero(BO->getOperand(1), DL, SE)) { - continue; + if (fitsS16(BO->getOperand(0), DL, SE) && + fitsS16(BO->getOperand(1), DL, SE)) { + Worklist.push_back({BO, Kind::Signed}); } - Worklist.push_back(BO); } if (Worklist.empty()) { @@ -151,16 +206,22 @@ bool W65816NarrowI32Mul::runOnFunction(Function &F) { return B.CreateTrunc(V, I16); }; - FunctionCallee Callee = getUmulhisi3(*M); + FunctionCallee UCallee = getMul16x16(*M, "__umulhisi3"); + FunctionCallee SCallee = getMul16x16(*M, "__mulhisi3"); SmallPtrSet MaybeDeadSet; - for (BinaryOperator *BO : Worklist) { + for (auto [BO, K] : Worklist) { IRBuilder<> B(BO); Value *AOp = BO->getOperand(0); Value *BOp = BO->getOperand(1); Value *A = narrowOperand(AOp, B); Value *Bv = narrowOperand(BOp, B); - Value *Call = B.CreateCall(Callee, {A, Bv}); - BO->replaceAllUsesWith(Call); + Value *Result; + if (K == Kind::Product16) { + Result = B.CreateZExt(B.CreateNUWMul(A, Bv), BO->getType()); + } else { + Result = B.CreateCall(K == Kind::Signed ? SCallee : UCallee, {A, Bv}); + } + BO->replaceAllUsesWith(Result); BO->eraseFromParent(); // If the original operands were zext/sext nodes, they may now be // dead. Add to a SET — `mul %x, %x` would push the same node diff --git a/src/llvm/lib/Target/W65816/W65816PromoteFiToImg.cpp b/src/llvm/lib/Target/W65816/W65816PromoteFiToImg.cpp index a75b41a..bbf37d3 100644 --- a/src/llvm/lib/Target/W65816/W65816PromoteFiToImg.cpp +++ b/src/llvm/lib/Target/W65816/W65816PromoteFiToImg.cpp @@ -261,7 +261,8 @@ bool W65816PromoteFiToImg::runOnMachineFunctionDisabled(MachineFunction &MF) { else continue; // libgcc.s multiply/divide/shift helpers — verified to only use // $E0..$E9 internally, no IMG0..7 touch. - if (Name == "__umulhisi3" || Name == "__mulhi3" || + if (Name == "__umulhisi3" || Name == "__mulhisi3" || + Name == "__mulhi3" || Name == "__mulsi3" || Name == "__udivhi3" || Name == "__umodhi3" || Name == "__divhi3" || Name == "__modhi3" || Name == "__udivsi3" || diff --git a/src/llvm/lib/Target/W65816/W65816RegisterInfo.cpp b/src/llvm/lib/Target/W65816/W65816RegisterInfo.cpp index 9d33620..775198c 100644 --- a/src/llvm/lib/Target/W65816/W65816RegisterInfo.cpp +++ b/src/llvm/lib/Target/W65816/W65816RegisterInfo.cpp @@ -49,6 +49,18 @@ static int imgRegToDP(Register R) { } } +// Whether A's value may still be read after MI, before something +// overwrites it: whether an LDAfi / STAfi that has to move a non-A +// register's value through A must save A around itself. Spill and reload +// code is inserted where the allocator has a value of its own in A (a +// compare's operand kept across a reloaded pointer's copy), so the old +// fixed `lda d,s ; sta dp` reload clobbered it, and the PHA / PLA every +// non-A spill paid is needed only here. +static bool aLiveAfter(const MachineInstr &MI) { + return W65816Helpers::aLiveFrom(*MI.getParent(), + std::next(MI.getIterator())); +} + // Far FI elim via DP frame-pointer ($F6/$F7). Called when an FI's // effective offset exceeds 0xFF and the function reserved an FP at // prologue time (StackSize > 200). Stack is always bank 0, so @@ -69,6 +81,14 @@ static bool expandFarFI(MachineInstr &MI, int FPOff, switch (Opc) { case W65816::LDAfi: { Register Dst = MI.getOperand(0).getReg(); + // A non-A destination is reached through A (an IMG one by `sta dp` + // after the load, which this path used to leave out: the reload landed + // in A only). Save A around it when A is read afterwards. + int DstDP = imgRegToDP(Dst); + bool SaveA = Dst != W65816::A && aLiveAfter(MI); + if (SaveA) + BuildMI(MBB, II, DL, TII.get(W65816::PHA)) + .addReg(W65816::A, RegState::Implicit); // Mark Y use as Undef: if Y is dead at this insertion point, the // value we save is "don't care" — we restore the same garbage byte // later. Without Undef, the verifier rejects when no def reaches @@ -89,6 +109,11 @@ static bool expandFarFI(MachineInstr &MI, int FPOff, BuildMI(MBB, II, DL, TII.get(W65816::TAX)); else if (Dst == W65816::Y) BuildMI(MBB, II, DL, TII.get(W65816::TAY)); + else if (DstDP >= 0) + BuildMI(MBB, II, DL, TII.get(W65816::STA_DP)).addImm(DstDP); + if (SaveA) + BuildMI(MBB, II, DL, TII.get(W65816::PLA)) + .addReg(W65816::A, RegState::ImplicitDefine); return true; } case W65816::STAfi: { @@ -101,7 +126,7 @@ static bool expandFarFI(MachineInstr &MI, int FPOff, // Without it here, an IMG-source STAfi in a >0xFF frame silently // clobbered A and the cleanup then deleted the *following* reload, // so the next `sta $E0` parked the source's value where the caller - // wanted something else. Live symptom: the LDAptr32S inserter + // wanted something else. Live symptom: the LDAptr32 inserter // built its far pointer with the pointer's BANK byte in the offset // half, and `lda [$E0],y` read a stray address -- Space Taxi's // stSimTick read a nonzero byte for sim->collisionPhase and never @@ -248,6 +273,8 @@ static bool expandFarFI(MachineInstr &MI, int FPOff, #define GET_REGINFO_TARGET_DESC #include "W65816GenRegisterInfo.inc" + + W65816RegisterInfo::W65816RegisterInfo() : W65816GenRegisterInfo(W65816::PC) {} const MCPhysReg * @@ -345,9 +372,15 @@ bool W65816RegisterInfo::eliminateFrameIndex(MachineBasicBlock::iterator II, } report_fatal_error("W65816: frame offset out of stack-relative range"); } + // A non-A destination is reached through A: save A around it when A + // is still read afterwards (see aLiveAfter). + bool SaveA = Dst != W65816::A && aLiveAfter(MI); + if (SaveA) + BuildMI(*MI.getParent(), II, MI.getDebugLoc(), TII.get(W65816::PHA)) + .addReg(W65816::A, RegState::Implicit); BuildMI(*MI.getParent(), II, MI.getDebugLoc(), TII.get(W65816::LDA_StackRel)) - .addImm(Offset) + .addImm(SaveA ? Offset + 2 : Offset) // PHA moved S down 2 .addReg(W65816::A, RegState::ImplicitDefine); if (Dst == W65816::X) { BuildMI(*MI.getParent(), II, MI.getDebugLoc(), TII.get(W65816::TAX)); @@ -387,6 +420,9 @@ bool W65816RegisterInfo::eliminateFrameIndex(MachineBasicBlock::iterator II, TII.get(W65816::STA_DP)).addImm(dstDP); } } + if (SaveA) + BuildMI(*MI.getParent(), II, MI.getDebugLoc(), TII.get(W65816::PLA)) + .addReg(W65816::A, RegState::ImplicitDefine); MI.eraseFromParent(); return true; } @@ -446,7 +482,25 @@ bool W65816RegisterInfo::eliminateFrameIndex(MachineBasicBlock::iterator II, // 16-bit load of the extra byte is a harmless read (the high byte of an i8 value is ignored), so // LDAfi is left 16-bit. The load side never corrupts anything. bool ByteSlot = MFI.getObjectSize(FI) == 1; - if (srcDP >= 0 || Src == W65816::X || Src == W65816::Y) { + if ((srcDP >= 0 || Src == W65816::X || Src == W65816::Y) && + !aLiveAfter(MI)) { + // A is overwritten before anything reads it: no need to keep it. + if (srcDP >= 0) { + BuildMI(*MI.getParent(), II, MI.getDebugLoc(), + TII.get(W65816::LDA_DP)).addImm(srcDP); + } else { + unsigned XferOp = (Src == W65816::X) ? W65816::TXA : W65816::TYA; + BuildMI(*MI.getParent(), II, MI.getDebugLoc(), TII.get(XferOp)); + } + if (ByteSlot) + BuildMI(*MI.getParent(), II, MI.getDebugLoc(), TII.get(W65816::SEP)).addImm(0x20); + BuildMI(*MI.getParent(), II, MI.getDebugLoc(), + TII.get(W65816::STA_StackRel)) + .addImm(Offset) + .addReg(W65816::A, RegState::Implicit); + if (ByteSlot) + BuildMI(*MI.getParent(), II, MI.getDebugLoc(), TII.get(W65816::REP)).addImm(0x20); + } else if (srcDP >= 0 || Src == W65816::X || Src == W65816::Y) { // STAfi with non-A source: must clobber A to land the value in // A and then `sta d,s`. PHA-bracket so A's incoming value is // preserved across the spill — without this, a regalloc-emitted diff --git a/src/llvm/lib/Target/W65816/W65816ReturnPopMerge.cpp b/src/llvm/lib/Target/W65816/W65816ReturnPopMerge.cpp new file mode 100644 index 0000000..3be51cd --- /dev/null +++ b/src/llvm/lib/Target/W65816/W65816ReturnPopMerge.cpp @@ -0,0 +1,325 @@ +//===-- W65816ReturnPopMerge.cpp - Merge stack releases into returns -----===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===---------------------------------------------------------------------===// +// +// Pre-emit: a path that ends in a return releases its stack in pieces. +// The last call's argument cleanup (N/2 PLY, or TSC/CLC/ADC/TCS) runs, +// then the epilogue's own frame release, often in a shared return block +// that branch folding tail-merged the two into, behind a BRA: +// +// jsl f jsl f +// ply tsc +// ply => clc +// bra .Lret adc #10 +// .Lret: tcs +// ply rtl +// ply +// ply +// rtl +// +// Here 25 cycles of pulls and 3 of branch become 9. For every block whose +// path to a return is nothing but stack releases (in the block's tail, +// then through blocks made only of releases and reached by fall-through +// or an unconditional branch), add the bytes up and, when one release +// sized for the total is cheaper, give the block its own copy of it and +// of the return. The release keeps what the return reads (A, X, Y by its +// implicit uses) and picks its form by W65816Helpers::emitStackRelease; +// flags are dead at a return. Return blocks left without predecessors +// are deleted. +// +// Kill switch: W65816_DISABLE_RETURN_POP_MERGE=1. +// +//===---------------------------------------------------------------------===// + +#include "W65816.h" +#include "W65816InstrInfo.h" +#include "W65816Subtarget.h" +#include "llvm/ADT/SmallPtrSet.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/CodeGen/MachineFunctionPass.h" +#include "llvm/CodeGen/MachineInstrBuilder.h" + +using namespace llvm; + +#define DEBUG_TYPE "w65816-return-pop-merge" + +namespace { + +class W65816ReturnPopMerge : public MachineFunctionPass { +public: + static char ID; + W65816ReturnPopMerge() : MachineFunctionPass(ID) {} + bool runOnMachineFunction(MachineFunction &MF) override; + StringRef getPassName() const override { + return "W65816 merge stack releases into returns"; + } +}; + +char W65816ReturnPopMerge::ID = 0; + +// What the return reads, which a merged release must keep. +struct KeepSet { + bool A = true; + bool X = true; + bool Y = true; +}; + +// A run of stack releases ending at a block position. +struct ReleaseRun { + MachineBasicBlock::iterator Begin; + int64_t Bytes = 0; + unsigned Cost = 0; +}; + +bool isImm(const MachineInstr &MI, unsigned Opc, int64_t Imm) { + return MI.getOpcode() == Opc && MI.getOperand(0).isImm() && + MI.getOperand(0).getImm() == Imm; +} + +// The TSC / CLC / ADC #k / TCS core ending at TCS (It), with its first +// instruction in Begin and k in Bytes. +bool matchAdcCore(MachineBasicBlock &MBB, MachineBasicBlock::iterator It, + MachineBasicBlock::iterator &Begin, int64_t &Bytes) { + if (It->getOpcode() != W65816::TCS || It == MBB.begin()) + return false; + MachineBasicBlock::iterator Adc = std::prev(It); + if (Adc->getOpcode() != W65816::ADC_Imm16 || !Adc->getOperand(0).isImm() || + Adc == MBB.begin()) + return false; + MachineBasicBlock::iterator Clc = std::prev(Adc); + if (Clc->getOpcode() != W65816::CLC || Clc == MBB.begin()) + return false; + MachineBasicBlock::iterator Tsc = std::prev(Clc); + if (Tsc->getOpcode() != W65816::TSC) + return false; + Begin = Tsc; + Bytes = Adc->getOperand(0).getImm(); + // A release, not an allocation written as a wrapping add. + return Bytes > 0 && Bytes < 0x8000; +} + +// Walk back from End over the stack releases emitStackRelease and the +// call cleanup produce: PLY / PLX (Y / X not kept), TSC/CLC/ADC/TCS, +// and that bracketed by TAY/TYA, TAX/TXA or STA $E0 / LDA $E0, costed +// by W65816Helpers::cycleByteCost as emitStackRelease's choices are. +ReleaseRun scanRunBack(MachineBasicBlock &MBB, MachineBasicBlock::iterator End, + const KeepSet &Keep) { + ReleaseRun Run; + Run.Begin = End; + while (Run.Begin != MBB.begin()) { + MachineBasicBlock::iterator It = std::prev(Run.Begin); + unsigned Opc = It->getOpcode(); + if ((Opc == W65816::PLY && !Keep.Y) || (Opc == W65816::PLX && !Keep.X)) { + Run.Begin = It; + Run.Bytes += 2; + Run.Cost += W65816Helpers::cycleByteCost(5, 1); + continue; + } + MachineBasicBlock::iterator Begin; + int64_t Bytes; + if (matchAdcCore(MBB, It, Begin, Bytes)) { + Run.Begin = Begin; + Run.Bytes += Bytes; + Run.Cost += W65816Helpers::cycleByteCost(9, 6); + continue; + } + // The bracket's two instructions: 4 cycles and 2 bytes for the + // transfers, 8 and 4 for the direct-page store and load. + unsigned Open; + unsigned Cycles; + unsigned Size; + if (Opc == W65816::TYA) { + Open = W65816::TAY; + Cycles = 9 + 4; + Size = 6 + 2; + } else if (Opc == W65816::TXA) { + Open = W65816::TAX; + Cycles = 9 + 4; + Size = 6 + 2; + } else if (isImm(*It, W65816::LDA_DP, 0xE0)) { + Open = W65816::STA_DP; + Cycles = 9 + 8; + Size = 6 + 4; + } else { + break; + } + if (It == MBB.begin() || !matchAdcCore(MBB, std::prev(It), Begin, Bytes) || + Begin == MBB.begin()) + break; + MachineBasicBlock::iterator O = std::prev(Begin); + if (O->getOpcode() != Open || + (Open == W65816::STA_DP && !isImm(*O, W65816::STA_DP, 0xE0))) + break; + Run.Begin = O; + Run.Bytes += Bytes; + Run.Cost += W65816Helpers::cycleByteCost(Cycles, Size); + } + return Run; +} + +// Whether M is 16-bit at the start of a run: the nearest SEP / REP above +// it in the block touching M is not a SEP (blocks start in 16-bit M). +bool sixteenBitMAt(MachineBasicBlock &MBB, MachineBasicBlock::iterator It) { + while (It != MBB.begin()) { + --It; + unsigned Opc = It->getOpcode(); + if ((Opc == W65816::SEP || Opc == W65816::REP) && + It->getOperand(0).isImm() && (It->getOperand(0).getImm() & 0x20)) + return Opc == W65816::REP; + } + return true; +} + +// Cost of reaching the next block: the unconditional branch's, or none +// for a fall-through. +unsigned branchCost(const MachineInstr *Br) { + if (!Br) + return 0; + switch (Br->getOpcode()) { + case W65816::BRL: + return W65816Helpers::cycleByteCost(4, 3); + case W65816::JMP_Abs: + return W65816Helpers::cycleByteCost(3, 3); + default: + return W65816Helpers::cycleByteCost(3, 2); + } +} + +// The block's lone successor and the unconditional branch (or nullptr +// for a fall-through) that reaches it, when that is all its terminators +// do. Null when the block returns, branches conditionally or has other +// successors. +MachineBasicBlock *loneSuccessor(MachineBasicBlock &MBB, MachineInstr *&Br) { + Br = nullptr; + if (MBB.succ_size() != 1) + return nullptr; + MachineBasicBlock *S = *MBB.succ_begin(); + MachineBasicBlock::iterator T = MBB.getFirstTerminator(); + if (T == MBB.end()) + return MBB.isLayoutSuccessor(S) ? S : nullptr; + if (std::next(T) != MBB.end()) + return nullptr; + unsigned Opc = T->getOpcode(); + if (Opc != W65816::BRA && Opc != W65816::BRL && Opc != W65816::JMP_Abs) + return nullptr; + if (!T->getOperand(0).isMBB() || T->getOperand(0).getMBB() != S) + return nullptr; + Br = &*T; + return S; +} + +KeepSet keepFor(const MachineInstr &Ret) { + const TargetRegisterInfo *TRI = + Ret.getMF()->getSubtarget().getRegisterInfo(); + KeepSet Keep; + Keep.A = Ret.readsRegister(W65816::A, TRI); + Keep.X = Ret.readsRegister(W65816::X, TRI); + Keep.Y = Ret.readsRegister(W65816::Y, TRI); + return Keep; +} + +MachineInstr *returnOf(MachineBasicBlock &MBB) { + MachineBasicBlock::iterator T = MBB.getFirstTerminator(); + if (T == MBB.end() || std::next(T) != MBB.end() || + T->getOpcode() != W65816::RTL) + return nullptr; + return &*T; +} + +} // namespace + +bool W65816ReturnPopMerge::runOnMachineFunction(MachineFunction &MF) { + if (getenv("W65816_DISABLE_RETURN_POP_MERGE")) + return false; + const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo(); + bool Changed = false; + SmallVector Orphans; + + for (MachineBasicBlock &MBB : MF) { + // The release-only chain from MBB to a return, if any. + MachineInstr *Ret = returnOf(MBB); + SmallVector Chain; + SmallVector Branches; + if (!Ret) { + MachineBasicBlock *B = &MBB; + while (!Ret && Chain.size() < 4) { + MachineInstr *Br; + MachineBasicBlock *S = loneSuccessor(*B, Br); + if (!S || S == &MBB || S->isEHPad() || S->hasAddressTaken()) + break; + Chain.push_back(S); + Branches.push_back(Br); + Ret = returnOf(*S); + B = S; + } + if (!Ret) + continue; + } + KeepSet Keep = keepFor(*Ret); + int64_t Bytes = 0; + unsigned OldCost = 0; + bool Pure = true; + for (unsigned I = 0; I < Chain.size(); ++I) { + MachineBasicBlock *S = Chain[I]; + ReleaseRun Run = scanRunBack(*S, S->getFirstTerminator(), Keep); + if (Run.Begin != S->begin()) { + Pure = false; + break; + } + Bytes += Run.Bytes; + OldCost += Run.Cost + branchCost(Branches[I]); + } + if (!Pure) + continue; + MachineBasicBlock::iterator End = MBB.getFirstTerminator(); + ReleaseRun Own = scanRunBack(MBB, End, Keep); + if (!sixteenBitMAt(MBB, Own.Begin)) + continue; + Bytes += Own.Bytes; + OldCost += Own.Cost; + // A block of its own pays one byte for its copy of the return. + unsigned NewCost = W65816Helpers::stackReleaseCost(Bytes, Keep.A, Keep.X, + Keep.Y) + + (Chain.empty() ? 0 : 1); + if (NewCost >= OldCost) + continue; + // MBB's tail from the run on goes (its branch, or its own return), + // replaced by the one release and a copy of the return. + DebugLoc DL = Ret->getDebugLoc(); + MachineInstr *NewRet = MF.CloneMachineInstr(Ret); + MBB.erase(Own.Begin, MBB.end()); + W65816Helpers::emitStackRelease(MBB, MBB.end(), DL, TII, Bytes, Keep.A, + Keep.X, Keep.Y); + MBB.insert(MBB.end(), NewRet); + if (!Chain.empty()) { + MBB.removeSuccessor(Chain.front()); + Orphans.push_back(Chain.front()); + } + Changed = true; + } + + // Delete the chain blocks nothing reaches any more. + SmallPtrSet Erased; + while (!Orphans.empty()) { + MachineBasicBlock *B = Orphans.pop_back_val(); + if (Erased.count(B) || !B->pred_empty() || B == &MF.front() || + B->hasAddressTaken() || B->isEHPad()) + continue; + Erased.insert(B); + SmallVector Succs(B->successors()); + for (MachineBasicBlock *S : Succs) { + B->removeSuccessor(S); + Orphans.push_back(S); + } + B->eraseFromParent(); + } + return Changed; +} + +FunctionPass *llvm::createW65816ReturnPopMerge() { + return new W65816ReturnPopMerge(); +} diff --git a/src/llvm/lib/Target/W65816/W65816SinkLoopInc.cpp b/src/llvm/lib/Target/W65816/W65816SinkLoopInc.cpp new file mode 100644 index 0000000..3069e34 --- /dev/null +++ b/src/llvm/lib/Target/W65816/W65816SinkLoopInc.cpp @@ -0,0 +1,274 @@ +//===-- W65816SinkLoopInc.cpp - Sink loop-carried increments -------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// Pre-RA, SSA: move a loop-carried increment below the last use of the +// value it increments, within its block. +// +// `while (p != end) s += *p++;` reaches register allocation as +// +// %p = PHI %p0, %bb.pre, %next, %bb.loop +// %next = ADCi16imm %p, 1 ; low half (the scheduler puts +// %nh = ADCEi16imm %ph, 0 ; high half the step first) +// ... load through %p:%ph ... +// ... compare %next:%nh with end ... +// +// %p is still live when %next is defined, so the coalescer cannot give the +// two one home: PHI elimination's back-edge copy (`%p = COPY %next`, four +// instructions for a far pointer) stays in every trip, and the step cannot +// become an in-place `inc dp ; bne ; inc dp+2` (W65816I32IncFold). Moved +// below the last use of %p, the step starts where %p ends. +// +// Conditions: the step is ADCi16imm / SBCi16imm by a constant, or INA / +// DEA, whose input is a PHI that takes the step's result back from this +// block or one it runs straight into; an ADCE / SBCE right behind it (the high half, reading the carry) +// moves with it and must have the same shape. Nothing between the step +// and its new place may read the step's result. The step lands after the +// old value's last use -- counting a far-pointer access that goes through +// a stack temporary or $E0 / $E2 staged from it -- and after any physical +// A / X / Y live range open there, and only where the flags are dead (the +// steps set them). +// +// Also an i16 counter's `i++` whose old value is read after it (an index, +// a call argument) stops overlapping its successor: Lua's luaV_execute +// allocates with greedy again (it needed the basic-allocator fallback). +// +//===----------------------------------------------------------------------===// + +#include "W65816.h" +#include "W65816InstrInfo.h" +#include "W65816Subtarget.h" +#include "llvm/CodeGen/MachineFunctionPass.h" +#include "llvm/CodeGen/MachineRegisterInfo.h" +#include "llvm/Support/Debug.h" + +using namespace llvm; + +#define DEBUG_TYPE "w65816-sink-loop-inc" + +namespace { + +class W65816SinkLoopInc : public MachineFunctionPass { +public: + static char ID; + W65816SinkLoopInc() : MachineFunctionPass(ID) {} + StringRef getPassName() const override { + return "W65816 sink loop-carried increments"; + } + void getAnalysisUsage(AnalysisUsage &AU) const override { + AU.setPreservesCFG(); + MachineFunctionPass::getAnalysisUsage(AU); + } + bool runOnMachineFunction(MachineFunction &MF) override; +}; + +} // namespace + +char W65816SinkLoopInc::ID = 0; + +INITIALIZE_PASS(W65816SinkLoopInc, DEBUG_TYPE, + "W65816 sink loop-carried increments", false, false) + +FunctionPass *llvm::createW65816SinkLoopInc() { + return new W65816SinkLoopInc(); +} + +// The step `Def = OP Cur, imm` (OP one of Opcs): Cur, or an invalid +// register when MI is not one or Cur is not a PHI taking Def back from MI's +// block. +static Register stepInput(const MachineInstr &MI, + std::initializer_list Opcs, + const MachineRegisterInfo &MRI) { + bool IncDec = MI.getOpcode() == W65816::INA_PSEUDO || + MI.getOpcode() == W65816::DEA_PSEUDO; + if (!llvm::is_contained(Opcs, MI.getOpcode()) || MI.getNumOperands() < 2 || + !MI.getOperand(0).isReg() || !MI.getOperand(1).isReg() || + (!IncDec && (MI.getNumOperands() < 3 || !MI.getOperand(2).isImm()))) + return Register(); + Register Def = MI.getOperand(0).getReg(); + Register Cur = MI.getOperand(1).getReg(); + if (!Def.isVirtual() || !Cur.isVirtual()) + return Register(); + const MachineInstr *Phi = MRI.getVRegDef(Cur); + if (!Phi || !Phi->isPHI()) + return Register(); + // The back edge may leave from a block MI's block runs straight into: + // a latch whose test was split in two (`while (*p && state != k)`, + // CoreMark's core_state_transition) ends MI's block on the first test + // and the loop on the second. + auto reachedFromMI = [&](const MachineBasicBlock *B) { + for (unsigned Hops = 0; B && Hops < 3; ++Hops) { + if (B == MI.getParent()) + return true; + B = B->pred_size() == 1 ? *B->pred_begin() : nullptr; + } + return false; + }; + for (unsigned I = 1; I + 1 < Phi->getNumOperands(); I += 2) + if (Phi->getOperand(I).getReg() == Def && + reachedFromMI(Phi->getOperand(I + 1).getMBB())) + return Cur; + return Register(); +} + +static bool readsP(const MachineInstr &MI) { + return MI.readsRegister(W65816::P, /*TRI=*/nullptr); +} + +static bool definesP(const MachineInstr &MI) { + return MI.definesRegister(W65816::P, /*TRI=*/nullptr); +} + +bool W65816SinkLoopInc::runOnMachineFunction(MachineFunction &MF) { + if (skipFunction(MF.getFunction())) + return false; + MachineRegisterInfo &MRI = MF.getRegInfo(); + if (!MRI.isSSA()) + return false; + bool Changed = false; + for (MachineBasicBlock &MBB : MF) { + SmallVector Steps; + for (MachineInstr &MI : MBB) + if (stepInput(MI, + {W65816::ADCi16imm, W65816::SBCi16imm, W65816::INA_PSEUDO, + W65816::DEA_PSEUDO}, + MRI)) + Steps.push_back(&MI); + for (MachineInstr *Step : Steps) { + // The group: the step, and the high half reading its carry. + SmallVector Group{Step}; + SmallVector Olds{Step->getOperand(1).getReg()}; + SmallVector News{Step->getOperand(0).getReg()}; + auto Next = std::next(Step->getIterator()); + while (Next != MBB.end() && Next->isDebugInstr()) + ++Next; + if (Next != MBB.end() && readsP(*Next)) { + Register Hi = + stepInput(*Next, {W65816::ADCEi16imm, W65816::SBCEi16imm}, MRI); + if (!Hi || !Next->registerDefIsDead(W65816::P, nullptr)) + continue; + Group.push_back(&*Next); + Olds.push_back(Hi); + News.push_back(Next->getOperand(0).getReg()); + } else if (!Step->registerDefIsDead(W65816::P, nullptr)) { + // Someone might read the step's own flags. + bool Read = false; + for (auto It = std::next(Step->getIterator()); It != MBB.end(); ++It) { + if (readsP(*It)) { + Read = true; + break; + } + if (definesP(*It)) + break; + } + if (Read) + continue; + } + // The last use of an old value, before any use of a new one. A far + // pointer access stores the pointer's halves to stack temporaries + // (STAfi), stages them into $E0 / $E2 from there (`$a = LDAfi ; + // STA_DP $E0`) and goes through [$E0],Y, so a read of a slot an old + // value was stored to, and an access through a direct-page word it + // was staged to, are uses of it too. + auto uses = [&](const MachineInstr &MI, ArrayRef Regs) { + for (Register R : Regs) + if (MI.readsVirtualRegister(R)) + return true; + return false; + }; + const TargetInstrInfo *TII = MF.getSubtarget().getInstrInfo(); + SmallVector OldSlots; + SmallVector OldDP; + bool AFromOld = false; + auto readsOldSlot = [&](const MachineInstr &MI) { + for (const MachineOperand &MO : MI.operands()) + if (MO.isFI() && llvm::is_contained(OldSlots, MO.getIndex())) + return true; + return false; + }; + auto throughOldDP = [&](const MachineInstr &MI) { + return TII->getName(MI.getOpcode()).contains("_DPInd") && + MI.getOperand(0).isImm() && + llvm::is_contained(OldDP, MI.getOperand(0).getImm()); + }; + MachineInstr *LastOld = nullptr; + for (auto It = std::next(Group.back()->getIterator()); + It != MBB.end() && !It->isTerminator(); ++It) { + if (uses(*It, News)) + break; + if (It->getOpcode() == W65816::STAfi && It->getOperand(1).isFI() && + uses(*It, Olds)) { + OldSlots.push_back(It->getOperand(1).getIndex()); + LastOld = &*It; + continue; + } + if (It->getOpcode() == W65816::STA_DP && AFromOld && + It->getOperand(0).isImm()) { + OldDP.push_back(It->getOperand(0).getImm()); + LastOld = &*It; + continue; + } + bool Old = uses(*It, Olds) || readsOldSlot(*It) || throughOldDP(*It); + if (It->definesRegister(W65816::A, nullptr)) + AFromOld = Old && It->getOpcode() == W65816::LDAfi; + if (Old) + LastOld = &*It; + } + if (!LastOld) + continue; + // Land where A, X and Y are not live as physical registers: the + // staging moves the pointer through A (`$a = LDAfi ; STA_DP $E0, + // implicit $a`), and a step needing A in between cannot be + // allocated. Advance past each such physical live range, never + // past a use of a new value or into the terminators. + const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo(); + auto physReader = [&](MachineBasicBlock::iterator At) -> MachineInstr * { + for (MCRegister R : {W65816::A, W65816::X, W65816::Y}) + for (auto It = At; It != MBB.end(); ++It) { + if (It->readsRegister(R, TRI)) + return &*It; + if (It->definesRegister(R, TRI)) + break; + } + return nullptr; + }; + auto IP = std::next(LastOld->getIterator()); + bool Blocked = false; + while (MachineInstr *Reader = physReader(IP)) { + for (auto It = IP; !Blocked && It != std::next(Reader->getIterator()); + ++It) + Blocked = It->isTerminator() || uses(*It, News); + if (Blocked) + break; + IP = std::next(Reader->getIterator()); + } + if (Blocked) + continue; + // The flags must be dead where the group lands (INA / DEA model no + // P def but set N / Z all the same). Flags that may flow out of the + // block count as live. + bool PLive = true; + for (auto It = IP; It != MBB.end(); ++It) { + if (readsP(*It)) + break; + if (definesP(*It)) { + PLive = false; + break; + } + } + if (PLive) + continue; + for (MachineInstr *MI : Group) + MBB.splice(IP, &MBB, MI->getIterator()); + for (Register R : Olds) + MRI.clearKillFlags(R); + LLVM_DEBUG(dbgs() << "sank step to after " << *LastOld); + Changed = true; + } + } + return Changed; +} diff --git a/src/llvm/lib/Target/W65816/W65816SpillToX.cpp b/src/llvm/lib/Target/W65816/W65816SpillToX.cpp index 7f14eb4..e6a78d6 100644 --- a/src/llvm/lib/Target/W65816/W65816SpillToX.cpp +++ b/src/llvm/lib/Target/W65816/W65816SpillToX.cpp @@ -40,6 +40,7 @@ #include "W65816.h" #include "W65816InstrInfo.h" #include "W65816Subtarget.h" +#include "llvm/ADT/SmallPtrSet.h" #include "llvm/ADT/SmallSet.h" #include "llvm/CodeGen/MachineFrameInfo.h" #include "llvm/CodeGen/MachineFunction.h" @@ -47,6 +48,7 @@ #include "llvm/CodeGen/MachineInstr.h" #include "llvm/CodeGen/MachineInstrBuilder.h" #include "llvm/CodeGen/TargetRegisterInfo.h" +#include "llvm/Support/CommandLine.h" using namespace llvm; @@ -57,7 +59,8 @@ namespace { class W65816SpillToX : public MachineFunctionPass { public: static char ID; - W65816SpillToX() : MachineFunctionPass(ID) {} + explicit W65816SpillToX(bool SharedSlots = false) + : MachineFunctionPass(ID), SharedSlots(SharedSlots) {} StringRef getPassName() const override { return "W65816 spill-to-X peephole"; } @@ -68,17 +71,31 @@ public: } bool runOnMachineFunction(MachineFunction &MF) override; + +private: + // Bridge spill slots that other values share too (see the cross-block + // use check). Only the run after W65816StackSlotCleanup does: that pass + // deletes many of the same spill/reload pairs outright, slot and all, + // which beats a bridge, and a bridge made first blocks it (JoeyLib's + // jlpGenericTileCopyMasked kept 36 frame slots instead of 26). + bool SharedSlots; }; } // namespace +// Run as the post-cleanup invocation (for tests driving the pass alone). +static cl::opt SpillToXSharedSlots( + "w65816-spill-to-x-shared-slots", cl::Hidden, cl::init(false), + cl::desc("W65816 spill-to-X: also bridge spill slots shared by " + "stack-slot coloring")); + char W65816SpillToX::ID = 0; INITIALIZE_PASS(W65816SpillToX, DEBUG_TYPE, "W65816 spill-to-X peephole", false, false) -FunctionPass *llvm::createW65816SpillToX() { - return new W65816SpillToX(); +FunctionPass *llvm::createW65816SpillToX(bool SharedSlots) { + return new W65816SpillToX(SharedSlots); } // Classifies how an MI interacts with X. @@ -157,6 +174,43 @@ static bool referencesSlot(const MachineInstr &MI, int FI) { return false; } +// True when the value spill slot FI holds just after From may still be +// read: some path from there reaches a reference to FI before a full +// overwrite (`STAfi $a, FI, 0`, a 16-bit store over the whole 2-byte +// slot). Every other reference counts as a read. +static bool slotLiveAfter(MachineInstr &From, int FI) { + // -1 for a read, 1 for a full overwrite, 0 for neither. + auto effect = [FI](const MachineInstr &MI) { + if (!referencesSlot(MI, FI)) + return 0; + return matchSTAfi(MI) == FI ? 1 : -1; + }; + MachineBasicBlock *MBB = From.getParent(); + for (auto I = std::next(From.getIterator()); I != MBB->end(); ++I) { + if (int E = effect(*I)) + return E < 0; + } + SmallVector Work(MBB->succ_begin(), + MBB->succ_end()); + SmallPtrSet Seen; + while (!Work.empty()) { + MachineBasicBlock *B = Work.pop_back_val(); + if (!Seen.insert(B).second) + continue; + int E = 0; + for (MachineInstr &MI : *B) { + E = effect(MI); + if (E) + break; + } + if (E < 0) + return true; + if (E == 0) + Work.append(B->succ_begin(), B->succ_end()); + } + return false; +} + bool W65816SpillToX::runOnMachineFunction(MachineFunction &MF) { const W65816Subtarget &STI = MF.getSubtarget(); const W65816InstrInfo *TII = STI.getInstrInfo(); @@ -197,6 +251,10 @@ bool W65816SpillToX::runOnMachineFunction(MachineFunction &MF) { // (which would invalidate the saved value). if (touchesX(MI2, TRI)) { xTouched = true; break; } if (referencesSlot(MI2, FI)) break; + // Unlike the STA, the TAX sets N/Z. A PHP in the window (the + // flag-save wraps exist by the second run) would save the TAX's + // flags instead of the test it brackets. + if (MI2.getOpcode() == W65816::PHP) break; gapEmpty = false; } @@ -212,13 +270,21 @@ bool W65816SpillToX::runOnMachineFunction(MachineFunction &MF) { // a libcall later in the block expects $x intact. Scan from just // past LDA to end-of-block; if any instr uses X, bail. bool xUsedAfter = false; + bool xRedefAfter = false; for (auto Scan = std::next(MachineBasicBlock::iterator(LdaMI)); Scan != MBB.end(); ++Scan) { const MachineInstr &MI3 = *Scan; if (MI3.isDebugInstr()) continue; XEffect eff = xEffect(MI3, TRI); if (eff & XReads) { xUsedAfter = true; break; } - if (eff & XDefs) break; // X redefined; no longer live + if (eff & XDefs) { xRedefAfter = true; break; } + } + // X still live at the end of the block: the TAX would clobber a + // value a successor reads. + if (!xUsedAfter && !xRedefAfter) { + for (MachineBasicBlock *Succ : MBB.successors()) { + if (Succ->isLiveIn(W65816::X)) { xUsedAfter = true; break; } + } } // Also bail if X is live-in to MBB and nothing has defined X // between MBB start and STA — the live-in value is needed past @@ -235,29 +301,41 @@ bool W65816SpillToX::runOnMachineFunction(MachineFunction &MF) { } if (xUsedAfter) continue; - // Cross-block use check: if the slot is referenced anywhere - // OUTSIDE the [STA, LDA] window (including other blocks), the - // STA we'd erase is feeding those other reads — eliding it - // would silently corrupt them. Caught by sumTable() returning - // a stale phi value because the loop's STA-to-merge-slot was - // eliminated; the merge block's LDA then read the bb.0-init 0 - // instead of the loop's accumulated sum. + // Cross-block use check: the STA we'd erase must feed no read but + // this LDA, or eliding it would silently corrupt the others. + // Caught by sumTable() returning a stale phi value because the + // loop's STA-to-merge-slot was eliminated; the merge block's LDA + // then read the bb.0-init 0 instead of the loop's accumulated sum. + // + // Strictly: no reference to the slot outside the window. In the + // shared-slot run, a spill slot may also have references in OTHER + // blocks when it is dead after the LDA: stack-slot coloring gives + // one slot to values with disjoint lives (two loops' counters, say), + // and those references never see this STA's value. A reference in + // the same block still bails: there the slot takes part in a pattern + // the later passes rewrite in place once StackRelToImg has moved it + // to a DP word -- an `i32 += 1` (I32IncFold's INC_HI_IF_CARRY), a + // pointer bump, a 32-bit shift's `lsr dp ; ror dp` -- and a bridge + // breaks it (popcount went from 1159 to 2285 cycles a call). Other + // slots never qualify: one whose address is taken can be read + // through a pointer, which no operand scan sees, and a setjmp's + // second return is no CFG edge. + bool Shared = (SharedSlots || SpillToXSharedSlots) && + MF.getFrameInfo().isSpillSlotObjectIndex(FI) && + !MF.exposesReturnsTwice(); bool externalUse = false; for (auto &OtherMBB : MF) { for (auto &OtherMI : OtherMBB) { if (&OtherMI == StaMI || &OtherMI == LdaMI) continue; - // Walk inside-window range and skip those refs. - if (&OtherMBB == &MBB) { - // We already verified the gap doesn't reference FI; only - // STA/LDA themselves are allowed users in this block. - } - if (referencesSlot(OtherMI, FI)) { + if (referencesSlot(OtherMI, FI) && (!Shared || &OtherMBB == &MBB)) { externalUse = true; break; } } if (externalUse) break; } + if (!externalUse && Shared) + externalUse = slotLiveAfter(*LdaMI, FI); if (externalUse) continue; // Replace STAfi with TAX, LDAfi with TXA. diff --git a/src/llvm/lib/Target/W65816/W65816SplitLoopExits.cpp b/src/llvm/lib/Target/W65816/W65816SplitLoopExits.cpp new file mode 100644 index 0000000..2724e8a --- /dev/null +++ b/src/llvm/lib/Target/W65816/W65816SplitLoopExits.cpp @@ -0,0 +1,131 @@ +//===-- W65816SplitLoopExits.cpp - Split critical loop-exit edges --------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===---------------------------------------------------------------------===// +// +// IR-level, the last pass before ISel: give each critical edge out of a +// loop into a block with PHIs a block of its own. +// +// PHI elimination puts a PHI's copy at the end of the predecessor the +// value comes from. When that predecessor is a loop block that also +// branches back into the loop, the copy runs on every trip, not just the +// one that leaves: a search loop's `return p` copies p (four instructions +// for a far pointer) and its `return NULL` a zero into the return value's +// home each time round (CoreMark's core_list_find spent a quarter of its +// trip on them). LLVM splits such edges in PHI elimination itself, but +// only through MachineBasicBlock::SplitCriticalEdge, which needs +// analyzeBranch to understand the predecessor's conditional branch, and +// W65816InstrInfo keeps conditional branches opaque on purpose (see +// analyzeBranch). Split here instead: the copy moves into the new block, +// which runs once per exit and ends in a branch to the old target. +// CodeGenPrepare, which would merge the new block back, has already run. +// +//===---------------------------------------------------------------------===// + +#include "W65816.h" +#include "llvm/Analysis/CFG.h" +#include "llvm/Analysis/LoopInfo.h" +#include "llvm/IR/Dominators.h" +#include "llvm/IR/Function.h" +#include "llvm/IR/Instructions.h" +#include "llvm/InitializePasses.h" +#include "llvm/Pass.h" +#include "llvm/Transforms/Utils/BasicBlockUtils.h" + +using namespace llvm; + +#define DEBUG_TYPE "w65816-split-loop-exits" + + +namespace { + + +class W65816SplitLoopExits : public FunctionPass { +public: + static char ID; + W65816SplitLoopExits() : FunctionPass(ID) {} + + StringRef getPassName() const override { + return "W65816 split critical loop-exit edges"; + } + + void getAnalysisUsage(AnalysisUsage &AU) const override { + AU.addRequired(); + AU.addRequired(); + } + + bool runOnFunction(Function &F) override; +}; + + +} // namespace + + +char W65816SplitLoopExits::ID = 0; + +INITIALIZE_PASS_BEGIN(W65816SplitLoopExits, DEBUG_TYPE, + "W65816 split critical loop-exit edges", false, false) +INITIALIZE_PASS_DEPENDENCY(LoopInfoWrapperPass) +INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass) +INITIALIZE_PASS_END(W65816SplitLoopExits, DEBUG_TYPE, + "W65816 split critical loop-exit edges", false, false) + + +// Whether V is what a header PHI of L takes over a back edge: the loop +// keeps it in that PHI's register already, so a copy of it on the way out +// coalesces into that register wherever PHI elimination puts it, and the +// split only moves the allocator's choices around (sumOfSquares' running +// total gained a store a trip from it). +static bool isCarriedValue(Value *V, Loop *L) { + for (PHINode &P : L->getHeader()->phis()) + for (unsigned K = 0, E = P.getNumIncomingValues(); K != E; ++K) + if (P.getIncomingValue(K) == V && L->contains(P.getIncomingBlock(K))) + return true; + return false; +} + + +bool W65816SplitLoopExits::runOnFunction(Function &F) { + if (skipFunction(F) || getenv("W65816_DISABLE_SPLIT_LOOP_EXITS")) + return false; + LoopInfo &LI = getAnalysis().getLoopInfo(); + DominatorTree &DT = getAnalysis().getDomTree(); + // Collect first: splitting adds blocks and edges. + SmallVector, 8> Edges; + for (BasicBlock &BB : F) { + Loop *L = LI.getLoopFor(&BB); + Instruction *TI = BB.getTerminator(); + if (!L || !TI || TI->getNumSuccessors() < 2 || + !(isa(TI) || isa(TI))) + continue; + for (unsigned I = 0, E = TI->getNumSuccessors(); I != E; ++I) { + BasicBlock *Succ = TI->getSuccessor(I); + if (L->contains(Succ) || !isa(Succ->begin()) || + Succ->getSinglePredecessor() || Succ->isEHPad()) + continue; + // Worth it only for a copy that would really run each trip: of a + // constant, or of a value the loop does not carry. + bool Worth = false; + for (PHINode &P : Succ->phis()) + if (!isCarriedValue(P.getIncomingValueForBlock(&BB), L)) + Worth = true; + if (Worth) + Edges.push_back({TI, I}); + } + } + bool Changed = false; + CriticalEdgeSplittingOptions Opts(&DT, &LI); + Opts.setMergeIdenticalEdges(); + for (auto [TI, I] : Edges) + if (isCriticalEdge(TI, I) && SplitCriticalEdge(TI, I, Opts)) + Changed = true; + return Changed; +} + + +FunctionPass *llvm::createW65816SplitLoopExits() { + return new W65816SplitLoopExits(); +} diff --git a/src/llvm/lib/Target/W65816/W65816StackRelToImg.cpp b/src/llvm/lib/Target/W65816/W65816StackRelToImg.cpp index e6e270b..c764026 100644 --- a/src/llvm/lib/Target/W65816/W65816StackRelToImg.cpp +++ b/src/llvm/lib/Target/W65816/W65816StackRelToImg.cpp @@ -48,11 +48,13 @@ #include "W65816Subtarget.h" #include "llvm/ADT/DenseMap.h" #include "llvm/ADT/DenseSet.h" +#include "llvm/ADT/PostOrderIterator.h" #include "llvm/ADT/SmallVector.h" #include "llvm/CodeGen/MachineFrameInfo.h" #include "llvm/CodeGen/MachineFunction.h" #include "llvm/CodeGen/MachineFunctionPass.h" #include "llvm/CodeGen/MachineInstrBuilder.h" +#include "llvm/CodeGen/MachineBlockFrequencyInfo.h" #include "llvm/CodeGen/MachineLoopInfo.h" #include "llvm/InitializePasses.h" #include "llvm/IR/DebugInfoMetadata.h" @@ -79,6 +81,7 @@ public: } void getAnalysisUsage(AnalysisUsage &AU) const override { AU.addRequired(); + AU.addRequired(); AU.setPreservesCFG(); MachineFunctionPass::getAnalysisUsage(AU); } @@ -94,6 +97,7 @@ char W65816StackRelToImg::ID = 0; INITIALIZE_PASS_BEGIN(W65816StackRelToImg, DEBUG_TYPE, "W65816 stack-rel to IMG", false, false) INITIALIZE_PASS_DEPENDENCY(MachineLoopInfoWrapperPass) +INITIALIZE_PASS_DEPENDENCY(MachineBlockFrequencyInfoWrapperPass) INITIALIZE_PASS_END(W65816StackRelToImg, DEBUG_TYPE, "W65816 stack-rel to IMG", false, false) @@ -135,7 +139,8 @@ static bool isImgSafeCall(const MachineInstr &MI) { if (MO.isGlobal()) Name = MO.getGlobal()->getName(); else if (MO.isSymbol()) Name = MO.getSymbolName(); else continue; - if (Name == "__umulhisi3" || Name == "__mulhi3" || + if (Name == "__umulhisi3" || Name == "__mulhisi3" || + Name == "__mulhi3" || Name == "__mulsi3" || Name == "__udivhi3" || Name == "__umodhi3" || Name == "__divhi3" || Name == "__modhi3" || Name == "__udivsi3" || @@ -569,15 +574,22 @@ static bool elidePlaPhaPair(MachineFunction &MF) { // Always-on: drop a redundant LDA when the prior LDA loaded the same -// source and the only intervening instruction was PHA (which reads A -// but doesn't modify it). Emerges from i64 arg-push sequences: +// word and the only intervening instruction was PHA (which reads A but +// doesn't modify it). A direct-page word keeps its address; a stack word +// is 2 bytes further from SP once the PHA has pushed: // // LDA off, s // PHA -// LDA off, s ; A still has this value — redundant -// PHA +// LDA off+2, s ; the same word: A still has this value -- redundant // -// Saves 1 instruction (3 bytes / 4 cyc) per match. +// `LDA off, s ; PHA ; LDA off, s` reads two different words. Matching +// equal offsets there deleted the load of ImgCalleeSave's restore, which +// reads its save slot inside a PHA bracket right after the return value +// was loaded from the word above it: libc's wmemcpy / wmemmove / rename, +// fminf and eleven Lua functions handed the caller back its callee-saved $C0 +// holding their own return value. +// +// Saves 1 instruction (2 bytes / 5 cyc) per match. static bool elideRedundantLdaAfterPha(MachineFunction &MF) { bool Changed = false; for (MachineBasicBlock &MBB : MF) { @@ -594,7 +606,7 @@ static bool elideRedundantLdaAfterPha(MachineFunction &MF) { MachineOperand &S1 = It->getOperand(0); MachineOperand &S2 = I2->getOperand(0); if (!S1.isImm() || !S2.isImm()) continue; - if (S1.getImm() != S2.getImm()) continue; + if (S2.getImm() != S1.getImm() + (IsLdaSr ? 2 : 0)) continue; ToErase.push_back(&*I2); } for (MachineInstr *MI : ToErase) { @@ -729,14 +741,20 @@ static bool elideDeadStaCarry(MachineFunction &MF) { // the return to the canonical body SP. static int spDelta(const MachineInstr &MI) { switch (MI.getOpcode()) { + // PUSH16X is the PHX a call argument already in X is pushed with: left + // out, every slot access between it and the call was taken for the + // neighbouring slot (luaT_init's promoted pointer words read the + // arguments it was pushing). case W65816::PHA: case W65816::PHX: case W65816::PHY: - case W65816::PUSH16: case W65816::PEA: + case W65816::PUSH16: case W65816::PUSH16X: case W65816::PEA: + case W65816::PHD: return -2; case W65816::PLA: case W65816::PLX: case W65816::PLY: + case W65816::PLD: return 2; - case W65816::PHP: + case W65816::PHP: case W65816::PHB: case W65816::PHK: return -1; - case W65816::PLP: + case W65816::PLP: case W65816::PLB: return 1; default: return 0; @@ -786,6 +804,62 @@ static bool entrySpAdj(MachineFunction &MF, } +// The logical offset (in entrySpAdj's terms) where the caller's stack +// arguments start: 4 bytes above the SP at function entry, past the 3-byte +// return address. Relative to the body SP that is FrameSize + 4, which is +// what entrySpAdj measures from when the prologue allocates with TCS (it +// resets the count). A small frame's prologue allocates with leading PHAs +// instead, and entrySpAdj counts those, so there logical offsets are +// relative to the entry SP and the arguments start at 4. Using +// FrameSize + 4 for both let a PHA-frame function promote its own incoming +// arguments to DP words nothing ever stored: libc's ungetc dereferenced a +// FILE pointer it never loaded. A VLA function's prologue saves SP before +// it allocates; take the lower bound there, which only promotes less. +// True when the function takes the address of a stack slot (TSX, or a +// TSC that is not a frame adjust TSC ; CLC|SEC ; ADC|SBC #n ; TCS): a +// pointer -- or a callee handed one -- may then read or write any slot. +// (ADDframe, the LEA, is already TSC ; CLC ; ADC here, after PEI.) +static bool takesStackAddr(const MachineFunction &MF) { + for (const MachineBasicBlock &MBB : MF) { + for (auto It = MBB.begin(), E = MBB.end(); It != E; ++It) { + if (It->getOpcode() == W65816::TSX) + return true; + if (It->getOpcode() != W65816::TSC) + continue; + auto N = std::next(It); + bool FrameAdjust = + N != E && + (N->getOpcode() == W65816::CLC || N->getOpcode() == W65816::SEC) && + ++N != E && + (N->getOpcode() == W65816::ADC_Imm16 || + N->getOpcode() == W65816::SBC_Imm16) && + ++N != E && N->getOpcode() == W65816::TCS; + if (!FrameAdjust) + return true; + } + } + return false; +} + + +static int argSlotMinOff(MachineFunction &MF) { + const MachineFrameInfo &MFI = MF.getFrameInfo(); + int FrameSize = static_cast(MFI.getStackSize()); + if (MFI.hasVarSizedObjects()) + return 4; + int Pushed = 0; + for (MachineInstr &MI : MF.front()) { + unsigned Opc = MI.getOpcode(); + if (Opc == W65816::REP || Opc == W65816::SEP) + continue; + if (Opc != W65816::PHA || Pushed >= FrameSize) + break; + Pushed += 2; + } + return FrameSize + 4 - Pushed; +} + + // After promotion, look for `LDA_DP X ; INA ; STA_DP X` // (or with DEA) patterns and fold to `INC_DP X` (DEC_DP X). These // are very common in loop body: `i++` and `counter--` both produce @@ -840,8 +914,10 @@ static bool foldDpIncDec(MachineFunction &MF, const TargetInstrInfo *TII) { } } if (Cur->isCall()) defsA = true; // caller-save - if (defsA) return true; + // A read comes first: a tied op (`$a = ANDi16imm $a, k`) + // both reads and writes A, and it reads the INA's result. if (usesA) return false; + if (defsA) return true; ++Cur; } return false; @@ -1094,9 +1170,69 @@ static bool aDeadAtEntry(MachineBasicBlock &MBB) { } +// True when MBB may end with the accumulator in 8-bit mode: its last +// SEP / REP of the M bit is a SEP. +static bool mayEndInM8(const MachineBasicBlock &MBB) { + for (auto It = MBB.rbegin(); It != MBB.rend(); ++It) { + unsigned Op = It->getOpcode(); + if ((Op == W65816::SEP || Op == W65816::REP) && + It->getOperand(0).isImm() && (It->getOperand(0).getImm() & 0x20)) + return Op == W65816::SEP; + } + return false; +} + + +// True when logical slot S may be read once E is entered: some path from +// E's start reaches an access to S before a 16-bit store over all of it. +// The caller guarantees no access overlaps S from a neighbouring word and +// that no slot's address is taken; blocks start in 16-bit M, as the loop +// copies assume. +static bool slotReadFrom(MachineBasicBlock *E, int64_t S, + const DenseMap &EntrySp) { + SmallVector Work{E}; + SmallPtrSet Seen; + while (!Work.empty()) { + MachineBasicBlock *B = Work.pop_back_val(); + if (!Seen.insert(B).second) + continue; + auto It = EntrySp.find(B); + if (It == EntrySp.end()) + return true; + int Sp = It->second; + bool M16 = true; + bool Killed = false; + for (MachineInstr &MI : *B) { + unsigned Op = MI.getOpcode(); + if (Op == W65816::TCS) { + Sp = 0; + continue; + } + if ((Op == W65816::SEP || Op == W65816::REP) && + MI.getOperand(0).isImm() && (MI.getOperand(0).getImm() & 0x20)) + M16 = Op == W65816::REP; + if ((isStackRelOp(Op) || carryDpOpcode(Op) || isStackRelIndYOp(Op)) && + MI.getNumOperands() >= 1 && MI.getOperand(0).isImm() && + MI.getOperand(0).getImm() + Sp == S) { + if (Op != W65816::STA_StackRel || !M16) + return true; + Killed = true; + break; + } + Sp += spDelta(MI); + } + if (!Killed) + Work.append(B->succ_begin(), B->succ_end()); + } + return false; +} + + // Loop-local promotion, for a function the whole-function promotion // below cannot touch because it makes calls: any call may clobber -// IMG0..7. An innermost loop that makes none can still keep its hottest +// IMG0..7. An innermost loop that makes none -- or only calls to the +// libgcc helpers isImgSafeCall lists, which never touch $D0..$DF (a +// multiply in CoreMark's matrix loops) -- can still keep its hottest // slots -- far pointers first, into adjacent words -- in IMG words the // function never uses: copied in at the end of the preheader, and the // ones the loop writes copied back at the start of each exit. The copies @@ -1107,12 +1243,18 @@ static bool aDeadAtEntry(MachineBasicBlock &MBB) { // access. (JoeyLib's sfx probe inlines its reference copy loop into // main, which calls: the whole-function promotion never reached it.) // -// Conservative throughout: a loop with a preheader and dedicated exits; -// nothing in it that calls, moves SP by more than a push, addresses the -// stack as a pointer, or runs in a frame reached through $F6; a slot no -// neighbour's access overlaps; the copy-in where A is dead and nothing -// later in the preheader writes the slot; exits whose A and flags are dead -// on entry. Anything else leaves the loop alone. +// Conservative throughout: a loop with a preheader; nothing in it that +// calls, moves SP by more than a push, addresses the stack as a pointer, +// or runs in a frame reached through $F6; a slot no neighbour's access +// overlaps; the copy-in where A is dead and nothing later in the preheader +// writes the slot. A written home is copied back at an exit only when the +// slot may be read after it; such a home needs a dedicated exit (no other +// way in), and an exit whose A or flags may be live on entry gets its +// copies inside PHP / PHA ... PLA / PLP. Loops failing those two exit +// tests used to be refused outright: a third of the corpus's in-loop far- +// pointer staging, JoeyLib's SCB upload in jlpPresent and every `while +// (n--)` copy (block placement merges the dedicated exit into the zero- +// trip guard's target) among them. Anything else leaves the loop alone. static bool promoteLoopLocal(MachineFunction &MF, MachineLoopInfo &MLI, const TargetInstrInfo *TII, uint8_t ImgBase) { // A plain slot pays its copies back at a cycle per access per trip; a @@ -1126,10 +1268,13 @@ static bool promoteLoopLocal(MachineFunction &MF, MachineLoopInfo &MLI, // Logical slots accessed anywhere (a candidate's neighbours must not // be: a 16-bit access at S +- 1 overlaps S), and the frame-pointer test. DenseSet AllSlots; + bool AddrTaken = false; for (MachineBasicBlock &MBB : MF) { int Sp = EntrySp.lookup(&MBB); for (MachineInstr &MI : MBB) { unsigned Op = MI.getOpcode(); + if (Op == W65816::ADDframe) + AddrTaken = true; if (Op == W65816::TCS) { Sp = 0; continue; @@ -1164,11 +1309,8 @@ static bool promoteLoopLocal(MachineFunction &MF, MachineLoopInfo &MLI, L->getUniqueExitBlocks(Exits); bool Ok = !Exits.empty(); for (MachineBasicBlock *E : Exits) { - if (!EntrySp.count(E) || !aDeadAtEntry(*E)) + if (!EntrySp.count(E)) Ok = false; - for (MachineBasicBlock *P : E->predecessors()) - if (!L->contains(P)) - Ok = false; } // The loop's accesses, by logical slot. DenseMap> Sites; @@ -1179,7 +1321,8 @@ static bool promoteLoopLocal(MachineFunction &MF, MachineLoopInfo &MLI, int Sp = EntrySp.lookup(B); for (MachineInstr &MI : *B) { unsigned Op = MI.getOpcode(); - if (MI.isCall() || MI.isInlineAsm() || isStackRelIndYOp(Op) || + if ((MI.isCall() && !isImgSafeCall(MI)) || MI.isInlineAsm() || + isStackRelIndYOp(Op) || Op == W65816::TCS || Op == W65816::TSC || Op == W65816::TSX || Op == W65816::TXS || Op == W65816::TCD || Op == W65816::PLD || Op == W65816::ADDframe || Op == W65816::ALLOCAfi) { @@ -1285,6 +1428,28 @@ static bool promoteLoopLocal(MachineFunction &MF, MachineLoopInfo &MLI, Home[S] = static_cast(ImgBase + 2 * Idx); Taken.set(Idx); } + // The homes each exit copies back: written in the loop, and maybe + // read after it. + auto copiedOut = [&](MachineBasicBlock *E, int64_t S) { + return Written.count(S) && + (AddrTaken || slotReadFrom(E, S, EntrySp)); + }; + // A home an exit must copy back stays on the stack when other paths + // enter that exit too (the copy would store a stale DP word there). + // Block placement merges LoopSimplify's dedicated exits into the + // zero-trip guard's target, so this is the common `while (n--)` shape; + // its pointers and counter are dead after the loop and keep their homes. + for (MachineBasicBlock *E : Exits) { + if (llvm::all_of(E->predecessors(), + [&](MachineBasicBlock *P) { return L->contains(P); })) + continue; + SmallVector Drop; + for (auto &KV : Home) + if (copiedOut(E, KV.first)) + Drop.push_back(KV.first); + for (int64_t S : Drop) + Home.erase(S); + } // Nothing after the copy-in point may write a home's slot. int Sp = SpIns; for (auto It = InsPt; It != Pre->end(); ++It) { @@ -1295,13 +1460,20 @@ static bool promoteLoopLocal(MachineFunction &MF, MachineLoopInfo &MLI, Sp = It->getOpcode() == W65816::TCS ? 0 : Sp + spDelta(*It); } // The copies address the slots from where they sit: the offsets must - // fit stack-relative addressing there. + // fit stack-relative addressing there (3 bytes further inside an + // exit's PHP / PHA bracket), and an exit's copies run in 16-bit M. for (auto &KV : Home) { int64_t Off = KV.first - SpIns; bool Fits = Off >= 0 && Off <= 0xFF; for (MachineBasicBlock *E : Exits) { - int64_t OffE = KV.first - EntrySp.lookup(E); - if (Written.count(KV.first) && (OffE < 0 || OffE > 0xFF)) + if (!copiedOut(E, KV.first)) + continue; + int64_t OffE = KV.first - EntrySp.lookup(E) + + (aDeadAtEntry(*E) ? 0 : 3); + if (OffE < 0 || OffE > 0xFF || + llvm::any_of(E->predecessors(), [](MachineBasicBlock *P) { + return mayEndInM8(*P); + })) Fits = false; } if (!Fits) @@ -1324,11 +1496,21 @@ static bool promoteLoopLocal(MachineFunction &MF, MachineLoopInfo &MLI, for (MachineInstr *MI : Sites[KV.first]) rewriteToDp(*MI, KV.second, TII); for (MachineBasicBlock *E : Exits) { - int SpE = EntrySp.lookup(E); + SmallVector, 8> Out; + for (auto &KV : Home) + if (copiedOut(E, KV.first)) + Out.push_back({KV.first, KV.second}); + if (Out.empty()) + continue; + bool Save = !aDeadAtEntry(*E); + int SpE = EntrySp.lookup(E) - (Save ? 3 : 0); auto At = E->begin(); - for (auto &KV : Home) { - if (!Written.count(KV.first)) - continue; + if (Save) { + BuildMI(*E, At, DL, TII->get(W65816::PHP)); + BuildMI(*E, At, DL, TII->get(W65816::PHA)) + .addReg(W65816::A, RegState::Implicit | RegState::Undef); + } + for (auto &KV : Out) { BuildMI(*E, At, DL, TII->get(W65816::LDA_DP)) .addImm(KV.second) .addReg(W65816::A, RegState::ImplicitDefine); @@ -1336,6 +1518,11 @@ static bool promoteLoopLocal(MachineFunction &MF, MachineLoopInfo &MLI, .addImm(KV.first - SpE) .addReg(W65816::A, RegState::Implicit); } + if (Save) { + BuildMI(*E, At, DL, TII->get(W65816::PLA)) + .addReg(W65816::A, RegState::ImplicitDefine); + BuildMI(*E, At, DL, TII->get(W65816::PLP)); + } } // Debug values in the loop that name a promoted slot would read the // stale stack copy: drop their location, as the whole-function @@ -1364,6 +1551,314 @@ static bool promoteLoopLocal(MachineFunction &MF, MachineLoopInfo &MLI, } +// The least a path through the function, entry to a return, saves: each +// block's own saving, loops taken once (back edges ignored). +static double minPathSaving( + MachineFunction &MF, + const std::function &Save) { + ReversePostOrderTraversal RPOT(&MF); + DenseMap Order; + DenseMap Dist; + unsigned N = 0; + for (MachineBasicBlock *MBB : RPOT) + Order[MBB] = N++; + double Best = -1; + Dist[&MF.front()] = 0; + for (MachineBasicBlock *MBB : RPOT) { + auto DIt = Dist.find(MBB); + if (DIt == Dist.end()) + continue; + double D = DIt->second + Save(*MBB); + if (MBB->succ_empty() && (Best < 0 || D < Best)) + Best = D; + for (MachineBasicBlock *S : MBB->successors()) { + if (Order.lookup(S) <= Order.lookup(MBB)) + continue; + auto It = Dist.find(S); + if (It == Dist.end() || D < It->second) + Dist[S] = D; + } + } + return Best < 0 ? 0 : Best; +} + + +// Whole-function homes in IMG8..IMG15 for a function that makes calls. A +// far pointer the function keeps in two stack slots is staged into $E0 / +// $E2 before every access (18 cycles); in two adjacent direct-page words +// W65816FarPtrStageReuse addresses it where it lives. IMG0..7 do not +// survive a call, but IMG8..15 do -- every function that writes one gives +// the caller its value back (W65816ImgCalleeSave) -- so a word this +// function never touches can carry a pointer across its calls as long as +// the function keeps the caller's value too. It does so in the pointer's +// own slots: the entry SWAPS each slot with its word (the caller's value +// waits in the slot, nothing else reads it), and every return swaps the +// caller's value back. No frame space, no change to the layout PEI fixed. +// +// The swaps cost every call ~60 cycles a pair on the way in and ~25 on +// the way out, so a pair is taken only where its accesses pay for them on +// every path (the cheapest path through the function, loops once), or +// pay twice over by block frequency and a quarter on every path: a big +// function driving loops of calls through a struct pointer (Space Taxi's +// stRenderFrame, -8% of its cycles), not one that returns early from a +// cache hit (spriteMasks) or switches to one of nine inlined hooks +// (hookPerTickHigh), which both ran slower when the expected saving alone +// decided. Loops without calls take their own IMG0..7 homes first +// (promoteLoopLocal, paid only when the loop runs). +// +// Conservative: every exit an RTL, every access a stack-relative op with +// a direct-page form, no stack address taken, no frame reached through +// $F6, no setjmp, slots no neighbouring access overlaps, M=16 and SP at a +// known depth where the swaps go. +static bool promoteCalleeSavedPairs(MachineFunction &MF, + const MachineBlockFrequencyInfo &MBFI, + const TargetInstrInfo *TII) { + static const bool Disabled = getenv("W65816_DISABLE_CALLEE_SAVED_HOMES"); + if (Disabled || MF.exposesReturnsTwice() || takesStackAddr(MF)) + return false; + DenseMap EntrySp; + if (!entrySpAdj(MF, EntrySp)) + return false; + DenseMap> Sites; + DenseSet AllSlots, Blocked; + // Estimated cycles a home saves: a cycle per access, 9 for each load + // that stages the pointer into $E0 / $E2 (the stage goes). Expected + // per call (block frequency), and per block for the cheapest path. + DenseMap Saving; + DenseMap, double> BlockSaving; + for (MachineBasicBlock &MBB : MF) { + auto EIt = EntrySp.find(&MBB); + if (EIt == EntrySp.end()) + continue; + int Sp = EIt->second; + double Freq = MBFI.getBlockFreqRelativeToEntryBlock(&MBB); + for (auto It = MBB.begin(); It != MBB.end(); ++It) { + MachineInstr &MI = *It; + unsigned Op = MI.getOpcode(); + if (Op == W65816::ALLOCAfi || MI.isInlineAsm()) + return false; + bool HasImm = MI.getNumOperands() >= 1 && MI.getOperand(0).isImm(); + if (HasImm && TII->getName(Op).contains("_DPInd") && + MI.getOperand(0).getImm() >= 0xF6 && + MI.getOperand(0).getImm() <= 0xF8) + return false; + if (Op == W65816::TCS) { + Sp = 0; + continue; + } + if (HasImm && (isStackRelOp(Op) || carryDpOpcode(Op))) { + int64_t Slot = MI.getOperand(0).getImm() + Sp; + Sites[Slot].push_back(&MI); + AllSlots.insert(Slot); + auto Next = std::next(It); + bool Stages = Op == W65816::LDA_StackRel && Next != MBB.end() && + Next->getOpcode() == W65816::STA_DP && + Next->getOperand(0).isImm() && + (Next->getOperand(0).getImm() == 0xE0 || + Next->getOperand(0).getImm() == 0xE2); + Saving[Slot] += Freq * (Stages ? 9.0 : 1.0); + BlockSaving[{&MBB, Slot}] += Stages ? 9.0 : 1.0; + if (!getDpOpcode(Op) && !carryDpOpcode(Op)) + Blocked.insert(Slot); + } else if (HasImm && isStackRelIndYOp(Op)) { + AllSlots.insert(MI.getOperand(0).getImm() + Sp); + Blocked.insert(MI.getOperand(0).getImm() + Sp); + } + Sp += spDelta(MI); + } + } + if (Sites.empty()) + return false; + + // M=16 at an insertion point: the block's last M switch before it is a + // REP (blocks start in M=16). + auto m16At = [](MachineBasicBlock &MBB, MachineBasicBlock::iterator Pt) { + bool M16 = true; + for (auto It = MBB.begin(); It != Pt; ++It) + if ((It->getOpcode() == W65816::SEP || + It->getOpcode() == W65816::REP) && + It->getOperand(0).isImm() && (It->getOperand(0).getImm() & 0x20)) + M16 = It->getOpcode() == W65816::REP; + return M16; + }; + // SP (logical) just before Pt. + auto spAt = [&](MachineBasicBlock &MBB, MachineBasicBlock::iterator Pt) { + int Sp = EntrySp.lookup(&MBB); + for (auto It = MBB.begin(); It != Pt; ++It) + Sp = It->getOpcode() == W65816::TCS ? 0 : Sp + spDelta(*It); + return Sp; + }; + + // The swap-in point: after the prologue's TCS, or after the leading REP + // and PHA run of a frame allocated by pushes. No access to a slot may + // come before it. + MachineBasicBlock &Entry = MF.front(); + MachineBasicBlock::iterator InPt = Entry.end(); + for (auto It = Entry.begin(); It != Entry.end(); ++It) + if (It->getOpcode() == W65816::TCS) { + InPt = std::next(It); + break; + } + if (InPt == Entry.end()) { + InPt = Entry.begin(); + while (InPt != Entry.end() && + (InPt->isDebugInstr() || InPt->getOpcode() == W65816::REP || + InPt->getOpcode() == W65816::PHA)) + ++InPt; + } + if (InPt == Entry.end() || !m16At(Entry, InPt)) + return false; + int SpIn = spAt(Entry, InPt); + DenseSet BeforeIn; + for (auto It = Entry.begin(); It != InPt; ++It) + BeforeIn.insert(&*It); + + // The swap-back points: every exit is an RTL, the swap goes in front of + // the frame release that leads to it. + SmallVector, 4> + OutPts; + double ExitFreq = 0; + for (MachineBasicBlock &MBB : MF) { + if (!MBB.succ_empty()) + continue; + auto Last = MBB.getLastNonDebugInstr(); + if (Last == MBB.end() || Last->getOpcode() != W65816::RTL) + return false; + if (!EntrySp.count(&MBB)) + continue; + MachineBasicBlock::iterator Pt = Last; + while (Pt != MBB.begin()) { + auto Prev = std::prev(Pt); + unsigned Op = Prev->getOpcode(); + if (Prev->isDebugInstr() || Op == W65816::TCS || Op == W65816::TSC || + Op == W65816::CLC || Op == W65816::ADC_Imm16 || + Op == W65816::TAY || Op == W65816::TYA || Op == W65816::TAX || + Op == W65816::TXA || Op == W65816::PLY || Op == W65816::PLX) { + Pt = Prev; + continue; + } + break; + } + if (!m16At(MBB, Pt)) + return false; + // A return in the entry block must swap back after it swapped in. + if (&MBB == &Entry && (Pt == InPt || BeforeIn.count(&*Pt))) + return false; + OutPts.push_back({&MBB, Pt}); + ExitFreq += MBFI.getBlockFreqRelativeToEntryBlock(&MBB); + } + if (OutPts.empty()) + return false; + + // Pairs, heaviest first, into adjacent free words. + BitVector Taken = usedImgWords(MF, 0xC0u); + DenseMap Home; + auto usable = [&](int64_t S) { + if (!Sites.count(S) || Blocked.count(S) || Home.count(S) || + AllSlots.count(S - 1) || AllSlots.count(S + 1)) + return false; + for (MachineInstr *MI : Sites[S]) + if (BeforeIn.count(MI)) + return false; + if (S - SpIn + 4 > 0xFF || S - SpIn + 2 < 1) + return false; + for (auto &OP : OutPts) { + int64_t Off = S - spAt(*OP.first, OP.second) + 2; + if (Off < 1 || Off > 0xFF) + return false; + } + return true; + }; + // Swap in ~61 cycles a pair (with the A save), back ~27 a return. + const double PairCost = 61.0 + 27.0 * ExitFreq; + for (const FarPtrPair &FP : collectFarPtrPairs(MF, EntrySp, 0xC0u)) { + if (FP.LoIsDp || FP.HiIsDp || FP.Lo == FP.Hi || !usable(FP.Lo) || + !usable(FP.Hi)) + continue; + double MinSave = minPathSaving(MF, [&](const MachineBasicBlock &B) { + return BlockSaving.lookup({&B, FP.Lo}) + BlockSaving.lookup({&B, FP.Hi}); + }); + double Expected = Saving[FP.Lo] + Saving[FP.Hi]; + if (MinSave < PairCost && + (Expected < 2.0 * PairCost || MinSave < PairCost / 4)) + continue; + for (unsigned Idx = 0; Idx + 1 < 8; ++Idx) { + if (!Taken.test(Idx) && !Taken.test(Idx + 1)) { + Home[FP.Lo] = static_cast(0xC0u + 2 * Idx); + Home[FP.Hi] = static_cast(0xC0u + 2 * Idx + 2); + Taken.set(Idx); + Taken.set(Idx + 1); + break; + } + } + } + if (Home.empty()) + return false; + + // Swap in: A kept on the stack, each word through it. + DebugLoc DL; + BuildMI(Entry, InPt, DL, TII->get(W65816::PHA)) + .addReg(W65816::A, RegState::Implicit | RegState::Undef); + for (auto &KV : Home) { + BuildMI(Entry, InPt, DL, TII->get(W65816::LDA_DP)) + .addImm(KV.second) + .addReg(W65816::A, RegState::ImplicitDefine); + BuildMI(Entry, InPt, DL, TII->get(W65816::PHA)) + .addReg(W65816::A, RegState::Implicit); + BuildMI(Entry, InPt, DL, TII->get(W65816::LDA_StackRel)) + .addImm(KV.first - SpIn + 4) + .addReg(W65816::A, RegState::ImplicitDefine); + BuildMI(Entry, InPt, DL, TII->get(W65816::STA_DP)) + .addImm(KV.second) + .addReg(W65816::A, RegState::Implicit); + BuildMI(Entry, InPt, DL, TII->get(W65816::PLA)) + .addReg(W65816::A, RegState::ImplicitDefine); + BuildMI(Entry, InPt, DL, TII->get(W65816::STA_StackRel)) + .addImm(KV.first - SpIn + 2) + .addReg(W65816::A, RegState::Implicit); + } + BuildMI(Entry, InPt, DL, TII->get(W65816::PLA)) + .addReg(W65816::A, RegState::ImplicitDefine); + // Swap back at each exit, the return value kept on the stack. + for (auto &OP : OutPts) { + int SpOut = spAt(*OP.first, OP.second); + BuildMI(*OP.first, OP.second, DL, TII->get(W65816::PHA)) + .addReg(W65816::A, RegState::Implicit | RegState::Undef); + for (auto &KV : Home) { + BuildMI(*OP.first, OP.second, DL, TII->get(W65816::LDA_StackRel)) + .addImm(KV.first - SpOut + 2) + .addReg(W65816::A, RegState::ImplicitDefine); + BuildMI(*OP.first, OP.second, DL, TII->get(W65816::STA_DP)) + .addImm(KV.second) + .addReg(W65816::A, RegState::Implicit); + } + BuildMI(*OP.first, OP.second, DL, TII->get(W65816::PLA)) + .addReg(W65816::A, RegState::ImplicitDefine); + } + for (auto &KV : Home) + for (MachineInstr *MI : Sites[KV.first]) + rewriteToDp(*MI, KV.second, TII); + // Debug values naming a promoted slot would read the swapped-out word. + for (MachineBasicBlock &MBB : MF) + for (MachineInstr &MI : MBB) { + if (!MI.isDebugValue()) + continue; + const DIExpression *Expr = MI.getDebugExpression(); + int64_t Lead = 0; + SmallVector Rest; + if (!Expr || !Expr->extractLeadingOffset(Lead, Rest) || + !Home.count(Lead)) + continue; + for (MachineOperand &Op : MI.debug_operands()) + if (Op.isReg()) { + Op.setReg(0); + Op.setSubReg(0); + } + } + return true; +} + + bool W65816StackRelToImg::runOnMachineFunction(MachineFunction &MF) { if (skipFunction(MF.getFunction())) return false; if (MF.getFunction().hasOptNone()) return false; @@ -1410,15 +1905,19 @@ bool W65816StackRelToImg::runOnMachineFunction(MachineFunction &MF) { } MachineLoopInfo &MLI = getAnalysis().getLI(); + const MachineBlockFrequencyInfo &MBFI = + getAnalysis().getMBFI(); if (isSelfCall) { ChangedEarly |= elideStackRelDeadStore(MF); ChangedEarly |= elideStoreForwarding(MF); ChangedEarly |= promoteLoopLocal(MF, MLI, STIp.getInstrInfo(), 0xD0u); + ChangedEarly |= promoteCalleeSavedPairs(MF, MBFI, STIp.getInstrInfo()); return ChangedEarly; } if (!isImgSafeCall(MI)) { ChangedEarly |= elideStoreForwarding(MF); ChangedEarly |= promoteLoopLocal(MF, MLI, STIp.getInstrInfo(), 0xD0u); + ChangedEarly |= promoteCalleeSavedPairs(MF, MBFI, STIp.getInstrInfo()); return ChangedEarly; } } @@ -1535,16 +2034,24 @@ bool W65816StackRelToImg::runOnMachineFunction(MachineFunction &MF) { if (AccessCount[Off] >= kThreshold) ++HotCount; } if (HotCount > kMaxHotSlots) { + // Too many hot slots to pick from across the whole function -- but + // its innermost loops each touch few, and every call in the function + // leaves IMG0..7 alone (checked above), so they can keep theirs in + // DP as a calling function's loops do. CoreMark's matrix kernels + // (a multiply per trip) landed here and kept every pointer and + // counter on the stack. ChangedEarly |= elideStoreForwarding(MF); + if (!getenv("W65816_DISABLE_LOOP_HOMES_HOT")) + ChangedEarly |= promoteLoopLocal( + MF, getAnalysis().getLI(), + STIp.getInstrInfo(), 0xD0u); return ChangedEarly; } DenseMap OffsetToDp; // logical offset -> DP byte BitVector Taken = UsedDp; // DP words spoken for - // Caller-passed arg slots live ABOVE the return address on the stack; - // post-frame-alloc, they're at offsets >= frame_size + 4. Don't - // promote them — caller wrote to the stack location, not to DP. - int FrameSize = static_cast(MF.getFrameInfo().getStackSize()); - int ArgSlotMinOff = FrameSize + 4; + // Caller-passed arg slots live ABOVE the return address on the stack. + // Don't promote them: the caller wrote to the stack location, not to DP. + int ArgSlotMinOff = argSlotMinOff(MF); // Slots Layer 2 ptr32 derefs read as a POINTER (their `(d,S),Y` uses // require stack-rel addressing) stay put (see PromoteBlocked). auto promotable = [&](int64_t Off) { @@ -2059,7 +2566,8 @@ bool W65816StackRelToImg::runOnMachineFunction(MachineFunction &MF) { // Helpers that take one i16 arg in A and one i16 arg at 4,s, // returning i32 in A:X. __mulhi3 returns i16 not i32 but reads // same arg layout. - if (Name == "__umulhisi3" || Name == "__mulhi3") + if (Name == "__umulhisi3" || Name == "__mulhisi3" || + Name == "__mulhi3") return true; return false; } @@ -2481,7 +2989,8 @@ bool W65816StackRelToImg::runOnMachineFunction(MachineFunction &MF) { (Ops[1]->getOpcode() == W65816::PHA || Ops[1]->getOpcode() == W65816::PUSH16) && Ops[2]->getOpcode() == W65816::LDA_StackRel && - Ops[3]->getOpcode() == W65816::JSLpseudo32 && + (Ops[3]->getOpcode() == W65816::JSLpseudo32 || + Ops[3]->getOpcode() == W65816::JSLlight32) && Ops[4]->getOpcode() == W65816::PLY && Ops[5]->getOpcode() == W65816::RTL) { // JSL target must be a commutative 16x16-multiply helper. @@ -2491,7 +3000,8 @@ bool W65816StackRelToImg::runOnMachineFunction(MachineFunction &MF) { if (MO.isGlobal()) Name = MO.getGlobal()->getName(); else if (MO.isSymbol()) Name = MO.getSymbolName(); else continue; - if (Name == "__umulhisi3" || Name == "__mulhi3") { + if (Name == "__umulhisi3" || Name == "__mulhisi3" || + Name == "__mulhi3") { isCommutativeMul = true; break; } @@ -2543,8 +3053,9 @@ bool W65816StackRelToImg::runOnMachineFunction(MachineFunction &MF) { } } if (Cur->isCall()) defsA = true; - if (defsA) return 1; + // A read first: a tied op both reads and writes A (Phase 6). if (usesA) return -1; + if (defsA) return 1; ++Cur; } return 0; @@ -3353,31 +3864,8 @@ bool W65816StackRelToImg::runOnMachineFunction(MachineFunction &MF) { // takes a stack address (TSX, or a TSC that is not a frame adjust) may // read H through a pointer and is left alone. { - bool TakesStackAddr = false; - for (MachineBasicBlock &MBB : MF) { - for (auto It = MBB.begin(), E = MBB.end(); It != E; ++It) { - if (It->getOpcode() == W65816::TSX) { - TakesStackAddr = true; - break; - } - if (It->getOpcode() != W65816::TSC) continue; - // Frame adjust: TSC ; CLC|SEC ; ADC|SBC #n ; TCS. - auto N = std::next(It); - bool FrameAdjust = - N != E && (N->getOpcode() == W65816::CLC || - N->getOpcode() == W65816::SEC) && - ++N != E && (N->getOpcode() == W65816::ADC_Imm16 || - N->getOpcode() == W65816::SBC_Imm16) && - ++N != E && N->getOpcode() == W65816::TCS; - if (!FrameAdjust) { - TakesStackAddr = true; - break; - } - } - if (TakesStackAddr) break; - } DenseMap SweepEntrySp; - if (!TakesStackAddr && entrySpAdj(MF, SweepEntrySp)) { + if (!takesStackAddr(MF) && entrySpAdj(MF, SweepEntrySp)) { DenseSet IncSlots; DenseSet ReadSlots; // INC_HI_IF_CARRY_StackRel and STA_StackRel sites, with their diff --git a/src/llvm/lib/Target/W65816/W65816StackSlotCleanup.cpp b/src/llvm/lib/Target/W65816/W65816StackSlotCleanup.cpp index 2b0c8d1..af8dea9 100644 --- a/src/llvm/lib/Target/W65816/W65816StackSlotCleanup.cpp +++ b/src/llvm/lib/Target/W65816/W65816StackSlotCleanup.cpp @@ -819,25 +819,39 @@ bool W65816StackSlotCleanup::runOnMachineFunction(MachineFunction &MF) { Opc == W65816::BVC || Opc == W65816::BVS) Branches.push_back(&MI); } + // Every store: a store sets no flag, and the byte-store pseudos' + // SEP / REP touch only the M bit. A store missing here was taken for + // the flag-defining test by isFlagDefining's default, and the walk + // stopped at it: `cmp #K ; ; ; lda #0 ; beq` got + // its wrap around the LDA alone, and the BEQ read the SRL8A's AND + // (llvm816's demos/randProbe.c compared rand() with 0x41C6 so). auto isFlagPreserving = [VLAFunc](unsigned Opc) { if (VLAFunc) { // FI store pseudos are flag-corrupting under VLA expansion. if (Opc == W65816::STAfi || Opc == W65816::STAfi_indY || - Opc == W65816::STA8fi) + Opc == W65816::STA8fi || Opc == W65816::STA8fi_indY) return false; } - return Opc == W65816::STA_StackRel || - Opc == W65816::STA_StackRelIndY || - Opc == W65816::STAfi || - Opc == W65816::STAfi_indY || - Opc == W65816::STA8fi || - Opc == W65816::STA_DP || - Opc == W65816::STA_Abs || - Opc == W65816::STA_Long || - Opc == W65816::STX_DP || - Opc == W65816::STX_Abs || - Opc == W65816::STY_DP || - Opc == W65816::STY_Abs; + switch (Opc) { + case W65816::STA_StackRel: case W65816::STA_StackRelIndY: + case W65816::STAfi: case W65816::STAfi_indY: + case W65816::STA8fi: case W65816::STA8fi_indY: + case W65816::STA_DP: case W65816::STA_DPX: + case W65816::STA_DPInd: case W65816::STA_DPIndX: + case W65816::STA_DPIndY: case W65816::STA_DPIndLong: + case W65816::STA_DPIndLongY: case W65816::STB_DPIndLongY: + case W65816::STA_Abs: case W65816::STA_AbsX: case W65816::STA_AbsY: + case W65816::STA_Long: case W65816::STA_LongX: + case W65816::STAabs: case W65816::STA8abs: case W65816::STA8absX: + case W65816::STA8long: + case W65816::STX_DP: case W65816::STX_DPY: case W65816::STX_Abs: + case W65816::STY_DP: case W65816::STY_DPX: case W65816::STY_Abs: + case W65816::STZ_DP: case W65816::STZ_DPX: case W65816::STZ_Abs: + case W65816::STZ_AbsX: + return true; + default: + return false; + } }; auto isFlagDefining = [](const MachineInstr &MI) { // Anything that physically writes A, X, Y, or P updates N/Z (or @@ -852,6 +866,7 @@ bool W65816StackSlotCleanup::runOnMachineFunction(MachineFunction &MF) { case W65816::TCS: case W65816::TXS: case W65816::TCD: case W65816::JSLpseudo: case W65816::JSLpseudo32: + case W65816::JSLlight: case W65816::JSLlight32: case W65816::JSL_Long: case W65816::JSR_Abs: case W65816::JMP_Abs: @@ -922,6 +937,7 @@ bool W65816StackSlotCleanup::runOnMachineFunction(MachineFunction &MF) { // Pseudo wrappers that lower to LDA #imm. Opc == W65816::LDAfi || Opc == W65816::LDAfi_indY || + Opc == W65816::LDAimg || // Pure-logic NZ-setters: AND / ORA / EOR against an immediate // or absolute operand. These physically set N/Z but // deliberately do NOT model Defs=[P] (see W65816InstrInfo.td: @@ -1066,8 +1082,21 @@ bool W65816StackSlotCleanup::runOnMachineFunction(MachineFunction &MF) { MachineInstr *LastCorrupt = nullptr; for (auto It = std::prev(Br->getIterator()); ; --It) { MachineInstr &MI = *It; + // An FI store of anything but A moves the value through A (`lda + // dp` / TXA / TYA, perhaps inside PHA ... PLA): it sets N and Z, + // whatever the other stores do. A spill of an IMG register + // between a compare and its BEQ was taken as flag-preserving. + bool NonAStore = + (MI.getOpcode() == W65816::STAfi || + MI.getOpcode() == W65816::STA8fi) && + MI.getOperand(0).isReg() && MI.getOperand(0).getReg() != W65816::A; if (!MI.isDebugInstr()) { - if (isFlagPreserving(MI.getOpcode())) { + if (NonAStore) { + if (TestsNZ) { + if (!LastCorrupt) LastCorrupt = &MI; + FirstCorrupt = &MI; + } + } else if (isFlagPreserving(MI.getOpcode())) { // skip } else if (isCarryCorrupting(MI.getOpcode()) || (TestsNZ && isLdaLike(MI.getOpcode()))) { @@ -2197,6 +2226,103 @@ bool W65816StackSlotCleanup::runOnMachineFunction(MachineFunction &MF) { Changed = true; } + // Pass 2.4: a temp slot that only carries an IMG word to A. + // STAfi $imgN, %T, 0 ... $a = LDAfi %T, 0 + // with nothing else touching T and imgN's DP word unchanged in between + // (no def, no call, no direct-page access to it): the reload becomes + // `lda dp` (a cycle less) and Pass 2.5 drops the store. The far-pointer staging inserters route a pointer word to + // $E0 / $E2 this way; with the word in an IMG register and A live, the + // store alone was `pha ; lda dp ; sta n,s ; pla` (400 such slots in + // Lua's luaV_execute, snprintf's format and CoreMark's list code). + { + const TargetInstrInfo *TII = MF.getSubtarget().getInstrInfo(); + MachineFrameInfo &FrameInfo = MF.getFrameInfo(); + DenseMap SlotRefs; + for (MachineBasicBlock &MBB : MF) + for (MachineInstr &MI : MBB) + if (!MI.isDebugInstr()) + for (const MachineOperand &MO : MI.operands()) + if (MO.isFI()) + ++SlotRefs[MO.getIndex()]; + for (MachineBasicBlock &MBB : MF) { + for (MachineInstr &St : MBB) { + if (St.getOpcode() != W65816::STAfi || !St.getOperand(0).isReg() || + !St.getOperand(2).isImm() || St.getOperand(2).getImm() != 0) + continue; + Register Img = St.getOperand(0).getReg(); + int Dp = W65816Helpers::imgDPAddr(Img); + int FI = St.getOperand(1).getIndex(); + if (Dp < 0 || FrameInfo.isFixedObjectIndex(FI) || SlotRefs[FI] != 2) + continue; + for (auto J = std::next(St.getIterator()); J != MBB.end(); ++J) { + if (J->isDebugInstr()) + continue; + if (J->getOpcode() == W65816::LDAfi && J->getOperand(1).isFI() && + J->getOperand(1).getIndex() == FI && + J->getOperand(2).isImm() && J->getOperand(2).getImm() == 0 && + J->getOperand(0).isReg() && + J->getOperand(0).getReg() == W65816::A) { + // LDAimg, not a COPY: see its definition. + BuildMI(MBB, J, J->getDebugLoc(), TII->get(W65816::LDAimg), + W65816::A) + .addReg(Img); + St.getOperand(0).setIsKill(false); + J->eraseFromParent(); + Changed = true; + break; + } + bool Touches = J->modifiesRegister(Img, TRI); + for (const MachineOperand &MO : J->operands()) + if (MO.isFI() && MO.getIndex() == FI) + Touches = true; + if (TII->getName(J->getOpcode()).contains("_DP")) + for (const MachineOperand &MO : J->operands()) + if (MO.isImm() && (MO.getImm() == Dp || MO.getImm() == Dp + 1 || + MO.getImm() + 1 == Dp || + MO.getImm() + 2 == Dp)) + Touches = true; + if (Touches) + break; + } + } + } + } + + // Pass 2.5: stores to a local stack object that nothing reads. The + // far-pointer deref inserters spill the pointer pair to a fresh object + // to stage it into $E0 / $E2 from there; when a later deref reuses the + // staging already in place (the consecutive-deref CSE), its reloads go + // but the stores stay -- and with the pointer in an IMG word and A + // live, each costs `pha ; lda dp ; sta n,s ; pla` (CoreMark's list + // reversal loop paid two such pairs per node). Any frame-index operand + // on anything but a plain store counts as a read (address taken + // included); fixed objects are the caller's. Pass 3 then drops the + // emptied objects from the frame. + { + MachineFrameInfo &FrameInfo = MF.getFrameInfo(); + DenseSet ReadSlots; + SmallVector SlotStores; + for (MachineBasicBlock &MBB : MF) + for (MachineInstr &MI : MBB) { + if (MI.isDebugInstr()) + continue; + bool PlainStore = MI.getOpcode() == W65816::STAfi || + MI.getOpcode() == W65816::STA8fi; + for (const MachineOperand &MO : MI.operands()) + if (MO.isFI() && !PlainStore) + ReadSlots.insert(MO.getIndex()); + if (PlainStore) + SlotStores.push_back(&MI); + } + for (MachineInstr *MI : SlotStores) { + int FI = MI->getOperand(1).getIndex(); + if (FrameInfo.isFixedObjectIndex(FI) || ReadSlots.count(FI)) + continue; + MI->eraseFromParent(); + Changed = true; + } + } + // Pass 3: zero-size unused local frame objects so the // PrologueEpilogue pass shrinks the prologue PHAs / TSC reservation. // Walk the MIR collecting which FIs are still referenced; any diff --git a/src/llvm/lib/Target/W65816/W65816TargetMachine.cpp b/src/llvm/lib/Target/W65816/W65816TargetMachine.cpp index da93672..48e8d42 100644 --- a/src/llvm/lib/Target/W65816/W65816TargetMachine.cpp +++ b/src/llvm/lib/Target/W65816/W65816TargetMachine.cpp @@ -61,10 +61,12 @@ LLVMInitializeW65816Target() { initializeW65816SjLjFinalizePass(PR); initializeW65816LowerWide32Pass(PR); initializeW65816CmpSinkPass(PR); + initializeW65816SinkLoopIncPass(PR); initializeW65816I32IncFoldPass(PR); initializeW65816ImgCalleeSavePass(PR); initializeW65816NarrowI32MulPass(PR); initializeW65816CmpPreparePass(PR); + initializeW65816SplitLoopExitsPass(PR); initializeW65816PromoteFiToImgPass(PR); initializeW65816StackSlotMergePass(PR); initializeW65816StackRelToImgPass(PR); @@ -211,6 +213,9 @@ void W65816PassConfig::addISelPrepare() { // only when optimizing: -O0 keeps the straight lowering. if (getOptLevel() != CodeGenOptLevel::None) addPass(createW65816CmpPrepare()); + // Loop-exit edges get their own blocks, for the PHI copies on them. + if (getOptLevel() != CodeGenOptLevel::None) + addPass(createW65816SplitLoopExits()); TargetPassConfig::addISelPrepare(); } @@ -239,6 +244,10 @@ void W65816PassConfig::addPreRegAlloc() { // pass walks Acc16/Idx16/Img16 vregs; running this first means they // see the decomposed halves uniformly. addPass(createW65816LowerWide32()); + // Loop-carried increments below the old value's last use, before PHI + // elimination and the coalescer see them (see W65816SinkLoopInc.cpp). + if (getOptLevel() != CodeGenOptLevel::None) + addPass(createW65816SinkLoopInc()); // Keep each compare next to its conditional branch once PHI elimination // has put the back-edge copies between them (see W65816CmpSink.cpp). if (getOptLevel() != CodeGenOptLevel::None) @@ -277,7 +286,9 @@ void W65816PassConfig::addPostRegAlloc() { // SpillToX converts STA/LDA pairs to TAX/TXA bridges; StackSlotCleanup // then deletes still-adjacent redundant spills. A second SpillToX // invocation collapses any TAX/TXA pair left adjacent by cleanup - // (e.g. when an inner copy between bridge endpoints went away). + // (e.g. when an inner copy between bridge endpoints went away), and + // bridges the pairs in spill slots that stack-slot coloring shares + // between values, once cleanup has deleted the pairs it can. // Diagnostic lever: W65816_DISABLE_STACKOPT=1 turns off the stack-slot // rewriting passes (SpillToX / StackSlotCleanup / StackSlotMerge / // StackRelToImg) wholesale. Used to bisect wrong-VALUES corruption in @@ -286,7 +297,7 @@ void W65816PassConfig::addPostRegAlloc() { if (!getenv("W65816_DISABLE_STACKOPT")) { addPass(createW65816SpillToX()); addPass(createW65816StackSlotCleanup()); - addPass(createW65816SpillToX()); + addPass(createW65816SpillToX(/*SharedSlots=*/true)); } // Disable MachineCopyPropagation: it eliminates `COPY $img = $a` // thinking the IMG dest is dead (no explicit physreg use of $img @@ -340,6 +351,9 @@ void W65816PassConfig::addPreEmitPass() { // because we deliberately KEEP ADCi16imm 1 so this pass can match // it; the subsequent SepRepCleanup will see only the residual // (non-fold-eligible) ADCi16imm cases. + // Merge the stack releases on a path into a return (call cleanup plus + // epilogue) into one, before the branch distances are measured. + addPass(createW65816ReturnPopMerge()); addPass(createW65816I32IncFold()); addPass(createW65816BranchExpand()); addPass(createW65816SepRepCleanup()); @@ -360,6 +374,9 @@ void W65816PassConfig::addPreEmitPass() { // stack slots promoted), so the far-pointer staging into $E0..$E3 can be // tracked across the whole function. Branch distances only shrink. addPass(createW65816FarPtrStageReuse()); + // A load whose only use is a commutative op with the word A was just + // stored to: fold it into the op. Branch distances only shrink. + addPass(createW65816CommuteReload()); } MachineFunctionInfo *W65816TargetMachine::createMachineFunctionInfo( diff --git a/src/llvm/test/CodeGen/W65816/callee-saved-homes.ll b/src/llvm/test/CodeGen/W65816/callee-saved-homes.ll new file mode 100644 index 0000000..6cba231 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/callee-saved-homes.ll @@ -0,0 +1,143 @@ +; Pin: a far pointer kept in two stack slots across calls gets a home in +; two IMG8..15 words ($C0..$CE survive calls), so W65816FarPtrStageReuse +; addresses it in place (`lda [0xc2], y`) instead of staging it into +; $E0/$E2 before every access. The function keeps its caller's words: +; the entry SWAPS each slot with its word, every return swaps back. +; Taken only where it pays on every path, or where it pays well on +; average and some on every path: a zero-trip guard in front of the loop +; leaves the pointers on the stack. A function that takes a stack +; address keeps them there too. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s +; RUN: env W65816_DISABLE_CALLEE_SAVED_HOMES=1 llc -mtriple=w65816 -O2 < %s | FileCheck %s --check-prefix=OFF + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" + +declare void @ext(i16) +declare void @sink(ptr) + +; q and r (stack arguments) swap with $C2/$C4 and $C6/$C8 after the +; frame is made, A kept; the loop reads them in place; the return swaps +; the caller's words back before the frame goes. +define void @walk(ptr %p, ptr %q, ptr %r, i16 %n) { +; CHECK-LABEL: walk: +; CHECK: tcs +; CHECK-NEXT: pha +; CHECK-NEXT: lda 0xc2 +; CHECK-NEXT: pha +; CHECK-NEXT: lda 0x14, s +; CHECK-NEXT: sta 0xc2 +; CHECK-NEXT: pla +; CHECK-NEXT: sta 0x12, s +; CHECK: lda 0xc8 +; CHECK-NEXT: pha +; CHECK-NEXT: lda 0x1a, s +; CHECK-NEXT: sta 0xc8 +; CHECK-NEXT: pla +; CHECK-NEXT: sta 0x18, s +; CHECK-NEXT: pla +; CHECK: jsl ext +; CHECK-NEXT: ldy #0x2 +; CHECK-NEXT: lda [0xc2], y +; CHECK: lda [0xc6], y +; CHECK: lda 0x12, s +; CHECK-NEXT: sta 0xc2 +; CHECK: lda 0x18, s +; CHECK-NEXT: sta 0xc8 +; CHECK-NEXT: pla +; CHECK-NEXT: tsc +; CHECK-NEXT: clc +; CHECK-NEXT: adc #0xc +; CHECK-NEXT: tcs +; CHECK-NEXT: rtl +; OFF-LABEL: walk: +; OFF-NOT: lda [0xc2], y +; OFF: rtl +entry: + br label %loop + +loop: + %i = phi i16 [ 0, %entry ], [ %i1, %loop ] + %a = load i16, ptr %p, align 2 + call void @ext(i16 %a) + %qb = getelementptr inbounds i16, ptr %q, i32 1 + %b = load i16, ptr %qb, align 2 + %b1 = add i16 %b, %i + store i16 %b1, ptr %qb, align 2 + %rc = getelementptr inbounds i16, ptr %r, i32 2 + %cc = load i16, ptr %rc, align 2 + %c1 = xor i16 %cc, %b1 + store i16 %c1, ptr %rc, align 2 + %pc = getelementptr inbounds i16, ptr %p, i32 2 + store i16 %c1, ptr %pc, align 2 + %i1 = add i16 %i, 1 + %e = icmp eq i16 %i1, %n + br i1 %e, label %done, label %loop + +done: + ret void +} + +; The same loop behind a zero-trip guard: the call that skips it would +; pay for the swaps and save nothing. +define void @guarded(ptr %p, ptr %q, ptr %r, i16 %n) { +; CHECK-LABEL: guarded: +; CHECK-NOT: lda [0xc2], y +; CHECK: rtl +entry: + %c0 = icmp sgt i16 %n, 0 + br i1 %c0, label %loop, label %done + +loop: + %i = phi i16 [ 0, %entry ], [ %i1, %loop ] + %a = load i16, ptr %p, align 2 + call void @ext(i16 %a) + %qb = getelementptr inbounds i16, ptr %q, i32 1 + %b = load i16, ptr %qb, align 2 + %b1 = add i16 %b, %i + store i16 %b1, ptr %qb, align 2 + %rc = getelementptr inbounds i16, ptr %r, i32 2 + %cc = load i16, ptr %rc, align 2 + %c1 = xor i16 %cc, %b1 + store i16 %c1, ptr %rc, align 2 + %pc = getelementptr inbounds i16, ptr %p, i32 2 + store i16 %c1, ptr %pc, align 2 + %i1 = add i16 %i, 1 + %e = icmp eq i16 %i1, %n + br i1 %e, label %done, label %loop + +done: + ret void +} + +; A stack address handed out: the callee may read or write any slot. +define void @addrTaken(ptr %p, ptr %q, ptr %r, i16 %n) { +; CHECK-LABEL: addrTaken: +; CHECK-NOT: lda [0xc2], y +; CHECK: rtl +entry: + %buf = alloca [4 x i16], align 2 + call void @sink(ptr %buf) + br label %loop + +loop: + %i = phi i16 [ 0, %entry ], [ %i1, %loop ] + %a = load i16, ptr %p, align 2 + call void @ext(i16 %a) + %qb = getelementptr inbounds i16, ptr %q, i32 1 + %b = load i16, ptr %qb, align 2 + %b1 = add i16 %b, %i + store i16 %b1, ptr %qb, align 2 + %rc = getelementptr inbounds i16, ptr %r, i32 2 + %cc = load i16, ptr %rc, align 2 + %c1 = xor i16 %cc, %b1 + store i16 %c1, ptr %rc, align 2 + %pc = getelementptr inbounds i16, ptr %p, i32 2 + store i16 %c1, ptr %pc, align 2 + %i1 = add i16 %i, 1 + %e = icmp eq i16 %i1, %n + br i1 %e, label %done, label %loop + +done: + ret void +} diff --git a/src/llvm/test/CodeGen/W65816/cmp-sink.ll b/src/llvm/test/CodeGen/W65816/cmp-sink.ll index eb8319b..e859a10 100644 --- a/src/llvm/test/CodeGen/W65816/cmp-sink.ll +++ b/src/llvm/test/CodeGen/W65816/cmp-sink.ll @@ -7,6 +7,12 @@ ; end of a CMP / SBC carry chain is better left next to its first half ; (cmp-prepare.ll's postinc_ult pins that shape). ; +; Since W65816SinkLoopInc the pointer's step lands below the load through +; it, so the pointer is bumped in place and only the sum's back-edge COPY +; is left above each compare. (The loop is what is pinned: since +; W65816SplitLoopExits the entry block's copy of the zero-trip result may +; land after its compare, a wrap paid once per call.) +; ; RUN: llc -mtriple=w65816 -O2 -print-after=w65816-cmp-sink -o /dev/null < %s 2>&1 | FileCheck %s --check-prefix=MIR ; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s @@ -14,18 +20,17 @@ target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16- define i16 @sum16(ptr %p, ptr %end) { ; MIR-LABEL: Machine code for function sum16 +; MIR: [[LO:%[0-9]+]]:acc16 = ADCi16imm [[LO]]:acc16(tied-def 0), 2 +; MIR-NEXT: [[HI:%[0-9]+]]:acc16 = ADCEi16imm [[HI]]:acc16(tied-def 0), 0 ; MIR: ADCfi ; MIR: COPY -; MIR-NEXT: COPY -; MIR-NEXT: COPY -; MIR-NEXT: CMPfi +; MIR-NEXT: CMPfi [[LO]]:acc16 ; MIR-NEXT: BNE ; MIR: COPY -; MIR-NEXT: COPY -; MIR-NEXT: COPY -; MIR-NEXT: CMPfi +; MIR-NEXT: CMPfi [[HI]]:acc16 ; MIR-NEXT: BNE ; CHECK-LABEL: sum16: +; CHECK: Loop Header ; CHECK-NOT: php ; CHECK: rtl entry: diff --git a/src/llvm/test/CodeGen/W65816/cmp-via-x.mir b/src/llvm/test/CodeGen/W65816/cmp-via-x.mir index 936dde9..e8aa854 100644 --- a/src/llvm/test/CodeGen/W65816/cmp-via-x.mir +++ b/src/llvm/test/CodeGen/W65816/cmp-via-x.mir @@ -5,7 +5,8 @@ # and C from X - k as CMP did from A - k. It needs X free from the test to # the end of the block and not live into a successor. # -# via_x: X free: TAX / CPX, no wrap. +# via_x: X free: TAX / CPX, no wrap. (bb.1 reads the stored slot so the +# store is not dropped as never read.) # x_live_out: X live into a branch target: the wrap stays. # x_in_region: the corrupter reads X (TXA): the wrap stays. # stack_operand: `cmp d,s` has no CPX form: the wrap stays. @@ -49,6 +50,7 @@ body: | BNE %bb.1, implicit $p BRA %bb.2 bb.1: + $a = LDAfi %stack.1, 0 RTL bb.2: RTL diff --git a/src/llvm/test/CodeGen/W65816/commute-reload.ll b/src/llvm/test/CodeGen/W65816/commute-reload.ll new file mode 100644 index 0000000..d48d464 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/commute-reload.ll @@ -0,0 +1,35 @@ +; Pin: W65816CommuteReload folds a load into the commutative op that +; follows it when the op's other operand is the word A was just stored to: +; `sta 0x1, s ; lda 0xd0 ; ora 0x1, s` is `sta 0x1, s ; ora 0xd0` (A +; already equals the slot). A far pointer's null test, at the top of +; nearly every JoeyLib entry point, is the common case: 4 cycles and 2 +; bytes each. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s +; RUN: env W65816_DISABLE_COMMUTE_RELOAD=1 llc -mtriple=w65816 -O2 < %s | FileCheck %s --check-prefix=OFF + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" + +declare void @f(ptr) + +; CHECK-LABEL: nullTest: +; CHECK: stx 0xd0 +; CHECK-NEXT: sta 0x1, s +; CHECK-NEXT: ora 0xd0 +; CHECK-NEXT: beq +; OFF-LABEL: nullTest: +; OFF: stx 0xd0 +; OFF-NEXT: sta 0x1, s +; OFF-NEXT: lda 0xd0 +; OFF-NEXT: ora 0x1, s +; OFF-NEXT: beq +define void @nullTest(ptr %p) { +entry: + %c = icmp eq ptr %p, null + br i1 %c, label %done, label %call +call: + call void @f(ptr %p) + br label %done +done: + ret void +} diff --git a/src/llvm/test/CodeGen/W65816/dp-inc-fold-tied-use.ll b/src/llvm/test/CodeGen/W65816/dp-inc-fold-tied-use.ll new file mode 100644 index 0000000..7dd4ca8 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/dp-inc-fold-tied-use.ll @@ -0,0 +1,59 @@ +; Pin: StackRelToImg folds `lda dp ; inc a ; sta dp` into `inc dp` only when +; nothing reads A after the store. Its A-liveness scan checked an +; instruction's def of A before its use, so a tied op that reads and +; writes A (`$a = ANDi16imm $a, 511`) counted as overwriting it, and a ring +; index's `head = (head + 1) & 511` became `inc dp ; and #0x1ff ; sta dp`, +; masking whatever A held (the byte just copied). JoeyLib's IIgs serial +; read returned that garbage as its new head and never caught up with the +; tail: SERLOSS hung in its read loop (verify-iigs-serial-loss, scc). The +; Phase 11 STZ peephole's scan had the same order. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S8" +target triple = "w65816" + +@gHead = external global i16, align 2 +@gTail = external global i16, align 2 +@gRing = external global [512 x i8], align 1 +@gRead = external global i16, align 2 + +; CHECK-LABEL: rd: +; CHECK: inc a +; CHECK-NEXT: and #0x1ff +; CHECK-NEXT: sta 0x[[HEAD:[0-9a-f]+]] +define i16 @rd(ptr %buf, i16 zeroext %max) { +entry: + %0 = load volatile i16, ptr @gHead, align 2 + %1 = load volatile i16, ptr @gTail, align 2 + %cmp12 = icmp ne i16 %max, 0 + %cmp113 = icmp ne i16 %0, %1 + %2 = select i1 %cmp12, i1 %cmp113, i1 false + br i1 %2, label %for.body, label %for.end + +for.body: + %n.015 = phi i16 [ %inc, %for.body ], [ 0, %entry ] + %head.014 = phi i16 [ %and, %for.body ], [ %0, %entry ] + %idxprom = zext i16 %head.014 to i32 + %arrayidx = getelementptr inbounds nuw i8, ptr @gRing, i32 %idxprom + %3 = load i8, ptr %arrayidx, align 1 + %idxprom2 = zext i16 %n.015 to i32 + %arrayidx3 = getelementptr inbounds nuw i8, ptr %buf, i32 %idxprom2 + store i8 %3, ptr %arrayidx3, align 1 + %add = add i16 %head.014, 1 + %and = and i16 %add, 511 + %inc = add nuw i16 %n.015, 1 + %cmp = icmp ult i16 %inc, %max + %cmp1 = icmp ne i16 %and, %1 + %4 = select i1 %cmp, i1 %cmp1, i1 false + br i1 %4, label %for.body, label %for.end + +for.end: + %head.0.lcssa = phi i16 [ %0, %entry ], [ %and, %for.body ] + %n.0.lcssa = phi i16 [ 0, %entry ], [ %inc, %for.body ] + store volatile i16 %head.0.lcssa, ptr @gHead, align 2 + %5 = load volatile i16, ptr @gRead, align 2 + %add4 = add i16 %5, %n.0.lcssa + store volatile i16 %add4, ptr @gRead, align 2 + ret i16 %n.0.lcssa +} diff --git a/src/llvm/test/CodeGen/W65816/far-index-y.ll b/src/llvm/test/CodeGen/W65816/far-index-y.ll new file mode 100644 index 0000000..a32b3a1 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/far-index-y.ll @@ -0,0 +1,65 @@ +; Pin: a far-pointer access at `p + zext(i) + K` puts i + K in Y when that +; sum provably fits 16 bits (`[dp],Y` adds Y to the whole 24-bit pointer, +; as the i32 add carried into the bank byte). It used to compute p + i +; with an i32 add (`clc ; adc ; sta ; lda ; adc #0 ; sta`) and stage that +; into $E0 / $E2. Space Taxi's glyph lookups (`sim->glyphSlot[ch]`, a u8 +; index at offset 0x215c) were the motivating case. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" + +; u8 index plus a field offset: Y = ch + 0x215c, no i32 add. +define i8 @byteIndexOffset(ptr %sim, i8 zeroext %ch) { +; CHECK-LABEL: byteIndexOffset: +; CHECK: and #0xff +; CHECK-NEXT: clc +; CHECK-NEXT: adc #0x215c +; CHECK-NEXT: tay +; CHECK-NOT: adc #0x0 +; CHECK: lda [0x{{[0-9a-f]+}}], y +; CHECK: rtl + %f = getelementptr inbounds i8, ptr %sim, i32 8540 + %i = zext i8 %ch to i32 + %p = getelementptr inbounds i8, ptr %f, i32 %i + %v = load i8, ptr %p, align 1 + ret i8 %v +} + +; A full u16 index at offset 0: Y = i. +define void @storeWord16(ptr %p, i16 %i, i16 %v) { +; CHECK-LABEL: storeWord16: +; CHECK-NOT: adc #0x0 +; CHECK: tay +; CHECK-NOT: adc #0x0 +; CHECK: sta [0x{{[0-9a-f]+}}], y +; CHECK: rtl + %i32 = zext i16 %i to i32 + %q = getelementptr inbounds i8, ptr %p, i32 %i32 + store i16 %v, ptr %q, align 1 + ret void +} + +; A u16 index plus a nonzero offset could carry past Y's 16 bits: the +; i32 add stays. +define i8 @word16IndexOffset(ptr %p, i16 %i) { +; CHECK-LABEL: word16IndexOffset: +; CHECK: adc #0x0 +; CHECK: rtl + %f = getelementptr inbounds i8, ptr %p, i32 4 + %i32 = zext i16 %i to i32 + %q = getelementptr inbounds i8, ptr %f, i32 %i32 + %v = load i8, ptr %q, align 1 + ret i8 %v +} + +; A signed index may be negative: no fold. +define i8 @signedIndex(ptr %p, i16 %i) { +; CHECK-LABEL: signedIndex: +; CHECK: adc +; CHECK: rtl + %i32 = sext i16 %i to i32 + %q = getelementptr inbounds i8, ptr %p, i32 %i32 + %v = load i8, ptr %q, align 1 + ret i8 %v +} diff --git a/src/llvm/test/CodeGen/W65816/fp-cmp-const-int.ll b/src/llvm/test/CodeGen/W65816/fp-cmp-const-int.ll new file mode 100644 index 0000000..e226fb2 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/fp-cmp-const-int.ll @@ -0,0 +1,154 @@ +; Pin: a float / double compare that the operand's bits alone decide is an +; integer compare, not a soft-float libcall (combineFPSetCCConst). +; InstCombine makes `(bits & ~sign) == 0` into `fcmp oeq x, 0.0` and a +; magnitude test against the exponent mask into `fcmp uno x, 0.0` or +; `fcmp olt fabs(x), C`; each used to call __eqsf2 / __unordsf2 / __ltsf2. +; +; x == +-0.0 (bits & 0x7fff..) == 0, the sign word masked +; x == C (C != 0) bits == bits(C) +; x uno / ord C (bits & 0x7fff..) > / <= bits(+inf) +; fabs(x) < C, C >= 0 (bits & 0x7fff..) < bits(C), unsigned +; x > 0.0 bits in [1, bits(+inf)]: (bits - 1) < 0x7f800000 +; x ult 0.0 (double) !(x >= 0): two compares (bits <= inf, bits == -0), +; still no libcall +; +; Compares of two variables, and relational compares against a constant +; other than a zero (x < 2.5), stay libcalls. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s + +target triple = "w65816" + +define i16 @f_oeq0(float %x) { +; CHECK-LABEL: f_oeq0: +; CHECK-NOT: jsl +; CHECK: and #0x7fff +; CHECK: rtl + %c = fcmp oeq float %x, 0.0 + %r = zext i1 %c to i16 + ret i16 %r +} + +define i16 @f_une_m0(float %x) { +; CHECK-LABEL: f_une_m0: +; CHECK-NOT: jsl +; CHECK: and #0x7fff +; CHECK: bne +; CHECK: rtl + %c = fcmp une float %x, -0.0 + %r = zext i1 %c to i16 + ret i16 %r +} + +define i16 @f_oeq1(float %x) { +; CHECK-LABEL: f_oeq1: +; CHECK-NOT: jsl +; CHECK: eor #0x3f80 +; CHECK: rtl + %c = fcmp oeq float %x, 1.0 + %r = zext i1 %c to i16 + ret i16 %r +} + +; isnan: |bits| >= 0x7f800001, one CMP/SBC chain. +define i16 @f_uno(float %x) { +; CHECK-LABEL: f_uno: +; CHECK-NOT: jsl +; CHECK: and #0x7fff +; CHECK: cmp #0x1 +; CHECK: sbc #0x7f80 +; CHECK-NEXT: lda #0x1 +; CHECK-NEXT: bcs +; CHECK: rtl + %c = fcmp uno float %x, 0.0 + %r = zext i1 %c to i16 + ret i16 %r +} + +define i16 @f_fabs_olt(float %x) { +; CHECK-LABEL: f_fabs_olt: +; CHECK-NOT: jsl +; CHECK: and #0x7fff +; CHECK: cmp #0x4020 +; CHECK-NEXT: bcc +; CHECK: rtl + %a = call float @llvm.fabs.f32(float %x) + %c = fcmp olt float %a, 2.5 + %r = zext i1 %c to i16 + ret i16 %r +} + +define i16 @f_ogt0(float %x) { +; CHECK-LABEL: f_ogt0: +; CHECK-NOT: jsl +; CHECK: adc #0xffff +; CHECK: adc #0xffff +; CHECK: cmp #0x7f80 +; CHECK-NEXT: bcc +; CHECK: rtl + %c = fcmp ogt float %x, 0.0 + %r = zext i1 %c to i16 + ret i16 %r +} + +define i16 @f_olt_var(float %x, float %y) { +; CHECK-LABEL: f_olt_var: +; CHECK: jsl __ltsf2 +; CHECK: rtl + %c = fcmp olt float %x, %y + %r = zext i1 %c to i16 + ret i16 %r +} + +define i16 @f_olt25(float %x) { +; CHECK-LABEL: f_olt25: +; CHECK: jsl __ltsf2 +; CHECK: rtl + %c = fcmp olt float %x, 2.5 + %r = zext i1 %c to i16 + ret i16 %r +} + +; The sign word (the 4th, on the stack) is masked; all four words ORed. +define i16 @d_oeq0(double %x) { +; CHECK-LABEL: d_oeq0: +; CHECK-NOT: jsl +; CHECK: lda #0x7fff +; CHECK-NEXT: and 0x{{[0-9a-f]+}}, s +; CHECK: ora 0x{{[0-9a-f]+}}, s +; CHECK: rtl + %c = fcmp oeq double %x, 0.0 + %r = zext i1 %c to i16 + ret i16 %r +} + +; !isnan: |bits| < 0x7ff0000000000001 as one four-word CMP/SBC chain. +define i16 @d_ord(double %x) { +; CHECK-LABEL: d_ord: +; CHECK-NOT: jsl +; CHECK: cmp #0x1 +; CHECK-NEXT: lda 0xd0 +; CHECK-NEXT: sbc #0x0 +; CHECK-NEXT: lda 0x6, s +; CHECK-NEXT: sbc #0x0 +; CHECK-NEXT: lda 0x1, s +; CHECK-NEXT: sbc #0x7ff0 +; CHECK-NEXT: bcc +; CHECK: rtl + %c = fcmp ord double %x, 0.0 + %r = zext i1 %c to i16 + ret i16 %r +} + +define i16 @d_ult0(double %x) { +; CHECK-LABEL: d_ult0: +; CHECK-NOT: jsl +; CHECK: eor #0x8000 +; CHECK: sbc #0x7ff0 +; CHECK: rtl + %c = fcmp ult double %x, 0.0 + %r = zext i1 %c to i16 + ret i16 %r +} + +declare float @llvm.fabs.f32(float) diff --git a/src/llvm/test/CodeGen/W65816/frame-order-density.mir b/src/llvm/test/CodeGen/W65816/frame-order-density.mir new file mode 100644 index 0000000..7ec671e --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/frame-order-density.mir @@ -0,0 +1,92 @@ +# Pin: W65816FrameLowering::orderFrameObjects lays a large frame out with +# its most-used objects nearest S. A slot more than 0xFF bytes above S is +# out of `off,s` reach, so every access to it goes through the $F6 frame +# pointer (`sty $fa ; ldy #off ; lda [$f6],y ; ldy $fa`, 18 cycles against +# 5). In creation order the 2-byte scalar below sat on top of the 300-byte +# array it was created before; by access density (direct accesses, 8x per +# loop level, per byte) it now sits at S+1 and the array, reached only +# through the address ADDframe builds, goes on top. +# +# pha_pinned: an object accessed past its end is the PHA-bracket idiom +# (W65816ImgCalleeSave saves IMG8..15 with `pha ; lda dp ; sta slot+2,s ; +# pla`), whose +2 holds only stack-relative: through $F6, which the PHA does +# not move, it addresses the neighbour. Such objects stay nearest S however +# rarely they are used. The first cut sorted runtime snprintf's IMG8 save +# slot out of reach: `format` restored IMG8 from its neighbour and fprintf +# wrote nothing (the smoke test's GS/OS fflush check). +# +# RUN: llc -mtriple=w65816 -run-pass=prologepilog -o - %s | FileCheck %s +# RUN: env W65816_DISABLE_FRAME_ORDER=1 llc -mtriple=w65816 -run-pass=prologepilog -o - %s | FileCheck %s --check-prefix=CREATION +--- | + target triple = "w65816" + define void @hot_near() { ret void } + define void @pha_pinned() { ret void } +... +--- +# CHECK-LABEL: name: hot_near +# CHECK-NOT: DPIndLongY 246 +# CHECK: STA_StackRel 1 +# CHECK-NOT: DPIndLongY 246 +# CHECK: LDA_StackRel 1 +# CREATION-LABEL: name: hot_near +# CREATION: STA_DPIndLongY 246 +name: hot_near +tracksRegLiveness: false +stack: + - { id: 0, size: 2, alignment: 1 } + - { id: 1, size: 300, alignment: 1 } +body: | + bb.0: + $a = ADDframe %stack.1, 0 + STAfi $a, %stack.0, 0, implicit-def $a + $a = LDAi16imm 0 + $a = LDAfi %stack.0, 0 + RTL +... +--- +# CHECK-LABEL: name: pha_pinned +# CHECK: PHA +# CHECK-NEXT: LDA_DP 192 +# CHECK-NEXT: STA_StackRel 3 +# CHECK-NEXT: PLA +# The array's 20 loop accesses outweigh the save slot's one (by density), +# which alone would sort the slot above the array, out of reach. +name: pha_pinned +tracksRegLiveness: false +stack: + - { id: 0, size: 2, alignment: 1 } + - { id: 1, size: 300, alignment: 1 } +body: | + bb.0: + successors: %bb.1 + PHA implicit $a + LDA_DP 192, implicit-def $a + STAfi $a, %stack.0, 2, implicit-def $a + PLA implicit-def $a + bb.1: + successors: %bb.1, %bb.2 + $a = LDAfi %stack.1, 0 + $a = LDAfi %stack.1, 2 + $a = LDAfi %stack.1, 4 + $a = LDAfi %stack.1, 6 + $a = LDAfi %stack.1, 8 + $a = LDAfi %stack.1, 10 + $a = LDAfi %stack.1, 12 + $a = LDAfi %stack.1, 14 + $a = LDAfi %stack.1, 16 + $a = LDAfi %stack.1, 18 + $a = LDAfi %stack.1, 20 + $a = LDAfi %stack.1, 22 + $a = LDAfi %stack.1, 24 + $a = LDAfi %stack.1, 26 + $a = LDAfi %stack.1, 28 + $a = LDAfi %stack.1, 30 + $a = LDAfi %stack.1, 32 + $a = LDAfi %stack.1, 34 + $a = LDAfi %stack.1, 36 + $a = LDAfi %stack.1, 38 + CMPi16imm $a, 0, implicit-def $p + BNE %bb.1, implicit $p + bb.2: + RTL +... diff --git a/src/llvm/test/CodeGen/W65816/global-index-abs.ll b/src/llvm/test/CodeGen/W65816/global-index-abs.ll new file mode 100644 index 0000000..690ed60 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/global-index-abs.ll @@ -0,0 +1,130 @@ +; Pin: an access to a global array at a variable index is `lda G+K,x` / +; `sta G+K,x` (DBR-relative absolute indexed, the bank every global lives +; in), not a far pointer `#G` + `lda $BE` staged into $E0/$E2 for +; `lda [$E0],y`. The access is based on G, so a valid program keeps +; the offset inside G (at most 64K): X is that offset's low word. A +; constant K stays in the operand only when the index is known not +; negative; otherwise it is added into X. A global of unknown size +; keeps the far pointer. Space Taxi's per-cell table lookups +; (kCellRow[cell >> 3], gRender.dirtyColMin[row]) were the motivating +; case. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" + +%struct.R = type { [25 x i8], [25 x i8], [10 x i16] } + +@tab = dso_local global [256 x i16] zeroinitializer, align 2 +@bytes = dso_local global [125 x i8] zeroinitializer, align 1 +@longs = dso_local global [16 x i32] zeroinitializer, align 2 +@rec = dso_local global %struct.R zeroinitializer, align 2 +@vbytes = dso_local global [8 x i8] zeroinitializer, align 1 +@unsized = external global [0 x i8], align 1 + +; u16 element, u16 index: X = i << 1. +define i16 @wordIndex(i16 zeroext %i) { +; CHECK-LABEL: wordIndex: +; CHECK: asl a +; CHECK-NEXT: tax +; CHECK-NEXT: lda tab, x +; CHECK-NOT: 0xbe +; CHECK: rtl + %idx = zext i16 %i to i32 + %p = getelementptr inbounds [256 x i16], ptr @tab, i32 0, i32 %idx + %v = load i16, ptr %p, align 2 + ret i16 %v +} + +; A byte reads the word and masks. The index is a right shift of a +; zero-extended value: computed in 16 bits. +define i16 @byteShiftIndex(i16 zeroext %cell) { +; CHECK-LABEL: byteShiftIndex: +; CHECK: lsr a +; CHECK-NEXT: lsr a +; CHECK-NEXT: lsr a +; CHECK-NEXT: tax +; CHECK-NEXT: lda bytes, x +; CHECK-NEXT: and #0xff +; CHECK-NOT: 0xbe +; CHECK: rtl + %c = zext i16 %cell to i32 + %s = lshr i32 %c, 3 + %p = getelementptr inbounds i8, ptr @bytes, i32 %s + %v = load i8, ptr %p, align 1 + %r = zext i8 %v to i16 + ret i16 %r +} + +; A field offset with a known non-negative index stays in the operand. +define void @fieldByteStore(i8 zeroext %r, i8 %v) { +; CHECK-LABEL: fieldByteStore: +; CHECK: tax +; CHECK: sep #0x20 +; CHECK-NEXT: sta rec+25, x +; CHECK-NEXT: rep #0x20 +; CHECK-NOT: 0xbe +; CHECK: rtl + %i = zext i8 %r to i32 + %f = getelementptr inbounds i8, ptr @rec, i32 25 + %p = getelementptr inbounds i8, ptr %f, i32 %i + store i8 %v, ptr %p, align 1 + ret void +} + +; A signed index may be negative: the field offset moves into X. +define i16 @fieldSignedIndex(i16 %i) { +; CHECK-LABEL: fieldSignedIndex: +; CHECK: asl a +; CHECK-NEXT: clc +; CHECK-NEXT: adc #0x32 +; CHECK-NEXT: tax +; CHECK-NEXT: lda rec, x +; CHECK: rtl + %s = sext i16 %i to i32 + %p = getelementptr inbounds %struct.R, ptr @rec, i32 0, i32 2, i32 %s + %v = load i16, ptr %p, align 2 + ret i16 %v +} + +; An i32 element: both halves off the same X. +define i32 @longIndex(i16 zeroext %i) { +; CHECK-LABEL: longIndex: +; CHECK: asl a +; CHECK-NEXT: asl a +; CHECK-NEXT: tax +; CHECK-NEXT: lda longs, x +; CHECK: lda longs+2, x +; CHECK-NOT: 0xbe +; CHECK: rtl + %idx = zext i16 %i to i32 + %p = getelementptr inbounds [16 x i32], ptr @longs, i32 0, i32 %idx + %v = load i32, ptr %p, align 2 + ret i32 %v +} + +; A volatile byte keeps its 8-bit access. +define i8 @volatileByte(i8 zeroext %i) { +; CHECK-LABEL: volatileByte: +; CHECK: tax +; CHECK-NEXT: sep #0x20 +; CHECK-NEXT: lda vbytes, x +; CHECK-NEXT: rep #0x20 +; CHECK: rtl + %idx = zext i8 %i to i32 + %p = getelementptr inbounds [8 x i8], ptr @vbytes, i32 0, i32 %idx + %v = load volatile i8, ptr %p, align 1 + ret i8 %v +} + +; No size, no bound: the far pointer stays. +define i8 @unsizedByte(i16 zeroext %i) { +; CHECK-LABEL: unsizedByte: +; CHECK: lda 0xbe +; CHECK: lda [0xe0], y +; CHECK: rtl + %idx = zext i16 %i to i32 + %p = getelementptr inbounds i8, ptr @unsized, i32 %idx + %v = load i8, ptr %p, align 1 + ret i8 %v +} diff --git a/src/llvm/test/CodeGen/W65816/i32-compare-chain.ll b/src/llvm/test/CodeGen/W65816/i32-compare-chain.ll index 95f2c0d..4e0c3ab 100644 --- a/src/llvm/test/CodeGen/W65816/i32-compare-chain.ll +++ b/src/llvm/test/CodeGen/W65816/i32-compare-chain.ll @@ -434,11 +434,13 @@ f: } ; Against 0 and -1, and constants whose low half is 0: no chain at all. +; (W65816CommuteReload folds `lda 0xd0 ; ora 0x1, s` after the store of +; A to 0x1, s into `ora 0xd0`.) define void @brk_eq0(i32 %a) { ; CHECK-LABEL: brk_eq0: ; CHECK-NOT: eor -; CHECK: lda 0xd0 -; CHECK-NEXT: ora 0x1, s +; CHECK: sta 0x1, s +; CHECK-NEXT: ora 0xd0 ; CHECK-NEXT: beq ; CHECK: rtl entry: diff --git a/src/llvm/test/CodeGen/W65816/i32-half-offset-fold.ll b/src/llvm/test/CodeGen/W65816/i32-half-offset-fold.ll new file mode 100644 index 0000000..a5d2ac6 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/i32-half-offset-fold.ll @@ -0,0 +1,82 @@ +; Pin: both halves of an i32 load or store through a far pointer address +; the pointer's base with the field offset in Y. The high half of an +; i32 load at `p + K` is a load at `(p + K) + 2`; peeling only the outer +; +2 left it addressed through the computed `p + K` (an i32 add and a +; staging of its result into $E0 / $E2 per access). The i32 store path +; did not peel at all: its high half always took `p + 2` through an i32 +; add. A `+ K` whose constant was already split into halves by +; LowerI32Constant now peels too. CoreMark's linked-list code +; (list->info, list->next) went -7% total time with this. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" + +; The pointer field at offset 4: low half at Y=4, high half at Y=6, no +; carry chain (the `adc #0x0` of an i32 address add). +define ptr @getInfo(ptr %n) { +; CHECK-LABEL: getInfo: +; CHECK-NOT: adc #0x0 +; CHECK-DAG: ldy #0x4 +; CHECK-DAG: ldy #0x6 +; CHECK-NOT: adc #0x0 +; CHECK: rtl + %f = getelementptr inbounds i8, ptr %n, i32 4 + %v = load ptr, ptr %f, align 2 + ret ptr %v +} + +; An i32 store at offset 8: Y=8 then Y=10 off the incoming pointer. +define void @setCnt(ptr %n, i32 %v) { +; CHECK-LABEL: setCnt: +; CHECK-NOT: adc #0x0 +; CHECK: ldy #0x8 +; CHECK-NEXT: sta [0x{{[0-9a-f]+}}], y +; CHECK-NOT: adc #0x0 +; CHECK: ldy #0xa +; CHECK-NEXT: sta [0x{{[0-9a-f]+}}], y +; CHECK-NOT: adc #0x0 +; CHECK: rtl + %f = getelementptr inbounds i8, ptr %n, i32 8 + store i32 %v, ptr %f, align 2 + ret void +} + +; An i32 store at offset 0: the high half is Y=2 off the same base. +define void @setFirst(ptr %n, i32 %v) { +; CHECK-LABEL: setFirst: +; CHECK-NOT: adc #0x0 +; CHECK: ldy #0x2 +; CHECK-NOT: adc #0x0 +; CHECK: rtl + store i32 %v, ptr %n, align 2 + ret void +} + +; A read-modify-write of an i32 element (CoreMark's transition_count[i]++): +; four [p],Y accesses at Y=8 / Y=10, the only add the increment's. +define void @bump(ptr %tc) { +; CHECK-LABEL: bump: +; CHECK-NOT: adc #0x2 +; CHECK-NOT: adc #0x8 +; CHECK-DAG: ldy #0x8 +; CHECK-DAG: ldy #0xa +; CHECK-NOT: adc #0x2 +; CHECK-NOT: adc #0x8 +; CHECK: rtl + %p = getelementptr inbounds i8, ptr %tc, i32 8 + %v = load i32, ptr %p, align 2 + %w = add i32 %v, 1 + store i32 %w, ptr %p, align 2 + ret void +} + +; Offsets that no longer fit Y fall back to the i32 add. +define void @setFar(ptr %n, i32 %v) { +; CHECK-LABEL: setFar: +; CHECK: adc #0xfffe +; CHECK: rtl + %f = getelementptr inbounds i8, ptr %n, i32 65534 + store i32 %v, ptr %f, align 2 + ret void +} diff --git a/src/llvm/test/CodeGen/W65816/i64-compare-chain.ll b/src/llvm/test/CodeGen/W65816/i64-compare-chain.ll new file mode 100644 index 0000000..4764ebe --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/i64-compare-chain.ll @@ -0,0 +1,125 @@ +; Pin: an i64 compare is one four-word CMP/SBC chain (lowerI32Cmp, reached +; from the type legalizer: SETCC / BR_CC / SELECT_CC are Custom for i64), +; not the generic expansion's per-half compares joined by selects. +; +; lda a.w0 ; cmp b.w0 ; lda a.w1 ; sbc b.w1 ; ... ; sbc b.w3 ; bcc/bcs +; +; Signed predicates bias both top words by 0x8000. Constant low words of 0 +; drop out of the chain (x < 0x100000000 compares words 2 and 3 only). +; eq / ne stay the type legalizer's own expansion, which ORs the four XORs +; together and tests once. Its expansion of the ordered predicates was ~50 +; instructions per compare, and the baseline compiler answered 63 of the +; i64 smoke check's value-form compares against constants wrongly +; (StackRelToImg promoted an incoming argument word to DP; see +; stackrel-to-img-arg-slot.ll). +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s + +target triple = "w65816" + +@sink = global i16 0 + +; a.w0 in A, a.w1 in X, the rest on the stack. +define void @br_ult(i64 %a, i64 %b) { +; CHECK-LABEL: br_ult: +; CHECK: stx 0xd0 +; CHECK-NEXT: sta 0xd2 +; CHECK-NEXT: cmp 0x8, s +; CHECK-NEXT: lda 0xd0 +; CHECK-NEXT: sbc 0xa, s +; CHECK-NEXT: lda 0x4, s +; CHECK-NEXT: sbc 0xc, s +; CHECK-NEXT: lda 0x6, s +; CHECK-NEXT: sbc 0xe, s +; CHECK-NEXT: bcs +; CHECK: rtl +entry: + %c = icmp ult i64 %a, %b + br i1 %c, label %t, label %f +t: + store volatile i16 1, ptr @sink + ret void +f: + store volatile i16 2, ptr @sink + ret void +} + +define i16 @va_slt(i64 %a, i64 %b) { +; CHECK-LABEL: va_slt: +; CHECK: lda #0x8000 +; CHECK: eor 0x12, s +; CHECK: eor 0xa, s +; CHECK: cmp 0xc, s +; CHECK-NEXT: lda 0xd0 +; CHECK-NEXT: sbc 0xe, s +; CHECK-NEXT: lda 0x8, s +; CHECK-NEXT: sbc 0x10, s +; CHECK-NEXT: lda 0xd4 +; CHECK-NEXT: sbc 0xd6 +; CHECK-NEXT: bcc +; CHECK: rtl + %c = icmp slt i64 %a, %b + %r = zext i1 %c to i16 + ret i16 %r +} + +; A select on the compare branches straight on the chain's carry. +define i16 @se_uge(i64 %a, i64 %b) { +; CHECK-LABEL: se_uge: +; CHECK: cmp 0x8, s +; CHECK: sbc 0xe, s +; CHECK-NEXT: bcs +; CHECK: lda #0x7 +; CHECK: lda #0x9 + %c = icmp uge i64 %a, %b + %r = select i1 %c, i16 9, i16 7 + ret i16 %r +} + +; x > 0x7ff0000000000000 is x >= 0x7ff0000000000001. +define i16 @va_ugtK(i64 %a) { +; CHECK-LABEL: va_ugtK: +; CHECK: cmp #0x1 +; CHECK-NEXT: lda 0xd0 +; CHECK-NEXT: sbc #0x0 +; CHECK-NEXT: lda 0x4, s +; CHECK-NEXT: sbc #0x0 +; CHECK-NEXT: lda 0x6, s +; CHECK-NEXT: sbc #0x7ff0 +; CHECK-NEXT: bcs +; CHECK: rtl + %c = icmp ugt i64 %a, 9218868437227405312 + %r = zext i1 %c to i16 + ret i16 %r +} + +; Two zero low words: only words 2 and 3 are compared. +define i16 @va_ultHiK(i64 %a) { +; CHECK-LABEL: va_ultHiK: +; CHECK-NOT: 0xd0 +; CHECK: lda 0x4, s +; CHECK-NEXT: cmp #0x1 +; CHECK-NEXT: txa +; CHECK-NEXT: sbc #0x0 +; CHECK-NEXT: bcc +; CHECK: rtl + %c = icmp ult i64 %a, 4294967296 + %r = zext i1 %c to i16 + ret i16 %r +} + +define i16 @va_eq(i64 %a, i64 %b) { +; CHECK-LABEL: va_eq: +; CHECK: tcs +; CHECK-NOT: sbc +; CHECK-DAG: eor 0x12, s +; CHECK-DAG: eor 0x10, s +; CHECK-DAG: eor 0x14, s +; CHECK-DAG: eor 0x16, s +; CHECK-NOT: sbc +; CHECK: beq +; CHECK: rtl + %c = icmp eq i64 %a, %b + %r = zext i1 %c to i16 + ret i16 %r +} diff --git a/src/llvm/test/CodeGen/W65816/i8-imm-wide.ll b/src/llvm/test/CodeGen/W65816/i8-imm-wide.ll new file mode 100644 index 0000000..5a110c9 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/i8-imm-wide.ll @@ -0,0 +1,33 @@ +; Pin: an i8 constant loaded into A is a 16-bit `lda #imm` (3 cycles), +; not `sep #0x20 ; lda #imm ; rep #0x20` (9): the byte's high half is +; nobody's business, and LDAi8imm is in the flag-wrap pass's isLdaLike +; list either way. A store's own SEP that follows still takes the 8-bit +; form and drops the REP/SEP pair between them. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s +; RUN: env W65816_DISABLE_I8_IMM16=1 llc -mtriple=w65816 -O2 < %s | FileCheck %s --check-prefix=OFF + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" + +declare i8 @takeByte(i8) + +define i8 @retByte() { +; CHECK-LABEL: retByte: +; CHECK: rep #0x30 +; CHECK-NEXT: lda #0x7 +; CHECK-NEXT: rtl +; OFF-LABEL: retByte: +; OFF: sep #0x20 +; OFF-NEXT: lda #0x7 +; OFF-NEXT: rep #0x20 + ret i8 7 +} + +define i8 @passByte() { +; CHECK-LABEL: passByte: +; CHECK: rep #0x30 +; CHECK-NEXT: lda #0xfb +; CHECK-NEXT: jsl takeByte + %r = call i8 @takeByte(i8 -5) + ret i8 %r +} diff --git a/src/llvm/test/CodeGen/W65816/lda-after-pha-stack-offset.mir b/src/llvm/test/CodeGen/W65816/lda-after-pha-stack-offset.mir new file mode 100644 index 0000000..3a61c0c --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/lda-after-pha-stack-offset.mir @@ -0,0 +1,75 @@ +# Pin: W65816StackRelToImg's redundant-LDA-after-PHA peephole drops the +# second load only when it reads the word the first one did. After the PHA +# a stack word sits 2 bytes further from SP, so `LDA off, s ; PHA ; +# LDA off+2, s` is redundant and `LDA off, s ; PHA ; LDA off, s` is not: +# it reads the word below. The equal-offset match deleted the load in +# W65816ImgCalleeSave's epilogue restore, +# +# lda 3,s (return value) ; pha ; lda 3,s (saved $C0) ; sta $c0 ; pla +# +# so wmemcpy, wmemmove, rename, fminf and eleven Lua functions returned with +# the caller's callee-saved $C0 replaced by their own return value. +# +# restore_after_return: different words; both loads stay. +# same_word: off, then off+2 after the push; the second goes. +# same_dp_word: a direct-page word does not move; the second goes. +# +# RUN: llc -mtriple=w65816 -run-pass=w65816-stackrel-to-img -o - %s | FileCheck %s +--- | + target triple = "w65816" + define void @restore_after_return() { ret void } + define void @same_word() { ret void } + define void @same_dp_word() { ret void } +... +--- +# CHECK-LABEL: name: restore_after_return +# CHECK: LDA_StackRel 3 +# CHECK-NEXT: PHA +# CHECK-NEXT: LDA_StackRel 3 +# CHECK-NEXT: STA_DP 192 +# CHECK-NEXT: PLA +name: restore_after_return +tracksRegLiveness: true +body: | + bb.0: + LDA_StackRel 3, implicit-def $a + PHA + LDA_StackRel 3, implicit-def $a + STA_DP 192, implicit $a + PLA implicit-def $a + RTL implicit $a +... +--- +# CHECK-LABEL: name: same_word +# CHECK: LDA_StackRel 3 +# CHECK-NEXT: PHA +# CHECK-NEXT: PHA +name: same_word +tracksRegLiveness: true +body: | + bb.0: + LDA_StackRel 3, implicit-def $a + PHA + LDA_StackRel 5, implicit-def $a + PHA + PLA implicit-def $a + PLA implicit-def $a + RTL implicit $a +... +--- +# CHECK-LABEL: name: same_dp_word +# CHECK: LDA_DP 208 +# CHECK-NEXT: PHA +# CHECK-NEXT: PHA +name: same_dp_word +tracksRegLiveness: true +body: | + bb.0: + LDA_DP 208, implicit-def $a + PHA + LDA_DP 208, implicit-def $a + PHA + PLA implicit-def $a + PLA implicit-def $a + RTL implicit $a +... diff --git a/src/llvm/test/CodeGen/W65816/light-call.ll b/src/llvm/test/CodeGen/W65816/light-call.ll new file mode 100644 index 0000000..230423d --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/light-call.ll @@ -0,0 +1,43 @@ +; Pin: a call to one of libgcc.s's hand-written helpers (multiply, +; divide, shift, memcpy / memmove / memset) is a JSLlight, which clobbers +; A, X, Y and $F0 only -- the helpers never touch IMG0..IMG7 ($D0..$DE). +; Values live across it stay in those words: here the pointer's high half +; stays in $D0 instead of moving to callee-saved $C0, which costs a save +; in the prologue and a restore before the return. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s +; RUN: env W65816_DISABLE_LIGHT_CALLS=1 llc -mtriple=w65816 -O2 < %s | FileCheck %s --check-prefix=OFF + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" + +define i16 @sumProducts(ptr %p, ptr %q, i16 %n) { +; CHECK-LABEL: sumProducts: +; CHECK-NOT: 0xc0 +; CHECK: stx 0xd0 +; CHECK-NOT: 0xc0 +; CHECK: jsl __mulhi3 +; CHECK-NOT: 0xc0 +; CHECK: rtl +; OFF-LABEL: sumProducts: +; OFF: lda 0xc0 +; OFF: stx 0xc0 +; OFF: jsl __mulhi3 +; OFF: sta 0xc0 +; OFF: rtl +entry: + br label %loop +loop: + %i = phi i16 [ 0, %entry ], [ %i1, %loop ] + %s = phi i16 [ 0, %entry ], [ %s1, %loop ] + %pa = getelementptr inbounds i16, ptr %p, i16 %i + %qa = getelementptr inbounds i16, ptr %q, i16 %i + %a = load i16, ptr %pa, align 2 + %b = load i16, ptr %qa, align 2 + %m = mul i16 %a, %b + %s1 = add i16 %s, %m + %i1 = add i16 %i, 1 + %e = icmp eq i16 %i1, %n + br i1 %e, label %done, label %loop +done: + ret i16 %s1 +} diff --git a/src/llvm/test/CodeGen/W65816/loop-homes-hot-slots.ll b/src/llvm/test/CodeGen/W65816/loop-homes-hot-slots.ll new file mode 100644 index 0000000..bda9a78 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/loop-homes-hot-slots.ll @@ -0,0 +1,221 @@ +; Pin: a function with more hot stack slots than the whole-function IMG +; promotion takes (kMaxHotSlots) still gives its innermost loops their own +; IMG0..7 homes (promoteLoopLocal): the byte-copy loop after twenty live +; accumulators walks both pointers in the direct page (`lda [0xd6], y`, +; `inc 0xd6`) instead of staging them through $E0/$E2 every byte. +; Generated from: +; unsigned mix(const unsigned *v, unsigned char *d, +; const unsigned char *s, unsigned n, unsigned m) +; twenty `aK = v[K]` mixed for n rounds, then `d[k] = s[k]` m times. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s +; RUN: env W65816_DISABLE_LOOP_HOMES_HOT=1 llc -mtriple=w65816 -O2 < %s | FileCheck %s --check-prefix=OFF + +; CHECK-LABEL: mix: +; CHECK: %for.body80.preheader +; CHECK: sta 0xd{{[0-9a-e]}} +; CHECK: %for.body80 +; CHECK-NOT: 0xe0 +; CHECK: lda [0xd{{[0-9a-e]}}], y +; CHECK-NOT: 0xe0 +; CHECK: sta [0xd{{[0-9a-e]}}], y +; CHECK-NOT: 0xe0 +; CHECK: inc 0xd{{[0-9a-e]}} +; CHECK: %for.end86 +; OFF-LABEL: mix: +; OFF: %for.body80 +; OFF: sta 0xe0 +; OFF: lda [0xe0], y + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" + +define dso_local i16 @mix(ptr noundef readonly captures(none) %v, ptr noundef writeonly captures(none) %d, ptr noundef readonly captures(none) %s, i16 noundef %n, i16 noundef %m) local_unnamed_addr { +entry: + %0 = load i16, ptr %v, align 2 + %arrayidx1 = getelementptr inbounds nuw i8, ptr %v, i32 2 + %1 = load i16, ptr %arrayidx1, align 2 + %arrayidx2 = getelementptr inbounds nuw i8, ptr %v, i32 4 + %2 = load i16, ptr %arrayidx2, align 2 + %arrayidx3 = getelementptr inbounds nuw i8, ptr %v, i32 6 + %3 = load i16, ptr %arrayidx3, align 2 + %arrayidx4 = getelementptr inbounds nuw i8, ptr %v, i32 8 + %4 = load i16, ptr %arrayidx4, align 2 + %arrayidx5 = getelementptr inbounds nuw i8, ptr %v, i32 10 + %5 = load i16, ptr %arrayidx5, align 2 + %arrayidx6 = getelementptr inbounds nuw i8, ptr %v, i32 12 + %6 = load i16, ptr %arrayidx6, align 2 + %arrayidx7 = getelementptr inbounds nuw i8, ptr %v, i32 14 + %7 = load i16, ptr %arrayidx7, align 2 + %arrayidx8 = getelementptr inbounds nuw i8, ptr %v, i32 16 + %8 = load i16, ptr %arrayidx8, align 2 + %arrayidx9 = getelementptr inbounds nuw i8, ptr %v, i32 18 + %9 = load i16, ptr %arrayidx9, align 2 + %arrayidx10 = getelementptr inbounds nuw i8, ptr %v, i32 20 + %10 = load i16, ptr %arrayidx10, align 2 + %arrayidx11 = getelementptr inbounds nuw i8, ptr %v, i32 22 + %11 = load i16, ptr %arrayidx11, align 2 + %arrayidx12 = getelementptr inbounds nuw i8, ptr %v, i32 24 + %12 = load i16, ptr %arrayidx12, align 2 + %arrayidx13 = getelementptr inbounds nuw i8, ptr %v, i32 26 + %13 = load i16, ptr %arrayidx13, align 2 + %arrayidx14 = getelementptr inbounds nuw i8, ptr %v, i32 28 + %14 = load i16, ptr %arrayidx14, align 2 + %arrayidx15 = getelementptr inbounds nuw i8, ptr %v, i32 30 + %15 = load i16, ptr %arrayidx15, align 2 + %arrayidx16 = getelementptr inbounds nuw i8, ptr %v, i32 32 + %16 = load i16, ptr %arrayidx16, align 2 + %arrayidx17 = getelementptr inbounds nuw i8, ptr %v, i32 34 + %17 = load i16, ptr %arrayidx17, align 2 + %arrayidx18 = getelementptr inbounds nuw i8, ptr %v, i32 36 + %18 = load i16, ptr %arrayidx18, align 2 + %arrayidx19 = getelementptr inbounds nuw i8, ptr %v, i32 38 + %19 = load i16, ptr %arrayidx19, align 2 + %cmp189.not = icmp eq i16 %n, 0 + br i1 %cmp189.not, label %for.cond78.preheader, label %for.body + +for.cond78.preheader: ; preds = %for.body, %entry + %a0.0.lcssa = phi i16 [ %0, %entry ], [ %add20, %for.body ] + %a1.0.lcssa = phi i16 [ %1, %entry ], [ %add23, %for.body ] + %a2.0.lcssa = phi i16 [ %2, %entry ], [ %add26, %for.body ] + %a3.0.lcssa = phi i16 [ %3, %entry ], [ %add29, %for.body ] + %a4.0.lcssa = phi i16 [ %4, %entry ], [ %add32, %for.body ] + %a5.0.lcssa = phi i16 [ %5, %entry ], [ %add35, %for.body ] + %a6.0.lcssa = phi i16 [ %6, %entry ], [ %add38, %for.body ] + %a7.0.lcssa = phi i16 [ %7, %entry ], [ %add41, %for.body ] + %a8.0.lcssa = phi i16 [ %8, %entry ], [ %add44, %for.body ] + %a9.0.lcssa = phi i16 [ %9, %entry ], [ %add47, %for.body ] + %a10.0.lcssa = phi i16 [ %10, %entry ], [ %add50, %for.body ] + %a11.0.lcssa = phi i16 [ %11, %entry ], [ %add53, %for.body ] + %a12.0.lcssa = phi i16 [ %12, %entry ], [ %add56, %for.body ] + %a13.0.lcssa = phi i16 [ %13, %entry ], [ %add59, %for.body ] + %a14.0.lcssa = phi i16 [ %14, %entry ], [ %add62, %for.body ] + %a15.0.lcssa = phi i16 [ %15, %entry ], [ %add65, %for.body ] + %a16.0.lcssa = phi i16 [ %16, %entry ], [ %add68, %for.body ] + %a17.0.lcssa = phi i16 [ %17, %entry ], [ %add71, %for.body ] + %a18.0.lcssa = phi i16 [ %18, %entry ], [ %add74, %for.body ] + %a19.0.lcssa = phi i16 [ %19, %entry ], [ %add77, %for.body ] + %cmp79230.not = icmp eq i16 %m, 0 + br i1 %cmp79230.not, label %for.end86, label %for.body80 + +for.body: ; preds = %entry, %for.body + %k.0210 = phi i16 [ %inc, %for.body ], [ 0, %entry ] + %a19.0209 = phi i16 [ %add77, %for.body ], [ %19, %entry ] + %a18.0208 = phi i16 [ %add74, %for.body ], [ %18, %entry ] + %a17.0207 = phi i16 [ %add71, %for.body ], [ %17, %entry ] + %a16.0206 = phi i16 [ %add68, %for.body ], [ %16, %entry ] + %a15.0205 = phi i16 [ %add65, %for.body ], [ %15, %entry ] + %a14.0204 = phi i16 [ %add62, %for.body ], [ %14, %entry ] + %a13.0203 = phi i16 [ %add59, %for.body ], [ %13, %entry ] + %a12.0202 = phi i16 [ %add56, %for.body ], [ %12, %entry ] + %a11.0201 = phi i16 [ %add53, %for.body ], [ %11, %entry ] + %a10.0200 = phi i16 [ %add50, %for.body ], [ %10, %entry ] + %a9.0199 = phi i16 [ %add47, %for.body ], [ %9, %entry ] + %a8.0198 = phi i16 [ %add44, %for.body ], [ %8, %entry ] + %a7.0197 = phi i16 [ %add41, %for.body ], [ %7, %entry ] + %a6.0196 = phi i16 [ %add38, %for.body ], [ %6, %entry ] + %a5.0195 = phi i16 [ %add35, %for.body ], [ %5, %entry ] + %a4.0194 = phi i16 [ %add32, %for.body ], [ %4, %entry ] + %a3.0193 = phi i16 [ %add29, %for.body ], [ %3, %entry ] + %a2.0192 = phi i16 [ %add26, %for.body ], [ %2, %entry ] + %a1.0191 = phi i16 [ %add23, %for.body ], [ %1, %entry ] + %a0.0190 = phi i16 [ %add20, %for.body ], [ %0, %entry ] + %mul = mul i16 %a0.0190, 3 + %add = add i16 %mul, %a1.0191 + %add20 = add i16 %add, %k.0210 + %mul21 = mul i16 %a1.0191, 3 + %add22 = add i16 %mul21, %a2.0192 + %add23 = add i16 %add22, %k.0210 + %mul24 = mul i16 %a2.0192, 3 + %add25 = add i16 %mul24, %a3.0193 + %add26 = add i16 %add25, %k.0210 + %mul27 = mul i16 %a3.0193, 3 + %add28 = add i16 %mul27, %a4.0194 + %add29 = add i16 %add28, %k.0210 + %mul30 = mul i16 %a4.0194, 3 + %add31 = add i16 %mul30, %a5.0195 + %add32 = add i16 %add31, %k.0210 + %mul33 = mul i16 %a5.0195, 3 + %add34 = add i16 %mul33, %a6.0196 + %add35 = add i16 %add34, %k.0210 + %mul36 = mul i16 %a6.0196, 3 + %add37 = add i16 %mul36, %a7.0197 + %add38 = add i16 %add37, %k.0210 + %mul39 = mul i16 %a7.0197, 3 + %add40 = add i16 %mul39, %a8.0198 + %add41 = add i16 %add40, %k.0210 + %mul42 = mul i16 %a8.0198, 3 + %add43 = add i16 %mul42, %a9.0199 + %add44 = add i16 %add43, %k.0210 + %mul45 = mul i16 %a9.0199, 3 + %add46 = add i16 %mul45, %a10.0200 + %add47 = add i16 %add46, %k.0210 + %mul48 = mul i16 %a10.0200, 3 + %add49 = add i16 %mul48, %a11.0201 + %add50 = add i16 %add49, %k.0210 + %mul51 = mul i16 %a11.0201, 3 + %add52 = add i16 %mul51, %a12.0202 + %add53 = add i16 %add52, %k.0210 + %mul54 = mul i16 %a12.0202, 3 + %add55 = add i16 %mul54, %a13.0203 + %add56 = add i16 %add55, %k.0210 + %mul57 = mul i16 %a13.0203, 3 + %add58 = add i16 %mul57, %a14.0204 + %add59 = add i16 %add58, %k.0210 + %mul60 = mul i16 %a14.0204, 3 + %add61 = add i16 %mul60, %a15.0205 + %add62 = add i16 %add61, %k.0210 + %mul63 = mul i16 %a15.0205, 3 + %add64 = add i16 %mul63, %a16.0206 + %add65 = add i16 %add64, %k.0210 + %mul66 = mul i16 %a16.0206, 3 + %add67 = add i16 %mul66, %a17.0207 + %add68 = add i16 %add67, %k.0210 + %mul69 = mul i16 %a17.0207, 3 + %add70 = add i16 %mul69, %a18.0208 + %add71 = add i16 %add70, %k.0210 + %mul72 = mul i16 %a18.0208, 3 + %add73 = add i16 %mul72, %a19.0209 + %add74 = add i16 %add73, %k.0210 + %mul75 = mul i16 %a19.0209, 3 + %add76 = add i16 %k.0210, %mul75 + %add77 = add i16 %add76, %add20 + %inc = add nuw i16 %k.0210, 1 + %exitcond.not = icmp eq i16 %inc, %n + br i1 %exitcond.not, label %for.cond78.preheader, label %for.body + +for.body80: ; preds = %for.cond78.preheader, %for.body80 + %k.1231 = phi i16 [ %inc85, %for.body80 ], [ 0, %for.cond78.preheader ] + %idxprom = zext i16 %k.1231 to i32 + %arrayidx81 = getelementptr inbounds nuw i8, ptr %s, i32 %idxprom + %20 = load i8, ptr %arrayidx81, align 1 + %arrayidx83 = getelementptr inbounds nuw i8, ptr %d, i32 %idxprom + store i8 %20, ptr %arrayidx83, align 1 + %inc85 = add nuw i16 %k.1231, 1 + %exitcond232.not = icmp eq i16 %inc85, %m + br i1 %exitcond232.not, label %for.end86, label %for.body80 + +for.end86: ; preds = %for.body80, %for.cond78.preheader + %xor = xor i16 %a1.0.lcssa, %a0.0.lcssa + %xor87 = xor i16 %xor, %a2.0.lcssa + %xor88 = xor i16 %xor87, %a3.0.lcssa + %xor89 = xor i16 %xor88, %a4.0.lcssa + %xor90 = xor i16 %xor89, %a5.0.lcssa + %xor91 = xor i16 %xor90, %a6.0.lcssa + %xor92 = xor i16 %xor91, %a7.0.lcssa + %xor93 = xor i16 %xor92, %a8.0.lcssa + %xor94 = xor i16 %xor93, %a9.0.lcssa + %xor95 = xor i16 %xor94, %a10.0.lcssa + %xor96 = xor i16 %xor95, %a11.0.lcssa + %xor97 = xor i16 %xor96, %a12.0.lcssa + %xor98 = xor i16 %xor97, %a13.0.lcssa + %xor99 = xor i16 %xor98, %a14.0.lcssa + %xor100 = xor i16 %xor99, %a15.0.lcssa + %xor101 = xor i16 %xor100, %a16.0.lcssa + %xor102 = xor i16 %xor101, %a17.0.lcssa + %xor103 = xor i16 %xor102, %a18.0.lcssa + %xor104 = xor i16 %xor103, %a19.0.lcssa + ret i16 %xor104 +} + + + diff --git a/src/llvm/test/CodeGen/W65816/loop-local-dp.ll b/src/llvm/test/CodeGen/W65816/loop-local-dp.ll index de52e61..a997bcd 100644 --- a/src/llvm/test/CodeGen/W65816/loop-local-dp.ll +++ b/src/llvm/test/CodeGen/W65816/loop-local-dp.ll @@ -9,9 +9,12 @@ ; every byte. ; ; walk: calls before and after the loop: promoted. -; call_in_loop: the loop calls: left alone (still staged through $E0). +; call_in_loop: the loop calls: no loop home (still staged through $E0) +; -- the pointer takes an IMG8..15 home for the whole function instead +; (callee-saved-homes.ll), so the pin needs that switched off. ; ; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s +; RUN: env W65816_DISABLE_CALLEE_SAVED_HOMES=1 llc -mtriple=w65816 -O2 < %s | FileCheck %s --check-prefix=NOSWAP target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" @@ -52,9 +55,12 @@ exit: define void @call_in_loop(ptr %p, i16 %n) { ; CHECK-LABEL: call_in_loop: -; CHECK: sta 0xe0 -; CHECK: lda [0xe0], y +; CHECK: lda [0xc0], y ; CHECK: jsl use +; NOSWAP-LABEL: call_in_loop: +; NOSWAP: sta 0xe0 +; NOSWAP: lda [0xe0], y +; NOSWAP: jsl use entry: br label %loop loop: diff --git a/src/llvm/test/CodeGen/W65816/lsr-nonneg-offsets.ll b/src/llvm/test/CodeGen/W65816/lsr-nonneg-offsets.ll new file mode 100644 index 0000000..a57f10f --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/lsr-nonneg-offsets.ll @@ -0,0 +1,69 @@ +; Pin: loop strength reduction keeps a pointer that walks an array of +; records at the record's start. isLegalAddressingMode used to allow any +; signed 16-bit offset, so LSR ran this loop on a pointer to `lastUse` +; and reached `used`, `ptr` and `color` at -4, -3 and -2: each of those +; loads was an i32 add (`adc #0xfffc ; ... adc #0xffff`) and a staging of +; the result into $E0 / $E2, since Y only adds an UNSIGNED offset. Now +; every field is `ldy #K ; lda [dp],y` off the walking pointer. (Space +; Taxi's sprite-cache scan.) +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s + +; CHECK-LABEL: find: +; CHECK-NOT: adc #0xff{{[0-9a-f]+}} +; CHECK: ldy #0x0 +; CHECK-NEXT: lda [0x{{[0-9a-f]+}}], y +; CHECK: ldy #0x1 +; CHECK-NEXT: lda [0x{{[0-9a-f]+}}], y +; CHECK: ldy #0x2 +; CHECK-NEXT: lda [0x{{[0-9a-f]+}}], y +; CHECK-NOT: adc #0xff{{[0-9a-f]+}} +; CHECK: rtl + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S8" +target triple = "w65816" + +define dso_local ptr @find(ptr noundef captures(none) %cache, i8 noundef zeroext %ptr, i8 noundef zeroext %color, i16 noundef zeroext %stamp, i8 noundef zeroext %n) local_unnamed_addr { +entry: + %cmp25.not = icmp eq i8 %n, 0 + br i1 %cmp25.not, label %cleanup15, label %for.body + +for.body: ; preds = %entry, %for.inc + %k.026 = phi i8 [ %inc, %for.inc ], [ 0, %entry ] + %idxprom = zext i8 %k.026 to i32 + %arrayidx = getelementptr inbounds nuw [10 x i8], ptr %cache, i32 %idxprom + %0 = load i8, ptr %arrayidx, align 2 + %tobool.not = icmp eq i8 %0, 0 + br i1 %tobool.not, label %for.inc, label %land.lhs.true + +land.lhs.true: ; preds = %for.body + %ptr4 = getelementptr inbounds nuw i8, ptr %arrayidx, i32 1 + %1 = load i8, ptr %ptr4, align 1 + %cmp7 = icmp eq i8 %1, %ptr + br i1 %cmp7, label %land.lhs.true9, label %for.inc + +land.lhs.true9: ; preds = %land.lhs.true + %color10 = getelementptr inbounds nuw i8, ptr %arrayidx, i32 2 + %2 = load i8, ptr %color10, align 2 + %cmp13 = icmp eq i8 %2, %color + br i1 %cmp13, label %cleanup, label %for.inc + +cleanup: ; preds = %land.lhs.true9 + %lastUse = getelementptr inbounds nuw i8, ptr %arrayidx, i32 4 + store i16 %stamp, ptr %lastUse, align 2 + %sprite = getelementptr inbounds nuw i8, ptr %arrayidx, i32 6 + %3 = load ptr, ptr %sprite, align 2 + br label %cleanup15 + +for.inc: ; preds = %land.lhs.true9, %land.lhs.true, %for.body + %inc = add nuw i8 %k.026, 1 + %exitcond.not = icmp eq i8 %inc, %n + br i1 %exitcond.not, label %cleanup15, label %for.body + +cleanup15: ; preds = %for.inc, %entry, %cleanup + %retval.2 = phi ptr [ %3, %cleanup ], [ null, %entry ], [ null, %for.inc ] + ret ptr %retval.2 +} + + + diff --git a/src/llvm/test/CodeGen/W65816/mul-const-shift-add.ll b/src/llvm/test/CodeGen/W65816/mul-const-shift-add.ll new file mode 100644 index 0000000..cc8be1e --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/mul-const-shift-add.ll @@ -0,0 +1,51 @@ +; Pin: LowerMUL_I32's strength reductions. A product known to fit 16 +; bits -- a u8 index times a 6-byte record's size, the address math of +; `sim->spr[idx]` -- is a 16-bit multiply, here shifts and adds +; ((i << 3) - (i << 1)), not a __umulhisi3 call per access. A 32-bit +; multiply by a constant of two signed digits (2^N +- 2^M: 6, 10, 12, 24, +; ...) is two inlined shifts and an add or subtract instead of __mulsi3; +; a constant of more digits keeps the call. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s +; RUN: env W65816_DISABLE_MUL_SHIFT_ADD=1 llc -mtriple=w65816 -O2 < %s | FileCheck %s --check-prefix=OFF + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" + +define zeroext i8 @recordField(ptr %s, i8 zeroext %idx) { +; CHECK-LABEL: recordField: +; CHECK-NOT: jsl +; CHECK: and #0xff +; CHECK-NEXT: asl a +; CHECK-NEXT: sta 0x1, s +; CHECK-NEXT: asl a +; CHECK-NEXT: asl a +; CHECK-NEXT: sec +; CHECK-NEXT: sbc 0x1, s +; CHECK: lda [0xe0], y +; CHECK: rtl +; OFF-LABEL: recordField: +; OFF: jsl __umulhisi3 + %i = zext i8 %idx to i32 + %rec = getelementptr inbounds [6 x i8], ptr %s, i32 %i + %f = getelementptr inbounds i8, ptr %rec, i32 2 + %v = load i8, ptr %f, align 1 + ret i8 %v +} + +define i32 @longTimes6(i32 %x) { +; CHECK-LABEL: longTimes6: +; CHECK-NOT: jsl +; CHECK: sbc +; CHECK: rtl +; OFF-LABEL: longTimes6: +; OFF: jsl __mulsi3 + %m = mul i32 %x, 6 + ret i32 %m +} + +define i32 @longTimes1000(i32 %x) { +; CHECK-LABEL: longTimes1000: +; CHECK: jsl __mulsi3 + %m = mul i32 %x, 1000 + ret i32 %m +} diff --git a/src/llvm/test/CodeGen/W65816/mul-narrow-product.ll b/src/llvm/test/CodeGen/W65816/mul-narrow-product.ll new file mode 100644 index 0000000..41d236e --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/mul-narrow-product.ll @@ -0,0 +1,50 @@ +; Pin: an i32 multiply whose PRODUCT fits 16 bits is a 16-bit multiply, +; zero-extended (W65816NarrowI32Mul): shifts and adds for a constant, else +; __mulhi3, and a high word of 0 -- not __umulhisi3's 16x16->32 call. +; CoreMark's matrix_mul_matrix_bitextract multiplies a 4-bit field by a +; 7-bit one; `(u32)byte * 6` is a 6-byte record's offset. Operands that +; only fit 16 bits each keep __umulhisi3. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s +; RUN: env W65816_DISABLE_NARROW_PRODUCT=1 llc -mtriple=w65816 -O2 < %s | FileCheck %s --check-prefix=OFF + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" + +define i32 @byteTimes6(i8 zeroext %x) { +; CHECK-LABEL: byteTimes6: +; CHECK-NOT: jsl +; CHECK: asl a +; CHECK: adc +; CHECK: ldx #0x0 +; CHECK-NEXT: ply +; OFF-LABEL: byteTimes6: +; OFF: jsl __umulhisi3 + %z = zext i8 %x to i32 + %m = mul nuw nsw i32 %z, 6 + ret i32 %m +} + +define i32 @fieldProduct(i16 %a, i16 %b) { +; CHECK-LABEL: fieldProduct: +; CHECK: and #0xf +; CHECK: jsl __mulhi3 +; CHECK-NOT: jsl +; CHECK: ldx #0x0 +; OFF-LABEL: fieldProduct: +; OFF: jsl __umulhisi3 + %x = and i16 %a, 15 + %y = and i16 %b, 127 + %xz = zext i16 %x to i32 + %yz = zext i16 %y to i32 + %m = mul nuw nsw i32 %xz, %yz + ret i32 %m +} + +define i32 @wideProduct(i16 %a, i16 %b) { +; CHECK-LABEL: wideProduct: +; CHECK: jsl __umulhisi3 + %xz = zext i16 %a to i32 + %yz = zext i16 %b to i32 + %m = mul nuw i32 %xz, %yz + ret i32 %m +} diff --git a/src/llvm/test/CodeGen/W65816/mul-narrow-signed.ll b/src/llvm/test/CodeGen/W65816/mul-narrow-signed.ll new file mode 100644 index 0000000..104730d --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/mul-narrow-signed.ll @@ -0,0 +1,47 @@ +; Pin: W65816NarrowI32Mul sends an i32 multiply of two sign-extended +; 16-bit values to __mulhisi3 (signed 16x16 -> 32), as it sends one of two +; zero-extended values to __umulhisi3. `(long)a * b` of two shorts went +; to __mulsi3, whose old loop took 32 steps for any negative operand +; (CoreMark's matrix_mul_matrix: ~330 cycles a call there, 8% of the run). +; A zero-extended byte fits a signed 16-bit value too; a zero-extended +; 16-bit value may not, so that pair keeps __mulsi3. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" + +define i32 @ss(i16 %a, i16 %b) { +; CHECK-LABEL: ss: +; CHECK: jsl __mulhisi3 + %x = sext i16 %a to i32 + %y = sext i16 %b to i32 + %m = mul i32 %x, %y + ret i32 %m +} + +define i32 @uu(i16 %a, i16 %b) { +; CHECK-LABEL: uu: +; CHECK: jsl __umulhisi3 + %x = zext i16 %a to i32 + %y = zext i16 %b to i32 + %m = mul i32 %x, %y + ret i32 %m +} + +define i32 @byteTimesShort(i8 %a, i16 %b) { +; CHECK-LABEL: byteTimesShort: +; CHECK: jsl __mulhisi3 + %x = zext i8 %a to i32 + %y = sext i16 %b to i32 + %m = mul i32 %x, %y + ret i32 %m +} + +define i32 @wordTimesShort(i16 %a, i16 %b) { +; CHECK-LABEL: wordTimesShort: +; CHECK: jsl __mulsi3 + %x = zext i16 %a to i32 + %y = sext i16 %b to i32 + %m = mul i32 %x, %y + ret i32 %m +} diff --git a/src/llvm/test/CodeGen/W65816/setjmp-img-clobber.ll b/src/llvm/test/CodeGen/W65816/setjmp-img-clobber.ll new file mode 100644 index 0000000..ada3abc --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/setjmp-img-clobber.ll @@ -0,0 +1,40 @@ +; Pin: a call that can return twice (setjmp) clobbers IMG8..IMG15 +; ($C0..$CE). The longjmp that comes back through it skips the returns +; that give each caller its IMG8..15 words back, so the function calling +; setjmp keeps no value of its own in them across the call and saves and +; restores all eight for its caller (W65816ImgCalleeSave sees the defs). +; tests/regress/setjmpImgSave.c is the run-time check. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" + +@jb = dso_local global [8 x i8] zeroinitializer, align 1 + +declare i16 @setjmp(ptr) returns_twice +declare void @work(ptr, ptr) + +define i16 @guard(ptr %p, ptr %q) { +; CHECK-LABEL: guard: +; CHECK: lda 0xc0 +; CHECK: lda 0xc2 +; CHECK: lda 0xc4 +; CHECK: lda 0xc6 +; CHECK: lda 0xc8 +; CHECK: lda 0xca +; CHECK: lda 0xcc +; CHECK: lda 0xce +; CHECK: jsl setjmp +; CHECK: sta 0xce +; CHECK: rtl +entry: + %r = call i16 @setjmp(ptr @jb) + %z = icmp eq i16 %r, 0 + br i1 %z, label %body, label %out +body: + call void @work(ptr %p, ptr %q) + br label %out +out: + %v = phi i16 [ 0, %body ], [ 1, %entry ] + ret i16 %v +} diff --git a/src/llvm/test/CodeGen/W65816/sink-loop-inc.ll b/src/llvm/test/CodeGen/W65816/sink-loop-inc.ll new file mode 100644 index 0000000..b8ab054 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/sink-loop-inc.ll @@ -0,0 +1,125 @@ +; Pin: W65816SinkLoopInc moves a loop-carried step below the last use of +; the value it steps, so the old and new values do not overlap and the +; coalescer gives them one home: no back-edge copy, and a far pointer's +; `p++` becomes the in-place `inc dp ; bne ; inc dp+2`. +; +; sumPtr (`while (p != end) s += *p++;`): the scheduler put the pointer's +; ADC / ADCE pair above the load through the old pointer. The pair must +; land after the load ([$E0],Y with $E0 / $E2 staged from the old value), +; and the loop then bumps the pointer's DP home in place. It used to step +; into a second pair of words and copy them back every trip (+8 +; instructions a trip; MAME: 169079 -> 115088 cycles for 2047 bytes). +; +; counterArg: an i16 `i++` whose old value is still passed after it. The +; INA lands after the old value's last use (the PUSH16 of the argument) +; and the counter's home is incremented in place. +; +; splitLatch: the back edge leaves from a second test block the step's +; block runs straight into (`while (*++p && state != 1)`, CoreMark's +; core_state_transition). The step was skipped because the PHI took it +; from that block; the pointer was then stepped into a second pair of +; words and copied back every trip. +; +; RUN: llc -mtriple=w65816 -O2 -stop-after=w65816-sink-loop-inc < %s | FileCheck %s --check-prefix=MIR +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s + +target triple = "w65816" + +@gOut16 = global i16 0 + +define i16 @sumPtr(ptr %p, ptr %end) { +; MIR-LABEL: name: sumPtr +; MIR: bb.2.while.body: +; MIR: PHI +; MIR-NEXT: [[LO:%[0-9]+]]:acc16 = PHI +; MIR-NEXT: [[HI:%[0-9]+]]:acc16 = PHI +; MIR: LDA_DPIndLongY 224 +; MIR: ADCi16imm [[LO]], 1 +; MIR-NEXT: ADCEi16imm [[HI]], 0 +; CHECK-LABEL: sumPtr: +; CHECK: [[LOOP:.LBB0_[0-9]+]]: ; %while.body{{$}} +; CHECK-NEXT: ; =>This Inner Loop Header +; CHECK-NEXT: lda [0x[[PLO:d[0-9a-f]]]], y +; CHECK: inc 0x[[PLO]] +; CHECK-NEXT: bne +; CHECK-NEXT: inc 0x{{d[0-9a-f]}} +; CHECK-NOT: sta 0x[[PLO]] +; CHECK: bne [[LOOP]] +entry: + %cmp.not4 = icmp eq ptr %p, %end + br i1 %cmp.not4, label %while.end, label %while.body + +while.body: + %s.06 = phi i16 [ %add, %while.body ], [ 0, %entry ] + %p.addr.05 = phi ptr [ %incdec.ptr, %while.body ], [ %p, %entry ] + %incdec.ptr = getelementptr inbounds nuw i8, ptr %p.addr.05, i32 1 + %0 = load i8, ptr %p.addr.05, align 1 + %conv = zext i8 %0 to i16 + %add = add i16 %s.06, %conv + %cmp.not = icmp eq ptr %incdec.ptr, %end + br i1 %cmp.not, label %while.end, label %while.body + +while.end: + %s.0.lcssa = phi i16 [ 0, %entry ], [ %add, %while.body ] + store volatile i16 %s.0.lcssa, ptr @gOut16, align 2 + ret i16 %s.0.lcssa +} + +define i16 @counterArg(i16 %n) { +; MIR-LABEL: name: counterArg +; MIR: [[I:%[0-9]+]]:acc16 = PHI +; MIR: $a = COPY [[I]] +; MIR-NEXT: PUSH16 +; MIR-NEXT: {{%[0-9]+}}:acc16 = INA_PSEUDO [[I]] +; CHECK-LABEL: counterArg: +; CHECK: lda 0x[[C:d[0-9a-f]]] +; CHECK-NEXT: pha +; CHECK-NEXT: inc a +; CHECK-NEXT: sta 0x[[C]] +; CHECK-NEXT: jsl __mulhi3 +entry: + br label %loop + +loop: + %i = phi i16 [ 0, %entry ], [ %i.next, %loop ] + %s = phi i16 [ 0, %entry ], [ %s.next, %loop ] + %i.next = add i16 %i, 1 + %t = mul i16 %i.next, %i + %s.next = add i16 %s, %t + %c = icmp ult i16 %i.next, %n + br i1 %c, label %loop, label %done + +done: + ret i16 %s.next +} + +define i16 @splitLatch(ptr %s, i16 %st0) { +; CHECK-LABEL: splitLatch: +; CHECK: Loop Header +; CHECK: lda [0x[[P:d[0-9a-f]]]], y +; CHECK-NEXT: tax +; CHECK-NEXT: inc 0x[[P]] +; CHECK-NOT: sta 0x[[P]] +; CHECK: rtl +entry: + %c0 = load i8, ptr %s, align 1 + %z0 = icmp eq i8 %c0, 0 + br i1 %z0, label %exit, label %body + +body: + %p = phi ptr [ %s, %entry ], [ %pn, %latch2 ] + %st = phi i16 [ %st0, %entry ], [ %stn, %latch2 ] + %stn = xor i16 %st, 3 + %pn = getelementptr inbounds i8, ptr %p, i32 1 + %c = load i8, ptr %pn, align 1 + %nz = icmp ne i8 %c, 0 + br i1 %nz, label %latch2, label %exit + +latch2: + %go = icmp ne i16 %stn, 1 + br i1 %go, label %body, label %exit + +exit: + %r = phi i16 [ %st0, %entry ], [ %stn, %body ], [ %stn, %latch2 ] + ret i16 %r +} diff --git a/src/llvm/test/CodeGen/W65816/slot-img-forward.mir b/src/llvm/test/CodeGen/W65816/slot-img-forward.mir new file mode 100644 index 0000000..7784dbe --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/slot-img-forward.mir @@ -0,0 +1,135 @@ +# Pin: W65816StackSlotCleanup's temp-slot forwarding (Pass 2.4) and its +# never-read store sweep (Pass 2.5). +# +# The far-pointer staging inserters move a pointer word to $E0 / $E2 through +# a fresh stack object: `STAfi src, T ; ... ; LDAfi T ; STA_DP $E2`. With the +# word in an IMG register and A live, the store alone cost +# `pha ; lda dp ; sta n,s ; pla`. The reload now reads the IMG word itself +# (LDAimg, `lda dp` after expansion) and the store, read by nothing any more, +# goes; so do stores to objects nothing ever read (consecutive-deref CSE left +# those behind: CoreMark's list reversal paid two pairs per node). +# +# forward: IMG word unchanged between store and reload: forwarded, +# store dropped. +# img_redefined: the IMG register is written in between: both stay. +# dp_touched: a direct-page access to the IMG word in between: both stay. +# two_reads: the slot is read twice: both stay. +# never_read: a store to an object nothing reads: dropped. +# addr_taken: an ADDframe of the object counts as a read: kept. +# +# RUN: llc -mtriple=w65816 -run-pass=w65816-stack-slot-cleanup -o - %s | FileCheck %s +--- | + target triple = "w65816" + define void @forward() { ret void } + define void @img_redefined() { ret void } + define void @dp_touched() { ret void } + define void @two_reads() { ret void } + define void @never_read() { ret void } + define void @addr_taken() { ret void } +... +--- +# CHECK-LABEL: name: forward +# CHECK: $img0 = COPY $x +# CHECK-NOT: STAfi +# CHECK-NOT: LDAfi +# CHECK: $a = LDAimg $img0 +# CHECK-NEXT: STA_DP 226 +name: forward +tracksRegLiveness: false +stack: + - { id: 0, size: 2, alignment: 1 } +body: | + bb.0: + $img0 = COPY $x + STAfi $img0, %stack.0, 0, implicit-def dead $a + $a = LDAi16imm 7 + STA_DP 224, implicit $a + dead $a = LDAfi %stack.0, 0 + STA_DP 226 + RTL +... +--- +# CHECK-LABEL: name: img_redefined +# CHECK: STAfi $img0, %stack.0, 0 +# CHECK: $a = LDAfi %stack.0, 0 +name: img_redefined +tracksRegLiveness: false +stack: + - { id: 0, size: 2, alignment: 1 } +body: | + bb.0: + $img0 = COPY $x + STAfi $img0, %stack.0, 0, implicit-def dead $a + $img0 = COPY $y + $a = LDAfi %stack.0, 0 + STA_DP 226, implicit $a + RTL +... +--- +# CHECK-LABEL: name: dp_touched +# CHECK: STAfi $img1, %stack.0, 0 +# CHECK: $a = LDAfi %stack.0, 0 +name: dp_touched +tracksRegLiveness: false +stack: + - { id: 0, size: 2, alignment: 1 } +body: | + bb.0: + $img1 = COPY $x + STAfi $img1, %stack.0, 0, implicit-def dead $a + INC_DP 210 + $a = LDAfi %stack.0, 0 + STA_DP 226, implicit $a + RTL +... +--- +# CHECK-LABEL: name: two_reads +# CHECK: STAfi $img0, %stack.0, 0 +# CHECK: $a = LDAfi %stack.0, 0 +# CHECK: $a = LDAfi %stack.0, 0 +name: two_reads +tracksRegLiveness: false +stack: + - { id: 0, size: 2, alignment: 1 } +body: | + bb.0: + $img0 = COPY $x + STAfi $img0, %stack.0, 0, implicit-def dead $a + $a = LDAfi %stack.0, 0 + STA_DP 224, implicit $a + $a = LDAi16imm 1 + STA_DP 228, implicit $a + $a = LDAfi %stack.0, 0 + STA_DP 226, implicit $a + RTL +... +--- +# CHECK-LABEL: name: never_read +# CHECK: $a = LDAi16imm 5 +# CHECK-NOT: STAfi +# CHECK: RTL +name: never_read +tracksRegLiveness: false +stack: + - { id: 0, size: 2, alignment: 1 } +body: | + bb.0: + $a = LDAi16imm 5 + STAfi $a, %stack.0, 0, implicit-def $a + RTL +... +--- +# CHECK-LABEL: name: addr_taken +# CHECK: STAfi $a, %stack.0, 0 +# CHECK: $a = ADDframe %stack.0, 0 +name: addr_taken +tracksRegLiveness: false +stack: + - { id: 0, size: 2, alignment: 1 } +body: | + bb.0: + $a = LDAi16imm 5 + STAfi $a, %stack.0, 0, implicit-def $a + $a = ADDframe %stack.0, 0 + RTL +... diff --git a/src/llvm/test/CodeGen/W65816/spill-a-live.mir b/src/llvm/test/CodeGen/W65816/spill-a-live.mir new file mode 100644 index 0000000..81878ee --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/spill-a-live.mir @@ -0,0 +1,87 @@ +# Pin: eliminateFrameIndex moves a non-A STAfi (a spill of an IMG word, X +# or Y) through A and keeps A around it -- PHA ... PLA, the store 2 bytes +# further up -- only when A's value is still read afterwards (aLiveAfter). +# Every such spill used to pay the PHA / PLA; the reloads, the other half +# of the change, kept none and clobbered A where the allocator had a value +# of its own in it (wide32-spill-subreg.ll pins that through CoreMark's +# core_list_mergesort: a non-A LDAfi is not legal MIR input here). +# +# spill_a_dead: A rewritten first: `lda dp ; sta d,s`, no bracket. +# spill_a_live: A read after: bracketed, offset + 2. +# nona_store_wrap: the store sets N and Z through A, so between a compare +# and its BNE it is wrapped in PHP / PLP (the wrap pass +# took every store for flag-preserving). +# +# RUN: llc -mtriple=w65816 -run-pass=prologepilog -o - %s | FileCheck %s +# RUN: llc -mtriple=w65816 -run-pass=w65816-stack-slot-cleanup -o - %s | FileCheck %s --check-prefix=WRAP +--- | + target triple = "w65816" + define void @spill_a_dead() { ret void } + define void @spill_a_live() { ret void } + define void @nona_store_wrap() { ret void } +... +--- +# CHECK-LABEL: name: spill_a_dead +# CHECK: TCS +# CHECK-NOT: PHA +# CHECK: LDA_DP 210 +# CHECK-NEXT: STA_StackRel 9 +# CHECK-NEXT: $a = LDAi16imm 0 +name: spill_a_dead +tracksRegLiveness: false +stack: + - { id: 0, size: 2, alignment: 1 } + - { id: 1, size: 8, alignment: 1 } +body: | + bb.0: + STAfi $img1, %stack.0, 0, implicit-def $a + $a = LDAi16imm 0 + STA_DP 224, implicit $a + RTL +... +--- +# CHECK-LABEL: name: spill_a_live +# CHECK: PHA +# CHECK-NEXT: LDA_DP 210 +# CHECK-NEXT: STA_StackRel 11 +# CHECK-NEXT: PLA +name: spill_a_live +tracksRegLiveness: false +stack: + - { id: 0, size: 2, alignment: 1 } + - { id: 1, size: 8, alignment: 1 } +body: | + bb.0: + $a = LDAi16imm 5 + STAfi $img1, %stack.0, 0, implicit-def $a + STA_DP 224, implicit $a + RTL +... +--- +# WRAP-LABEL: name: nona_store_wrap +# WRAP: CMPi16imm killed $a, 3, implicit-def $p +# WRAP-NEXT: PHP +# WRAP-NEXT: STAfi $img1, %stack.1, 0 +# WRAP-NEXT: PLP +# WRAP-NEXT: BNE %bb.1, implicit $p +name: nona_store_wrap +tracksRegLiveness: false +stack: + - { id: 0, size: 2, alignment: 1 } + - { id: 1, size: 2, alignment: 1 } +body: | + bb.0: + successors: %bb.1, %bb.2 + $a = LDAfi %stack.0, 0 + CMPi16imm killed $a, 3, implicit-def $p + STAfi $img1, %stack.1, 0, implicit-def $a + BNE %bb.1, implicit $p + BRA %bb.2 + bb.1: + liveins: $x + $a = LDAfi %stack.1, 0 + RTL + bb.2: + liveins: $x + RTL +... diff --git a/src/llvm/test/CodeGen/W65816/spill-stafi-no-a-def.mir b/src/llvm/test/CodeGen/W65816/spill-stafi-no-a-def.mir new file mode 100644 index 0000000..bd980d0 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/spill-stafi-no-a-def.mir @@ -0,0 +1,74 @@ +# Pin: a spill STAfi the register allocator inserts names no def of A. +# STAfi carries Defs = [A] as an over-approximation (it never changes A: +# eliminateFrameIndex moves an IMG / X / Y source through A inside PHA / +# PLA when A is still read), but the spiller places spills without +# checking A's interference. Here ten values outlive a call that keeps +# only IMG8..15, so two go to the stack through $img0 while the incoming +# A is still live (it is copied again right after the spill). With the +# implicit def on the spill, A looked clobbered in the middle of its own +# live range, and post-RA MachineCopyPropagation deleted an earlier +# `$a = COPY` it took as overwritten -- snprintf's format() lost its +# format pointer's bank and stopped at the first conversion. +# +# RUN: llc -mtriple=w65816 -run-pass=greedy,virtregrewriter -o - %s | FileCheck %s + +# CHECK-LABEL: name: f +# CHECK: $img0 = COPY $a +# CHECK-NEXT: STAfi killed $img0, %stack.{{[0-9]+}}, 0{{$}} +# CHECK-NEXT: $img0 = COPY killed $a +# CHECK-NEXT: STAfi killed $img0, %stack.{{[0-9]+}}, 0{{$}} +# CHECK: $img0 = LDAfi %stack.{{[0-9]+}}, 0 +--- | + target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S8" + target triple = "w65816" + declare void @ext() + define i16 @f(i16 %a) { ret i16 %a } +... +--- +name: f +tracksRegLiveness: true +registers: + - { id: 0, class: acc16 } + - { id: 1, class: acc16 } + - { id: 10, class: img16 } + - { id: 11, class: img16 } + - { id: 12, class: img16 } + - { id: 13, class: img16 } + - { id: 14, class: img16 } + - { id: 15, class: img16 } + - { id: 16, class: img16 } + - { id: 17, class: img16 } + - { id: 18, class: img16 } + - { id: 19, class: img16 } +stack: + - { id: 0, size: 2, alignment: 2 } +body: | + bb.0: + liveins: $a + %0:acc16 = COPY $a + %10:img16 = COPY %0 + %11:img16 = COPY %0 + %12:img16 = COPY %0 + %13:img16 = COPY %0 + %14:img16 = COPY %0 + %15:img16 = COPY %0 + %16:img16 = COPY %0 + %17:img16 = COPY %0 + %18:img16 = COPY %0 + %19:img16 = COPY %0 + ADJCALLSTACKDOWN 0, 0, implicit-def dead $sp, implicit $sp + JSLpseudo32 @ext, implicit-def dead $a, implicit-def dead $x, implicit-def dead $y, implicit-def dead $dpf0, implicit-def dead $img0, implicit-def dead $img1, implicit-def dead $img2, implicit-def dead $img3, implicit-def dead $img4, implicit-def dead $img5, implicit-def dead $img6, implicit-def dead $img7 + ADJCALLSTACKUP 0, 0, implicit-def dead $sp, implicit $sp + STAfi %10, %stack.0, 0, implicit-def dead $a + STAfi %11, %stack.0, 0, implicit-def dead $a + STAfi %12, %stack.0, 0, implicit-def dead $a + STAfi %13, %stack.0, 0, implicit-def dead $a + STAfi %14, %stack.0, 0, implicit-def dead $a + STAfi %15, %stack.0, 0, implicit-def dead $a + STAfi %16, %stack.0, 0, implicit-def dead $a + STAfi %17, %stack.0, 0, implicit-def dead $a + STAfi %18, %stack.0, 0, implicit-def dead $a + STAfi %19, %stack.0, 0, implicit-def dead $a + $a = COPY %0 + RTL implicit $a +... diff --git a/src/llvm/test/CodeGen/W65816/spill-to-x-merged-slot.mir b/src/llvm/test/CodeGen/W65816/spill-to-x-merged-slot.mir new file mode 100644 index 0000000..bed0408 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/spill-to-x-merged-slot.mir @@ -0,0 +1,199 @@ +# Pin: W65816SpillToX bridges a spill through X (TAX ... TXA) when stack- +# slot coloring has given the slot to other values too, as long as the +# stored value is dead after the reload. +# +# merged_dead: `for (..) gOut = gV; for (..) gOut = gW;`. Coloring shares +# slot 1 between loop 1's counter and gW's copy, and slot 0 +# between n and loop 2's counter. The pass used to bail on +# any reference outside the STA..LDA window, so both counters +# went through memory (sta/lda, 8 cycles a trip, after +# StackRelToImg) instead of tax/txa (4). Both are dead after +# their reloads: the next reference on every path is a store. +# merged_live: the reload is followed by another read of the slot in a +# successor, so the store must stay. +# x_live_out: X, set in the block, is live into the successor; the TAX +# would clobber it (only reads within the block were checked). +# php_window: a PHP between the store and the reload would save the TAX's +# N/Z (the store sets none) in place of the test it brackets. +# +# Shared slots are bridged only by the run after StackSlotCleanup (which +# deletes many such pairs outright, a better result than a bridge); the +# option makes a lone -run-pass behave like that run. The first run keeps +# merged_dead's spills (FIRST). +# +# RUN: llc -mtriple=w65816 -run-pass=w65816-spill-to-x -w65816-spill-to-x-shared-slots -o - %s | FileCheck %s +# RUN: llc -mtriple=w65816 -run-pass=w65816-spill-to-x -o - %s | FileCheck %s --check-prefix=FIRST +--- | + target triple = "w65816" + @gV = global i16 0 + @gW = global i16 0 + @gOut = global i16 0 + define void @merged_dead() { ret void } + define void @merged_live() { ret void } + define void @x_live_out() { ret void } + define void @php_window() { ret void } +... +--- +# FIRST-LABEL: name: merged_dead +# FIRST: bb.3: +# FIRST-NOT: TAX +# FIRST: bb.4: +# FIRST-NOT: TAX +# FIRST: RTL +# CHECK-LABEL: name: merged_dead +# CHECK: bb.3: +# CHECK: TAX +# CHECK-NEXT: $a = LDAfi %stack.2, 0 +# CHECK-NEXT: STAabs killed $a, @gOut +# CHECK-NEXT: TXA +# CHECK: bb.4: +# CHECK: TAX +# CHECK-NEXT: $a = LDAfi %stack.1, 0 +# CHECK-NEXT: STAabs killed $a, @gOut +# CHECK-NEXT: TXA +name: merged_dead +tracksRegLiveness: true +liveins: + - { reg: '$a' } +stack: + - { id: 0, type: spill-slot, size: 2, alignment: 2 } + - { id: 1, type: spill-slot, size: 2, alignment: 2 } + - { id: 2, type: spill-slot, size: 2, alignment: 2 } +body: | + bb.0: + successors: %bb.1, %bb.5 + liveins: $a + STAfi $a, %stack.0, 0, implicit-def $a + $a = LDAfi %stack.0, 0 + $a = EORi16imm killed $a, -32768 + CMPi16imm killed $a, -32767, implicit-def $p + dead $a = LDAfi %stack.0, 0 + BCC %bb.5, implicit $p + BRA %bb.1 + + bb.1: + successors: %bb.3 + $a = LDAabs @gV + STAfi killed $a, %stack.2, 0, implicit-def $a + $a = LDAfi %stack.0, 0 + BRA %bb.3 + + bb.2: + successors: %bb.4 + $a = LDAabs @gW + STAfi killed $a, %stack.1, 0, implicit-def $a + $a = LDAfi %stack.0, 0 + BRA %bb.4 + + bb.3: + successors: %bb.2, %bb.3 + liveins: $a + STAfi killed $a, %stack.1, 0, implicit-def $a + $a = LDAfi %stack.2, 0 + STAabs killed $a, @gOut + $a = LDAfi %stack.1, 0 + $a = DEA_PSEUDO killed $a + CMPi16imm $a, 0, implicit-def $p + BEQ %bb.2, implicit $p + BRA %bb.3 + + bb.4: + successors: %bb.5, %bb.4 + liveins: $a + STAfi killed $a, %stack.0, 0, implicit-def $a + $a = LDAfi %stack.1, 0 + STAabs killed $a, @gOut + $a = LDAfi %stack.0, 0 + $a = DEA_PSEUDO killed $a + CMPi16imm $a, 0, implicit-def $p + BNE %bb.4, implicit $p + BRA %bb.5 + + bb.5: + RTL +... +--- +# CHECK-LABEL: name: merged_live +# CHECK: bb.0: +# CHECK-NOT: TAX +# CHECK: STAfi $a, %stack.0, 0 +# CHECK: $a = LDAfi %stack.0, 0 +# CHECK: bb.1: +name: merged_live +tracksRegLiveness: true +liveins: + - { reg: '$a' } +stack: + - { id: 0, type: spill-slot, size: 2, alignment: 2 } +body: | + bb.0: + successors: %bb.1 + liveins: $a + STAfi $a, %stack.0, 0, implicit-def $a + $a = LDAabs @gV + STAabs killed $a, @gOut + $a = LDAfi %stack.0, 0 + STAabs killed $a, @gW + BRA %bb.1 + + bb.1: + $a = LDAfi %stack.0, 0 + STAabs killed $a, @gOut + RTL +... +--- +# CHECK-LABEL: name: x_live_out +# CHECK: bb.0: +# CHECK-NOT: TAX +# CHECK: STAfi $a, %stack.0, 0 +# CHECK: bb.1: +name: x_live_out +tracksRegLiveness: true +liveins: + - { reg: '$a' } +stack: + - { id: 0, type: spill-slot, size: 2, alignment: 2 } +body: | + bb.0: + successors: %bb.1 + liveins: $a + $x = LDXi16imm 7 + STAfi $a, %stack.0, 0, implicit-def $a + $a = LDAabs @gV + STAabs killed $a, @gOut + $a = LDAfi %stack.0, 0 + BRA %bb.1 + + bb.1: + liveins: $a, $x + STAabs killed $a, @gW + TXA implicit-def $a, implicit $x + STAabs killed $a, @gOut + RTL +... +--- +# CHECK-LABEL: name: php_window +# CHECK: bb.0: +# CHECK-NOT: TAX +# CHECK: STAfi $a, %stack.0, 0 +# CHECK: RTL +name: php_window +tracksRegLiveness: true +liveins: + - { reg: '$a' } +stack: + - { id: 0, type: spill-slot, size: 2, alignment: 2 } +body: | + bb.0: + liveins: $a + STAfi $a, %stack.0, 0, implicit-def $a + $a = LDAabs @gV + CMPi16imm killed $a, 5, implicit-def $p + PHP + $a = LDAabs @gW + STAabs killed $a, @gOut + PLP + $a = LDAfi %stack.0, 0 + STAabs killed $a, @gW + RTL +... diff --git a/src/llvm/test/CodeGen/W65816/split-loop-exits.ll b/src/llvm/test/CodeGen/W65816/split-loop-exits.ll new file mode 100644 index 0000000..d4eef5a --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/split-loop-exits.ll @@ -0,0 +1,83 @@ +; Pin: W65816SplitLoopExits gives a critical edge out of a loop into a +; block with PHIs a block of its own, so PHI elimination's copies run once +; per exit instead of on every trip. LLVM does that split in PHI +; elimination only through MachineBasicBlock::SplitCriticalEdge, which +; needs analyzeBranch to understand the conditional branch -- this target +; keeps conditional branches opaque. A search loop's two exits (found: +; return p; end of list: return NULL) copied p into the return value's +; words and zero into them on every trip (CoreMark's core_list_find: +; -27% with the copies gone). +; +; RUN: llc -mtriple=w65816 -O2 -print-after=w65816-split-loop-exits -o /dev/null < %s 2>&1 | FileCheck %s --check-prefix=IR +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" + +%struct.node = type { ptr, ptr } + +; IR-LABEL: define {{.*}}@find( +; IR: br i1 %cmp.not, label %land.rhs.while.end_crit_edge, label %while.body +; IR: land.rhs.while.end_crit_edge: +; IR-NEXT: br label %while.end +; IR: br i1 %tobool.not, label %while.body.while.end_crit_edge, label %land.rhs +; IR: while.body.while.end_crit_edge: +; IR-NEXT: br label %while.end +; IR: %r = phi ptr [ null, %entry ], [ %list.addr.05, %land.rhs.while.end_crit_edge ], [ null, %while.body.while.end_crit_edge ] + +; The loop keeps neither the return value's copy of p nor its zero; the +; copies sit in the exit blocks. +; CHECK-LABEL: find: +; CHECK: Loop Header +; CHECK-NOT: lda #0x0 +; CHECK: bne .LBB0_[[LOOP:[0-9]+]] +; CHECK-NEXT: ; %bb.{{[0-9]+}}: ; %while.body.while.end_crit_edge +; CHECK-NEXT: lda #0x0 +define ptr @find(ptr %list, i16 %want) { +entry: + %tobool.not4 = icmp eq ptr %list, null + br i1 %tobool.not4, label %while.end, label %land.rhs + +land.rhs: + %list.addr.05 = phi ptr [ %next, %while.body ], [ %list, %entry ] + %info = getelementptr inbounds nuw i8, ptr %list.addr.05, i32 4 + %0 = load ptr, ptr %info, align 2 + %idx = getelementptr inbounds nuw i8, ptr %0, i32 2 + %1 = load i16, ptr %idx, align 2 + %cmp.not = icmp eq i16 %1, %want + br i1 %cmp.not, label %while.end, label %while.body + +while.body: + %next = load ptr, ptr %list.addr.05, align 2 + %tobool.not = icmp eq ptr %next, null + br i1 %tobool.not, label %while.end, label %land.rhs + +while.end: + %r = phi ptr [ null, %entry ], [ null, %while.body ], [ %list.addr.05, %land.rhs ] + ret ptr %r +} + +; A running total leaves the loop as the value the loop carries: that +; copy coalesces into the loop's own register wherever it sits, so the +; edge stays as it is (splitting it cost sumOfSquares a store a trip). +; IR-LABEL: define {{.*}}@total( +; IR-NOT: crit_edge +; IR: ret i32 +define i32 @total(i16 %n) { +entry: + %z = icmp eq i16 %n, 0 + br i1 %z, label %done, label %loop + +loop: + %i = phi i16 [ 1, %entry ], [ %i1, %loop ] + %t = phi i32 [ 0, %entry ], [ %t1, %loop ] + %w = zext i16 %i to i32 + %sq = mul nuw i32 %w, %w + %t1 = add i32 %t, %sq + %i1 = add i16 %i, 1 + %c = icmp ugt i16 %i1, %n + br i1 %c, label %done, label %loop + +done: + %r = phi i32 [ 0, %entry ], [ %t1, %loop ] + ret i32 %r +} diff --git a/src/llvm/test/CodeGen/W65816/srl8a-flag-corrupt-wrap.mir b/src/llvm/test/CodeGen/W65816/srl8a-flag-corrupt-wrap.mir index 2f11547..a3400f4 100644 --- a/src/llvm/test/CodeGen/W65816/srl8a-flag-corrupt-wrap.mir +++ b/src/llvm/test/CodeGen/W65816/srl8a-flag-corrupt-wrap.mir @@ -23,6 +23,9 @@ # The wrapped reload keeps its offset: eliminateFrameIndex adds the byte # PHP pushed (php-wrap-frame-index.mir). # +# srl8a reads %stack.2 after the branch so its store survives the pass's +# never-read slot sweep. +# # The branch targets take X live in, so the wrap cannot be replaced by the # TAX ... CPX #k form that StackSlotCleanup uses when X is free (pinned in # cmp-via-x.mir); these cases keep testing the PHP / PLP wrap itself. @@ -61,6 +64,7 @@ body: | BRA %bb.2 bb.1: liveins: $x + $a = LDAfi %stack.2, 0 RTL bb.2: liveins: $x diff --git a/src/llvm/test/CodeGen/W65816/stack-release.ll b/src/llvm/test/CodeGen/W65816/stack-release.ll new file mode 100644 index 0000000..c3c651a --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/stack-release.ll @@ -0,0 +1,132 @@ +; Pin: stack releases (a call's argument cleanup, the epilogue's frame +; release) and the prologue's frame allocation take the cheapest form for +; what must survive them, by W65816Helpers::cycleByteCost (a cycle is worth +; two bytes): N/2 PLY (5 cycles each), TSC/CLC/ADC/TCS (9) when A is free, +; that inside TAY/TYA (13) when A holds a value. The old fixed thresholds +; pulled up to seven PLYs after a call (35 cycles) and kept A through +; TAY/TYA in every epilogue, void ones included. +; +; W65816ReturnPopMerge folds the releases on a path into a return into one: +; `tailish` pays 9 cycles where it paid 18 (and, before the cost model, 15 +; plus 15 in PLYs). +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s +; RUN: env W65816_DISABLE_RETURN_POP_MERGE=1 llc -mtriple=w65816 -O2 < %s | FileCheck %s --check-prefix=NOMERGE + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S16" + +declare void @take3(i16, i16, i16, i16) +declare i16 @get3(i16, i16, i16, i16) +declare i16 @get4(i16, i16, i16, i16, i16) +declare void @use(ptr) +@sink = global i16 0 + +; A void call: A is dead after it, so three words go in one TSC/CLC/ADC/TCS. +; CHECK-LABEL: voidCall: +; CHECK: jsl take3 +; CHECK-NEXT: tsc +; CHECK-NEXT: clc +; CHECK-NEXT: adc #0x6 +; CHECK-NEXT: tcs +; CHECK-NEXT: lda #0x1 +define void @voidCall(i16 %a, i16 %b) { +entry: + call void @take3(i16 %a, i16 %b, i16 %a, i16 %b) + store volatile i16 1, ptr @sink + ret void +} + +; The call's value is used: three words are three PLYs (15 cycles, 3 bytes +; against 13 and 8) ... +; CHECK-LABEL: valCall3: +; CHECK: jsl get3 +; CHECK-NEXT: ply +; CHECK-NEXT: ply +; CHECK-NEXT: ply +; CHECK-NEXT: sta sink +define i16 @valCall3(i16 %a, i16 %b) { +entry: + %r = call i16 @get3(i16 %a, i16 %b, i16 %a, i16 %b) + store volatile i16 %r, ptr @sink + ret i16 %r +} + +; ... four go through TAY ... TYA. +; CHECK-LABEL: valCall4: +; CHECK: jsl get4 +; CHECK-NEXT: tay +; CHECK-NEXT: tsc +; CHECK-NEXT: clc +; CHECK-NEXT: adc #0x8 +; CHECK-NEXT: tcs +; CHECK-NEXT: tya +define i16 @valCall4(i16 %a, i16 %b) { +entry: + %r = call i16 @get4(i16 %a, i16 %b, i16 %a, i16 %b, i16 %a) + store volatile i16 %r, ptr @sink + ret i16 %r +} + +; A void function's epilogue keeps nothing in A; its argument does come in +; in A, so the prologue keeps it. +; CHECK-LABEL: voidFrame: +; CHECK: tay +; CHECK-NEXT: tsc +; CHECK-NEXT: sec +; CHECK-NEXT: sbc #0xa +; CHECK-NEXT: tcs +; CHECK-NEXT: tya +; CHECK: sta sink +; CHECK-NEXT: tsc +; CHECK-NEXT: clc +; CHECK-NEXT: adc #0xa +; CHECK-NEXT: tcs +; CHECK-NEXT: rtl +define void @voidFrame(i16 %a) { +entry: + %buf = alloca [8 x i8], align 1 + call void @use(ptr %buf) + store volatile i16 %a, ptr @sink + ret void +} + +; Call, then return: one release for the arguments and the frame. +; CHECK-LABEL: tailish: +; CHECK: jsl take3 +; CHECK-NEXT: tsc +; CHECK-NEXT: clc +; CHECK-NEXT: adc #0x12 +; CHECK-NEXT: tcs +; CHECK-NEXT: rtl +; NOMERGE-LABEL: tailish: +; NOMERGE: jsl take3 +; NOMERGE-NEXT: tsc +; NOMERGE-NEXT: clc +; NOMERGE-NEXT: adc #0x6 +; NOMERGE-NEXT: tcs +; NOMERGE-NEXT: tsc +; NOMERGE-NEXT: clc +; NOMERGE-NEXT: adc #0xc +; NOMERGE-NEXT: tcs +; NOMERGE-NEXT: rtl +define void @tailish(i16 %a) { +entry: + %buf = alloca [6 x i8], align 1 + store volatile i16 %a, ptr %buf + call void @take3(i16 %a, i16 %a, i16 %a, i16 %a) + ret void +} + +; No argument in A: no TAY / TYA in the prologue either. +; CHECK-LABEL: noArgFrame: +; CHECK: rep #0x30 +; CHECK-NEXT: tsc +; CHECK-NEXT: sec +; CHECK-NEXT: sbc #0x8 +; CHECK-NEXT: tcs +define void @noArgFrame() { +entry: + %buf = alloca [8 x i8], align 1 + call void @use(ptr %buf) + ret void +} diff --git a/src/llvm/test/CodeGen/W65816/stackrel-to-img-arg-slot.ll b/src/llvm/test/CodeGen/W65816/stackrel-to-img-arg-slot.ll new file mode 100644 index 0000000..b2e0713 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/stackrel-to-img-arg-slot.ll @@ -0,0 +1,52 @@ +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s +; +; W65816StackRelToImg must not promote the caller's stack arguments to DP +; words. It keys slots by "logical" offsets that count the PHAs of a +; small-frame prologue, so there the arguments start at logical offset 4, +; not at FrameSize + 4 as with a TCS prologue. Measuring from the body SP +; in both cases let this function (libc's ungetc: a 4-byte PHA frame, the +; FILE pointer read four times) move its pointer argument to $D0/$D2 as a +; far-pointer home that nothing ever stored, then write through it. +; +; The pointer must be read from its stack slots (0x8,s / 0xa,s here) every +; time, and no DP word may stand in for it. + +target triple = "w65816" + +define i16 @ungetcLike(i16 %c, ptr %stream) { +; CHECK-LABEL: ungetcLike: +; CHECK: pha +; CHECK-NEXT: pha +; CHECK-NOT: tcs +; CHECK: lda 0x8, s +; CHECK-NEXT: ora 0xa, s +; CHECK: lda 0x8, s +; CHECK-NEXT: sta 0xe0 +; CHECK-NEXT: lda 0xa, s +; CHECK-NEXT: sta 0xe2 +; CHECK: lda [0xe0], y +; CHECK-NOT: [0xd{{[0-9a-f]}}], y +; CHECK: rtl +entry: + %tobool = icmp eq ptr %stream, null + %cmp = icmp slt i16 %c, 0 + %or.cond = or i1 %cmp, %tobool + br i1 %or.cond, label %return, label %if.end + +if.end: + %unget = getelementptr inbounds nuw i8, ptr %stream, i32 22 + %0 = load i16, ptr %unget, align 2 + %cmp1 = icmp sgt i16 %0, -1 + br i1 %cmp1, label %return, label %if.end3 + +if.end3: + %and = and i16 %c, 255 + store i16 %and, ptr %unget, align 2 + %eof = getelementptr inbounds nuw i8, ptr %stream, i32 2 + store i8 0, ptr %eof, align 2 + br label %return + +return: + %retval.0 = phi i16 [ %and, %if.end3 ], [ -1, %entry ], [ -1, %if.end ] + ret i16 %retval.0 +} diff --git a/src/llvm/test/CodeGen/W65816/stafi-far-nona-pha-bracket.mir b/src/llvm/test/CodeGen/W65816/stafi-far-nona-pha-bracket.mir index 3b645c0..9d2a0eb 100644 --- a/src/llvm/test/CodeGen/W65816/stafi-far-nona-pha-bracket.mir +++ b/src/llvm/test/CodeGen/W65816/stafi-far-nona-pha-bracket.mir @@ -7,7 +7,7 @@ # invalidating its "A still holds slot N" fact, on the strength of that bracket. # The far path used to emit a bare `LDA dp`, so an IMG-source STAfi in a >0xFF # frame silently clobbered A and the cleanup then deleted the FOLLOWING reload. -# Live symptom: the LDAptr32S inserter built its far pointer with the pointer's +# Live symptom: the LDAptr32 inserter built its far pointer with the pointer's # BANK byte in the offset half, so `lda [$E0],y` read a stray address -- JoeyLib's # Space Taxi read a nonzero byte for a zero struct field and never ran its # physics. @@ -17,7 +17,14 @@ # far_acc: offset > 0xFF, A source -> NO bracket (A is the source; the # +7 cycles would be pure cost). # -# RUN: llc -mtriple=w65816 -run-pass=prologepilog -o - %s | FileCheck %s +# The returns read A (a value-returning function), so A is live across the +# store; the bracket is left out where nothing reads A after it. +# +# The frame layout is pinned with W65816_DISABLE_FRAME_ORDER: the far cases +# need their 2-byte slot above the big array, where creation order puts it +# and the access-density order would not. +# +# RUN: env W65816_DISABLE_FRAME_ORDER=1 llc -mtriple=w65816 -run-pass=prologepilog -o - %s | FileCheck %s --- | target triple = "w65816" define void @far_img() { ret void } @@ -41,7 +48,7 @@ stack: body: | bb.0: STAfi $img8, %stack.0, 0, implicit-def $a - RTL + RTL implicit $a ... --- # CHECK-LABEL: name: near_img @@ -57,7 +64,7 @@ stack: body: | bb.0: STAfi $img8, %stack.0, 0, implicit-def $a - RTL + RTL implicit $a ... --- # A is already the source, so the far store needs no bracket at all. diff --git a/src/llvm/test/CodeGen/W65816/store-flag-preserving-wrap.mir b/src/llvm/test/CodeGen/W65816/store-flag-preserving-wrap.mir new file mode 100644 index 0000000..dd86790 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/store-flag-preserving-wrap.mir @@ -0,0 +1,119 @@ +# Pin: every store counts as flag-PRESERVING in the PHP/PLP wrap +# (W65816StackSlotCleanup Pass -2.5): the backward walk from a branch steps +# over it to the real test, and wraps every corrupter in between. +# +# A store sets no flag (the byte-store pseudos' SEP / REP touch only the M +# bit), but only the 16-bit STA / STX / STY forms were listed. Any other +# store fell to isFlagDefining's default and was taken as THE test, so the +# walk stopped at it and left the corrupters before it unwrapped: +# +# cmp #$41C6 ; xba ; and #$ff ; sep ; sta $71 ; rep ; php ; lda #0 ; plp ; beq +# +# -- llvm816's demos/randProbe.c: `(r1 == 0x41C6u) ? 0 : 0x99` next to a +# byte store of r1 >> 8. The BEQ read the AND's Z, not the compare's. +# +# byte_store_long: the randProbe shape (SRL8A, STA8long, LDAi8imm). +# byte_store_ptr: a far-pointer byte store (STB_DPIndLongY) between a +# reload and the branch. +# stz: an STZ_Abs between the corrupter and the branch (it stays +# outside the wrap, which ends after the last corrupter). +# +# RUN: llc -mtriple=w65816 -run-pass=w65816-stack-slot-cleanup -o - %s | FileCheck %s +--- | + target triple = "w65816" + @g = global i16 0 + define void @byte_store_long() { ret void } + define void @byte_store_ptr() { ret void } + define void @stz() { ret void } +... +--- +# CHECK-LABEL: name: byte_store_long +# CHECK: CMPi16imm $a, 16838, implicit-def $p +# CHECK-NEXT: PHP +# CHECK-NEXT: $a = SRL8A killed $a +# CHECK-NEXT: STA8long killed $a, 113 +# CHECK-NEXT: $a = LDAi8imm 0 +# CHECK-NEXT: PLP +# CHECK-NEXT: BEQ %bb.2, implicit $p +name: byte_store_long +tracksRegLiveness: false +body: | + bb.0: + successors: %bb.1, %bb.2 + liveins: $a + STA8long $a, 112 + CMPi16imm $a, 16838, implicit-def $p + $a = SRL8A killed $a + STA8long killed $a, 113 + $a = LDAi8imm 0 + BEQ %bb.2, implicit $p + BRA %bb.1 + bb.1: + liveins: $x + RTL + bb.2: + liveins: $x + RTL +... +--- +# CHECK-LABEL: name: byte_store_ptr +# CHECK: CMPi16imm killed $a, 7, implicit-def $p +# CHECK-NEXT: PHP +# CHECK-NEXT: $a = LDAfi %stack.1, 0 +# CHECK-NEXT: LDY_Imm16 2 +# CHECK-NEXT: STB_DPIndLongY 224 +# CHECK-NEXT: $a = LDAi16imm 1 +# CHECK-NEXT: PLP +# CHECK-NEXT: BNE %bb.1, implicit $p +name: byte_store_ptr +tracksRegLiveness: false +stack: + - { id: 0, size: 2, alignment: 1 } + - { id: 1, size: 2, alignment: 1 } +body: | + bb.0: + successors: %bb.1, %bb.2 + $a = LDAfi %stack.0, 0 + CMPi16imm killed $a, 7, implicit-def $p + $a = LDAfi %stack.1, 0 + LDY_Imm16 2, implicit-def $y + STB_DPIndLongY 224, implicit-def $p, implicit $a, implicit $y + $a = LDAi16imm 1 + BNE %bb.1, implicit $p + BRA %bb.2 + bb.1: + liveins: $x + RTL + bb.2: + liveins: $x + RTL +... +--- +# CHECK-LABEL: name: stz +# CHECK: CMPi16imm killed $a, 9, implicit-def $p +# CHECK-NEXT: PHP +# CHECK-NEXT: $a = LDAfi %stack.1, 0 +# CHECK-NEXT: PLP +# CHECK-NEXT: STZ_Abs @g +# CHECK-NEXT: BEQ %bb.1, implicit $p +name: stz +tracksRegLiveness: false +stack: + - { id: 0, size: 2, alignment: 1 } + - { id: 1, size: 2, alignment: 1 } +body: | + bb.0: + successors: %bb.1, %bb.2 + $a = LDAfi %stack.0, 0 + CMPi16imm killed $a, 9, implicit-def $p + $a = LDAfi %stack.1, 0 + STZ_Abs @g + BEQ %bb.1, implicit $p + BRA %bb.2 + bb.1: + liveins: $x + RTL + bb.2: + liveins: $x + RTL +... diff --git a/src/llvm/test/CodeGen/W65816/weak-fn-address-bank.ll b/src/llvm/test/CodeGen/W65816/weak-fn-address-bank.ll new file mode 100644 index 0000000..e8ba557 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/weak-fn-address-bank.ll @@ -0,0 +1,65 @@ +; Pin: the bank half of `&sym` for an external-weak symbol is the program +; bank when the symbol is linked and 0 when it is not, decided at run time +; from the low half the linker resolved (0 for an undefined weak): +; +; lda #weakFn ; beq L ; lda $BE ; L: +; +; It used to be a literal 0, which made an undefined weak a null pointer +; but sent every call through a stored pointer to a DEFINED weak function +; to bank 0: the runtime's `&gsosFlush` hook crashed under GS/OS, where the +; program does not run in bank 0. Other symbols -- a weak definition, a +; strong declaration, a local function -- read the program bank from $BE. +; +; RUN: llc -mtriple=w65816 -O2 < %s | FileCheck %s + +target triple = "w65816" + +@gWeak = global ptr null +@gWeakDef = global ptr null +@gStrong = global ptr null +@gStatic = global ptr null + +declare extern_weak void @weakFn() +declare void @strongFn() + +define weak void @weakDef() { + ret void +} + +define internal void @staticFn() { + ret void +} + +define void @take() { +; CHECK-LABEL: take: +; CHECK: lda #weakFn +; CHECK-NEXT: sta gWeak{{$}} +; CHECK: lda #weakFn +; CHECK-NEXT: beq [[NULL:.Ltmp[0-9]+]] +; CHECK-NEXT: lda 0xbe +; CHECK-NEXT: [[NULL]]: +; CHECK-NEXT: sta gWeak+2 +; CHECK: rtl + store ptr @weakFn, ptr @gWeak + store ptr @weakDef, ptr @gWeakDef + store ptr @strongFn, ptr @gStrong + store ptr @staticFn, ptr @gStatic + ret void +} + +; Each of the other three gets its bank from $BE alone. +define void @takeOthers() { +; CHECK-LABEL: takeOthers: +; CHECK-NOT: beq +; CHECK-NOT: lda #0x0 +; CHECK-DAG: sta gWeakDef+2 +; CHECK-DAG: sta gStrong+2 +; CHECK-DAG: sta gStatic+2 +; CHECK-NOT: beq +; CHECK-NOT: lda #0x0 +; CHECK: rtl + store ptr @weakDef, ptr @gWeakDef + store ptr @strongFn, ptr @gStrong + store ptr @staticFn, ptr @gStatic + ret void +} diff --git a/src/llvm/test/CodeGen/W65816/wide32-spill-subreg.ll b/src/llvm/test/CodeGen/W65816/wide32-spill-subreg.ll new file mode 100644 index 0000000..a04e541 --- /dev/null +++ b/src/llvm/test/CodeGen/W65816/wide32-spill-subreg.ll @@ -0,0 +1,156 @@ +; Pin: a VIRTUAL Wide32 register (one W65816LowerWide32 could not split: +; the pointer PHIs of a loop nest) spills and reloads both halves, by +; sub-register. load/storeRegToStackSlot took a virtual register for the +; unpaired-physreg case and moved the low half only (`STAfi $a` / +; `%n:wide32 = COPY $a`), so a reloaded pointer kept whatever bank its +; register pair last held, and the reload went through A behind the +; allocator's back. CoreMark's core_list_mergesort (this function) walked +; off its list once its loop exits got blocks of their own; with the old +; code that only happened to work. +; +; The reload of a pointer into an IMG pair goes through A, which it keeps +; (PHA ... PLA) where A may still be read after it: spill-a-live.mir pins +; that. +; +; RUN: llc -mtriple=w65816 -O2 -stop-after=greedy -o - < %s | FileCheck %s + +; CHECK-LABEL: name: core_list_mergesort +; CHECK-NOT: wide32 = COPY $a +; CHECK: STAfi %{{[0-9]+}}.sub_lo, %stack.[[S:[0-9]+]], 0 +; CHECK-NEXT: STAfi %{{[0-9]+}}.sub_hi, %stack.[[S]], 2 +; CHECK-NOT: wide32 = COPY $a +; CHECK: undef %[[R:[0-9]+]].sub_hi:wide32 = LDAfi %stack.{{[0-9]+}}, 2 +; CHECK-NEXT: %[[R]].sub_lo:wide32 = LDAfi %stack.{{[0-9]+}}, 0 +; CHECK-NOT: wide32 = COPY $a + +target datalayout = "e-m:e-p:32:16-i16:16-i32:16-i64:16-f32:16-f64:16-a:8-n8:16-S8" +target triple = "w65816" + +%struct.list_data_s = type { i16, i16 } + +define dso_local ptr @core_list_mergesort(ptr noundef %list, ptr noundef readonly captures(none) %cmp, ptr noundef %res) local_unnamed_addr { +entry: + br label %while.cond + +while.cond: ; preds = %while.end37, %entry + %list.addr.0 = phi ptr [ %list, %entry ], [ %list.addr.2.ph, %while.end37 ] + %insize.0 = phi i32 [ 1, %entry ], [ %mul, %while.end37 ] + call void @llvm.assume(i1 true) [ "nonnull"(ptr %list.addr.0) ] + %cmp374 = icmp sgt i32 %insize.0, 0 + br label %while.body2 + +while.cond1.loopexit: ; preds = %lor.rhs + %tobool.not = icmp eq ptr %q.2, null + br i1 %tobool.not, label %while.end37, label %while.body2 + +while.body2: ; preds = %while.cond, %while.cond1.loopexit + %nmerges.085 = phi i32 [ 0, %while.cond ], [ %inc, %while.cond1.loopexit ] + %tail.084 = phi ptr [ null, %while.cond ], [ %tail.1, %while.cond1.loopexit ] + %p.083 = phi ptr [ %list.addr.0, %while.cond ], [ %q.2, %while.cond1.loopexit ] + %list.addr.182 = phi ptr [ null, %while.cond ], [ %list.addr.2.ph, %while.cond1.loopexit ] + %inc = add nuw nsw i32 %nmerges.085, 1 + br i1 %cmp374, label %for.body, label %while.cond7.preheader + +for.cond: ; preds = %for.body + %cmp3 = icmp slt i32 %inc6, %insize.0 + br i1 %cmp3, label %for.body, label %while.cond7.preheader + +for.body: ; preds = %while.body2, %for.cond + %i.077 = phi i32 [ %inc6, %for.cond ], [ 0, %while.body2 ] + %q.075 = phi ptr [ %0, %for.cond ], [ %p.083, %while.body2 ] + %inc6 = add nuw nsw i32 %i.077, 1 + %0 = load ptr, ptr %q.075, align 2 + %tobool5.not = icmp eq ptr %0, null + br i1 %tobool5.not, label %while.cond7.preheader, label %for.cond + +while.cond7.preheader: ; preds = %for.cond, %for.body, %while.body2 + %q.2.ph = phi ptr [ %p.083, %while.body2 ], [ %0, %for.cond ], [ null, %for.body ] + %psize.2.ph = phi i32 [ 0, %while.body2 ], [ %inc6, %for.body ], [ %inc6, %for.cond ] + br label %while.cond7.outer + +while.cond7.outer: ; preds = %if.end31, %while.cond7.preheader + %list.addr.2.ph = phi ptr [ %list.addr.182, %while.cond7.preheader ], [ %e.0, %if.end31 ] + %p.1.ph = phi ptr [ %p.083, %while.cond7.preheader ], [ %p.2, %if.end31 ] + %q.2.ph88 = phi ptr [ %q.2.ph, %while.cond7.preheader ], [ %q.3, %if.end31 ] + %tail.1.ph = phi ptr [ %tail.084, %while.cond7.preheader ], [ %e.0, %if.end31 ] + %psize.2.ph89 = phi i32 [ %psize.2.ph, %while.cond7.preheader ], [ %psize.3, %if.end31 ] + %qsize.0.ph = phi i32 [ %insize.0, %while.cond7.preheader ], [ %qsize.1, %if.end31 ] + br label %while.cond7 + +while.cond7: ; preds = %while.cond7.outer, %if.then33 + %p.1 = phi ptr [ %p.2, %if.then33 ], [ %p.1.ph, %while.cond7.outer ] + %q.2 = phi ptr [ %q.3, %if.then33 ], [ %q.2.ph88, %while.cond7.outer ] + %tail.1 = phi ptr [ %e.0, %if.then33 ], [ %tail.1.ph, %while.cond7.outer ] + %psize.2 = phi i32 [ %psize.3, %if.then33 ], [ %psize.2.ph89, %while.cond7.outer ] + %qsize.0 = phi i32 [ %qsize.1, %if.then33 ], [ %qsize.0.ph, %while.cond7.outer ] + %cmp8 = icmp sgt i32 %psize.2, 0 + br i1 %cmp8, label %if.else, label %lor.rhs + +lor.rhs: ; preds = %while.cond7 + %cmp9 = icmp sgt i32 %qsize.0, 0 + %tobool10 = icmp ne ptr %q.2, null + %1 = select i1 %cmp9, i1 %tobool10, i1 false + br i1 %1, label %while.body11, label %while.cond1.loopexit + +while.body11: ; preds = %lor.rhs + %cmp12 = icmp eq i32 %psize.2, 0 + br i1 %cmp12, label %if.then13, label %if.else + +if.then13: ; preds = %while.body11 + %2 = load ptr, ptr %q.2, align 2 + %dec = add nsw i32 %qsize.0, -1 + br label %if.end31 + +if.else: ; preds = %while.cond7, %while.body11 + %cmp15 = icmp ne i32 %qsize.0, 0 + %tobool16 = icmp ne ptr %q.2, null + %or.cond = select i1 %cmp15, i1 %tobool16, i1 false + br i1 %or.cond, label %if.else20, label %if.then17 + +if.then17: ; preds = %if.else + %3 = load ptr, ptr %p.1, align 2 + %dec19 = add nsw i32 %psize.2, -1 + br label %if.end31 + +if.else20: ; preds = %if.else + %info = getelementptr inbounds nuw i8, ptr %p.1, i32 4 + %4 = load ptr, ptr %info, align 2 + %info21 = getelementptr inbounds nuw i8, ptr %q.2, i32 4 + %5 = load ptr, ptr %info21, align 2 + %call = tail call i32 %cmp(ptr noundef %4, ptr noundef %5, ptr noundef %res) + %cmp22 = icmp slt i32 %call, 1 + br i1 %cmp22, label %if.then23, label %if.else26 + +if.then23: ; preds = %if.else20 + %6 = load ptr, ptr %p.1, align 2 + %dec25 = add nsw i32 %psize.2, -1 + br label %if.end31 + +if.else26: ; preds = %if.else20 + %7 = load ptr, ptr %q.2, align 2 + %dec28 = add nsw i32 %qsize.0, -1 + br label %if.end31 + +if.end31: ; preds = %if.then17, %if.else26, %if.then23, %if.then13 + %p.2 = phi ptr [ %p.1, %if.then13 ], [ %6, %if.then23 ], [ %p.1, %if.else26 ], [ %3, %if.then17 ] + %q.3 = phi ptr [ %2, %if.then13 ], [ %q.2, %if.then23 ], [ %7, %if.else26 ], [ %q.2, %if.then17 ] + %e.0 = phi ptr [ %q.2, %if.then13 ], [ %p.1, %if.then23 ], [ %q.2, %if.else26 ], [ %p.1, %if.then17 ] + %psize.3 = phi i32 [ 0, %if.then13 ], [ %dec25, %if.then23 ], [ %psize.2, %if.else26 ], [ %dec19, %if.then17 ] + %qsize.1 = phi i32 [ %dec, %if.then13 ], [ %qsize.0, %if.then23 ], [ %dec28, %if.else26 ], [ %qsize.0, %if.then17 ] + %tobool32.not = icmp eq ptr %tail.1, null + br i1 %tobool32.not, label %while.cond7.outer, label %if.then33 + +if.then33: ; preds = %if.end31 + store ptr %e.0, ptr %tail.1, align 2 + br label %while.cond7 + +while.end37: ; preds = %while.cond1.loopexit + store ptr null, ptr %tail.1, align 2 + %cmp39 = icmp eq i32 %nmerges.085, 0 + %mul = shl nsw i32 %insize.0, 1 + br i1 %cmp39, label %if.then40, label %while.cond + +if.then40: ; preds = %while.end37 + ret ptr %list.addr.2.ph +} + diff --git a/tests/coremark/coreMark.manifest.json b/tests/coremark/coreMark.manifest.json index bca31e9..12b2dc7 100644 --- a/tests/coremark/coreMark.manifest.json +++ b/tests/coremark/coreMark.manifest.json @@ -7,7 +7,7 @@ "num": 1, "name": "SEG1", "base": "0x001000", - "size": 44791, + "size": 43655, "bss_start": "0x00d800", "bss_size": 2034, "image": "coreMark.bin", @@ -17,7 +17,7 @@ "num": 2, "name": "SEG2", "base": "0x040000", - "size": 10368, + "size": 4571, "bss_start": "0x000000", "bss_size": 0, "image": "coreMark.seg2.bin", diff --git a/tests/fpNan/genFpCmpConst.py b/tests/fpNan/genFpCmpConst.py new file mode 100644 index 0000000..776cd6b --- /dev/null +++ b/tests/fpNan/genFpCmpConst.py @@ -0,0 +1,217 @@ +#!/usr/bin/env python3 +# genFpCmpConst.py - emits the C source of the smoke check "float/double +# compares against constants", with kExp (the expected answers) computed +# here from Python's IEEE 754 compares. +# +# python3 tests/fpNan/genFpCmpConst.py f # float +# python3 tests/fpNan/genFpCmpConst.py d 1 2 # double, first half +# python3 tests/fpNan/genFpCmpConst.py d 2 2 # double, second half +# +# PART / NPARTS split the tests so each binary stays well below the IIgs +# I/O window at $C000 (the whole double table is ~54 KB of code). +# +# W65816's combineFPSetCCConst turns a compare the operand's bit pattern +# alone decides (x against a zero, an infinity or any non-NaN constant for +# == / !=, fabs(x) against a constant >= 0, x against itself, the ordered / +# unordered tests) into integer compares instead of a soft-float libcall. +# This check runs every C comparison operator, its negation (LLVM's +# unordered predicates), isunordered and islessgreater and their negations +# against a set of constants, on x and on fabs(x), plus the classification +# builtins, each as a branch and as a 0/1 value, on 34 values: signed +# zeros, subnormals, normals and their neighbours one ulp away, the largest +# finite values, infinities and quiet, signaling and all-ones NaNs of both +# signs. Compares the combine leaves alone run the libcall, so the table +# also pins the helpers' NaN contract. +# +# Mailbox: 0x025000 = mismatches, 0x025002 = first failing case +# (function index * NV + value index), 0x025004 = 0xC0DE when done. + +import math +import struct +import sys + +KIND = sys.argv[1] if len(sys.argv) > 1 else "f" +PART = int(sys.argv[2]) if len(sys.argv) > 2 else 1 +NPARTS = int(sys.argv[3]) if len(sys.argv) > 3 else 1 +if KIND not in ("f", "d") or not 1 <= PART <= NPARTS: + sys.exit("usage: genFpCmpConst.py f|d [PART NPARTS]") + +if KIND == "f": + T = "float" + BITS_T = "u32" + SUF = "UL" + FABS = "__builtin_fabsf" + VALS = [ + 0x00000000, 0x80000000, 0x00000001, 0x80000001, 0x007FFFFF, + 0x807FFFFF, 0x00800000, 0x80800000, 0x3F7FFFFF, 0x3F800000, + 0x3F800001, 0xBF7FFFFF, 0xBF800000, 0xBF800001, 0x401FFFFF, + 0x40200000, 0x40200001, 0xC01FFFFF, 0xC0200000, 0xC0200001, + 0x7F7FFFFF, 0xFF7FFFFF, 0x7F800000, 0xFF800000, 0x7FC00000, + 0xFFC00000, 0x7F800001, 0xFF800001, 0x7FFFFFFF, 0xFFFFFFFF, + 0x7F7FFFFE, 0x00000002, 0x7F000000, 0xFF000000, + ] + + def decode(b): + return struct.unpack(" B", lambda a, b: a > b), + ("A >= B", lambda a, b: a >= b), + ("A == B", lambda a, b: a == b), + ("A != B", lambda a, b: a != b), + ("!(A < B)", lambda a, b: not a < b), + ("!(A <= B)", lambda a, b: not a <= b), + ("!(A > B)", lambda a, b: not a > b), + ("!(A >= B)", lambda a, b: not a >= b), + ("__builtin_isunordered(A, B)", isun), + ("!__builtin_isunordered(A, B)", lambda a, b: not isun(a, b)), + ("__builtin_islessgreater(A, B)", lambda a, b: a < b or a > b), + ("!__builtin_islessgreater(A, B)", lambda a, b: not (a < b or a > b)), +] + +# Every test: (C expression of a, Python predicate of a). +tests = [] +for c in CONSTS: + for expr, fn in PREDS: + tests.append((expr.replace("A", "a").replace("B", lit(c)), + (lambda fn, c: lambda a: fn(a, c))(fn, c))) +for c in ABS_CONSTS: + for expr, fn in PREDS: + tests.append((expr.replace("A", FABS + "(a)").replace("B", lit(c)), + (lambda fn, c: lambda a: fn(abs(a), c))(fn, c))) +for expr, fn in PREDS: + tests.append((expr.replace("A", "a").replace("B", "a"), + (lambda fn: lambda a: fn(a, a))(fn))) +tests += [ + ("__builtin_isnan(a)", lambda a: math.isnan(a)), + ("!__builtin_isnan(a)", lambda a: not math.isnan(a)), + ("__builtin_isinf(a)", lambda a: math.isinf(a)), + ("__builtin_isfinite(a)", lambda a: math.isfinite(a)), +] + +per = (len(tests) + NPARTS - 1) // NPARTS +tests = tests[(PART - 1) * per:PART * per] +nv = len(VALS) +nf = 2 * len(tests) +bits = [] +for expr, fn in tests: + for form in range(2): + for v in VALS: + bits.append(1 if fn(decode(v)) else 0) +exp = bytearray((len(bits) + 7) // 8) +for i, b in enumerate(bits): + if b: + exp[i >> 3] |= 1 << (i & 7) + +out = [] +w = out.append +w("// Float / double compares against constants, generated by") +w("// tests/fpNan/genFpCmpConst.py %s %d %d; see there." % (KIND, PART, NPARTS)) +w("typedef unsigned char u8;") +w("typedef unsigned short u16;") +w("typedef unsigned long u32;") +w("typedef unsigned long long u64;") +w('#define BAR() __asm__ volatile("" ::: "memory")') +w("#define NOI __attribute__((noinline)) static") +w("#define NV %d" % nv) +w("#define NF %d" % nf) +w("typedef %s T;" % T) +w("static const %s kBits[NV] = {" % BITS_T) +for i in range(0, nv, 4): + w(" " + " ".join("0x%X%s," % (v, SUF) for v in VALS[i:i + 4])) +w("};") +w("static const u8 kExp[%d] = {" % len(exp)) +for i in range(0, len(exp), 16): + w(" " + " ".join("0x%02X," % b for b in exp[i:i + 16])) +w("};") +for k, (expr, fn) in enumerate(tests): + w("NOI u16 b%d(T a) { if (%s) { BAR(); return 1u; } return 0u; }" + % (k, expr)) + w("NOI u16 v%d(T a) { return (u16)(%s); }" % (k, expr)) +w("typedef u16 (*Fn)(T);") +w("static Fn const kFn[NF] = {") +for k in range(len(tests)): + w(" b%d, v%d," % (k, k)) +w("};") +w("static u16 gBad;") +w("static u16 gFirst = 0xFFFFu;") +w("static T val(u16 i) {") +w(" union { %s u; T x; } c;" % BITS_T) +w(" c.u = kBits[i];") +w(" return c.x;") +w("}") +w("int main(void) {") +w(" u16 f;") +w(" u16 i;") +w(" u16 bit = 0u;") +w(" for (f = 0u; f < NF; f++) {") +w(" for (i = 0u; i < NV; i++) {") +w(" u16 want = (u16)((kExp[bit >> 3] >> (bit & 7u)) & 1u);") +w(" if ((kFn[f](val(i)) != 0u) != want) {") +w(" if (gBad == 0u) {") +w(" gFirst = bit;") +w(" }") +w(" gBad++;") +w(" }") +w(" bit++;") +w(" }") +w(" }") +w(" *(volatile u16 *)0x025000UL = gBad;") +w(" *(volatile u16 *)0x025002UL = gFirst;") +w(" *(volatile u16 *)0x025004UL = 0xC0DEu;") +w(" for (;;) {") +w(" }") +w(" return 0;") +w("}") +print("\n".join(out)) diff --git a/tests/fpNan/genFpTables.py b/tests/fpNan/genFpTables.py index d11ab4f..bbf5a8a 100644 --- a/tests/fpNan/genFpTables.py +++ b/tests/fpNan/genFpTables.py @@ -5,6 +5,7 @@ # # python3 tests/fpNan/genFpTables.py cmp # kFBits / kDBits / kExp # python3 tests/fpNan/genFpTables.py special # kCases +# python3 tests/fpNan/genFpTables.py edge # kEdge # # Comparisons come from Python's own float compares, which are IEEE 754 # (every ordered compare with a NaN is false). Float operands are widened @@ -23,6 +24,14 @@ # pow with a non-integer exponent, sinh, cbrt, ...) are only ~1e-6 # accurate, so their ordinary-value cases carry an `approx` flag: the check # accepts a result within 1e-6 relative of the host's value. +# +# The edge table is math.c's exact and bit-level functions, checked bit for +# bit against Python's (C99 Annex F) answers: fmod and remainder must be +# exact for any quotient, modf / modff keep the sign of zero and split the +# infinities, atan2 has every signed-zero and infinity case, fpclassify / +# isnormal tell subnormals apart in both widths, hypot scales instead of +# overflowing or underflowing, frexp / ldexp handle subnormals and huge +# exponents, and asinh / acosh / atanh keep signs and do not overflow. import math import struct @@ -583,7 +592,177 @@ def annexFCases(): return out +# ---- math.c edge table ------------------------------------------------- + +EDGE_OPS = ['modfFrac', 'modfInt', 'modffFrac', 'modffInt', 'atan2', + 'clsD', 'clsF', 'hypot', 'fmod', 'remainder', 'frexpM', 'frexpE', + 'ldexp', 'asinh', 'acosh', 'atanh'] + +FP_NAN_CLS, FP_INFINITE_CLS, FP_NORMAL_CLS, FP_SUBNORMAL_CLS, FP_ZERO_CLS = 0, 1, 2, 3, 4 +D_MIN_SUB = 5e-324 +F_MIN_SUB_BITS = 0x00000001 +F_MAX_SUB_BITS = 0x007FFFFF +D_MAX_SUB = dval(0x000FFFFFFFFFFFFF) + + +def clsBits(bits, expBits, mantBits): + e = (bits >> mantBits) & ((1 << expBits) - 1) + m = bits & ((1 << mantBits) - 1) + if e == (1 << expBits) - 1: + c = FP_NAN_CLS if m else FP_INFINITE_CLS + elif e: + c = FP_NORMAL_CLS + else: + c = FP_SUBNORMAL_CLS if m else FP_ZERO_CLS + return c | ((1 if c == FP_NORMAL_CLS else 0) << 4) + + +def cmodf(x): + if math.isnan(x): + return (NAN, NAN) + return math.modf(x) + + +def cfmodExact(x, y): + if math.isnan(x) or math.isnan(y) or math.isinf(x) or y == 0: + return NAN + if math.isinf(y): + return x + return math.fmod(x, y) + + +def catan2(y, x): + return NAN if math.isnan(x) or math.isnan(y) else math.atan2(y, x) + + +def chypot(x, y): + if math.isinf(x) or math.isinf(y): + return INF + return math.hypot(x, y) + + +def cldexp(x, n): + try: + return math.ldexp(x, n) + except OverflowError: + return math.copysign(INF, x) + + +def cacosh(x): + return NAN if math.isnan(x) or x < 1 else math.acosh(x) + + +def catanh(x): + if math.isnan(x) or abs(x) > 1: + return NAN + if abs(x) == 1: + return math.copysign(INF, x) + return math.atanh(x) + + +# (op, a, b, approx); a and b are doubles, or floats for the float ops. +EDGE_CASES = [ + ('modfFrac', -2.0, 0, 0), ('modfInt', -2.0, 0, 0), ('modfFrac', -0.0, 0, 0), + ('modfInt', -0.0, 0, 0), ('modfFrac', INF, 0, 0), ('modfInt', INF, 0, 0), + ('modfFrac', -INF, 0, 0), ('modfInt', -INF, 0, 0), ('modfFrac', NAN, 0, 0), + ('modfInt', NAN, 0, 0), ('modfFrac', -3.75, 0, 0), ('modfInt', -3.75, 0, 0), + ('modfFrac', 0.5, 0, 0), ('modfInt', 0.5, 0, 0), ('modfFrac', 1e300, 0, 0), + ('modfInt', 4503599627370497.5, 0, 0), ('modfFrac', 4503599627370497.5, 0, 0), + ('modfFrac', D_MIN_SUB, 0, 0), ('modfInt', -D_MIN_SUB, 0, 0), + ('modffFrac', -2.0, 0, 0), ('modffInt', -2.0, 0, 0), ('modffFrac', -0.0, 0, 0), + ('modffFrac', INF, 0, 0), ('modffInt', -INF, 0, 0), ('modffFrac', NAN, 0, 0), + ('modffInt', NAN, 0, 0), ('modffFrac', -3.75, 0, 0), ('modffInt', -3.75, 0, 0), + ('modffFrac', 8388609.5, 0, 0), ('modffFrac', 1e-45, 0, 0), + ('atan2', 0.0, 0.0, 0), ('atan2', 0.0, -0.0, 0), ('atan2', -0.0, 0.0, 0), + ('atan2', -0.0, -0.0, 0), ('atan2', 0.0, -1.0, 0), ('atan2', -0.0, -1.0, 0), + ('atan2', 0.0, 1.0, 0), ('atan2', -0.0, 1.0, 0), ('atan2', 1.0, 0.0, 0), + ('atan2', -1.0, -0.0, 0), ('atan2', 1.0, -INF, 0), ('atan2', -1.0, -INF, 0), + ('atan2', 1.0, INF, 0), ('atan2', -1.0, INF, 0), ('atan2', INF, 1.0, 0), + ('atan2', -INF, -1.0, 0), ('atan2', INF, -INF, 0), ('atan2', -INF, -INF, 0), + ('atan2', INF, INF, 0), ('atan2', -INF, INF, 0), ('atan2', NAN, 1.0, 0), + ('atan2', 1.0, NAN, 0), ('atan2', 1.0, 1.0, 1), ('atan2', -1.0, -2.0, 1), + ('atan2', 2.0, -1.0, 1), ('atan2', 1e300, 1e-300, 0), ('atan2', 1e-300, -1e300, 0), + ('clsD', 1.0, 0, 0), ('clsD', -0.0, 0, 0), ('clsD', D_MIN_SUB, 0, 0), + ('clsD', -D_MAX_SUB, 0, 0), ('clsD', 2.2250738585072014e-308, 0, 0), + ('clsD', INF, 0, 0), ('clsD', NAN, 0, 0), ('clsD', -1e300, 0, 0), + ('clsF', 1.0, 0, 0), ('clsF', -0.0, 0, 0), ('clsF', fval(F_MIN_SUB_BITS), 0, 0), + ('clsF', -fval(F_MAX_SUB_BITS), 0, 0), ('clsF', 1.1754943508222875e-38, 0, 0), + ('clsF', INF, 0, 0), ('clsF', NAN, 0, 0), ('clsF', -3e38, 0, 0), + ('hypot', 1e200, 1e200, 1), ('hypot', 3e200, -4e200, 1), ('hypot', 1e-200, 1e-200, 1), + ('hypot', -3e-300, 4e-300, 1), ('hypot', 1e308, 1e308, 0), ('hypot', 1e300, 1.0, 0), + ('hypot', D_MIN_SUB, D_MIN_SUB, 0), ('hypot', -0.0, -0.0, 0), ('hypot', -0.0, -5.0, 0), + ('hypot', -INF, NAN, 0), ('hypot', NAN, 2.0, 0), ('hypot', 3.0, 4.0, 1), + ('fmod', 1e300, 3.0, 0), ('fmod', -1e300, 7.0, 0), ('fmod', 1e20, 0.1, 0), + ('fmod', 5.5, -2.0, 0), ('fmod', -5.5, -2.0, 0), ('fmod', -6.0, 3.0, 0), + ('fmod', 6.0, -3.0, 0), ('fmod', 1.0, 3.0, 0), ('fmod', 3 * D_MIN_SUB, 2 * D_MIN_SUB, 0), + ('fmod', 1.5, D_MIN_SUB, 0), ('fmod', 1.7976931348623157e308, 1e-300, 0), + ('fmod', 2.0 ** 60 + 2.0 ** 8, 2.0 ** -40, 0), ('fmod', 7.5, 2.0, 0), + ('fmod', -0.0, 3.0, 0), ('fmod', 1.0, -INF, 0), ('fmod', INF, 1.0, 0), + ('fmod', NAN, 1.0, 0), ('fmod', 1.0, 0.0, 0), + ('remainder', 1e300, 3.0, 0), ('remainder', -1e300, 7.0, 0), ('remainder', 1e20, 0.1, 0), + ('remainder', -4.0, 2.0, 0), ('remainder', 4.0, -2.0, 0), ('remainder', 3.0, 2.0, 0), + ('remainder', -3.0, 2.0, 0), ('remainder', 5.0, -2.0, 0), ('remainder', 1.5, 1e-300, 0), + ('remainder', 3 * D_MIN_SUB, 2 * D_MIN_SUB, 0), ('remainder', 1.7976931348623157e308, -1.7e308, 0), + ('remainder', 1.0, INF, 0), ('remainder', INF, 1.0, 0), ('remainder', 1.0, -0.0, 0), + ('frexpM', D_MIN_SUB, 0, 0), ('frexpE', D_MIN_SUB, 0, 0), ('frexpM', -D_MAX_SUB, 0, 0), + ('frexpE', -D_MAX_SUB, 0, 0), ('frexpM', 1e300, 0, 0), ('frexpE', 1e300, 0, 0), + ('frexpM', -0.0, 0, 0), ('frexpE', -0.0, 0, 0), ('frexpM', -INF, 0, 0), + ('frexpM', 0.75, 0, 0), ('frexpE', 0.75, 0, 0), + ('ldexp', 1.0, 1023, 0), ('ldexp', 1.0, 1024, 0), ('ldexp', -1.0, 32767, 0), + ('ldexp', 1.0, -32768, 0), ('ldexp', 0.0, 30000, 0), ('ldexp', -0.0, -30000, 0), + ('ldexp', 1.0, -1074, 0), ('ldexp', 1.5, -1074, 0), ('ldexp', 1.0, -1075, 0), + ('ldexp', 3.0, -1075, 0), ('ldexp', D_MIN_SUB, 1074, 0), ('ldexp', -D_MAX_SUB, 1, 0), + ('ldexp', 0.75, -1022, 0), ('ldexp', INF, -5, 0), ('ldexp', NAN, 5, 0), + ('asinh', -0.0, 0, 0), ('asinh', 0.0, 0, 0), ('asinh', -INF, 0, 0), ('asinh', INF, 0, 0), + ('asinh', NAN, 0, 0), ('asinh', -1e10, 0, 1), ('asinh', 1e200, 0, 1), ('asinh', -1e300, 0, 1), + ('asinh', 0.5, 0, 1), ('asinh', -2.0, 0, 1), + ('acosh', 1e200, 0, 1), ('acosh', INF, 0, 0), ('acosh', 1.0, 0, 0), ('acosh', 0.5, 0, 0), + ('acosh', -INF, 0, 0), ('acosh', NAN, 0, 0), ('acosh', 2.0, 0, 1), + ('atanh', -0.0, 0, 0), ('atanh', 0.0, 0, 0), ('atanh', -0.5, 0, 1), ('atanh', 1.0, 0, 0), +] + +EDGE_MODEL = { + 'modfFrac': lambda x, y: cmodf(x)[0], 'modfInt': lambda x, y: cmodf(x)[1], + 'atan2': catan2, 'hypot': chypot, 'fmod': cfmodExact, 'remainder': cremainder, + 'frexpM': lambda x, y: x if not math.isfinite(x) else math.frexp(x)[0], + 'ldexp': lambda x, n: x if not math.isfinite(x) else cldexp(x, n), + 'asinh': lambda x, y: math.asinh(x), 'acosh': lambda x, y: cacosh(x), + 'atanh': lambda x, y: catanh(x), +} + + +def edgeCase(name, x, y, approx): + op = EDGE_OPS.index(name) + if name in ('modffFrac', 'modffInt'): + xf = fround32(x) + r = cmodf(xf)[0 if name == 'modffFrac' else 1] + return (op, fbits(xf), 0, nanFold(fbits(fround32(r)), True), approx) + if name == 'clsF': + b = fbits(fround32(x)) if not math.isnan(x) else F_DNAN + return (op, b, 0, clsBits(b, 8, 23), approx) + if name == 'clsD': + b = dbits(x) if not math.isnan(x) else D_DNAN + return (op, b, 0, clsBits(b, 11, 52), approx) + if name == 'frexpE': + return (op, dbits(x), 0, math.frexp(x)[1] & 0xFFFF, approx) + if name == 'ldexp': + return (op, dbits(x), y & 0xFFFF, nanFold(dbits(EDGE_MODEL[name](x, y)), False), approx) + want = EDGE_MODEL[name](x, y) + return (op, dbits(x), dbits(y), nanFold(dbits(want), False), approx) + + +def emitEdge(): + table = [edgeCase(*c) for c in EDGE_CASES] + print('#define NE %d' % len(table)) + print('static const CaseT kEdge[NE] = {') + for op, a, b, want, approx in table: + print(' { %-14s 0x%016XULL, 0x%016XULL, 0x%016XULL, %d },' % + ('E_' + EDGE_OPS[op].upper() + ',', a, b, want, approx)) + print('};') + + if __name__ == '__main__': - if len(sys.argv) != 2 or sys.argv[1] not in ('cmp', 'special'): - sys.exit('usage: genFpTables.py cmp|special') - emitCmp() if sys.argv[1] == 'cmp' else emitSpecial() + modes = {'cmp': emitCmp, 'special': emitSpecial, 'edge': emitEdge} + if len(sys.argv) != 2 or sys.argv[1] not in modes: + sys.exit('usage: genFpTables.py cmp|special|edge') + modes[sys.argv[1]]() diff --git a/tests/lua/luaTest.manifest.json b/tests/lua/luaTest.manifest.json index 4d6e11e..f673ddb 100644 --- a/tests/lua/luaTest.manifest.json +++ b/tests/lua/luaTest.manifest.json @@ -7,9 +7,9 @@ "num": 1, "name": "SEG1", "base": "0x001000", - "size": 44531, + "size": 45004, "bss_start": "0x00e300", - "bss_size": 4248, + "bss_size": 4252, "image": "luaTest.bin", "entry_offset": "0x0000" }, @@ -17,7 +17,7 @@ "num": 2, "name": "SEG2", "base": "0x040000", - "size": 44940, + "size": 44949, "bss_start": "0x000000", "bss_size": 0, "image": "luaTest.seg2.bin", @@ -27,21 +27,11 @@ "num": 3, "name": "SEG3", "base": "0x050000", - "size": 41839, + "size": 32420, "bss_start": "0x000000", "bss_size": 0, "image": "luaTest.seg3.bin", "entry_offset": "0x0000" - }, - { - "num": 4, - "name": "SEG4", - "base": "0x060000", - "size": 28814, - "bss_start": "0x000000", - "bss_size": 0, - "image": "luaTest.seg4.bin", - "entry_offset": "0x0000" } ] } diff --git a/tests/memBlock/memBlockIrq.c b/tests/memBlock/memBlockIrq.c new file mode 100644 index 0000000..5be2dd9 --- /dev/null +++ b/tests/memBlock/memBlockIrq.c @@ -0,0 +1,122 @@ +// MVN / MVP stay correct when an interrupt lands mid-move and its handler +// moves memory too. memBlockIrq.lua injects a native-mode interrupt (to +// irqEntry, memBlockIrqEntry.s) whenever the CPU is executing an armed +// outer call's block-move stub on the stack; the handler runs irqNested, +// whose memcpy / memmove / memset use other banks and their own stubs, and +// after RTI the outer move resumes. A move whose bank operands lived in +// shared code would resume with the handler's banks. +// Mailbox: 0x025000 = bad outer results, 0x025002 = bad nested results, +// 0x025004 = 0xC0DE done, 0x025006 = 1 when at least MIN_SHOTS interrupts +// landed mid-move and every handler finished, 0x025020 = irqEntry (set +// here), 0x025022 = armed, 0x025024 = interrupts injected (Lua), 0x025026 +// = handlers completed, 0x025028 = a handler is running (Lua sets it, the +// handler clears it). +#include "memPattern.h" + +#define MB_BAD (*(volatile u16 *)0x025000UL) +#define MB_BAD_IN (*(volatile u16 *)0x025002UL) +#define MB_DONE (*(volatile u16 *)0x025004UL) +#define MB_SHOT_OK (*(volatile u16 *)0x025006UL) +#define MB_IRQ (*(volatile u16 *)0x025020UL) +#define MB_ARM (*(volatile u16 *)0x025022UL) +#define MB_SHOTS (*(volatile u16 *)0x025024UL) +#define MB_HANDLED (*(volatile u16 *)0x025026UL) +#define MB_BUSY (*(volatile u16 *)0x025028UL) + +#define MB_DONE_MARK 0xC0DEu +#define ROUNDS 2u +#define MIN_SHOTS 8u +#define OUTER_LEN 20000UL +#define INNER_LEN 300UL +#define SEED_OUT 0x3Cu +#define SEED_IN 0x96u +#define MOVE_SHIFT 0x123UL +#define IN_SET_VAL 0xA7 + +// Outer: one MVN (bank 6 to 7), several runs (the source across 08/09, the +// destination across 0A/0B), a backward overlapping MVP across 0C/0D. +#define OUT_SRC1 0x061000UL +#define OUT_DST1 0x072000UL +#define OUT_SRC2 (0x090000UL - 11111UL) +#define OUT_DST2 (0x0B0000UL - 7777UL) +#define OUT_MOVE (0x0D0000UL - 15000UL) +// Inner: banks 0E-11, apart from the outer ones. +#define IN_SRC (0x0F0000UL - 500UL) +#define IN_DST 0x0E4000UL +#define IN_MOVE (0x110000UL - 700UL) +#define IN_SET 0x103000UL + +static int filledWith(u32 a, u32 n, u8 c); +void irqEntry(void); +void irqNested(void); + +static u16 gBadIn; + + +static int filledWith(u32 a, u32 n, u8 c) { + const u8 *p = (const u8 *)a; + u16 i; + for (i = 0; i < (u16)n; i++) { + if (p[i] != c) { + return 0; + } + } + return 1; +} + + +// Called from irqEntry, on the interrupted program's stack. +void irqNested(void) { + fill(IN_SRC, INNER_LEN, SEED_IN); + memset((void *)IN_DST, 0, INNER_LEN); + if (memcpy((void *)IN_DST, (const void *)IN_SRC, INNER_LEN) != (void *)IN_DST || + !holds(IN_DST, IN_SRC, INNER_LEN, SEED_IN)) { + gBadIn++; + } + fill(IN_MOVE, INNER_LEN + MOVE_SHIFT, SEED_IN); + memmove((void *)(IN_MOVE + MOVE_SHIFT), (const void *)IN_MOVE, INNER_LEN); + if (!holds(IN_MOVE + MOVE_SHIFT, IN_MOVE, INNER_LEN, SEED_IN)) { + gBadIn++; + } + memset((void *)IN_SET, IN_SET_VAL, INNER_LEN); + if (!filledWith(IN_SET, INNER_LEN, (u8)IN_SET_VAL)) { + gBadIn++; + } + MB_BAD_IN = gBadIn; + MB_HANDLED = (u16)(MB_HANDLED + 1u); + MB_BUSY = 0u; +} + + +int main(void) { + u16 bad = 0u; + u16 r; + MB_IRQ = (u16)(u32)irqEntry; + fill(OUT_SRC1, OUTER_LEN, SEED_OUT); + fill(OUT_SRC2, OUTER_LEN, SEED_OUT); + for (r = 0; r < ROUNDS; r++) { + memset((void *)OUT_DST1, 0, OUTER_LEN); + memset((void *)OUT_DST2, 0, OUTER_LEN); + fill(OUT_MOVE, OUTER_LEN + MOVE_SHIFT, SEED_OUT); + MB_ARM = 1u; + memcpy((void *)OUT_DST1, (const void *)OUT_SRC1, OUTER_LEN); + memcpy((void *)OUT_DST2, (const void *)OUT_SRC2, OUTER_LEN); + memmove((void *)(OUT_MOVE + MOVE_SHIFT), (const void *)OUT_MOVE, OUTER_LEN); + MB_ARM = 0u; + if (!holds(OUT_DST1, OUT_SRC1, OUTER_LEN, SEED_OUT)) { + bad++; + } + if (!holds(OUT_DST2, OUT_SRC2, OUTER_LEN, SEED_OUT)) { + bad++; + } + if (!holds(OUT_MOVE + MOVE_SHIFT, OUT_MOVE, OUTER_LEN, SEED_OUT)) { + bad++; + } + MB_BAD = bad; + } + MB_SHOT_OK = (u16)(MB_SHOTS >= MIN_SHOTS && MB_HANDLED == MB_SHOTS); + MB_DONE = MB_DONE_MARK; + for (;;) { + } + return 0; +} diff --git a/tests/memBlock/memBlockIrq.lua b/tests/memBlock/memBlockIrq.lua new file mode 100644 index 0000000..c8c2bda --- /dev/null +++ b/tests/memBlock/memBlockIrq.lua @@ -0,0 +1,31 @@ +-- Inject a native-mode interrupt into memBlockIrq while an armed outer +-- move is executing its block-move stub on the stack (PB 0, PC below the +-- program at $1000), never into a handler and at most every third frame, +-- so the outer moves make progress between handlers. +local cpu = manager.machine.devices[":maincpu"] +local mem = cpu.spaces["program"] +local irqFrame = 0 +emu.register_frame_done(function() + irqFrame = irqFrame + 1 + if irqFrame % 3 ~= 0 then + return + end + if mem:read_u16(0x025022) ~= 1 or mem:read_u16(0x025028) ~= 0 then + return + end + local pc = cpu.state["PC"].value + if cpu.state["PB"].value ~= 0 or pc < 0x0100 or pc >= 0x1000 then + return + end + local s = cpu.state["S"].value + local p = cpu.state["P"].value + mem:write_u8(s, cpu.state["PB"].value) + mem:write_u8(s - 1, (pc >> 8) & 0xFF) + mem:write_u8(s - 2, pc & 0xFF) + mem:write_u8(s - 3, p) + cpu.state["S"].value = s - 4 + cpu.state["P"].value = (p | 0x04) & 0xF7 + cpu.state["PC"].value = mem:read_u16(0x025020) + mem:write_u16(0x025028, 1) + mem:write_u16(0x025024, mem:read_u16(0x025024) + 1) +end) diff --git a/tests/memBlock/memBlockIrqEntry.s b/tests/memBlock/memBlockIrqEntry.s new file mode 100644 index 0000000..8f99c56 --- /dev/null +++ b/tests/memBlock/memBlockIrqEntry.s @@ -0,0 +1,28 @@ +; irqEntry -- the interrupt memBlockIrq.lua injects: a native-mode handler +; that saves every register, gives the C code its own direct page (any +; interrupt code that runs C must: C keeps its scratch in the direct +; page) and DBR 0, runs irqNested and returns with RTI. + +IRQ_D = 0x0200 + + .text + .globl irqEntry +irqEntry: + rep #0x30 + pha + phx + phy + phb + phd + pea 0 + plb + plb + lda #IRQ_D + tcd + jsl irqNested + pld + plb + ply + plx + pla + rti diff --git a/tests/memBlock/memBlockRegs.s b/tests/memBlock/memBlockRegs.s new file mode 100644 index 0000000..df3aa37 --- /dev/null +++ b/tests/memBlock/memBlockRegs.s @@ -0,0 +1,124 @@ +; memBlockRegs -- memcpy / memmove / memset keep the caller's DBR and D. +; +; unsigned short memBlockRegs(void) calls each routine down each of its +; paths (short loop, one MVN / MVP, several runs; memset short, one pass +; and over 64 KB) with DBR = $05 and D = $0100, so the routines' direct- +; page scratch lands at $01E0, and sets one bit at $02:5008 for each call +; that came back with DBR, D and dst in A:X intact. It returns that word: +; 0x0FFF, all 12 bits, is a pass. + +REGS_DBR = 0x05 +REGS_D = 0x0100 +REGS_OUT = 0x025008 + + .text + .globl memBlockRegs + +; The caller pushed the arguments after dst (nWords words); call fn with +; dst in A:X, drop the arguments and check what came back. + .macro regsCall fn, dstLo, dstHi, nWords, bit + lda #\dstLo + ldx #\dstHi + jsl \fn + tay + tsc + clc + adc #(\nWords * 2) + tcs + cpy #\dstLo + bne .Lskip\bit + cpx #\dstHi + bne .Lskip\bit + tdc + cmp #REGS_D + bne .Lskip\bit + phb + sep #0x20 + .a8 + pla + cmp #REGS_DBR + rep #0x20 + .a16 + bne .Lskip\bit + lda #(1 << \bit) + ora REGS_OUT + sta REGS_OUT +.Lskip\bit: + .endm + +memBlockRegs: + rep #0x30 + phb + phd + lda #0 + sta REGS_OUT + pea REGS_DBR * 0x101 ; DBR twice: plb, plb + plb + plb + lda #REGS_D + tcd + ; memcpy: short, one MVN, runs (source across 03/04) + pea 0 + pea 20 + pea 0x0002 + pea 0x6200 + regsCall memcpy, 0x6100, 0x0002, 4, 0 + pea 0 + pea 300 + pea 0x0002 + pea 0x6200 + regsCall memcpy, 0x6800, 0x0002, 4, 1 + pea 0 + pea 600 + pea 0x0003 + pea 0xff00 + regsCall memcpy, 0x7000, 0x0002, 4, 2 + ; memmove backward: short, one MVP, runs (both across 03/04) + pea 0 + pea 20 + pea 0x0002 + pea 0x6100 + regsCall memmove, 0x6105, 0x0002, 4, 3 + pea 0 + pea 300 + pea 0x0002 + pea 0x6100 + regsCall memmove, 0x6105, 0x0002, 4, 4 + pea 0 + pea 600 + pea 0x0003 + pea 0xff00 + regsCall memmove, 0xff40, 0x0003, 4, 5 + ; memmove forward: short, one MVN, runs + pea 0 + pea 20 + pea 0x0002 + pea 0x6105 + regsCall memmove, 0x6100, 0x0002, 4, 6 + pea 0 + pea 300 + pea 0x0002 + pea 0x6105 + regsCall memmove, 0x6100, 0x0002, 4, 7 + pea 0 + pea 600 + pea 0x0003 + pea 0xff40 + regsCall memmove, 0xff00, 0x0003, 4, 8 + ; memset: short, one pass, over 64 KB + pea 0 + pea 5 + pea 0x77 + regsCall memset, 0x6100, 0x0002, 3, 9 + pea 0 + pea 300 + pea 0x77 + regsCall memset, 0x6100, 0x0002, 3, 10 + pea 1 + pea 0x10 + pea 0x77 + regsCall memset, 0x0000, 0x0004, 3, 11 + pld + plb + lda REGS_OUT + rtl diff --git a/tests/memBlock/memBlockTest.c b/tests/memBlock/memBlockTest.c new file mode 100644 index 0000000..1fe6149 --- /dev/null +++ b/tests/memBlock/memBlockTest.c @@ -0,0 +1,196 @@ +// memcpy / memmove / memset against a byte-exact model: every length +// class (0, 1, short, both sides of the MVN threshold, long, over 64 KB), +// odd and even source and destination addresses, buffers that straddle a +// bank boundary (source, destination, both), overlapping memmove in both +// directions (short and long, across a bank edge), memset of each class +// and of a value above 255, the returned pointer, and guard bytes on both +// sides; first (memBlockRegs.s) the caller's DBR and D across every path. +// Mailbox: 0x025000 = mismatches, 0x025002 = first failing case (0xFFFF: +// none), 0x025004 = 0xC0DE done, 0x025006 = cases run, 0x025008 = 0x0FFF +// when DBR and D survived all 12 register probes; the first failure's +// dst / src (or c) / n | op << 28 at 0x025010 / 0x025014 / 0x025018. +#include "memPattern.h" + +#define MB_BAD (*(volatile u16 *)0x025000UL) +#define MB_FIRST (*(volatile u16 *)0x025002UL) +#define MB_DONE (*(volatile u16 *)0x025004UL) +#define MB_CASES (*(volatile u16 *)0x025006UL) +#define MB_DST (*(volatile u32 *)0x025010UL) +#define MB_SRC (*(volatile u32 *)0x025014UL) +#define MB_N (*(volatile u32 *)0x025018UL) + +#define MB_DONE_MARK 0xC0DEu +#define MB_NO_FAIL 0xFFFFu +#define GUARD 8u +#define SEED_SRC 0xC3u +#define SEED_DST 0x5Au +#define SET_BYTE 0xA5 +#define SET_BIG 0x3C +#define SET_HIGH 0x100 +#define BIG_LEN 70000UL +#define N_LENS 27u +#define N_SHIFTS 10u +#define N_ALIGNS 2u +#define OP_COPY 0x10000000UL +#define OP_MOVE 0x20000000UL +#define OP_SET 0x30000000UL + +// Plain buffers in bank 2; bank edges at 03/04, 04/05, 05/06, 06/07 and +// 07/08 for the straddling ones. +#define PLAIN_DST 0x026000UL +#define PLAIN_SRC 0x027000UL +#define PLAIN_MOVE 0x026200UL +#define EDGE_SRC 0x040000UL +#define EDGE_DST 0x050000UL +#define EDGE_MOVE 0x060000UL +#define EDGE_SET 0x070000UL +#define BIG_SRC 0x058000UL +#define BIG_DST 0x074000UL +#define BIG_UP 0x058123UL +#define BIG_DOWN 0x057F00UL +#define BIG_SET 0x09FFFFUL + +static void begin(u32 dst, u32 src, u32 n); +static void caseCopy(u32 dst, u32 src, u32 n); +static void caseMove(u32 dst, u32 src, u32 n); +static void caseSet(u32 dst, int c, u32 n); +static void fail(void); +static int guarded(u32 a, u32 n, u8 seed); +static int holdsByte(u32 a, u32 n, u8 c); +u16 memBlockRegs(void); + +static const u16 kLens[N_LENS] = { + 0, 1, 2, 3, 4, 5, 7, 8, 15, 16, 17, 22, 23, 24, 25, 26, 31, 32, 33, 34, + 35, 64, 100, 255, 256, 257, 1000 +}; +static const short kShifts[N_SHIFTS] = { -100, -7, -3, -2, -1, 1, 2, 3, 7, 100 }; + +static u16 gBad; +static u16 gCase; +static u32 gDst; +static u32 gSrc; +static u32 gN; + + +static void begin(u32 dst, u32 src, u32 n) { + gDst = dst; + gSrc = src; + gN = n; + gCase++; + MB_CASES = gCase; +} + + +static void caseCopy(u32 dst, u32 src, u32 n) { + begin(dst, src, n | OP_COPY); + fill(dst - GUARD, n + 2u * GUARD, SEED_DST); + fill(src, n, SEED_SRC); + if (memcpy((void *)dst, (const void *)src, n) != (void *)dst || + !holds(dst, src, n, SEED_SRC) || !guarded(dst, n, SEED_DST)) { + fail(); + } +} + + +// One region holds both blocks: after the move [dst, dst + n) holds what +// [src, src + n) held and the rest of the window is untouched. +static void caseMove(u32 dst, u32 src, u32 n) { + u32 lo = (dst < src) ? dst : src; + u32 hi = ((dst < src) ? src : dst) + n; + begin(dst, src, n | OP_MOVE); + fill(lo - GUARD, hi - lo + 2u * GUARD, SEED_SRC); + if (memmove((void *)dst, (const void *)src, n) != (void *)dst || + !holds(dst, src, n, SEED_SRC) || + !holds(lo - GUARD, lo - GUARD, dst - (lo - GUARD), SEED_SRC) || + !holds(dst + n, dst + n, hi + GUARD - (dst + n), SEED_SRC)) { + fail(); + } +} + + +static void caseSet(u32 dst, int c, u32 n) { + begin(dst, (u32)c, n | OP_SET); + fill(dst - GUARD, n + 2u * GUARD, SEED_DST); + if (memset((void *)dst, c, n) != (void *)dst || + !holdsByte(dst, n, (u8)c) || !guarded(dst, n, SEED_DST)) { + fail(); + } +} + + +static void fail(void) { + if (gBad == 0u) { + MB_FIRST = gCase; + MB_DST = gDst; + MB_SRC = gSrc; + MB_N = gN; + } + gBad++; + MB_BAD = gBad; +} + + +// The GUARD bytes on each side of [a, a + n) still hold their own pattern. +static int guarded(u32 a, u32 n, u8 seed) { + return holds(a - GUARD, a - GUARD, GUARD, seed) && holds(a + n, a + n, GUARD, seed); +} + + +static int holdsByte(u32 a, u32 n, u8 c) { + const u8 *p = (const u8 *)a; + while (n != 0u) { + u16 run = (n > 0x8000UL) ? 0x8000u : (u16)n; + u16 i; + for (i = 0; i < run; i++) { + if (p[i] != c) { + return 0; + } + } + p += run; + n -= run; + } + return 1; +} + + +int main(void) { + u16 li; + u16 d; + u16 s; + u16 k; + // DBR and D survive every path (memBlockRegs.s writes 0x025008 itself). + (void)memBlockRegs(); + for (li = 0; li < N_LENS; li++) { + u32 n = kLens[li]; + u32 half = n / 2u; + for (d = 0; d < N_ALIGNS; d++) { + for (s = 0; s < N_ALIGNS; s++) { + // Plain, source across 03/04, destination across 04/05, + // both across (at different points of the run). + caseCopy(PLAIN_DST + d, PLAIN_SRC + s, n); + caseCopy(PLAIN_DST + d, EDGE_SRC - half + s, n); + caseCopy(EDGE_DST - half + d, PLAIN_SRC + s, n); + caseCopy(EDGE_DST - half + d, EDGE_SRC - n / 3u + s, n); + } + } + for (k = 0; k < N_SHIFTS; k++) { + caseMove(PLAIN_MOVE + (long)kShifts[k], PLAIN_MOVE, n); + caseMove(EDGE_MOVE - half + (long)kShifts[k], EDGE_MOVE - half, n); + } + caseSet(PLAIN_DST + (li & 1u), SET_BYTE, n); + caseSet(EDGE_SET - half, SET_HIGH + li, n); + } + // Over 64 KB: several MVN / MVP runs, every bank byte stepping. + caseCopy(BIG_DST, BIG_SRC, BIG_LEN); + caseMove(BIG_UP, BIG_SRC, BIG_LEN); + caseMove(BIG_DOWN, BIG_SRC, BIG_LEN); + caseSet(BIG_SET, SET_BIG, BIG_LEN); + MB_BAD = gBad; + if (gBad == 0u) { + MB_FIRST = MB_NO_FAIL; + } + MB_DONE = MB_DONE_MARK; + for (;;) { + } + return 0; +} diff --git a/tests/memBlock/memPattern.h b/tests/memBlock/memPattern.h new file mode 100644 index 0000000..e63aaf0 --- /dev/null +++ b/tests/memBlock/memPattern.h @@ -0,0 +1,75 @@ +// memPattern.h - the byte pattern the memBlock tests fill and check. +// +// The byte at address a is a's low byte xor a mix of its page and bank, so +// a copy off by one byte, one page or one bank reads wrong. Walked a page +// at a time so the per-byte work stays 16-bit. +#ifndef MEM_PATTERN_H +#define MEM_PATTERN_H + +typedef unsigned char u8; +typedef unsigned short u16; +typedef unsigned long u32; +typedef unsigned long size_t; + +#define PAGE_BYTES 256u + +static void fill(u32 a, u32 n, u8 seed); +static int holds(u32 a, u32 from, u32 n, u8 seed); +void *memcpy(void *dst, const void *src, size_t n); +void *memmove(void *dst, const void *src, size_t n); +void *memset(void *dst, int c, size_t n); +static u8 mixOf(u32 a, u8 seed); + + +// [a, a + n) = the pattern of its own addresses. +static void fill(u32 a, u32 n, u8 seed) { + u8 *p = (u8 *)a; + while (n != 0u) { + u8 lo = (u8)a; + u8 m = mixOf(a, seed); + u16 run = (u16)(PAGE_BYTES - lo); + u16 i; + if ((u32)run > n) { + run = (u16)n; + } + for (i = 0; i < run; i++) { + p[i] = (u8)(lo ^ m); + lo++; + } + p += run; + a += run; + n -= run; + } +} + + +// 1 when [a, a + n) holds the pattern of [from, from + n). +static int holds(u32 a, u32 from, u32 n, u8 seed) { + const u8 *p = (const u8 *)a; + while (n != 0u) { + u8 lo = (u8)from; + u8 m = mixOf(from, seed); + u16 run = (u16)(PAGE_BYTES - lo); + u16 i; + if ((u32)run > n) { + run = (u16)n; + } + for (i = 0; i < run; i++) { + if (p[i] != (u8)(lo ^ m)) { + return 0; + } + lo++; + } + p += run; + from += run; + n -= run; + } + return 1; +} + + +static u8 mixOf(u32 a, u8 seed) { + return (u8)((u8)((u8)(a >> 8) * 7u) ^ (u8)((u8)(a >> 16) * 13u) ^ seed); +} + +#endif diff --git a/tests/regress/genI64Cmp.py b/tests/regress/genI64Cmp.py new file mode 100644 index 0000000..58cc70a --- /dev/null +++ b/tests/regress/genI64Cmp.py @@ -0,0 +1,148 @@ +#!/usr/bin/env python3 +# Emits the C source of the smoke check "i64 compares", with kExp holding +# the expected results computed here (C semantics). +# +# Every i64 predicate (eq ne ult ule ugt uge slt sle sgt sge) on values +# that sit on each 16-bit word boundary and both sides of the sign bit: as +# a branch, as a 0/1 value and as an i16 select against registers, and as +# a branch and a value against ten constants (some with zero low words, +# which lowerI32Cmp drops from its CMP/SBC chain), then loops whose i64 +# counters cross a 32-bit boundary. Pins the four-word CMP/SBC chain +# (SETCC / BR_CC / SELECT_CC on i64 are lowered by lowerI32Cmp). +# +# Mailbox: 0x025000 = mismatches, 0x025002 = first failing case, +# 0x025004 = 0xC0DE when done. +V = [ + 0x0000000000000000, 0x0000000000000001, 0x000000000000FFFF, + 0x0000000000010000, 0x00000000FFFFFFFF, 0x0000000100000000, + 0x0000000100000001, 0x0000FFFFFFFFFFFF, 0x0001000000000000, + 0x0000800000000000, 0x7FF0000000000000, 0x7FF0000000000001, + 0x0001FFFF00000000, 0x7FFFFFFFFFFFFFFF, 0x8000000000000000, + 0x8000000000000001, 0xFFFF000000000000, 0xFFFFFFFF00000000, + 0xFFFFFFFFFFFF0000, 0xFFFFFFFFFFFFFFFF, +] +K = [ + 0x0000000000000000, 0x0000000000000001, 0x00000000FFFFFFFF, + 0x0000000100000000, 0x7FF0000000000000, 0x7FF0000000000001, + 0x7FFFFFFFFFFFFFFF, 0x8000000000000000, 0xFFFF000000000000, + 0xFFFFFFFFFFFFFFFF, +] +PREDS = [ + ("eq", "u64", "=="), ("ne", "u64", "!="), ("ult", "u64", "<"), + ("ule", "u64", "<="), ("ugt", "u64", ">"), ("uge", "u64", ">="), + ("slt", "s64", "<"), ("sle", "s64", "<="), ("sgt", "s64", ">"), + ("sge", "s64", ">="), +] + + +def s64(x): + return x - (1 << 64) if x & (1 << 63) else x + + +def ev(t, op, a, b): + if t == "s64": + a, b = s64(a), s64(b) + return {"==": a == b, "!=": a != b, "<": a < b, "<=": a <= b, + ">": a > b, ">=": a >= b}[op] + + +bits = [] +for _, t, op in PREDS: + for a in V: + for b in V: + bits.append(ev(t, op, a, b)) +for _, t, op in PREDS: + for k in K: + for a in V: + bits.append(ev(t, op, a, k)) +exp = bytearray((len(bits) + 7) // 8) +for i, b in enumerate(bits): + if b: + exp[i >> 3] |= 1 << (i & 7) + +o = [] +w = o.append +w("// i64 compares, generated by tests/regress/genI64Cmp.py; see there.") +w("typedef unsigned char u8;") +w("typedef unsigned short u16;") +w("typedef unsigned long long u64;") +w("typedef long long s64;") +w("#define NV %du" % len(V)) +w("#define NK %du" % len(K)) +w("#define NP %du" % len(PREDS)) +w('#define BAR() __asm__ volatile("" ::: "memory")') +w("#define NOI __attribute__((noinline)) static") +w("static volatile u16 gSink;") +w("static const u64 kV[NV] = {") +for i in range(0, len(V), 3): + w(" " + " ".join("0x%016XULL," % v for v in V[i:i + 3])) +w("};") +w("static const u8 kExp[%d] = {" % len(exp)) +for i in range(0, len(exp), 16): + w(" " + " ".join("0x%02X," % b for b in exp[i:i + 16])) +w("};") +for name, t, op in PREDS: + w("NOI u16 br_%s(u64 a, u64 b) { if ((%s)a %s (%s)b) { BAR(); return 1u; } return 0u; }" % (name, t, op, t)) + w("NOI u16 va_%s(u64 a, u64 b) { return (u16)((%s)a %s (%s)b); }" % (name, t, op, t)) + w("NOI u16 se_%s(u64 a, u64 b) { return (%s)a %s (%s)b ? 0x5A5Au : 0xA5A5u; }" % (name, t, op, t)) + for i, k in enumerate(K): + w("NOI u16 bk_%s_%d(u64 a) { if ((%s)a %s (%s)0x%016XULL) { BAR(); return 1u; } return 0u; }" % (name, i, t, op, t, k)) + w("NOI u16 vk_%s_%d(u64 a) { return (u16)((%s)a %s (%s)0x%016XULL); }" % (name, i, t, op, t, k)) +w("typedef u16 (*Fn2)(u64, u64);") +w("typedef u16 (*Fn1)(u64);") +for p in ("br", "va", "se"): + w("static Fn2 const k%s[NP] = { %s };" % (p.capitalize(), ", ".join("%s_%s" % (p, n) for n, _, _ in PREDS))) +for p in ("bk", "vk"): + w("static Fn1 const k%s[NP * NK] = {" % p.capitalize()) + for n, _, _ in PREDS: + w(" " + " ".join("%s_%s_%d," % (p, n, i) for i in range(len(K)))) + w("};") +w("NOI u16 loopUlt(u64 s, u64 e) { u16 n = 0u; u64 i; for (i = s; i < e; i++) { gSink = (u16)i; n++; } return n; }") +w("NOI u16 loopSge(s64 s, s64 e) { u16 n = 0u; s64 i; for (i = e; i >= s; i--) { gSink = (u16)i; n++; } return n; }") +w("NOI u16 loopUgtDown(u64 s, u64 e) { u16 n = 0u; u64 i; for (i = e; i > s; i--) { gSink = (u16)i; n++; } return n; }") +w("static u16 gBad;") +w("static u16 gFirst = 0xFFFFu;") +w("static void check(u16 got, u16 want, u16 id) {") +w(" if (got != want) {") +w(" if (gBad == 0u) {") +w(" gFirst = id;") +w(" }") +w(" gBad++;") +w(" }") +w("}") +w("int main(void) {") +w(" u16 p;") +w(" u16 i;") +w(" u16 j;") +w(" u16 bit = 0u;") +w(" for (p = 0u; p < NP; p++) {") +w(" for (i = 0u; i < NV; i++) {") +w(" for (j = 0u; j < NV; j++) {") +w(" u16 want = (u16)((kExp[bit >> 3] >> (bit & 7u)) & 1u);") +w(" check(kBr[p](kV[i], kV[j]), want, bit);") +w(" check(kVa[p](kV[i], kV[j]), want, (u16)(bit | 0x4000u));") +w(" check(kSe[p](kV[i], kV[j]) == 0x5A5Au, want, (u16)(bit | 0x8000u));") +w(" bit++;") +w(" }") +w(" }") +w(" }") +w(" for (p = 0u; p < NP * NK; p++) {") +w(" for (i = 0u; i < NV; i++) {") +w(" u16 want = (u16)((kExp[bit >> 3] >> (bit & 7u)) & 1u);") +w(" check(kBk[p](kV[i]), want, bit);") +w(" check(kVk[p](kV[i]), want, (u16)(bit | 0x4000u));") +w(" bit++;") +w(" }") +w(" }") +w(" check(loopUlt(0x00000000FFFFFFF0ULL, 0x0000000100000010ULL), 32u, 0xC000u);") +w(" check(loopUlt(0x7FFFFFFFFFFFFFF0ULL, 0x8000000000000010ULL), 32u, 0xC001u);") +w(" check(loopSge(-5LL, 5LL), 11u, 0xC002u);") +w(" check(loopUgtDown(0x0000FFFFFFFFFFF8ULL, 0x0001000000000008ULL), 16u, 0xC003u);") +w(" *(volatile u16 *)0x025000UL = gBad;") +w(" *(volatile u16 *)0x025002UL = gFirst;") +w(" *(volatile u16 *)0x025004UL = 0xC0DEu;") +w(" for (;;) {") +w(" }") +w(" return 0;") +w("}") +print("\n".join(o)) diff --git a/tests/regress/genMulDiv.py b/tests/regress/genMulDiv.py new file mode 100755 index 0000000..6625b36 --- /dev/null +++ b/tests/regress/genMulDiv.py @@ -0,0 +1,240 @@ +#!/usr/bin/env python3 +# Emits the C source of the smoke check "mul/div libcalls", with kExp +# holding the expected checksums computed here (C semantics). +# +# Every multiply and divide helper in runtime/src/libgcc.s on N operand +# pairs per operation: an xorshift32 stream (shifts and XORs only, so the +# generator needs no helper it tests), each operand masked to a size class +# (full 32 bits, 16, 9, 8 bits, 4 bits, sign-extended 16 / 8 bits) so every +# fast path and its boundaries are taken, plus the word-boundary values in +# kEdge. Results fold into one checksum per operation. +# +# Mailbox: 0x025000 = bitmask of operations whose checksum differs, +# 0x025002 = 0xC0DE when done. +import sys + +N = int(sys.argv[1]) if len(sys.argv) > 1 else 300 +EDGE = [0x00000000, 0x00000001, 0x00000002, 0x00000003, 0x0000007F, + 0x00000080, 0x000000FF, 0x00000100, 0x00000101, 0x000001FF, + 0x00007FFF, 0x00008000, 0x00008001, 0x0000FFFF, 0x00010000, + 0x00010001, 0x0001FFFF, 0x00FFFFFF, 0x7FFFFFFF, 0x80000000, + 0x80000001, 0xFFFF0000, 0xFFFF7FFF, 0xFFFF8000, 0xFFFFFF80, + 0xFFFFFFFE, 0xFFFFFFFF, 0x12345678, 0xFEDCBA98, 0x0000ABCD] +M32 = 0xFFFFFFFF + + +def xs(x): + x ^= (x << 13) & M32 + x ^= x >> 17 + x ^= (x << 5) & M32 + return x + + +def cls(v, c): + # Size class c (0..7) of the raw 32-bit value v. + if c == 0: + return v + if c == 1: + return v & 0xFFFF + if c == 2: + return v & 0x1FF + if c == 3: + return v & 0xFF + if c == 4: + return v & 0xF + if c == 5: + return (v & 0xFFFF) | (0xFFFF0000 if v & 0x8000 else 0) + if c == 6: + return (v & 0xFF) | (0xFFFFFF00 if v & 0x80 else 0) + return v & 0x7FFFFFFF + + +def s32(x): + x &= M32 + return x - (1 << 32) if x & 0x80000000 else x + + +def s16(x): + x &= 0xFFFF + return x - 0x10000 if x & 0x8000 else x + + +def tdiv(a, b): + q = abs(a) // abs(b) + return -q if (a < 0) != (b < 0) else q + + +def tmod(a, b): + return a - tdiv(a, b) * b + + +OPS = ['mulU32', 'mulU16x', 'mulS16x', 'mulU16', 'divU32', 'modU32', + 'divS32', 'modS32', 'divU16', 'modU16', 'divS16', 'modS16'] + + +def ev(op, a, b): + if op == 'mulU32': + return (a * b) & M32 + if op == 'mulU16x': + return ((a & 0xFFFF) * (b & 0xFFFF)) & M32 + if op == 'mulS16x': + return (s16(a) * s16(b)) & M32 + if op == 'mulU16': + return (a * b) & 0xFFFF + if op == 'divU32': + return a // b + if op == 'modU32': + return a % b + if op == 'divS32': + return tdiv(s32(a), s32(b)) & M32 + if op == 'modS32': + return tmod(s32(a), s32(b)) & M32 + if op == 'divU16': + return (a & 0xFFFF) // (b & 0xFFFF) + if op == 'modU16': + return (a & 0xFFFF) % (b & 0xFFFF) + if op == 'divS16': + return tdiv(s16(a), s16(b)) & 0xFFFF + return tmod(s16(a), s16(b)) & 0xFFFF + + +def fix(op, a, b): + # The divisor must not be zero, and INT_MIN / -1 is undefined. + if op.startswith('div') or op.startswith('mod'): + w16 = op.endswith('16') + if (b & 0xFFFF if w16 else b) == 0: + b |= 1 + if op.endswith('S32') and a == 0x80000000 and b == M32: + b = 1 + if op.endswith('S16') and (a & 0xFFFF) == 0x8000 and (b & 0xFFFF) == 0xFFFF: + b = 1 + return a, b + + +def rot(h): + return ((h << 5) | (h >> 27)) & M32 + + +exp = [] +for op in OPS: + h = 0 + x = 0x2545F491 + pairs = [(a, b) for a in EDGE for b in EDGE] + for i in range(N): + x = xs(x) + a = cls(x, (x >> 8) & 7) + x = xs(x) + b = cls(x, (x >> 8) & 7) + pairs.append((a, b)) + for a, b in pairs: + a, b = fix(op, a, b) + h = rot(h) ^ ev(op, a, b) + exp.append(h) + +print('''// Generated by tests/regress/genMulDiv.py: every multiply / divide helper +// on the word-boundary values and an xorshift32 stream of size-classed +// operands; one checksum per operation against kExp (host C semantics). +// Mailbox: 0x025000 = bitmask of mismatching operations, 0x025002 = +// 0xC0DE when done. int is 16 bits and long 32 here. +typedef unsigned short u16; +typedef short s16; +typedef unsigned long u32; +typedef long s32; +#define NOI __attribute__((noinline)) static +#define NOPS %d +#define NRAND %d +#define NEDGE %d +static const u32 kEdge[NEDGE] = { %s }; +static const u32 kExp[NOPS] = { %s }; +NOI u32 mulU32(u32 a, u32 b) { return a * b; } +NOI u32 mulU16x(u16 a, u16 b) { return (u32)a * (u32)b; } +NOI s32 mulS16x(s16 a, s16 b) { return (s32)a * (s32)b; } +NOI u16 mulU16(u16 a, u16 b) { return (u16)(a * b); } +NOI u32 divU32(u32 a, u32 b) { return a / b; } +NOI u32 modU32(u32 a, u32 b) { return a %% b; } +NOI s32 divS32(s32 a, s32 b) { return a / b; } +NOI s32 modS32(s32 a, s32 b) { return a %% b; } +NOI u16 divU16(u16 a, u16 b) { return a / b; } +NOI u16 modU16(u16 a, u16 b) { return a %% b; } +NOI s16 divS16(s16 a, s16 b) { return a / b; } +NOI s16 modS16(s16 a, s16 b) { return a %% b; } +static u32 xs(u32 x) { + x ^= x << 13; + x ^= x >> 17; + x ^= x << 5; + return x; +} +static u32 cls(u32 v, u16 c) { + switch (c) { + case 0: return v; + case 1: return v & 0xFFFFUL; + case 2: return v & 0x1FFUL; + case 3: return v & 0xFFUL; + case 4: return v & 0xFUL; + case 5: return (v & 0xFFFFUL) | ((v & 0x8000UL) ? 0xFFFF0000UL : 0UL); + case 6: return (v & 0xFFUL) | ((v & 0x80UL) ? 0xFFFFFF00UL : 0UL); + default: return v & 0x7FFFFFFFUL; + } +} +static u32 ev(u16 op, u32 a, u32 b) { + if (op >= 4) { + u16 w16 = (op >= 8); + if ((w16 ? (b & 0xFFFFUL) : b) == 0UL) { + b |= 1UL; + } + if ((op == 6 || op == 7) && a == 0x80000000UL && b == 0xFFFFFFFFUL) { + b = 1UL; + } + if ((op == 10 || op == 11) && (a & 0xFFFFUL) == 0x8000UL && (b & 0xFFFFUL) == 0xFFFFUL) { + b = 1UL; + } + } + switch (op) { + case 0: return mulU32(a, b); + case 1: return mulU16x((u16)a, (u16)b); + case 2: return (u32)mulS16x((s16)a, (s16)b); + case 3: return mulU16((u16)a, (u16)b); + case 4: return divU32(a, b); + case 5: return modU32(a, b); + case 6: return (u32)divS32((s32)a, (s32)b); + case 7: return (u32)modS32((s32)a, (s32)b); + case 8: return divU16((u16)a, (u16)b); + case 9: return modU16((u16)a, (u16)b); + case 10: return (u16)divS16((s16)a, (s16)b); + default: return (u16)modS16((s16)a, (s16)b); + } +} +int main(void) { + u16 bad = 0; + u16 op; + *(volatile u16 *)0x025002UL = 0u; + for (op = 0; op < NOPS; op++) { + u32 h = 0UL; + u32 x = 0x2545F491UL; + u16 i; + u16 j; + for (i = 0; i < NEDGE; i++) { + for (j = 0; j < NEDGE; j++) { + h = ((h << 5) | (h >> 27)) ^ ev(op, kEdge[i], kEdge[j]); + } + } + for (i = 0; i < NRAND; i++) { + u32 a; + u32 b; + x = xs(x); + a = cls(x, (u16)((x >> 8) & 7UL)); + x = xs(x); + b = cls(x, (u16)((x >> 8) & 7UL)); + h = ((h << 5) | (h >> 27)) ^ ev(op, a, b); + } + if (h != kExp[op]) { + bad |= (u16)(1u << op); + } + } + *(volatile u16 *)0x025000UL = bad; + *(volatile u16 *)0x025002UL = 0xC0DEu; + for (;;) { + } + return 0; +}''' % (len(OPS), N, len(EDGE), ', '.join('0x%08XUL' % v for v in EDGE), + ', '.join('0x%08XUL' % v for v in exp))) diff --git a/tests/regress/setjmpImgSave.c b/tests/regress/setjmpImgSave.c new file mode 100644 index 0000000..d758364 --- /dev/null +++ b/tests/regress/setjmpImgSave.c @@ -0,0 +1,88 @@ +// Regression test: a longjmp must not hand a caller back IMG8..IMG15 +// ($C0..$CE) words that a function it unwound overwrote. +// +// IMG8..IMG15 survive calls because every function that writes one gives +// its caller the old value back on return (W65816ImgCalleeSave, and the +// entry/return swaps of W65816StackRelToImg's IMG8..15 pointer homes). A +// longjmp skips those returns. walk() keeps q and r in $C2..$C8 and +// longjmps out of ext(); outer() keeps ITS q and r in the same words +// across the call to f0(). Before the fix outer() came back from f0() +// walking walk()'s r (bufF) instead of its own (bufE). +// +// Fix: a call that can return twice (setjmp) clobbers IMG8..IMG15 +// (W65816TargetLowering::AdjustInstrPostInstrSelection), so f0() keeps +// nothing in them across setjmp and saves and restores all eight for +// outer(). +// +// outer(bufB, bufC, bufE, 2) = 2 + bufC[1] (5) + bufE[2] (9) = 16; the +// broken build gave 9, 5, 8. + +#include + +static jmp_buf gJb; +volatile int gSink; + +int bufA[4] = {5, 0, 0, 0}; +int bufB[4] = {0, 1, 2, 3}; +int bufC[4] = {0, 4, 5, 6}; +int bufD[4] = {9, 9, 9, 9}; +int bufE[4] = {0, 0, 8, 0}; +int bufF[4] = {0, 0, 0, 0}; + +__attribute__((noinline)) void ext(int v) { + gSink = v; + if (v == 7) { + longjmp(gJb, 1); + } +} + + +// q and r live across the calls to ext in IMG8..15 homes. +__attribute__((noinline)) void walk(int *p, int *q, int *r, int n) { + int i = 0; + + do { + ext(p[0]); + q[1] += i; + r[2] ^= q[1]; + p[2] = r[2]; + p[0]++; + } while (++i != n); +} + + +__attribute__((noinline)) int f0(int *p, int *q, int *r) { + if (setjmp(gJb)) { + return 1; + } + walk(p, q, r, 3); + return 0; +} + + +// Its own q and r in the same IMG8..15 words across the call to f0. +__attribute__((noinline)) int outer(int *p, int *q, int *r, int n) { + int i = 0; + int t = 0; + + do { + t += f0(bufA, bufD, bufF); + q[1] += i; + r[2] ^= q[1]; + p[3] = r[2]; + } while (++i != n); + return t + q[1] + r[2]; +} + + +int main(void) { + volatile int *out = (volatile int *)0x025000; + + out[1] = outer(bufB, bufC, bufE, 2); // 0x0010 + out[2] = bufC[1]; // 0x0005 + out[3] = bufE[2]; // 0x0009 + out[0] = 0xC0DE; + while (1) { + } + return 0; +}