diff --git a/docs/adr/2026-07-30-isa-parameterized-host-emu-engine.md b/docs/adr/2026-07-30-isa-parameterized-host-emu-engine.md index 64c853fc9..1a0687f7d 100644 --- a/docs/adr/2026-07-30-isa-parameterized-host-emu-engine.md +++ b/docs/adr/2026-07-30-isa-parameterized-host-emu-engine.md @@ -134,9 +134,15 @@ type parameter, not a trait object over emulators, and not a second engine. word-mirrored, so `build_xt_image` rejects it explicitly. A classic host target needs that layout reworked. **Revisit when:** an LX6 host execution target is wanted. -- Shader code shares the 112 KiB text region with ~84 KiB of builtins, leaving +- ~~Shader code shares the 112 KiB text region with ~84 KiB of builtins, leaving ~28 KiB. Overflow is an explicit error naming the budget. **Revisit when:** a - real shader hits it — the fix is the linker script's split, not the host region. + real shader hits it — the fix is the linker script's split, not the host + region.~~ **Closed 2026-08-01, and the stated fix was the wrong one.** Both + now live where they live on the device: the builtins image links as + flash-resident firmware (IROM/DROM) and the shader gets the *whole* SRAM code + region — 128 KiB, unchanged in size. Splitting the linker script differently + would have preserved the model error. See + `docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md`. - No measured Xtensa cycle model; `CycleModel::InstructionCount` remains the honest default (rv32 has a measured C6 table). **Revisit when:** the perf column needs Xtensa numbers to mean something. diff --git a/docs/adr/2026-07-30-xtensa-host-shared-memory.md b/docs/adr/2026-07-30-xtensa-host-shared-memory.md index 6e8e9051a..c40a037e2 100644 --- a/docs/adr/2026-07-30-xtensa-host-shared-memory.md +++ b/docs/adr/2026-07-30-xtensa-host-shared-memory.md @@ -1,11 +1,23 @@ # ADR: Host-shared guest memory for the Xtensa emulator -- **Status:** Accepted +- **Status:** Accepted (amended 2026-08-01 — the address changed, the decision did not) - **Date:** 2026-07-30 - **Deciders:** Photomancer - **Supersedes:** None - **Superseded by:** None +> **Amendment, 2026-08-01.** `SHARED_DBUS_BASE` is now **`0x3000_0000`**, not +> `0x3F40_0000`. This ADR's own reasoning predicted it: "the assertion, not the +> address, is the decision that matters. A comment claiming an address is free +> rots the moment a profile changes." Both profiles gained modeled flash windows +> and `0x3F40_0000` turned out to *be* classic's DROM base (esp-hal +> `ld/esp32/memory.x`, `drom_seg`). The `add_shared` assertion failed at the +> moment of the mistake, exactly as designed. The replacement is chosen on +> stronger ground than "no profile maps it": `0x3000_0000` is below the lowest +> address either chip's data bus decodes, so no future profile can reach it. +> Everything else below stands as written; read `0x3F40_0000` as the historical +> value. + ## Context The host-side shader execution path (`lpvm-native`'s `rt_emu`) does not copy data diff --git a/docs/adr/2026-08-01-host-emulator-models-flash.md b/docs/adr/2026-08-01-host-emulator-models-flash.md new file mode 100644 index 000000000..afa3a6dc6 --- /dev/null +++ b/docs/adr/2026-08-01-host-emulator-models-flash.md @@ -0,0 +1,135 @@ +# ADR: The host Xtensa emulator models flash, and firmware links into it + +- **Status:** Accepted +- **Date:** 2026-08-01 +- **Deciders:** Photomancer +- **Supersedes:** None +- **Superseded by:** None + +## Context + +`lp-xt-emu` modeled one executable region per board — SRAM1, 128 KiB on the S3 +and 92 KiB on classic — and everything went in it: the linked `lps-builtins-xt-app` +image *and* the shader code `rt_emu` JIT-compiles. `lps-builtins-xt-app/link.ld` +split that region 112 KiB text / 16 KiB data, and `rt_emu::xt_image` placed +compiled shader functions in whatever IRAM the image's `.text` left over. + +On silicon neither of those things is true. Every ESP32 executes its application +`.text` **from flash through the cache** (XIP; the boot log's `vaddr=0x4200_0020 +map` lines are the cache MMU being programmed). Internal SRAM holds the stack, +`.data`/`.bss`, the heap, and — for this product — code a JIT produced at +runtime. Firmware and JIT output do not share an address region; they are not +even in the same address quadrant. + +Three consequences arrived together on 2026-08-01, when `--features float-f32` +was tried as the Xtensa builtins image's default +(`docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md`): + +1. The f32 family took `.text` from 66,300 B to 113,757 B, leaving **931 bytes** + of the region for shader code. The `xtn.q32` filetest suite fell from 849/849 + files to 522/849 on link failures. The image and the shader were competing + for bytes that on hardware they never share. +2. It produced a wrong product conclusion — read as "f32 doesn't fit on classic + ESP32" — by conflating the shader-JIT budget with the builtins' footprint. + Classic's 92 KiB bounds JIT'd shader code and says nothing about whether the + f32 builtins fit, because those are flash-resident there too. +3. It hid a live bug class. A shader→builtin call spans SRAM→flash, ~29 MB on + the S3, far outside `CALL8`'s ±512 KiB displacement — which is why the + emitter reaches every call target through a literal-pool slot and `CALLX8`. + In a single small region an accidentally-direct call would have been *in + range on the host* and out of range on silicon: passing tests, failing + hardware. + +The defect entry proposed three resolutions: widen the modeled region, give the +shader its own region, or split the f32 builtin feature so less of it links. +**All three preserve the wrong map**, and the first and third also spend real +budget to do so. + +## Decision + +Model flash. `BoardProfile` gains, per chip: + +| | ESP32-S3 | classic ESP32 | +| --- | --- | --- | +| IROM — flash instruction window, read-only, `AliasRule::Identity` | `0x4200_0000` | `0x400D_0000` | +| DROM — flash data window, read-only, not fetchable | `0x3C00_0000` | `0x3F40_0000` | +| image `.data`/`.bss` — plain internal SRAM | `0x3FCA_8000` | `0x3FFD_0000` | + +`lps-builtins-xt-app` links as flash-resident firmware into those windows. +`rt_emu::xt_image` places each `PT_LOAD` segment by classifying its `p_vaddr` +against the profile, and compiled shader code gets the **whole** SRAM code +region. Guest stores into either flash window fault; the loader paths write them +anyway, because putting an image into flash is what a flasher does. + +Three properties are deliberate: + +1. **The SRAM code region did not grow.** 128 KiB on the S3, 92 KiB on classic, + unchanged. The fix is not more room; it is that the builtins left. Widening + it would have been the same mistake with a bigger number, and it would have + made the modeled region stop corresponding to anything. +2. **Bases are hardware; lengths are the model's.** The S3's real `irom_seg` is + 32 MB and the emulator allocates its regions once per host call, so the + modeled windows are 256 KiB / 64 KiB / 32 KiB — sized to the image with + headroom. A segment past the end is a loud load error, never a silent wrap. +3. **Provenance is graded and stated.** `board.rs`'s SRAM numbers are + hardware-measured (FINDINGS E2, the classic C1–C5 ladder). The flash numbers + are *documented and observed*: esp-hal 1.1.1's MIT-or-Apache-2.0 + `ld/esp32{,s3}/memory.x`, corroborated by `lpc-shared`'s backtrace validator + and by S3 boot logs. They are labelled as the weaker grade rather than blended + in, because the difference matters if one turns out wrong. No GPL source + (binutils, GCC, QEMU) was consulted — see + `docs/adr/2026-07-29-license-provenance-discipline.md`. + +`--features float-f32` becomes the Xtensa builtins image's unconditional +default; `LP_XT_BUILTINS_F32` is deleted. + +## Consequences + +**Good.** `xtn.q32` is 849/849 files (6336/6336 tests) *with* f32, and +`xtlpn.q32` 849/849 (6385/6385). The largest compilable shader no longer depends +on the size of the builtins — the error message that used to name the builtins' +footprint now names the shader's own size. The classic-ESP32 f32 question is +correctly reframed as a flash-budget and unprobed-LX6-FPU question. And the +SRAM→flash call reach is now testable, which `lp-xt/lp-xt-emu/tests/call_range.rs` +does. + +That test also found something worth having: on **classic**, IROM sits only +~192 KiB above the SRAM1 I-bus window, comfortably inside `CALL8`'s range. A +direct call from JIT'd code to a builtin would work there. The emitter must stay +indirect because the S3 requires it, **not** because every Xtensa target does — +pinned so it is not "optimized away" after testing on one board. + +**Costs.** Every emulator now installs five regions instead of two (~352 KiB +more zeroed allocation per host call, `alloc_zeroed`-backed) and the per-call +load copies the image's ~130 KiB of flash instead of the region's 128 KiB — a +wash in bytes. `Memory` now asserts every region is mutually disjoint in both +address views, not just the shared window; that assertion immediately earned its +keep by catching `SHARED_DBUS_BASE = 0x3F40_0000`, which **is** classic's DROM +base. It moved to `0x3000_0000`, below the lowest address either chip's data bus +decodes (`docs/adr/2026-07-30-xtensa-host-shared-memory.md`, amended). + +**Unchanged.** `lp-xt/fixtures/link.ld` still links its guests into the SRAM code +region: those are raw payloads a runner writes into RAM, which is a different +thing from a flash-resident image. Every raw-payload suite — `fp_conformance`, +`conformance`, the metrics and trap tests — is untouched, and `lp-xt-elf`'s +`DATA_BASE = CODE_DBUS_BASE + CODE_REGION_LEN/2` convention is untouched with it. +rv32 is untouched: `GuestImage` gained an empty `regions` list there. + +## Alternatives rejected + +- **Widen the modeled SRAM code region.** Cheapest, and it makes the number stop + corresponding to hardware. The 128 KiB figure is the emulator's claim about + what a device has available for JIT output; inflating it to fit a resident + image would quietly retire that claim. +- **Give the shader a second SRAM region, builtins keeping the first.** Removes + the coupling without fixing the map. The call-range bug class would still be + invisible, because both regions would still be a short hop apart. +- **Split the `float-f32` feature** so only the arithmetic family M7 D4 routes + to is linked, leaving the generative-noise builtins (most of the 47 KB) + rv32-only. This spends a real feature contract to buy back space that the + device never needed spent, and it would have to be re-litigated the next time + the image grows. +- **Model flash at its true length (32 MB windows).** Correct and unaffordable: + regions are `Vec`-backed and allocated per host call. Truncating the length + while keeping the base is the honest compromise, and it is documented as such + on `MODELED_IROM_LEN`. diff --git a/docs/adr/README.md b/docs/adr/README.md index 328983143..6891c2993 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -84,7 +84,7 @@ holds the full context. | Compute-tick / shader-init fuel blame route (traps abort bounded but bypass the panic/blame ledger) | `2026-07-20-lpvm-native-fuel` | Runaway compute shaders show up in practice | | Budgeted/async shader compile (spread the ~194 ms device compile across frames instead of one long frame per apply) | `2026-07-14-shader-auto-apply` | The per-apply frame stall matters in practice | | Classic-ESP32 (LX6) host execution: the Xtensa guest image is one flat code-region buffer, correct only under an **offset** I-bus alias; classic's SRAM1 alias is word-mirrored and `build_xt_image` rejects it | `2026-07-30-isa-parameterized-host-emu-engine` | An LX6 host execution target is wanted | -| Xtensa shader-code budget: ~28 KiB free in the 112 KiB text region after ~84 KiB of builtins; overflow is an explicit error | `2026-07-30-isa-parameterized-host-emu-engine` | A real shader hits it — the fix is `lps-builtins-xt-app/link.ld`'s split, not the host region | +| ~~Xtensa shader-code budget: ~28 KiB free in the 112 KiB text region after ~84 KiB of builtins~~ — **closed 2026-08-01**: the builtins image is flash-resident, so the shader has the whole 128 KiB SRAM code region | `2026-07-30-isa-parameterized-host-emu-engine` | Closed; the budget is now the shader's own size | | Measured Xtensa cycle model (`CycleModel::InstructionCount` is the honest default; rv32 has a measured C6 table) | `2026-07-30-isa-parameterized-host-emu-engine`; `2026-07-28-emu-core-crate-family` | The filetest perf column needs Xtensa numbers to mean something | | Sim-worker recovery layer 2 (timeout-streak detection → terminate+respawn preserving the unsaved-overlay mirror; NotResponding sim roster card; PreviewHost in-flight deadline) | `2026-07-23-sim-wasm-fuel`; `2026-07-24-runtime-pool` | Next sim-runtime lifecycle work (the pool landed without it; requirements in the M4 plan notes) | | Live sim-card frames: core-owned present service for pool sessions sharing PreviewHost's CPU blit seam + the gallery routing rule (sim frames instead of a preview lease) | `2026-07-24-runtime-pool` | The roadmap's live-thumbnails item is prioritized | diff --git a/docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md b/docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md new file mode 100644 index 000000000..0b932247d --- /dev/null +++ b/docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md @@ -0,0 +1,109 @@ +--- +status: fixed +found: 2026-08-01 # how: filetests (xtn.q32 dropped 849/849 -> 522/849 on the gate flip) +fixed: this change # the host emulator now models flash; the image is firmware +area: lp-xt/lps-builtins-xt-app, lp-shader/lpvm-native (rt_emu/xt_image), lp-xt-emu +class: model-conflation # presented as `capacity`; that was the symptom, not the class +related: + - docs/defects/2026-08-01-xtensa-backend-cannot-select-float-constant-pool.md +--- +# With `float-f32` in, the Xtensa builtins image fills the emulator's code region and leaves no room for shader code + +**Symptom** — building the Xtensa builtins image with `--features float-f32` +succeeds, and then most of the Xtensa filetest suite fails to *link*: + +``` +xtn.q32 without float-f32: 6336/6336 tests, 849/849 files +xtn.q32 with float-f32: 2917/6336 tests, 522/849 files (327 files fail) +``` + +Failures report as `compile-fail`, not as wrong answers. Small shaders pass and +large ones fail, which is the tell. + +**Cause** — it is capacity, not codegen. `lp-xt/lps-builtins-xt-app/link.ld` +gives `.text` 112 KiB (114,688 B) of the emulator's 128 KiB code region, and +`.rodata`/`.data`/`.bss` the remaining 16 KiB (16,384 B). Measured with +`xtensa-esp32s3-elf-size -A`: + +| build | `.text` | free IRAM | `.rodata` | free DRAM | +| --- | --- | --- | --- | --- | +| default (Q32 only) | 66,300 B | 48,388 B | 12,176 B | 4,208 B | +| `--features float-f32` | 113,757 B | **931 B** | 16,156 B | **228 B** | + +`rt_emu::xt_image::build_xt_image` places compiled shader code 4-aligned after +the image's `.text` and requires it to end before the image's data segments — +i.e. in exactly that free-IRAM gap. At 931 bytes, only the smallest shaders +fit; everything else fails with + +``` +compiled shader code does not fit the Xtensa code region: N bytes of shader +after 113757 bytes of builtins ... +``` + +The image *links* because 113,757 < 114,688. It links with 0.8% headroom, so +the f32 family very nearly does not fit its own region either. + +**This is a different defect from the constant-pool one it was found behind.** +That one was "the f32 family does not compile for Xtensa"; it is worked around +and the image now builds. This one is "the f32 family does not *fit* alongside +shader code". Fixing the first exposed the second. + +## Root cause — the capacity number was a symptom + +Everything above is accurate and none of it is the cause. The cause is that +**the emulator modeled flash-resident firmware as if it lived in SRAM**. + +On real ESP32 silicon the application's `.text` executes from flash through the +cache (XIP; boot logs show `vaddr=0x4200_0020 map` segments). Only code a JIT +produces at runtime lives in SRAM. The emulator had one modeled SRAM code +region and put both in it, so the builtins image and compiled shader code were +competing for bytes that on hardware they never share. "112 KiB of `.text` +leaves 931 bytes" was a true statement about a false map. + +**Fix** — `BoardProfile` gained read-only IROM/DROM windows at each chip's real +flash-cache bases, plus an internal-SRAM region for an image's `.data`/`.bss`; +`lps-builtins-xt-app` links as flash-resident firmware; `rt_emu::xt_image` +places each segment by classifying its `p_vaddr`, and the compiled shader gets +the **whole** SRAM code region. The region did not grow — 128 KiB on the S3, +92 KiB on classic, unchanged — the builtins simply left it. + +`--features float-f32` is now unconditional for the Xtensa builtins image; +`LP_XT_BUILTINS_F32` is gone. + +**Regression coverage** — `scripts/filetests.sh -t xtn.q32` is back to 849/849 +files (6336/6336 tests) *with* float-f32, and `xtlpn.q32` to 849/849 +(6385/6385); `scripts/build-builtins-xt.sh` now asserts the image's segment +addresses, so a `link.ld` regression fails at the source rather than as a +loader error 300 files later; `lp-xt/lp-xt-emu/tests/call_range.rs` covers the +bug class below. + +## The corrected classic-ESP32 conclusion + +The reading this defect invited — "f32 doesn't fit on classic ESP32" — was +**wrong, and wrong in a way worth naming**: it conflated two unrelated budgets. + +Classic's 92 KiB code region bounds **JIT'd shader code only**. It says nothing +about whether the f32 builtins fit, because on classic as on the S3 the +builtins are flash-resident (`irom_seg` at `0x400D_0000`, 3 MB). Classic f32 +viability is therefore a **flash-budget** question plus an **unprobed LX6 FPU** +question, not an SRAM one — and the flash budget is the one with room to spare. + +## Lesson + +A capacity failure reported in the units of the thing that overflowed is +usually a modeling failure reported in the wrong units. The number (931 bytes) +was measured, reproducible, and led straight to three candidate resolutions — +widen the region, split the region, split the feature — **all three of which +would have preserved the wrong map**. The question that dissolved it was not +"how do we make it fit" but "why are these two things in the same region at +all, when on the device they are not". + +The same model gap was also hiding a live bug class. A shader→builtin call +spans SRAM→flash, tens of megabytes on the S3 and far outside `call8`'s +±512 KiB — which is why the JIT patches indirect calls. In a single small +region an accidentally-direct call would have been *in range on the host* and +out of range on silicon: passing tests, failing hardware. It is now pinned in +`call_range.rs`, along with the finding that classic's IROM sits only ~192 KiB +above its SRAM1 I-bus window, so a direct call *would* work there. The emitter +must stay indirect because the S3 requires it, not because every Xtensa target +does. diff --git a/docs/defects/2026-08-01-xtensa-backend-cannot-select-float-constant-pool.md b/docs/defects/2026-08-01-xtensa-backend-cannot-select-float-constant-pool.md index 98ad6c296..eff8818ea 100644 --- a/docs/defects/2026-08-01-xtensa-backend-cannot-select-float-constant-pool.md +++ b/docs/defects/2026-08-01-xtensa-backend-cannot-select-float-constant-pool.md @@ -1,10 +1,12 @@ --- -status: open +status: open # the backend is still broken; our source avoids it (see "The workaround") found: 2026-08-01 # how: build (M7 P4, flipping the Xtensa builtins image to float-f32) +worked-around: 2026-08-01 # rgb2hsv_f32.rs; f32 image and filetests unblocked area: lp-xt/lps-builtins-xt-app, lp-shader/lps-builtins, esp Rust toolchain class: upstream-toolchain-limitation related: - docs/design/float.md + - docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md --- # The esp Xtensa backend cannot select a float constant pool, so `lps-builtins/float-f32` does not compile for Xtensa @@ -30,57 +32,111 @@ same way, only earlier — in `lps-builtins` (lib) itself rather than in the final binary. So the failure is the backend's, at any optimization level, and is not confined to the image crate. -**Trigger** — `lps-builtins/src/builtins/lpfn/generative/snoise/snoise2_f32.rs` -has an 8-entry gradient lookup table written as a `match` returning -`[f32; 2]`. LLVM promotes the arms to constant-pool entries, and the Xtensa -backend has no selection rule for `PCREL_WRAPPER` over a `TargetConstantPool` -node — it can materialize the *address* of ordinary data, but not of a -constant-pool entry. The reported constant, `[0.0, -1.0]`, is that table's -arm 3. Other `[f32; N]` tables in the generative-noise family -(`worley2/3`, `gnoise3_tile`, `psrdnoise2/3`, `snoise3`, `rgb2hsv`) are -plausible further instances; only the first to be selected is reported. +**Trigger** — a *constant aggregate* of floats. The backend can materialize +the address of ordinary data but has no selection rule for `PCREL_WRAPPER` +over a `TargetConstantPool` node, so any `[N x float]` constant LLVM chooses to +put in the constant pool is fatal. Only the first such constant to be selected +is reported, which made the trigger easy to misattribute (see below). **rv32 is unaffected** — `scripts/build-builtins.sh` has passed `--features float-f32` since M5 and builds the same source cleanly. This is specific to the Xtensa target's backend. **It is also load-bearing for the Q32 path.** `scripts/filetests.sh` builds -this image before running the `xtn.*` / `xtlpn.*` targets, so a script that -passes `--features float-f32` unconditionally takes the entire Xtensa filetest -suite down with it. The feature is therefore **opt-in** — the crate wiring is -in place and `LP_XT_BUILTINS_F32=1` requests it — rather than a default that -would trade a blocked f32 path for a broken fixed-point one. - -## Consequence - -The Xtensa builtins image **cannot currently carry M5's f32 symbols**. M7's -hardware-float lowering inlines the single-instruction family but routes -divide, sqrt, the rounding family, min/max, the float→int conversions and -every transcendental to those symbols (M7 D4), so on the host emulation path -those calls have nothing to resolve against. - -What this does *not* block: everything M7 P3 emits inline — `fadd`/`fsub`/ -`fmul`, the sign-bit ops, all six compares, float select, `itof`, float -load/store and the `wfr`/`rfr` boundary transfers. Those are the majority of -the emitted subset and are covered end to end by -`lp-shader/lpvm-native/tests/xt_pipeline_f32.rs`, whose one builtin-calling -case is marked `#[ignore]` pointing here. - -It is also a live risk for **M7 P5**: `fw-esp32s3` naming -`lps-builtins/float-f32` will hit the same wall at firmware build time. - -## Candidate resolutions (not chosen here — out of M7 P4's scope) - -1. **Rewrite the affected LUTs** so LLVM does not form a constant pool — e.g. - return a tuple, index a `const` array through a `static`, or compute the - gradient arithmetically. Cheapest, but it is a workaround living in shared - builtin source with no local explanation, and it must be rediscovered for - every new `[f32; N]` table. -2. **Report upstream** to `esp-rs/rust` and pin the toolchain once fixed. -3. **Split the f32 builtin feature** so the generative-noise family is - separately gated, letting Xtensa link the arithmetic builtins M7 D4 - actually needs while the noise family stays rv32/host-only. - -(3) is the closest fit to what M7 needs — the routed-to-builtin list in D4 is -arithmetic, not noise — but it changes a feature contract M5 owns, so it is a -decision for Yona rather than for this phase. +this image before running the `xtn.*` / `xtlpn.*` targets, so a compile failure +here takes the entire Xtensa filetest suite down with it — not just the f32 +half. + +## What was actually wrong (2026-08-01) + +The first draft of this record named +`lps-builtins/src/builtins/lpfn/generative/snoise/snoise2_f32.rs`'s 8-entry +`[f32; 2]` gradient LUT as the trigger, inferring it from the reported +constant `[0.0, -1.0]` — which happens to be that table's arm 3. **That was +wrong.** The compiler names the containing function, and it is +`__lp_lpfn_rgb2hsv_f32`: + +```rust +// lps-builtins/src/builtins/lpfn/color/space/rgb2hsv_f32.rs +let p: [f32; 4] = if g < b { + [b, g, -1.0, 2.0 / 3.0] +} else { + [g, b, 0.0, -1.0 / 3.0] +}; +``` + +The two constant lanes of Hocevar's `p` term are selected together, and LLVM +materializes the pair as `[2 x float] [0.0, -1.0]` — the same bytes the +snoise2 arm would have produced, from an entirely different function. + +**Exactly one site needed changing.** With `rgb2hsv` fixed the image builds, +and every other suspect this record named — `snoise2`'s and `snoise3`'s +gradient LUTs, `worley2/3`'s offset tables, `gnoise3_tile`'s `CORNERS`, +`psrdnoise2/3` — compiles unchanged and is present in the built ELF. The +pattern that trips the backend is narrower than "a float lookup table": it is a +*fully constant* aggregate that survives to instruction selection. A LUT +indexed by a runtime value becomes an ordinary global; a `match` whose arms are +runtime-scaled (`worley`'s `[diag, diag, 0.0]`) is not a constant aggregate at +all. + +## The workaround (shipped) + +`rgb2hsv_f32.rs` selects the two constant lanes as **integer bit patterns** and +bitcasts afterwards, so no float constant aggregate is ever formed: + +```rust +const P_Z_IF: u32 = (-1.0f32).to_bits(); // const-evaluated from the +const P_W_IF: u32 = (2.0f32 / 3.0).to_bits(); // original literals +... +f32::from_bits(if g_lt_b { P_Z_IF } else { P_Z_ELSE }) +``` + +Chosen over computing the gradient arithmetically because it is *provably* +bit-identical rather than argued: the constants are the literals, put through +`f32::to_bits` at compile time. Sign-of-zero is the specific hazard an +arithmetic rewrite would have introduced (`0.0 * -1.0` is `-0.0`, and the +literal table's zero lane is `+0.0` on both sides). + +The site carries a comment naming this file and stating it is an upstream +limitation, not a preference, with a slot for the upstream issue URL. The +pre-workaround form is kept verbatim in the test module as an oracle: +`hocevar_p_is_bit_identical_to_the_literal_form` compares the two lane-by-lane +over a grid that crosses `g == b`, plus `-0.0` and NaN. These builtins are +shared with rv32 and wasm, where the original compiles fine, so drift would be +a silent behaviour change on targets that never had the problem. + +`LP_XT_BUILTINS_F32=1 scripts/build-builtins-xt.sh` now succeeds — the thing +this defect blocked — and +`lp-shader/lpvm-native/tests/xt_pipeline_f32.rs`'s +`a_builtin_routed_float_op_resolves_and_runs` is no longer `#[ignore]`d: it +links the shader against the builtins base image and resolves `ffloor` to the +real `__lp_lpir_ffloor_f32` on the M6 emulator. + +**The feature was briefly still not the default, for an unrelated reason.** +Making it unconditional was attempted and reverted on measurement: with the +family in, the image's `.text` was 113,757 B against link.ld's 112 KiB, leaving +931 bytes of the code region for shader code, and the xtn.q32 filetest suite +dropped from 849/849 files to 522/849 on link failures. That was a separate +defect — +`docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md` +— and fixing this one is what exposed it. It is now fixed in turn (the image is +flash-resident, so it never shared the code region to begin with), and +`--features float-f32` **is** the Xtensa builtins image's unconditional +default. + +## Why this stays open + +The backend limitation is untouched. A new `[f32; N]` constant aggregate +anywhere in `lps-builtins` will fail the same way, with the same one-function- +at-a-time reporting, and **M7 P5's `fw-esp32s3` naming `lps-builtins/float-f32` +is exposed to it**. It closes when the esp fork can select +`PCREL_WRAPPER(TargetConstantPool)` and the toolchain is pinned past that. + +## Remaining resolutions + +1. **Report upstream** to `esp-rs/rust` and pin the toolchain once fixed. Not + yet filed; the comment in `rgb2hsv_f32.rs` and the one in + `build-builtins-xt.sh` are shaped for the URL to be dropped in. +2. **Split the f32 builtin feature** so the generative-noise family is + separately gated. No longer needed — this was the fallback for "Xtensa + cannot have the noise family at all", and it can. diff --git a/docs/defects/README.md b/docs/defects/README.md index 9b40e6817..893e58450 100644 --- a/docs/defects/README.md +++ b/docs/defects/README.md @@ -105,6 +105,13 @@ genuinely fits none of these, and define it here in one line. - **`unsynchronized-shared-artifact`** — two steps share a filesystem artifact, but the lock that would order them is scoped narrower than the artifact, so a reader observes a writer's intermediate state. +- **`model-conflation`** — a model represents two things the real system + keeps separate as one resource, so they contend for something that on + hardware they never share. Presents as a *capacity* failure in the units + of the conflated resource, which is why it invites resolutions that all + preserve the wrong model (make it bigger, split it, shrink what goes in + it). The diagnostic question is not "how do we make it fit" but "why are + these in the same place here when they are not on the device". - **`invented-encoding`** — a binary format's numbering (an instruction encoding, a relocation type) is inferred from the shape of its neighbours instead of read from the spec, and the invented value @@ -169,6 +176,8 @@ sentence (arguments in, returns out; registers and stack). | Class | Date | Entry | Status | Area | | --- | --- | --- | --- | --- | +| model-conflation | 2026-08-01 | [xt-f32-builtins-exhaust-the-emulator-code-region](2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md) | fixed | lp-xt-emu (board/memory) + lps-builtins-xt-app + lpvm-native/rt_emu | +| upstream-toolchain-limitation | 2026-08-01 | [xtensa-backend-cannot-select-float-constant-pool](2026-08-01-xtensa-backend-cannot-select-float-constant-pool.md) | **open** (worked around) | lps-builtins + esp Rust toolchain | | invented-encoding | 2026-07-31 | [zexth-encoding-steals-xori-128](2026-07-31-zexth-encoding-steals-xori-128.md) | fixed | lp-riscv-inst (encode/decode) + lp-riscv-emu (executor) | | invented-encoding | 2026-07-31 | [elf-loader-riscv-reloc-numbering](2026-07-31-elf-loader-riscv-reloc-numbering.md) | **open** | lp-riscv-elf (relocations) | | partial-knowledge-loss | 2026-07-31 | [elf-loader-drops-relocation-addends](2026-07-31-elf-loader-drops-relocation-addends.md) | fixed | lp-riscv-elf (relocations) | diff --git a/lp-shader/lps-builtins/src/builtins/lpfn/color/space/rgb2hsv_f32.rs b/lp-shader/lps-builtins/src/builtins/lpfn/color/space/rgb2hsv_f32.rs index 1eebd8281..c167fc919 100644 --- a/lp-shader/lps-builtins/src/builtins/lpfn/color/space/rgb2hsv_f32.rs +++ b/lp-shader/lps-builtins/src/builtins/lpfn/color/space/rgb2hsv_f32.rs @@ -17,16 +17,49 @@ //! //! **Tolerance:** exact against the canonical f32. +// The `p.zw` constants of Hocevar's formulation, as f32 *bit patterns*. +// +// **This is an upstream-LLVM workaround, not a preference.** Written the +// natural way — `if g < b { [b, g, -1.0, 2.0 / 3.0] } else { [g, b, 0.0, +// -1.0 / 3.0] }` — LLVM materialises the constant lane pair as a +// `[2 x float]` constant-pool entry, and the esp Xtensa backend cannot select +// `XtensaISD::PCREL_WRAPPER` over a `TargetConstantPool`, so the whole f32 +// builtins family fails to build for `xtensa-esp32s3-none-elf` at every +// optimisation level. Selecting the integer bit patterns and bitcasting after +// the fact keeps the constants out of the float constant pool. +// +// Defect: docs/defects/2026-08-01-xtensa-backend-cannot-select-float-constant-pool.md +// Upstream: not yet filed. +// +// `f32::to_bits` is const-evaluated, so these are the same floats the literals +// were; `hocevar_p_is_bit_identical_to_the_literal_form` pins that, and is the +// test to keep green if this is ever reverted. +const P_Z_IF: u32 = (-1.0f32).to_bits(); +const P_W_IF: u32 = (2.0f32 / 3.0).to_bits(); +const P_Z_ELSE: u32 = (0.0f32).to_bits(); +const P_W_ELSE: u32 = (-1.0f32 / 3.0).to_bits(); + +/// Hocevar's `p` term: the green/blue ordering step. +/// +/// Split out of `rgb2hsv` only so the constant-pool workaround above can be +/// tested against the literal form it replaced. +#[inline(always)] +fn hocevar_p(g: f32, b: f32) -> [f32; 4] { + let g_lt_b = g < b; + [ + if g_lt_b { b } else { g }, + if g_lt_b { g } else { b }, + f32::from_bits(if g_lt_b { P_Z_IF } else { P_Z_ELSE }), + f32::from_bits(if g_lt_b { P_W_IF } else { P_W_ELSE }), + ] +} + /// Rust-facing form. #[inline] fn rgb2hsv(r: f32, g: f32, b: f32) -> [f32; 3] { const EPSILON: f32 = 1.0 / 65536.0; - let p: [f32; 4] = if g < b { - [b, g, -1.0, 2.0 / 3.0] - } else { - [g, b, 0.0, -1.0 / 3.0] - }; + let p = hocevar_p(g, b); let q: [f32; 4] = if r < p[0] { [p[0], p[1], p[3], r] } else { @@ -123,6 +156,56 @@ mod tests { } } + /// The pre-workaround form of `hocevar_p`, kept verbatim as the oracle. + /// + /// This is the code the Xtensa constant-pool workaround replaced. It stays + /// here so the rewrite's bit-equivalence is *pinned* rather than argued — + /// these builtins are shared with rv32 and wasm, where the original form + /// compiles fine and any drift would be a silent behaviour change. + fn hocevar_p_literal(g: f32, b: f32) -> [f32; 4] { + if g < b { + [b, g, -1.0, 2.0 / 3.0] + } else { + [g, b, 0.0, -1.0 / 3.0] + } + } + + #[test] + fn hocevar_p_is_bit_identical_to_the_literal_form() { + // A grid that crosses g == b in both directions, plus the cases where + // a sign-of-zero or NaN difference would hide: -0.0/+0.0 must select + // the same lanes, and `g < b` is false for any NaN operand. + const SAMPLES: [f32; 11] = [ + -1.0, + -0.75, + -0.5, + -0.25, + 0.0, + 0.25, + 0.5, + 0.75, + 1.0, + -0.0, + f32::NAN, + ]; + + for &g in SAMPLES.iter() { + for &b in SAMPLES.iter() { + let got = hocevar_p(g, b); + let want = hocevar_p_literal(g, b); + for k in 0..4 { + assert_eq!( + got[k].to_bits(), + want[k].to_bits(), + "lane {k} for g={g} b={b}: {:#010x} vs {:#010x}", + got[k].to_bits(), + want[k].to_bits() + ); + } + } + } + } + #[test] fn alpha_passes_through_the_vec4_form_untouched() { let mut out = [f32::NAN; 4]; diff --git a/lp-shader/lpvm-native/README.md b/lp-shader/lpvm-native/README.md index 1c6a8ba6d..184048b7b 100644 --- a/lp-shader/lpvm-native/README.md +++ b/lp-shader/lpvm-native/README.md @@ -243,10 +243,15 @@ Without it, `lps_builtins_xt_image::is_available()` is false and Xtensa consumers skip with a loud note rather than failing — the workspace must build and test on a machine with no esp toolchain. -Xtensa shader code shares the image's 112 KiB text region with ~84 KiB of -builtins, leaving **~28 KiB**. Overflowing it is an explicit error naming the -budget, never a silent write past the region; the fix would be -`lp-xt/lps-builtins-xt-app/link.ld`'s split, not the host region size. +The Xtensa builtins image is **flash-resident firmware**, exactly as on the +device: `rt_emu/xt_image` places its `.text` in the emulator's modeled IROM +window and its `.rodata` in DROM, and compiled shader code gets the **whole** +SRAM code region — 128 KiB on the S3 — with nothing resident in front of it. +Overflowing that is an explicit error naming the budget, never a silent write +past the region, and the budget is now the shader's own size rather than +whatever the builtins left over. (It was the latter until 2026-08-01; see +`docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md` +for why that was a modeling bug and not a capacity one.) `isa/xt/` and the `lp-xt-*` crates it builds on contain material derived from LLVM under Apache-2.0-WITH-LLVM-exception and carry per-file provenance diff --git a/lp-shader/lpvm-native/src/rt_emu/image.rs b/lp-shader/lpvm-native/src/rt_emu/image.rs index 2d09b2f09..2f140be1e 100644 --- a/lp-shader/lpvm-native/src/rt_emu/image.rs +++ b/lp-shader/lpvm-native/src/rt_emu/image.rs @@ -1,9 +1,9 @@ //! [`GuestImage`] — the ISA-neutral loaded image `rt_emu` executes. //! -//! `rt_emu` needs exactly four things from a linked image: the code bytes, the -//! RAM bytes, a name→address symbol map to resolve entry points with, and where -//! the code ends (for `code_size_bytes`). Nothing about those four is -//! ISA-specific, but the type that used to carry them — +//! `rt_emu` needs a handful of things from a linked image: the code bytes, the +//! RAM bytes, any further preloaded regions, a name→address symbol map to +//! resolve entry points with, and where the code ends (for `code_size_bytes`). +//! Nothing about those is ISA-specific, but the type that used to carry them — //! `lp_riscv_elf::ElfLoadInfo` — lives in an rv32 crate, which was the last //! rv32-shaped thing in the module/instance plumbing. //! @@ -16,6 +16,16 @@ use alloc::collections::BTreeMap; use alloc::string::String; use alloc::vec::Vec; +/// One preloaded region of a guest image beyond `code`/`ram`: `bytes` are +/// written at `base`, which must be a region base in the emulator's map. +#[derive(Clone, Debug)] +pub struct ImageRegion { + /// Guest address of `bytes[0]`. + pub base: u32, + /// Bytes to place. May be shorter than the region — the rest stays zero. + pub bytes: Vec, +} + /// A linked guest image ready to execute: code, RAM, symbols. /// /// Addresses are guest addresses in whatever map the target ISA's emulator @@ -25,11 +35,18 @@ pub struct GuestImage { /// Code region bytes: `code[0]` lives at [`code_base`](Self::code_base). pub code: Vec, /// Guest address of `code[0]`. Zero on rv32, whose code region starts at - /// address 0; the Xtensa code region sits in SRAM1. + /// address 0; the Xtensa code region sits in SRAM1 and holds **only** the + /// compiled shader — the builtins are in [`regions`](Self::regions). pub code_base: u32, - /// RAM region bytes, starting at the RAM base address. Empty on Xtensa, - /// whose data segments live inside the code region. + /// RAM region bytes, starting at the RAM base address. Empty on Xtensa. pub ram: Vec, + /// Further regions to preload before entry, each at its own base. + /// + /// Empty on rv32, which links one executable. Xtensa loads a + /// **flash-resident** builtins image alongside the shader — its `.text` in + /// the IROM window, its `.rodata` in DROM, its `.data`/`.bss` in internal + /// SRAM — which is three regions the shader's code region is not. + pub regions: Vec, /// Symbol name → absolute guest address. pub symbol_map: BTreeMap, /// Address one past the last code byte, for `code_size_bytes` reporting. @@ -51,6 +68,7 @@ impl From for GuestImage { // lp-riscv-elf lays the code region out from address 0. code_base: 0, ram: load.ram, + regions: Vec::new(), symbol_map: load.symbol_map.into_iter().collect(), code_end: load.code_end, } diff --git a/lp-shader/lpvm-native/src/rt_emu/instance.rs b/lp-shader/lpvm-native/src/rt_emu/instance.rs index d742e0d33..0d8b5b3ba 100644 --- a/lp-shader/lpvm-native/src/rt_emu/instance.rs +++ b/lp-shader/lpvm-native/src/rt_emu/instance.rs @@ -572,6 +572,12 @@ impl NativeEmuInstance { emu.step_budget = EMU_CALL_INSTRUCTION_LIMIT; emu.mem .add_shared(lp_xt_emu::SHARED_DBUS_BASE, self.module.arena.storage_arc()); + // The flash-resident builtins image first (IROM/DROM, plus its SRAM + // .data/.bss when it has any), then the shader's own SRAM code region. + // Both are per-call: the emulator is built fresh for every entry. + for r in &self.module.load.regions { + emu.mem.load_region(r.base, &r.bytes); + } emu.mem .load_region(self.module.load.code_base, &self.module.load.code); diff --git a/lp-shader/lpvm-native/src/rt_emu/mod.rs b/lp-shader/lpvm-native/src/rt_emu/mod.rs index 4a4b80644..967db34dd 100644 --- a/lp-shader/lpvm-native/src/rt_emu/mod.rs +++ b/lp-shader/lpvm-native/src/rt_emu/mod.rs @@ -24,6 +24,6 @@ pub mod module; pub mod xt_image; pub use engine::NativeEmuEngine; -pub use image::GuestImage; +pub use image::{GuestImage, ImageRegion}; pub use instance::NativeEmuInstance; pub use module::NativeEmuModule; diff --git a/lp-shader/lpvm-native/src/rt_emu/xt_image.rs b/lp-shader/lpvm-native/src/rt_emu/xt_image.rs index 18694ab85..5d1aa6215 100644 --- a/lp-shader/lpvm-native/src/rt_emu/xt_image.rs +++ b/lp-shader/lpvm-native/src/rt_emu/xt_image.rs @@ -1,5 +1,5 @@ -//! Building the Xtensa [`GuestImage`]: the builtins base image with compiled -//! shader code placed after it and call relocations resolved. +//! Building the Xtensa [`GuestImage`]: the flash-resident builtins base image, +//! compiled shader code in SRAM, and call relocations resolved between them. //! //! ## Why this shape //! @@ -7,7 +7,7 @@ //! `lpvm_cranelift::link_object_with_builtins`. Xtensa takes the route the M3b //! spike verified as the lower-risk one: load the **already-linked** builtins //! executable with `lp-xt-elf`'s proven base loader, place the shader's compiled -//! functions after its `.text`, and patch each call relocation against the +//! functions in the code region, and patch each call relocation against the //! merged symbol map. That is the flow `lpvm-native/tests/xt_pipeline.rs` proves //! for shader-to-shader calls, extended with the builtins symbols. //! @@ -15,18 +15,25 @@ //! deliberately *not* used: its own docs call it a stretch prototype, and it is //! reserved for the on-device builtins-link path. //! -//! ## The single-region layout +//! ## The layout: firmware in flash, JIT output in SRAM //! -//! `lps-builtins-xt-app`'s linker script puts both of its segments inside the -//! emulator's 128 KiB code region: `.text` in the low 112 KiB (I-bus -//! `0x4037_8000`) and `.rodata`/`.data`/`.bss` in the top 16 KiB (D-bus -//! `0x3FCA_4000`). So one flat buffer covering the whole region reproduces the -//! image exactly, and a run costs one `load_bytes` — the same shape as rv32's -//! per-call `code`/`ram` clone. +//! This mirrors the device. `lps-builtins-xt-app` is firmware, so it links into +//! the flash windows the cache maps (`.text` at IROM, `.rodata` at DROM, +//! `.data`/`.bss` in internal SRAM); the shader is JIT output, so it gets the +//! **whole** SRAM code region — 128 KiB on the S3, 92 KiB on classic — with +//! nothing resident in front of it. //! -//! The IRAM/DRAM split is **not** hardcoded here. It is derived from the image: -//! the shader-code limit is the lowest data-segment offset, so changing the -//! linker script moves the limit automatically. +//! Until 2026-08-01 both lived in the one SRAM code region, shader code +//! starting wherever the image's `.text` happened to end. That coupled the +//! largest compilable shader to the size of the builtins: turning on the f32 +//! builtin family left 931 bytes for shader code and took the `xtn.q32` +//! filetest suite from 849/849 files to 522/849. It also made a +//! shader→builtin call look like a short local hop when on silicon it spans +//! SRAM→flash, far outside any direct-call displacement. +//! +//! Nothing here hardcodes the split. Every segment is placed by classifying its +//! `p_vaddr` against [`BoardProfile`], so moving a section in +//! `lps-builtins-xt-app/link.ld` moves it here. use alloc::format; use alloc::string::{String, ToString}; @@ -41,14 +48,14 @@ use crate::compile::CompiledModule; use crate::error::NativeError; use crate::isa::IsaTarget; -use super::GuestImage; +use super::{GuestImage, ImageRegion}; /// Build the Xtensa guest image for `compiled` against the builtins base image -/// `builtins_elf`, laid out for `profile`'s code region. +/// `builtins_elf`, laid out for `profile`'s memory map. /// -/// Returns an image whose `symbol_map` holds **I-bus execute** addresses for -/// every builtin and every compiled function, so `rt_emu` can use a resolved -/// symbol directly as an entry PC. +/// Returns an image whose `symbol_map` holds **execute** addresses for every +/// builtin (in flash) and every compiled function (in SRAM), so `rt_emu` can +/// use a resolved symbol directly as an entry PC. pub fn build_xt_image( compiled: &CompiledModule, builtins_elf: &[u8], @@ -75,75 +82,83 @@ pub fn build_xt_image( let elf = XtensaElf::parse(builtins_elf) .map_err(|e| NativeError::Internal(format!("Xtensa builtins image: {e}")))?; - let region_base = profile.code_dbus_base; - let region_len = profile.code_region_len; - let mut code = vec![0u8; region_len]; + // --- the builtins image: one buffer per region it occupies --- + // + // Each buffer is trimmed to the bytes actually used, not to the region + // length: `rt_emu` reloads them into a fresh emulator on every host call, + // so the copy is on the hot path. + let mut irom = SegmentBuffer::new(profile.irom_base, profile.irom_len, "IROM (flash .text)"); + let mut drom = SegmentBuffer::new(profile.drom_base, profile.drom_len, "DROM (flash .rodata)"); + let mut data = SegmentBuffer::new( + profile.image_data_base, + profile.image_data_len, + "DRAM (image .data/.bss)", + ); - // Place every PT_LOAD segment at its own address, tracking where executable - // bytes end and where data begins. - let mut text_end = 0usize; - let mut data_start = region_len; for seg in elf .segments() .map_err(|e| NativeError::Internal(format!("Xtensa builtins segments: {e}")))? { - let (offset, executable) = region_offset_of(seg.vaddr, profile).ok_or_else(|| { - NativeError::Internal(format!( - "builtins segment at {:#010x} falls outside the {} code region \ - {region_base:#010x}..{:#010x}", - seg.vaddr, - profile.name, - region_base + region_len as u32 - )) - })?; - let end = offset + seg.memsz as usize; - if end > region_len { + let target = if let Some(off) = profile.irom_offset(seg.vaddr) { + (&mut irom, off) + } else if let Some(off) = profile.drom_offset(seg.vaddr) { + (&mut drom, off) + } else if let Some(off) = profile.image_data_offset(seg.vaddr) { + (&mut data, off) + } else { return Err(NativeError::Internal(format!( - "builtins segment at {:#010x} ({} bytes) overruns the code region by {} bytes", + "builtins segment at {:#010x} ({} bytes) is in none of the {} \ + image regions: IROM {:#010x}+{:#x}, DROM {:#010x}+{:#x}, \ + DRAM {:#010x}+{:#x}. The image must link as flash-resident \ + firmware — see lp-xt/lps-builtins-xt-app/link.ld.", seg.vaddr, seg.memsz, - end - region_len + profile.name, + profile.irom_base, + profile.irom_len, + profile.drom_base, + profile.drom_len, + profile.image_data_base, + profile.image_data_len, ))); - } - code[offset..offset + seg.data.len()].copy_from_slice(seg.data); - // The p_memsz tail (.bss) stays zero — the buffer started zeroed. - if executable { - text_end = text_end.max(end); - } else { - data_start = data_start.min(offset); - } + }; + let (buf, offset) = target; + buf.place(seg.vaddr, offset, seg.data, seg.memsz as usize)?; } + // --- the shader: the whole SRAM code region, nothing resident in front --- + let region_base = profile.code_dbus_base; + let region_len = profile.code_region_len; + let mut code = vec![0u8; region_len]; + let alias = profile.alias; let ibus_of = |offset: usize| alias.dbus_to_ibus(region_base + offset as u32); - // Shader code starts 4-aligned after the builtins text and must stay below - // the image's own data segments. - let shader_start = text_end.next_multiple_of(4); let mut symbol_map: alloc::collections::BTreeMap = elf.symbols().into_iter().collect(); let mut offsets = Vec::with_capacity(compiled.functions.len()); - let mut cursor = shader_start; + let mut cursor = 0usize; for f in &compiled.functions { offsets.push(cursor); symbol_map.insert(f.name.to_string(), ibus_of(cursor)); cursor += f.code.len(); cursor = cursor.next_multiple_of(4); } - if cursor > data_start { + if cursor > region_len { return Err(NativeError::Internal(format!( - "compiled shader code does not fit the Xtensa code region: {} bytes of \ - shader after {} bytes of builtins needs {} bytes, but only {} are free \ - before the image's data segments at region offset {:#x} \ - (see lp-xt/lps-builtins-xt-app/link.ld)", - cursor - shader_start, - text_end, - cursor - shader_start, - data_start - shader_start, - data_start + "compiled shader code does not fit the Xtensa code region: {} bytes \ + of shader code, but the {} SRAM code region is {} bytes \ + ({:#010x}..{:#010x}). The builtins image is not in this region — \ + it is flash-resident — so this is the shader's own size.", + cursor, + profile.name, + region_len, + region_base, + region_base + region_len as u32, ))); } + code.truncate(cursor); // Copy the function bodies in, then resolve every call relocation. Patching // happens after placement so a forward call to a later function resolves. @@ -180,26 +195,64 @@ pub fn build_xt_image( code, code_base: region_base, ram: Vec::new(), + regions: [irom, drom, data] + .into_iter() + .filter_map(SegmentBuffer::into_region) + .collect(), symbol_map, code_end: ibus_of(cursor), }) } -/// Offset of `vaddr` within the profile's code region, and whether it was named -/// through the executable (I-bus) view. -/// -/// The builtins image addresses `.text` at its I-bus alias and its data segments -/// at their D-bus addresses — both inside the same region — so both views must -/// resolve to the one flat buffer. -fn region_offset_of(vaddr: u32, profile: &BoardProfile) -> Option<(usize, bool)> { - let base = profile.code_dbus_base; - let len = profile.code_region_len as u32; - if vaddr >= base && vaddr < base + len { - return Some(((vaddr - base) as usize, false)); +/// Accumulates the PT_LOAD segments that land in one modeled region. +struct SegmentBuffer { + base: u32, + len: usize, + what: &'static str, + bytes: Vec, +} + +impl SegmentBuffer { + fn new(base: u32, len: usize, what: &'static str) -> SegmentBuffer { + SegmentBuffer { + base, + len, + what, + bytes: Vec::new(), + } } - let dbus = profile.alias.ibus_to_dbus(vaddr); - if dbus >= base && dbus < base + len { - return Some(((dbus - base) as usize, true)); + + /// Place one segment's `data` at `offset`, growing the buffer to cover it. + /// `memsz` beyond `data` (the `.bss` tail) is left zero — the emulator's + /// regions start zeroed, so it costs nothing to carry. + fn place( + &mut self, + vaddr: u32, + offset: usize, + data: &[u8], + memsz: usize, + ) -> Result<(), NativeError> { + if offset + memsz > self.len { + return Err(NativeError::Internal(format!( + "builtins segment at {vaddr:#010x} ({memsz} bytes) overruns the \ + modeled {} window ({} bytes) by {} bytes", + self.what, + self.len, + offset + memsz - self.len, + ))); + } + let end = offset + data.len(); + if self.bytes.len() < end { + self.bytes.resize(end, 0); + } + self.bytes[offset..end].copy_from_slice(data); + Ok(()) + } + + fn into_region(self) -> Option { + (!self.bytes.is_empty()).then_some(ImageRegion { + base: self.base, + bytes: self.bytes, + }) } - None } diff --git a/lp-shader/lpvm-native/tests/xt_builtins_image.rs b/lp-shader/lpvm-native/tests/xt_builtins_image.rs index b4c45b543..3f033b334 100644 --- a/lp-shader/lpvm-native/tests/xt_builtins_image.rs +++ b/lp-shader/lpvm-native/tests/xt_builtins_image.rs @@ -92,21 +92,36 @@ fn image_exports_every_builtin_and_they_execute() { } } +/// The builtins are **flash-resident firmware**, so every one of them must +/// live in the modeled IROM window — and, just as load-bearing, none of them +/// may live in the SRAM code region, which belongs to JIT'd shader code. +/// +/// This assertion used to read the other way round, back when the image shared +/// the code region with the shader. Both halves matter: a symbol outside every +/// modeled region would fetch-fault at run time rather than fail here, and a +/// symbol back inside the code region would silently re-couple the largest +/// compilable shader to the size of the builtins +/// (`docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md`). #[test] -fn builtins_land_inside_the_modeled_code_region() { +fn builtins_land_in_flash_and_never_in_the_shader_code_region() { let Some((_emu, bytes)) = load_image() else { return; }; let elf = XtensaElf::parse(&bytes).unwrap(); - // The emulator maps 128 KiB of code at I-bus 0x40378000; a symbol outside - // it would fetch-fault at run time rather than fail here, so pin it. - let lo = 0x4037_8000u32; - let hi = lo + 0x0002_0000; + let p = lp_xt_emu::board::BoardProfile::esp32s3(); for name in ["__lps_sin_q32", "__lps_cos_q32", "__lps_pow_q32"] { let a = elf.symbol(name).unwrap_or_else(|| panic!("exports {name}")); assert!( - (lo..hi).contains(&a), - "{name} at {a:#x} is outside the modeled code region {lo:#x}..{hi:#x}" + p.irom_offset(a).is_some(), + "{name} at {a:#x} is outside the modeled IROM window {:#x}..{:#x} \ + — see lp-xt/lps-builtins-xt-app/link.ld", + p.irom_base, + p.irom_base + p.irom_len as u32, + ); + assert!( + p.code_region_offset(a).is_none(), + "{name} at {a:#x} is inside the SRAM code region, which belongs to \ + JIT'd shader code" ); } } diff --git a/lp-shader/lpvm-native/tests/xt_pipeline_f32.rs b/lp-shader/lpvm-native/tests/xt_pipeline_f32.rs index 96183fe2d..001db98a4 100644 --- a/lp-shader/lpvm-native/tests/xt_pipeline_f32.rs +++ b/lp-shader/lpvm-native/tests/xt_pipeline_f32.rs @@ -32,6 +32,14 @@ //! two-instruction preamble that does what P5's board init will do. //! `unarmed_float_code_faults_with_a_coprocessor_trap` is the negative //! control: the same code without the preamble must take EXCCAUSE 32. +//! +//! # Two link paths +//! +//! Most tests here link only the module's own functions ([`link_blob`]) — +//! everything M7 P3 emits inline needs nothing else. The one case that calls a +//! routed builtin ([`run_f32_with_builtins`]) links against the Xtensa builtins +//! base image instead, and skips loudly when that cross-target artifact has not +//! been built. use lp_collection::VecMap; use lpir::builder::FunctionBuilder; @@ -41,7 +49,8 @@ use lpvm_native::compile::compile_module; use lpvm_native::isa::IsaTarget; use lpvm_native::native_options::NativeCompileOptions; -use lp_xt_emu::{Emulator, RunOutcome}; +use lp_xt_elf::XtensaElf; +use lp_xt_emu::{CallOutcome, Emulator, RunOutcome}; use lp_xt_inst::{Inst, NullaryNarrowOp, Reg, SpecialReg, SrOp, encode}; /// EXCCAUSE 32 — `Coprocessor0Disabled`, what an FP instruction takes when @@ -108,7 +117,11 @@ fn link_blob( let mut order: Vec = (0..funcs.len()).collect(); order.sort_by_key(|&i| i != entry_idx); - let mut code = if arm_fpu { arm_fpu_preamble() } else { Vec::new() }; + let mut code = if arm_fpu { + arm_fpu_preamble() + } else { + Vec::new() + }; let entry_off = code.len() as u32; let mut entries = VecMap::::new(); let mut func_offsets = vec![0usize; funcs.len()]; @@ -151,6 +164,169 @@ fn run_f32(ir: &LpirModule, sig: &LpsModuleSig, entry_name: &str, args: &[u32]) emu.run_with_args(&code, entry_off, args) } +/// The Xtensa builtins base image **carrying the f32 family**, or `None` with a +/// loud note. +/// +/// Two ways to come up empty, both skips rather than failures: +/// +/// - the image has not been built at all — a gitignored cross-target artifact +/// (`scripts/build-builtins-xt.sh`, esp toolchain), the same contract +/// `xt_builtins_image.rs` uses; +/// - it was built by an older checkout, before `float-f32` became the image's +/// unconditional default. It is no longer opt-in: the capacity problem that +/// kept it so was the emulator modeling flash-resident firmware as if it +/// shared SRAM with the JIT, and the flash model closed it +/// (`docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md`). +/// A stale artifact is still a rebuild rather than a red suite. +fn builtins_image_with_f32() -> Option> { + let p = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("../../lp-xt/fixtures/elf/lps-builtins-xt-app.elf"); + let Ok(bytes) = std::fs::read(&p) else { + eprintln!( + "SKIP: {} not found — run scripts/build-builtins-xt.sh (esp toolchain) first", + p.display() + ); + return None; + }; + let has_f32 = { + let elf = XtensaElf::parse(&bytes).expect("builtins image parses as Xtensa ELF32"); + elf.symbol("__lp_lpir_ffloor_f32").is_some() + }; + if !has_f32 { + eprintln!( + "SKIP: {} carries no native-f32 builtins (stale artifact) — \ + rerun scripts/build-builtins-xt.sh", + p.display() + ); + return None; + } + Some(bytes) +} + +/// Compile in `FloatMode::F32`, link against the **builtins base image**, and +/// run with the FPU armed. +/// +/// [`run_f32`] links only the module's own functions and loads the blob at the +/// code region's base. That is enough for everything M7 P3 emits inline, but a +/// `sym_call` into `__lp_lpir_*_f32` (M7 D4) needs the builtins image resident +/// *and* its symbols in the relocation map. +/// +/// The layout is `rt_emu::xt_image`'s: the builtins image is **flash-resident** +/// (`.text` in IROM, `.rodata` in DROM), so the shader keeps the SRAM code +/// region to itself and starts at its base — exactly where [`run_f32`] puts it. +/// The only thing this adds over that path is the image and its symbols. It is +/// hand-rolled here rather than calling `build_xt_image` because that sits +/// behind the `emu` feature, and the other tests in this file run at *default* +/// features. +/// +/// The call this exercises therefore crosses SRAM → flash, which is the point: +/// on silicon it always did, and until 2026-08-01 the emulator could not tell. +fn run_f32_with_builtins( + ir: &LpirModule, + sig: &LpsModuleSig, + entry_name: &str, + args: &[u32], + image: &[u8], +) -> RunOutcome { + assert!( + args.len() <= 5, + "the arming preamble clobbers the sixth argument register" + ); + let mut emu = Emulator::new(); + let elf = XtensaElf::parse(image).expect("builtins image parses as Xtensa ELF32"); + elf.load_into(&mut emu) + .expect("image loads into emulator memory"); + + // The image is firmware: every one of its segments must have landed in a + // flash window, never in the SRAM code region the shader is about to use. + // Assert that rather than assume it — a link.ld regression would otherwise + // show up as a mysterious wrong answer. + let p = emu.profile; + for seg in elf.segments().expect("image segments decode") { + assert!( + p.irom_offset(seg.vaddr).is_some() + || p.drom_offset(seg.vaddr).is_some() + || p.image_data_offset(seg.vaddr).is_some(), + "builtins segment at {:#010x} is outside the modeled flash/image \ + regions — see lp-xt/lps-builtins-xt-app/link.ld", + seg.vaddr, + ); + assert!( + p.code_region_offset(seg.vaddr).is_none(), + "builtins segment at {:#010x} is inside the SRAM code region, which \ + belongs to JIT'd shader code", + seg.vaddr, + ); + } + let shader_base = p.code_ibus_base(); + let shader_limit = shader_base + p.code_region_len as u32; + + let opts = NativeCompileOptions { + float_mode: FloatMode::F32, + fuel: false, + ..Default::default() + }; + let module = compile_module(ir, sig, FloatMode::F32, opts, IsaTarget::Xtensa) + .expect("xt f32 compile should succeed"); + + // Arming preamble, then the entry function, then everything else. The + // preamble is a multiple of 4 bytes long, so the entry's ENTRY is the next + // instruction executed and no jump is needed. + let funcs: Vec<_> = module.functions.iter().collect(); + let entry_idx = funcs + .iter() + .position(|f| f.name == entry_name) + .expect("entry function exists"); + let mut order: Vec = (0..funcs.len()).collect(); + order.sort_by_key(|&i| i != entry_idx); + + // Execution starts at the preamble, not at the entry function — entering at + // the entry skips the arming and the first FP instruction takes EXCCAUSE 32. + let mut code = arm_fpu_preamble(); + let entry = shader_base; + let mut symbols: std::collections::BTreeMap = elf.symbols().into_iter().collect(); + let mut func_addrs = vec![0u32; funcs.len()]; + for &i in &order { + let at = shader_base + code.len() as u32; + func_addrs[i] = at; + symbols.insert(funcs[i].name.clone(), at); + code.extend_from_slice(&funcs[i].code); + while code.len() % 4 != 0 { + code.push(0); + } + } + assert!( + shader_base + code.len() as u32 <= shader_limit, + "shader code overruns the SRAM code region" + ); + + for (fi, f) in funcs.iter().enumerate() { + for reloc in &f.relocs { + assert_eq!( + reloc.r_type, + IsaTarget::Xtensa.call_reloc_type(), + "unexpected reloc type" + ); + let target = *symbols.get(&reloc.symbol).unwrap_or_else(|| { + panic!( + "unresolved symbol {} — not a builtin in the base image nor a \ + function in this module", + reloc.symbol + ) + }); + let slot = (func_addrs[fi] - shader_base) as usize + reloc.offset; + code[slot..slot + 4].copy_from_slice(&target.to_le_bytes()); + } + } + + emu.mem.load_bytes(shader_base, &code); + let mut tracer = lp_xt_emu::NoopTracer; + match emu.run_loaded_with_args(entry, args, &mut tracer, None) { + CallOutcome::Ok { lo, .. } => RunOutcome::Ok(lo), + CallOutcome::Trap(t) => RunOutcome::Trap(t), + } +} + fn expect_ok(out: RunOutcome) -> u32 { match out { RunOutcome::Ok(v) => v, @@ -217,10 +393,7 @@ fn run_float_binop( build(&mut fb, x, y, out); fb.push_return(&[out]); let ir = one_function_module(fb.finish()); - let sig = one_function_sig( - vec![float_param("a"), float_param("b")], - LpsType::Float, - ); + let sig = one_function_sig(vec![float_param("a"), float_param("b")], LpsType::Float); expect_ok(run_f32(&ir, &sig, "f", &[0, bits(a), bits(b)])) } @@ -437,7 +610,11 @@ fn itof_signed_and_unsigned_through_the_pipeline() { let ir = one_function_module(fb.finish()); let sig = one_function_sig(vec![int_param("x")], LpsType::Float); let got = expect_ok(run_f32(&ir, &sig, "f", &[0, arg])); - assert_eq!(f32::from_bits(got), want, "itof signed={signed} of {arg:#x}"); + assert_eq!( + f32::from_bits(got), + want, + "itof signed={signed} of {arg:#x}" + ); } } @@ -789,13 +966,7 @@ fn float_recursion_at_depth_100_preserves_every_live_value() { imports: vec![], functions: VecMap::from([(FuncId(0), rec), (FuncId(1), f)]), }; - let params = || { - vec![ - int_param("n"), - float_param("a"), - float_param("b"), - ] - }; + let params = || vec![int_param("n"), float_param("a"), float_param("b")]; let sig = LpsModuleSig { functions: vec![ LpsFnSig { @@ -1018,30 +1189,46 @@ fn unarmed_float_code_faults_with_a_coprocessor_trap() { // Builtin routing (M7 D4) // --------------------------------------------------------------------------- -/// **Blocked on an upstream toolchain limitation** — see -/// `docs/defects/2026-08-01-xtensa-backend-cannot-select-float-constant-pool.md`. -/// /// M7 D4 routes the non-inlinable float operations (divide, sqrt, the rounding /// family, min/max, float→int, every transcendental) to M5's `_f32` builtins /// via `sym_call`. Resolving those on the host emulation path needs -/// `lp-xt/lps-builtins-xt-app` built with `float-f32` — and that build fails in -/// the esp Rust backend, which cannot select a float constant pool, at any -/// optimization level. The image therefore carries none of the f32 family yet. +/// `lp-xt/lps-builtins-xt-app` built with `float-f32`. +/// +/// This was `#[ignore]`d while that build failed outright in the esp Rust +/// backend, which cannot select a float constant pool +/// (`docs/defects/2026-08-01-xtensa-backend-cannot-select-float-constant-pool.md` +/// — still open upstream, worked around in our source), and then ran only +/// against an opt-in image while the family did not *fit* alongside shader code +/// (`docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md`). +/// Both are closed: the family is unconditional, and the call it makes crosses +/// SRAM → flash exactly as it does on silicon. It still skips rather than fails +/// when the cross-target artifact is absent — see [`builtins_image_with_f32`]. /// -/// The wiring is done (`Cargo.toml`, and `scripts/build-builtins-xt.sh` behind -/// `LP_XT_BUILTINS_F32=1`); only the build is blocked. Un-ignore this, and -/// make the feature that script's default, when the defect is resolved. /// `ffloor` is deliberately chosen over `fdiv`/`fsqrt`: it reaches no /// `div0.s`/`const.s` estimate helper, so it does not additionally depend on /// M6-P6's unresolved policy fields. #[test] -#[ignore = "blocked: lps-builtins/float-f32 does not compile for Xtensa (see docs/defects/2026-08-01-xtensa-backend-cannot-select-float-constant-pool.md)"] fn a_builtin_routed_float_op_resolves_and_runs() { - let r = run_float_binop( - |fb, x, _y, out| fb.push(LpirOp::Ffloor { dst: out, src: x }), - 3.75, - 0.0, - ); + let Some(image) = builtins_image_with_f32() else { + return; + }; + + let mut fb = FunctionBuilder::new("f", &[IrType::F32, IrType::F32]); + let x = fb.add_param(IrType::F32); + let _y = fb.add_param(IrType::F32); + let out = fb.alloc_vreg(IrType::F32); + fb.push(LpirOp::Ffloor { dst: out, src: x }); + fb.push_return(&[out]); + let ir = one_function_module(fb.finish()); + let sig = one_function_sig(vec![float_param("a"), float_param("b")], LpsType::Float); + + let r = expect_ok(run_f32_with_builtins( + &ir, + &sig, + "f", + &[0, bits(3.75), bits(0.0)], + &image, + )); assert_eq!(f32::from_bits(r), 3.0); } diff --git a/lp-xt/lp-xt-elf/src/loader.rs b/lp-xt/lp-xt-elf/src/loader.rs index 929688f36..a82c9e803 100644 --- a/lp-xt/lp-xt-elf/src/loader.rs +++ b/lp-xt/lp-xt-elf/src/loader.rs @@ -168,22 +168,23 @@ impl<'d> XtensaElf<'d> { /// Copy every `PT_LOAD` segment into the emulator's memory at its /// `p_vaddr`, zero-filling the `p_memsz` tail (`.bss`). Fails without /// side-channel panics if a segment falls outside the modeled regions. + /// + /// Goes through `Memory`'s **loader** path, not guest stores: an image's + /// `.text`/`.rodata` land in the read-only flash windows, where a guest + /// store faults by design. Placing an image there is what a flasher does. pub fn load_into(&self, emu: &mut Emulator) -> Result<(), ElfError> { for seg in self.segments()? { let unmapped = |_| ElfError::Unmapped { vaddr: seg.vaddr, memsz: seg.memsz, }; - for (i, &b) in seg.data.iter().enumerate() { - emu.mem - .write_u8(seg.vaddr.wrapping_add(i as u32), b) - .map_err(unmapped)?; - } - for i in seg.data.len() as u32..seg.memsz { - emu.mem - .write_u8(seg.vaddr.wrapping_add(i), 0) - .map_err(unmapped)?; - } + emu.mem + .try_load_bytes(seg.vaddr, seg.data) + .map_err(unmapped)?; + let tail = seg.memsz.saturating_sub(seg.data.len() as u32); + emu.mem + .try_zero(seg.vaddr.wrapping_add(seg.data.len() as u32), tail) + .map_err(unmapped)?; } Ok(()) } diff --git a/lp-xt/lp-xt-emu/README.md b/lp-xt/lp-xt-emu/README.md index 7cd941686..1a3b75392 100644 --- a/lp-xt/lp-xt-emu/README.md +++ b/lp-xt/lp-xt-emu/README.md @@ -110,6 +110,39 @@ out. Not modeled: classic's *word-only* data access to I-bus addresses (byte stores to SRAM0/I-bus fault EXCCAUSE=3 on hardware) — the emulator is more permissive there; the device-side writer honors the constraint. +### Flash — where firmware actually executes from + +The code region above is **SRAM**, and on an ESP32 that is not where firmware +lives. The application's `.text` executes from flash through the cache (XIP; a +boot log shows `vaddr=0x4200_0020 map` segments); only code a JIT produces at +runtime is in SRAM. So each profile also models two **read-only** flash +windows and a small internal-SRAM region for a flash-resident image's +`.data`/`.bss`: + +| | ESP32-S3 | classic ESP32 | +| --- | --- | --- | +| IROM (executable, `AliasRule::Identity`) | `0x4200_0000` | `0x400D_0000` | +| DROM (data only) | `0x3C00_0000` | `0x3F40_0000` | +| image `.data`/`.bss` (plain SRAM) | `0x3FCA_8000` | `0x3FFD_0000` | + +Guest stores into either flash window fault; the loader paths +(`load_bytes`/`load_region`) write them anyway, because placing an image in +flash is what a flasher does, not what the guest does. The **lengths** are the +model's, not hardware's (256 KiB / 64 KiB / 32 KiB against 32 MB of real +`irom_seg` on the S3) — regions are allocated per emulator, and one emulator is +built per host call. The **bases** are hardware, sourced in `board.rs` from +esp-hal's MIT/Apache-2.0 `ld/esp32{,s3}/memory.x` with in-repo corroboration, +and labelled *documented/observed* rather than measured, unlike the SRAM +numbers around them. + +Until 2026-08-01 the resident builtins image shared the SRAM code region with +JIT'd shader code. That coupled the largest compilable shader to the size of +the builtins — turning on the f32 builtin family left 931 bytes and took the +`xtn.q32` filetest suite from 849/849 files to 522/849 — and it hid the fact +that a shader→builtin call spans SRAM→flash, far outside `call8`'s ±512 KiB +direct reach. `tests/call_range.rs` pins both halves of that now. See +`docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md`. + ### D-bus / I-bus dual mapping The runner firmware writes payloads via the D-bus view and *executes them at @@ -130,10 +163,14 @@ globals, snapshot region, texture buffers — and reads results back out. The window is **data only**: it carries no `AliasRule`, so an instruction fetch from it faults (`EXC_INSTR_FETCH_ERROR`) exactly as jumping into a data region -does. `SHARED_DBUS_BASE` (`0x3F40_0000`) is the base both board profiles leave -free; `add_shared` **asserts** the range does not overlap any installed region, -in either its D-bus range or its I-bus image, so the choice cannot rot silently -as profiles change. +does. `SHARED_DBUS_BASE` (`0x3000_0000`) is below the lowest address either +chip's data bus decodes, so no future profile can grow into it; `add_shared` +**asserts** the range does not overlap any installed region, in either its +D-bus range or its I-bus image, so the choice cannot rot silently as profiles +change. It did rot once, which is how we know: the base used to be +`0x3F40_0000` on the weaker ground that no *profile* mapped it, and that +address turned out to be classic's DROM base the moment flash was modeled. The +assertion caught it. This address is **host-emulator fiction** — no ESP32 has a host-shared region, and the on-device JIT never needs one. See diff --git a/lp-xt/lp-xt-emu/src/board.rs b/lp-xt/lp-xt-emu/src/board.rs index 9dd124ead..c615e2f32 100644 --- a/lp-xt/lp-xt-emu/src/board.rs +++ b/lp-xt/lp-xt-emu/src/board.rs @@ -8,10 +8,27 @@ //! that; [`crate::Emulator::with_profile`] builds an emulator on any profile, //! and [`crate::Emulator::new`] keeps the ESP32-S3 profile as the default. //! -//! Every number below is hardware-measured, never recalled: the S3 map from -//! the original spike (FINDINGS E2, `fw/spike-esp32s3`), the classic map from -//! the C1–C5 ladder (FINDINGS "classic ESP32 (LX6)" section, `fw/spike-esp32`, -//! run 2026-07-28 on rev v3.0 silicon). +//! Every SRAM number below is hardware-measured, never recalled: the S3 map +//! from the original spike (FINDINGS E2, `fw/spike-esp32s3`), the classic map +//! from the C1–C5 ladder (FINDINGS "classic ESP32 (LX6)" section, +//! `fw/spike-esp32`, run 2026-07-28 on rev v3.0 silicon). +//! +//! The **flash** numbers are a weaker but still stated grade of evidence: +//! *documented and observed*, not probed by us. Each carries its source +//! inline — an MIT/Apache-2.0 linker script, an in-repo citation, or a boot +//! log. They are labelled as such rather than blended in with the measured +//! ones, because the difference matters when one of them turns out wrong. +//! +//! # Why flash is modeled at all +//! +//! On every ESP32 the application's code executes from **flash through the +//! cache** (XIP; a boot log shows `vaddr=0x4200_0020 map` segments). Only +//! code a JIT produces at runtime lives in SRAM. Collapsing the two into one +//! SRAM region — which this emulator did until 2026-08-01 — makes the +//! resident builtins image compete with JIT'd shader code for the same bytes, +//! and hides the fact that a shader→builtin call crosses from SRAM to flash, +//! tens of megabytes away and far outside any direct-call displacement. See +//! `docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md`. use crate::memory::{AliasRule, IBUS_ALIAS_OFFSET, Memory, SRAM1_DBUS_END, SRAM1_DBUS_START}; @@ -42,13 +59,66 @@ pub struct BoardProfile { pub dbus_window_start: u32, /// D-bus end (exclusive) of the dual-mapped window. pub dbus_window_end: u32, + /// Base of the flash **instruction** window (XIP `.text`), read-only and + /// directly executable — the flash cache presents instruction addresses + /// with no separate data view to alias from, so the region carries + /// [`AliasRule::Identity`]. + pub irom_base: u32, + /// Modeled length of the flash instruction window. See + /// [`Self::MODELED_IROM_LEN`] for why this is not the hardware length. + pub irom_len: usize, + /// Base of the flash **data** window (XIP `.rodata`), read-only and not + /// fetchable. + pub drom_base: u32, + /// Modeled length of the flash data window. + pub drom_len: usize, + /// Base of the internal-SRAM data region a flash-resident image's + /// `.data`/`.bss` live in — the emulator's `dram_seg`. Plain data RAM: no + /// I-bus view, so fetching there faults. + pub image_data_base: u32, + /// Length of that region. + pub image_data_len: usize, } impl BoardProfile { - /// ESP32-S3 (LX7). The numbers are the spike's measured map (FINDINGS E2): - /// SRAM1 dual-mapped D-bus `0x3FC8_8000..0x3FCF_0000`, executable alias a - /// constant `+0x6F_0000`. Code and stack are both SRAM1 regions, exactly - /// as [`crate::Emulator::new`] has always laid them out. + /// Modeled length of a flash instruction window. + /// + /// Hardware gives far more — 32 MB of `irom_seg` on the S3, 3 MB on + /// classic — but the emulator *allocates* its regions, and one is built per + /// host call, so the modeled window is sized to the job: the resident + /// builtins image plus generous headroom. The base addresses are hardware; + /// only the lengths are the model's, and a segment past the end is a loud + /// load error, never a silent wrap. + pub const MODELED_IROM_LEN: usize = 0x0004_0000; // 256 KiB + /// Modeled length of a flash data window. Same reasoning as + /// [`Self::MODELED_IROM_LEN`]. + pub const MODELED_DROM_LEN: usize = 0x0001_0000; // 64 KiB + /// Length of the internal-SRAM region holding a flash-resident image's + /// `.data`/`.bss`. The builtins image currently emits **neither** (its + /// only segments are `.text` and `.rodata`), so this is headroom for the + /// day one appears rather than space in use. + pub const IMAGE_DATA_LEN: usize = 0x0000_8000; // 32 KiB + + /// ESP32-S3 (LX7). The SRAM numbers are the spike's measured map (FINDINGS + /// E2): SRAM1 dual-mapped D-bus `0x3FC8_8000..0x3FCF_0000`, executable + /// alias a constant `+0x6F_0000`. Code and stack are both SRAM1 regions, + /// exactly as [`crate::Emulator::new`] has always laid them out. + /// + /// The flash numbers are **documented, not probed**: + /// + /// - IROM `0x4200_0000` — esp-hal 1.1.1 (MIT OR Apache-2.0), + /// `ld/esp32s3/memory.x`: `irom_seg ORIGIN = 0x42000020, len = 32M - 0x20` + /// (the `0x20` is an image-header convenience, not a hardware boundary). + /// Corroborated in-repo: `lpc-shared`'s backtrace validator already + /// accepts S3 text at `0x4200_0000..0x4400_0000` + /// (`lp-core/lpc-shared/src/backtrace.rs`), and S3 boot logs show + /// `vaddr=0x42000020 map`. + /// - DROM `0x3C00_0000` — same file, `drom_seg ORIGIN = 0x3C000020`. + /// + /// `image_data_base` `0x3FCA_8000` is the free SRAM1 span between the code + /// region (ends `0x3FCA_8000`) and the stack region (starts `0x3FCC_0000`) + /// inside the measured `0x3FC8_8000..0x3FCF_0000` window — internal data + /// RAM by the same measurement that placed the other two. pub fn esp32s3() -> BoardProfile { BoardProfile { name: "esp32s3", @@ -60,6 +130,12 @@ impl BoardProfile { alias: AliasRule::Offset(IBUS_ALIAS_OFFSET), dbus_window_start: SRAM1_DBUS_START, dbus_window_end: SRAM1_DBUS_END, + irom_base: 0x4200_0000, + irom_len: Self::MODELED_IROM_LEN, + drom_base: 0x3C00_0000, + drom_len: Self::MODELED_DROM_LEN, + image_data_base: 0x3FCA_8000, + image_data_len: Self::IMAGE_DATA_LEN, } } @@ -81,6 +157,18 @@ impl BoardProfile { /// stack comes from ordinary data RAM (C5 measured 98 304 B heap free, /// so a 64 KiB stack arena fits with headroom); modeling it as a plain /// region also reproduces "SRAM2 is NOT executable" (C2g). + /// + /// The flash numbers are **documented, not probed**, from esp-hal 1.1.1 + /// (MIT OR Apache-2.0), `ld/esp32/memory.x`: `irom_seg ORIGIN = 0x400D0020, + /// len = 3M - 0x20` and `drom_seg ORIGIN = 0x3F400020, len = 4M - 0x20` + /// (the `0x20` is an image-header convenience, not a hardware boundary). + /// Classic's IROM sits *above* SRAM1's I-bus window `0x400A_0000.. + /// 0x400C_0000`, so it neither overlaps the measured mirror nor disturbs + /// it — the word-mirrored alias is untouched by this profile's flash. + /// + /// `image_data_base` `0x3FFD_0000` is SRAM2 immediately above the stack + /// region, inside the same `dram_seg` (`0x3FFA_E000 + 192 KB`) the stack + /// was measured into. pub fn esp32() -> BoardProfile { BoardProfile { name: "esp32", @@ -95,6 +183,12 @@ impl BoardProfile { }, dbus_window_start: 0x3FFE_0000, dbus_window_end: 0x4000_0000, + irom_base: 0x400D_0000, + irom_len: Self::MODELED_IROM_LEN, + drom_base: 0x3F40_0000, + drom_len: Self::MODELED_DROM_LEN, + image_data_base: 0x3FFD_0000, + image_data_len: Self::IMAGE_DATA_LEN, } } @@ -120,10 +214,47 @@ impl BoardProfile { self.stack_dbus_base + self.stack_region_len as u32 - 16 } + /// Offset of `vaddr` within the SRAM code region, named through **either** + /// view — its D-bus address or its executable I-bus alias. + /// + /// This region belongs to JIT-produced code. A resident image landing here + /// is the bug the flash model exists to prevent, so consumers check. + pub fn code_region_offset(&self, vaddr: u32) -> Option { + offset_in(vaddr, self.code_dbus_base, self.code_region_len).or_else(|| { + offset_in( + self.alias.ibus_to_dbus(vaddr), + self.code_dbus_base, + self.code_region_len, + ) + }) + } + + /// Whether `vaddr` falls in the modeled flash instruction window, and its + /// offset there. + pub fn irom_offset(&self, vaddr: u32) -> Option { + offset_in(vaddr, self.irom_base, self.irom_len) + } + + /// Whether `vaddr` falls in the modeled flash data window, and its offset. + pub fn drom_offset(&self, vaddr: u32) -> Option { + offset_in(vaddr, self.drom_base, self.drom_len) + } + + /// Whether `vaddr` falls in the image's internal-SRAM data region, and its + /// offset. + pub fn image_data_offset(&self, vaddr: u32) -> Option { + offset_in(vaddr, self.image_data_base, self.image_data_len) + } + /// Install this profile's regions into `mem`, validating that anything /// dual-mapped actually sits inside the dual-mapped window (the /// profile-relative form of the assert `Memory::add_sram1` applies for /// the S3). + /// + /// Five regions: the SRAM code region (JIT'd code only — the resident + /// image is in flash), the stack, the image's SRAM `.data`/`.bss`, and the + /// two read-only flash windows. `Memory` asserts they are mutually + /// disjoint in both address views. pub fn install(&self, mem: &mut Memory) { self.assert_in_window("code", self.code_dbus_base, self.code_region_len); mem.add_executable(self.code_dbus_base, self.code_region_len, self.alias); @@ -133,6 +264,11 @@ impl BoardProfile { } else { mem.add_ram(self.stack_dbus_base, self.stack_region_len); } + mem.add_ram(self.image_data_base, self.image_data_len); + // Flash: read-only both ways. The instruction window is fetchable at + // its own addresses (Identity); the data window is not fetchable at all. + mem.add_rom(self.irom_base, self.irom_len, Some(AliasRule::Identity)); + mem.add_rom(self.drom_base, self.drom_len, None); } fn assert_in_window(&self, what: &str, base: u32, len: usize) { @@ -147,3 +283,8 @@ impl BoardProfile { ); } } + +/// Offset of `vaddr` within `[base, base + len)`, if it lands there. +fn offset_in(vaddr: u32, base: u32, len: usize) -> Option { + (vaddr >= base && (vaddr as u64) < base as u64 + len as u64).then(|| (vaddr - base) as usize) +} diff --git a/lp-xt/lp-xt-emu/src/memory.rs b/lp-xt/lp-xt-emu/src/memory.rs index 1e8a5f61d..002676f65 100644 --- a/lp-xt/lp-xt-emu/src/memory.rs +++ b/lp-xt/lp-xt-emu/src/memory.rs @@ -13,6 +13,12 @@ //! 0x3FFE_0000)` — the two windows run in opposite directions at word //! granularity, bytes within each word verbatim (C2b, 5 sentinels). //! +//! Not every region is SRAM. Firmware `.text` executes from **flash through +//! the cache** (XIP), which the address space exposes as two read-only windows +//! — an executable IROM window and a data-only DROM window — so +//! [`Memory::add_rom`] adds regions that fault on a store. See +//! [`crate::board::BoardProfile`] for the per-chip addresses. +//! //! Original code; no derivation from QEMU/binutils (see the repo license ADR). use std::sync::{Arc, Mutex}; @@ -28,12 +34,17 @@ use crate::error::{Trap, TrapKind}; /// like this; the address exists only so a host engine can hand the guest a /// pointer into host memory. /// -/// Chosen unmapped on **both** board profiles and outside the `0x4xxx_xxxx` -/// I-bus quadrant: the S3 installs SRAM1 code/stack at `0x3FC8_8000` / -/// `0x3FCC_0000` (I-bus images `0x4037_8000+`), and classic installs -/// `0x3FFE_8000` / `0x3FFC_0000` (I-bus image `0x400A_1000..0x400B_8000`). -/// `0x3F40_0000` is in the external-memory-mapped range on both chips, which no -/// profile models. +/// Chosen **reserved on both chips**, not merely unused by a profile: +/// `0x3000_0000` is below the lowest address either data bus decodes (the S3's +/// external-memory window opens at `0x3C00_0000`; classic's DROM0 opens at +/// `0x3F40_0000`), so no future profile can grow into it. +/// +/// It used to be `0x3F40_0000`, on the weaker ground that no *profile* mapped +/// it. That stopped being true the moment the profiles gained modeled flash +/// windows: `0x3F40_0000` **is** classic's DROM base +/// ([`crate::board::BoardProfile::esp32`]). The `add_shared` overlap assertion +/// caught it, which is exactly what it is for; the fix is a base that is not a +/// hardware address at all. /// /// It is deliberately **not** `lp_emu_core::DEFAULT_SHARED_START` (the rv32 /// engine's `0x4000_0000`): that address is [`crate::SENTINEL_PC`], the @@ -42,7 +53,7 @@ use crate::error::{Trap, TrapKind}; /// and silently undermine the "chosen unmapped" property that harness relies /// on. The two ISAs therefore use different shared bases, which costs nothing — /// the guest reaches the region only through a pointer argument. -pub const SHARED_DBUS_BASE: u32 = 0x3F40_0000; +pub const SHARED_DBUS_BASE: u32 = 0x3000_0000; /// ESP32-S3 SRAM1 D-bus window start (data view). pub const SRAM1_DBUS_START: u32 = 0x3FC8_8000; @@ -145,7 +156,6 @@ impl Region { /// it has no alias. Conservative: computed from the endpoints and rounded /// out to word boundaries, because a word-mirrored alias runs *downward* /// and so maps the region's low D-bus address to its image's high end. - /// Used only by [`Memory::add_shared`]'s overlap check. fn ibus_bounds(&self) -> Option<(u32, u32)> { let rule = self.alias?; let last = self.dbus_start.wrapping_add(self.data.len() as u32 - 1); @@ -153,6 +163,22 @@ impl Region { let b = rule.dbus_to_ibus(last); Some((a.min(b) & !3, (a.max(b) | 3))) } + + /// Every inclusive address range this region answers to, each with the name + /// of the view it comes from: its D-bus range, plus its I-bus image when it + /// has one. Used by the overlap checks — an address in *either* view + /// resolves to this region, so both count. + /// + /// Under [`AliasRule::Identity`] the two coincide and the duplicate is + /// harmless: the checks are pure comparisons. + fn address_views(&self) -> impl Iterator { + let dbus = ( + "D-bus range", + self.dbus_start, + self.dbus_start.wrapping_add(self.data.len() as u32 - 1), + ); + core::iter::once(dbus).chain(self.ibus_bounds().map(|(lo, hi)| ("I-bus image", lo, hi))) + } } /// A region whose bytes are owned by the **host**, mapped into the guest's data @@ -216,12 +242,7 @@ impl Memory { /// Add a plain read/write data region with no executable alias. pub fn add_ram(&mut self, dbus_start: u32, len: usize) { - self.regions.push(Region { - dbus_start, - alias: None, - data: vec![0u8; len], - writable: true, - }); + self.push_region(dbus_start, len, None, true); } /// Add a writable region whose bytes are also fetchable at the I-bus view @@ -229,12 +250,74 @@ impl Memory { /// board profile validates its regions against its own dual-mapped window /// (see [`crate::board::BoardProfile::install`]). pub fn add_executable(&mut self, dbus_start: u32, len: usize, rule: AliasRule) { - self.regions.push(Region { + self.push_region(dbus_start, len, Some(rule), true); + } + + /// Add a **read-only** region: guest loads and fetches behave as for any + /// other region, guest stores fault with [`EXC_LOAD_STORE_ERROR`]. + /// + /// This is how flash-resident firmware is modeled. On every ESP32 the + /// application's `.text` and `.rodata` execute and load *from flash through + /// the cache* (XIP), which the address space exposes as read-only windows; + /// a store there is a bug on hardware and is a bug here. + /// + /// `alias` is `Some(AliasRule::Identity)` for an executable (IROM) window — + /// flash instruction addresses are already the fetch addresses, there is no + /// separate data view to alias from — and `None` for a data-only (DROM) + /// window, which then faults on fetch exactly as any data region does. + /// + /// The loader paths ([`load_bytes`](Self::load_bytes), + /// [`load_region`](Self::load_region)) deliberately ignore the read-only + /// flag: placing an image into flash is what a flasher does, not what the + /// guest does. + pub fn add_rom(&mut self, dbus_start: u32, len: usize, alias: Option) { + self.push_region(dbus_start, len, alias, false); + } + + /// Install a region, asserting it overlaps no existing one — in either + /// address view. + /// + /// The board profiles now install five regions apiece across three address + /// quadrants (SRAM code, SRAM data, stack, flash IROM, flash DROM), and + /// [`resolve`](Self::resolve) is first-match-wins, so an overlap would not + /// fault — it would silently shadow. [`add_shared`](Self::add_shared) has + /// asserted this for its one window since it was added; the regions + /// themselves now get the same guarantee from the same check. + fn push_region( + &mut self, + dbus_start: u32, + len: usize, + alias: Option, + writable: bool, + ) { + assert!(len > 0, "region at {dbus_start:#x} is empty"); + let region = Region { dbus_start, - alias: Some(rule), + alias, data: vec![0u8; len], - writable: true, - }); + writable, + }; + for (view, lo, hi) in region.address_views() { + self.assert_free((lo, hi), "new region's", view); + } + self.regions.push(region); + } + + /// Assert `[lo, hi]` overlaps no installed region, in either view. + /// + /// `what`/`new_view` only name the range in the panic message, so they stay + /// `&str`: this runs on every region add, and an emulator is built per host + /// call. Nothing is formatted unless the assertion actually fires. + fn assert_free(&self, (lo, hi): (u32, u32), what: &str, new_view: &str) { + for r in &self.regions { + for (view, r_lo, r_hi) in r.address_views() { + assert!( + hi < r_lo || lo > r_hi, + "{what} {new_view} {lo:#x}..={hi:#x} overlaps the {view} \ + {r_lo:#x}..={r_hi:#x} of an installed region" + ); + } + } } /// S3 convenience: add a region backing the ESP32-S3 SRAM1 dual mapping — @@ -279,22 +362,7 @@ impl Memory { assert!(len > 0, "shared backing is empty"); let lo = dbus_start; let hi = dbus_start.wrapping_add(len as u32 - 1); - for r in &self.regions { - let r_lo = r.dbus_start; - let r_hi = r.dbus_start.wrapping_add(r.data.len() as u32 - 1); - assert!( - hi < r_lo || lo > r_hi, - "shared region {lo:#x}..={hi:#x} overlaps the D-bus range \ - {r_lo:#x}..={r_hi:#x} of an installed region" - ); - if let Some((i_lo, i_hi)) = r.ibus_bounds() { - assert!( - hi < i_lo || lo > i_hi, - "shared region {lo:#x}..={hi:#x} overlaps the I-bus image \ - {i_lo:#x}..={i_hi:#x} of an installed region" - ); - } - } + self.assert_free((lo, hi), "shared region", ""); self.shared = Some(SharedRegion { dbus_start, len, @@ -332,6 +400,18 @@ impl Memory { /// under a word-mirrored alias (where the backing D-bus image is not /// contiguous). pub fn load_bytes(&mut self, addr: u32, bytes: &[u8]) { + if let Err(a) = self.try_load_bytes(addr, bytes) { + panic!("load_bytes: address {a:#x} not mapped"); + } + } + + /// Fallible [`load_bytes`](Self::load_bytes): on failure returns the first + /// unmapped address instead of panicking. + /// + /// An image loader is given whatever addresses its ELF names, and "this + /// segment is not in the modeled map" is a diagnosable condition its caller + /// should report — not a panic from inside the memory model. + pub fn try_load_bytes(&mut self, addr: u32, bytes: &[u8]) -> Result<(), u32> { if let Some(s) = &self.shared { if let Some(idx) = s.index(addr) { let mut v = s.backing.lock().expect("shared backing lock"); @@ -342,16 +422,30 @@ impl Memory { bytes.len() ); v[idx..end].copy_from_slice(bytes); - return; + return Ok(()); } } for (i, b) in bytes.iter().enumerate() { let a = addr.wrapping_add(i as u32); - let (ri, idx) = self - .resolve(a, Access::Data) - .unwrap_or_else(|| panic!("load_bytes: address {a:#x} not mapped")); + let (ri, idx) = self.resolve(a, Access::Data).ok_or(a)?; self.regions[ri].data[idx] = *b; } + Ok(()) + } + + /// Zero `len` bytes at `addr` through the loader path (the `p_memsz` tail of + /// a `PT_LOAD` segment). Fallible for the same reason + /// [`try_load_bytes`](Self::try_load_bytes) is. + pub fn try_zero(&mut self, addr: u32, len: u32) -> Result<(), u32> { + const CHUNK: usize = 4096; + let zeros = [0u8; CHUNK]; + let mut done = 0u32; + while done < len { + let n = ((len - done) as usize).min(CHUNK); + self.try_load_bytes(addr.wrapping_add(done), &zeros[..n])?; + done += n as u32; + } + Ok(()) } /// Copy `bytes` into the single region that starts at `dbus_start`, in one diff --git a/lp-xt/lp-xt-emu/tests/board_profile.rs b/lp-xt/lp-xt-emu/tests/board_profile.rs index 25cf22d24..72d5e8870 100644 --- a/lp-xt/lp-xt-emu/tests/board_profile.rs +++ b/lp-xt/lp-xt-emu/tests/board_profile.rs @@ -163,6 +163,86 @@ fn s3_default_still_runs_gv1() { ); } +/// Both profiles model flash, and none of the five regions collide. +/// +/// `install` is what asserts disjointness, so building the emulator is the +/// test; the explicit checks below are about *which* region answers to what, +/// which a silent shadowing bug would get wrong without panicking. +#[test] +fn both_profiles_install_disjoint_flash_and_sram_regions() { + for p in [BoardProfile::esp32s3(), BoardProfile::esp32()] { + let emu = Emulator::with_profile(p); + + // Flash is readable and, for the instruction window, fetchable. + assert!( + emu.mem.read_u32(p.irom_base).is_ok(), + "{}: IROM read", + p.name + ); + assert!( + emu.mem.read_u32(p.drom_base).is_ok(), + "{}: DROM read", + p.name + ); + let mut out = [0u8; 3]; + assert!( + emu.mem.fetch(p.irom_base, &mut out).is_ok(), + "{}: IROM must be executable at its own address", + p.name + ); + assert_eq!( + emu.mem.fetch(p.drom_base, &mut out).unwrap_err().cause, + EXC_INSTR_FETCH_ERROR, + "{}: DROM is data only", + p.name + ); + + // The image's .data/.bss region is plain SRAM: writable, not fetchable. + let mut emu = emu; + assert!( + emu.mem.write_u32(p.image_data_base, 0x1234).is_ok(), + "{}: image DRAM write", + p.name + ); + assert_eq!(emu.mem.read_u32(p.image_data_base).unwrap(), 0x1234); + assert_eq!( + emu.mem + .fetch(p.image_data_base, &mut out) + .unwrap_err() + .cause, + EXC_INSTR_FETCH_ERROR, + "{}: image DRAM is not executable", + p.name + ); + + // No flash address is mistaken for the JIT's code region, in either view. + assert!(p.code_region_offset(p.irom_base).is_none(), "{}", p.name); + assert!(p.code_region_offset(p.drom_base).is_none(), "{}", p.name); + assert!( + p.code_region_offset(p.code_ibus_base()).is_some(), + "{}: the code region must recognise its own I-bus base", + p.name + ); + } +} + +/// The flash bases are the ones the linker script and `lpc-shared`'s backtrace +/// validator already use. Pinned so a profile edit cannot drift away from the +/// image it has to load. +#[test] +fn flash_bases_match_the_documented_chip_maps() { + let s3 = BoardProfile::esp32s3(); + assert_eq!(s3.irom_base, 0x4200_0000, "S3 irom_seg (esp-hal memory.x)"); + assert_eq!(s3.drom_base, 0x3C00_0000, "S3 drom_seg (esp-hal memory.x)"); + + let classic = BoardProfile::esp32(); + assert_eq!(classic.irom_base, 0x400D_0000, "classic irom_seg"); + assert_eq!(classic.drom_base, 0x3F40_0000, "classic drom_seg"); + + // Classic's IROM begins above SRAM1's I-bus window rather than inside it. + assert!(classic.irom_base >= 0x400C_0000); +} + /// Offset and Identity rules stay trivial. #[test] fn offset_and_identity_rules() { diff --git a/lp-xt/lp-xt-emu/tests/call_range.rs b/lp-xt/lp-xt-emu/tests/call_range.rs new file mode 100644 index 000000000..bcf0d794d --- /dev/null +++ b/lp-xt/lp-xt-emu/tests/call_range.rs @@ -0,0 +1,210 @@ +//! Calling **from SRAM into flash** — the reach a JIT'd shader needs, and the +//! reach a direct call does not have. +//! +//! On hardware, firmware `.text` executes from flash through the cache and +//! JIT-produced code lives in SRAM, tens of megabytes apart. A `CALL8` encodes +//! an 18-bit signed word displacement — ±512 KiB — so it *cannot* express that +//! jump, which is why `isa::xt`'s emitter reaches every call target through a +//! literal-pool slot and `CALLX8`. +//! +//! While the emulator modeled both in one 128 KiB region, that constraint was +//! invisible: an accidentally-direct call would have been in range on the host +//! and out of range on silicon — passing tests, failing hardware. These tests +//! pin both halves now that the two live where they live on the device. + +use lp_xt_emu::board::BoardProfile; +use lp_xt_emu::emu::CallOutcome; +use lp_xt_emu::memory::EXC_INSTR_FETCH_ERROR; +use lp_xt_emu::{Emulator, NoopTracer, TrapKind}; +use lp_xt_inst::{CallOp, Inst, NullaryOp, Reg, encode}; + +/// Callee frame size. Any multiple of 8 works. +const FRAME: u32 = 32; +/// What the flash-resident callee returns. Fits `movi`'s 12-bit signed +/// immediate, so the callee is four instructions with no literal pool of its +/// own — the value only has to be distinguishable from zero and from a wrapped +/// address. +const ANSWER: u32 = 0x7EF; + +fn a(n: u8) -> Reg { + Reg::new(n) +} + +fn asm(insts: &[Inst]) -> Vec { + insts.iter().flat_map(encode).collect() +} + +/// A windowed callee: `entry a1,32; movi a2,ANSWER; retw`. Placed in flash. +fn flash_callee() -> Vec { + asm(&[ + Inst::Entry(a(1), FRAME), + Inst::Movi(a(2), ANSWER as i32), + Inst::Nullary(NullaryOp::Retw), + ]) +} + +/// The word displacement a `CALL` at `pc` would need to reach `target`, per the +/// executor's own rule: `target = (pc & !3) + (offset << 2) + 4`. +fn call_displacement(pc: u32, target: u32) -> i64 { + (target as i64 - (pc & !3) as i64 - 4) / 4 +} + +/// The 18-bit signed field a `CALL` carries. +const CALL_OFFSET_MIN: i64 = -(1 << 17); +const CALL_OFFSET_MAX: i64 = (1 << 17) - 1; + +/// The range fact, stated as arithmetic on the real addresses — and it is +/// **not the same fact on both boards**, which is the more useful half. +/// +/// On the S3, IROM (`0x4200_0000`) is ~29 MB above the SRAM1 I-bus window: a +/// direct call cannot name it, by three orders of magnitude. +/// +/// On classic, flash IROM (`0x400D_0000`) sits directly above the SRAM1 I-bus +/// window (`0x400A_0000..0x400C_0000`) — ~192 KiB away, comfortably inside the +/// ±512 KiB field. A direct call from JIT'd code to a builtin would *work* +/// there. That is a trap for anyone who "optimizes" the indirect form away +/// after testing on one board, so it is pinned rather than left to be +/// rediscovered: the emitter must stay indirect because the S3 requires it, not +/// because every Xtensa target does. +#[test] +fn only_the_s3_puts_flash_out_of_direct_call_range() { + let s3 = BoardProfile::esp32s3(); + let d = call_displacement(s3.code_ibus_base(), s3.irom_base); + assert!( + d > CALL_OFFSET_MAX, + "S3: CALL8 to IROM {:#010x} needs displacement {d}, which fits the \ + 18-bit field — the range constraint this suite exists for would not bite", + s3.irom_base, + ); + assert!( + d.unsigned_abs() > 1_000_000, + "S3: displacement {d} is suspiciously close to in-range" + ); + + let classic = BoardProfile::esp32(); + let d = call_displacement(classic.code_ibus_base(), classic.irom_base); + assert!( + (CALL_OFFSET_MIN..=CALL_OFFSET_MAX).contains(&d), + "classic: IROM {:#010x} was expected within direct-call reach of the \ + SRAM1 I-bus window (displacement {d}); if the map moved, the comment \ + above needs rewriting, not this assertion relaxing", + classic.irom_base, + ); +} + +/// Executed rather than computed: a `CALL8` whose offset field holds the +/// truncated displacement lands somewhere that is **not** the callee. +/// +/// This is the shape of the bug the old single-region model hid. The +/// displacement does not fit, so the field wraps; the emulator jumps where the +/// encoding actually says, which is unmapped, and the run traps on fetch. +#[test] +fn a_direct_call8_toward_flash_traps_instead_of_arriving() { + let p = BoardProfile::esp32s3(); + let mut emu = Emulator::with_profile(p); + + let callee = p.irom_base + 0x100; + emu.mem.load_bytes(callee, &flash_callee()); + + let entry = p.code_ibus_base(); + // `entry a1,32` is 3 bytes, so the CALL8 sits at entry+3. + let call_pc = entry + 3; + let wanted = call_displacement(call_pc, callee); + assert!( + wanted > CALL_OFFSET_MAX, + "expected the displacement to overflow the field, got {wanted}" + ); + // What the encoder actually emits: the low 18 bits, sign-extended on decode. + let truncated = ((wanted as i32) << 14) >> 14; + let code = asm(&[ + Inst::Entry(a(1), FRAME), + Inst::Call(CallOp::Call8, truncated), + Inst::Nullary(NullaryOp::Retw), + ]); + emu.mem.load_bytes(entry, &code); + + match emu.run_loaded_with_args(entry, &[], &mut NoopTracer, None) { + CallOutcome::Ok { lo, .. } => panic!( + "a direct CALL8 must not reach a flash callee, but the run returned \ + {lo:#x} (callee returns {ANSWER:#x})" + ), + CallOutcome::Trap(t) => { + assert_eq!(t.kind, TrapKind::Exception); + assert_eq!( + t.cause, EXC_INSTR_FETCH_ERROR, + "the wrapped target should be unmapped, not merely wrong" + ); + } + } +} + +/// The form the JIT actually emits — `l32r` a literal holding the absolute +/// address, then `callx8` — reaches the same flash callee and returns its value. +/// +/// `l32r` is backward-only, so the literal precedes the code that loads it; +/// that is exactly how `isa::xt`'s emitter lays out its pools. +#[test] +fn an_indirect_callx8_reaches_a_flash_callee() { + let p = BoardProfile::esp32s3(); + let mut emu = Emulator::with_profile(p); + + let callee = p.irom_base + 0x100; + emu.mem.load_bytes(callee, &flash_callee()); + + // Literal at the code region base, entry one word later. + let lit = p.code_ibus_base(); + emu.mem.load_bytes(lit, &callee.to_le_bytes()); + let entry = lit + 4; + + // `l32r a8, ` — the field is the one's-complement word distance the + // disassembler's own helper computes, so build it by search rather than by + // re-deriving the formula here. + let l32r_pc = entry + 3; // after `entry a1,32` + let field = (0u16..=0xFFFF) + .find(|&f| lp_xt_inst::disasm::l32r_target(l32r_pc, f) == lit) + .expect("the literal is within l32r's backward reach"); + + // `callx8` rotates by 8, so the callee's `a2` result arrives in *our* `a10`; + // move it to `a2` so our own `retw` returns it. + let code = asm(&[ + Inst::Entry(a(1), FRAME), + Inst::L32r(a(8), field), + Inst::Callx(lp_xt_inst::CallxOp::Callx8, a(8)), + Inst::Addi(a(2), a(10), 0), + Inst::Nullary(NullaryOp::Retw), + ]); + emu.mem.load_bytes(entry, &code); + + match emu.run_loaded_with_args(entry, &[], &mut NoopTracer, None) { + CallOutcome::Ok { lo, .. } => assert_eq!(lo, ANSWER), + CallOutcome::Trap(t) => panic!("indirect call into flash trapped: {t:?}"), + } +} + +/// Flash is read-only: a guest store into either window faults, as it does on +/// hardware, while the loader path (a flasher, not the guest) may write it. +#[test] +fn guest_stores_into_flash_fault() { + let p = BoardProfile::esp32s3(); + let mut emu = Emulator::with_profile(p); + + assert!( + emu.mem.write_u32(p.irom_base, 0xDEAD).is_err(), + "IROM store" + ); + assert!( + emu.mem.write_u32(p.drom_base, 0xDEAD).is_err(), + "DROM store" + ); + + // Loads work in both windows; a fetch works only in the instruction one. + emu.mem.load_bytes(p.drom_base, &0xC0DEu32.to_le_bytes()); + assert_eq!(emu.mem.read_u32(p.drom_base).unwrap(), 0xC0DE); + let mut out = [0u8; 3]; + assert!(emu.mem.fetch(p.irom_base, &mut out).is_ok(), "IROM fetch"); + assert_eq!( + emu.mem.fetch(p.drom_base, &mut out).unwrap_err().cause, + EXC_INSTR_FETCH_ERROR, + "DROM is data only" + ); +} diff --git a/lp-xt/lps-builtins-xt-app/link.ld b/lp-xt/lps-builtins-xt-app/link.ld index dee2cc7d5..7c608b6bc 100644 --- a/lp-xt/lps-builtins-xt-app/link.ld +++ b/lp-xt/lps-builtins-xt-app/link.ld @@ -1,45 +1,63 @@ /* Linker script for the Xtensa builtins image. * - * Same address model as fixtures/link.ld (see its header for the full - * rationale) — the emulator's 128 KiB code region at D-bus - * 0x3FC88000..0x3FCA8000, I-bus alias +0x6F0000 — but with a **different - * split of that region**. + * This image is **firmware**, and firmware on an ESP32 executes from flash + * through the cache (XIP). So it links exactly where a real app's flash + * segments go, and nowhere near the SRAM the JIT writes into: * - * The fixtures script splits it evenly (64 KiB text / 64 KiB data), which - * matches lp-xt-elf's `DATA_BASE = CODE_DBUS_BASE + CODE_REGION_LEN/2` - * convention its reloc engine relies on. This image is the opposite shape: - * it is nearly all code (every `__lps_*` builtin — ~71 KiB, which overflows - * a 64 KiB text region) and carries almost no data. So it takes 112 KiB of - * text and 16 KiB of data, staying inside the same 128 KiB region. + * .text -> IROM, the flash instruction window (read-only, XIP) + * .rodata -> DROM, the flash data window (read-only) + * .data/.bss -> DRAM, internal SRAM * - * Nothing hardcodes this split: `rt_emu_xt` loads PT_LOAD segments at their - * p_vaddr straight out of this ELF, so the addresses come from the image - * itself. The fixtures' even split and the reloc engine's DATA_BASE are - * untouched. + * The addresses are lp-xt-emu's BoardProfile::esp32s3() — see board.rs for the + * provenance of each (esp-hal's MIT/Apache-2.0 ld/esp32s3/memory.x, plus + * in-repo corroboration). Lengths here are the *modeled* window lengths, not + * the hardware ones (hardware gives 32 MB of each flash window); a segment + * that outgrew them would fail to link here rather than fail to load later. + * + * Contrast fixtures/link.ld, which links its guests into the SRAM code region: + * those are raw payloads a runner writes into RAM, which is a different thing + * from a flash-resident image and stays as it is. + * + * No boot-time copy machinery, and LMA = VMA throughout: the emulator's loader + * writes each PT_LOAD segment straight to its p_vaddr, the way a flasher plus a + * configured cache MMU would leave memory. There is no second-stage bootloader + * here to copy .data out of flash, which is why .data/.bss get an SRAM VMA + * directly. + * + * Nothing downstream hardcodes this split: `rt_emu::xt_image` classifies each + * segment by its p_vaddr against the board profile, so moving a section here + * moves it there. */ ENTRY(_start) MEMORY { - IRAM (x) : ORIGIN = 0x40378000, LENGTH = 112K /* code region bytes 0..112K */ - DRAM (rw) : ORIGIN = 0x3FCA4000, LENGTH = 16K /* code region bytes 112K..128K */ + /* Flash, mapped through the cache. Read-only in the emulator's model, as on + * hardware. 256K/64K are BoardProfile::MODELED_IROM_LEN / MODELED_DROM_LEN. */ + IROM (rx) : ORIGIN = 0x42000000, LENGTH = 256K + DROM (r) : ORIGIN = 0x3C000000, LENGTH = 64K + + /* Internal SRAM between the emulator's code region (ends 0x3FCA8000) and its + * stack region (starts 0x3FCC0000) — BoardProfile::image_data_base. */ + DRAM (rw) : ORIGIN = 0x3FCA8000, LENGTH = 32K } SECTIONS { - /* l32r is backward-only, so literal pools must precede the text using them. */ + /* l32r is backward-only, so literal pools must precede the text using them. + * Both live in IROM: a literal is loaded from the instruction stream. */ .text : ALIGN(4) { *(.literal .literal.*) *(.text .text.*) - } > IRAM + } > IROM .rodata : ALIGN(4) { *(.rodata .rodata.*) *(.srodata .srodata.*) - } > DRAM + } > DROM .data : ALIGN(4) { diff --git a/scripts/build-builtins-xt.sh b/scripts/build-builtins-xt.sh index df43f7825..307d06332 100755 --- a/scripts/build-builtins-xt.sh +++ b/scripts/build-builtins-xt.sh @@ -29,6 +29,11 @@ if [[ -z "$NM" ]]; then echo "error: xtensa-esp32s3-elf-nm not found alongside the gcc above" >&2 exit 1 fi +READELF="$(command -v xtensa-esp32s3-elf-readelf)" +if [[ -z "$READELF" ]]; then + echo "error: xtensa-esp32s3-elf-readelf not found alongside the gcc above" >&2 + exit 1 +fi OUT_DIR="$ROOT/lp-xt/fixtures/elf" OUT="$OUT_DIR/lps-builtins-xt-app.elf" @@ -42,28 +47,27 @@ export CARGO_TARGET_DIR="$PWD/target" # and staleness is not harmless here — on esp Rust 1.88 `lpc-model` genuinely # fails to compile (70x E0716 from the Slotted derive's const-promotion of a # temporary), which is what blocked the firmware crates. Fail loudly instead. -# The native-f32 builtin family is **not** built in by default, and that is a -# blocker rather than a preference: `lps-builtins/float-f32` does not currently -# compile for Xtensa at all. The esp Rust backend fails with -# rustc-LLVM ERROR: Cannot select: XtensaISD::PCREL_WRAPPER -# TargetConstantPool <[2 x float] [0.0, -1.0]> -# at every optimization level, on a `[f32; 2]` gradient table in the -# generative-noise builtins. Full record and candidate fixes: -# docs/defects/2026-08-01-xtensa-backend-cannot-select-float-constant-pool.md # -# Passing the feature unconditionally would take the *whole* Xtensa filetest -# suite down with it (`scripts/filetests.sh` builds this image before running -# xtn/xtlpn), so it is opt-in until the defect is resolved. The crate-level -# feature is already wired (`lps-builtins-xt-app/Cargo.toml`), so closing this -# is one environment variable and then one default flip. -F32_REQUESTED="${LP_XT_BUILTINS_F32:-0}" -if [[ "$F32_REQUESTED" == "1" ]]; then - BUILD_CMD=(cargo build --release -p lps-builtins-xt-app --features float-f32) -else - BUILD_CMD=(cargo build --release -p lps-builtins-xt-app) -fi - -if ! "${BUILD_CMD[@]}"; then +# The native-f32 builtin family is **unconditional**. It was opt-in twice, for +# two different reasons, and both are now closed: +# +# 1. `lps-builtins/float-f32` did not compile for Xtensa at all — the backend +# cannot select a float constant pool. Worked around on our side, with a +# bit-equivalence test, in lps-builtins' rgb2hsv_f32.rs. +# docs/defects/2026-08-01-xtensa-backend-cannot-select-float-constant-pool.md +# 2. It then did not *fit*: link.ld gave .text 112 KiB of the emulator's 128 KiB +# code region and `rt_emu::xt_image` placed compiled shader code in whatever +# was left after it — 931 bytes with float-f32 in, which dropped the xtn.q32 +# suite from 849/849 files to 522/849. That was an artifact of modeling +# flash-resident firmware as if it lived in SRAM. It doesn't, and now it +# doesn't here either: the image links into IROM/DROM, and the whole SRAM +# code region is the shader's. +# docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md +# +# This is load-bearing beyond the f32 path: scripts/filetests.sh builds this +# image before running the xtn/xtlpn targets, so anything wrong here takes the +# whole Xtensa filetest suite down with it. +if ! cargo build --release -p lps-builtins-xt-app --features float-f32; then echo >&2 echo "error: build failed. If that was an MSRV error, the esp toolchain is stale:" >&2 echo " installed: $(rustc +esp --version 2>/dev/null || echo unknown)" >&2 @@ -85,13 +89,29 @@ fi # The f32 family is what M7's hardware-float lowering calls for everything it # does not inline — divide, sqrt, the rounding family, min/max, the saturating # conversions and every transcendental (M7 D4). An image without those symbols -# cannot resolve any of it, so the count is reported either way and asserted -# when the family was actually requested (mirroring the rv32 script, where a -# missing family fails 800+ filetests with one opaque message). +# cannot resolve any of it (mirroring the rv32 script, where a missing family +# fails 800+ filetests with one opaque message). F32_COUNT="$("$NM" "$OUT" | grep -c '_f32$' || true)" -if [[ "$F32_REQUESTED" == "1" && "$F32_COUNT" -eq 0 ]]; then +if [[ "$F32_COUNT" -eq 0 ]]; then echo "error: $OUT contains no native-f32 builtins despite --features float-f32" >&2 exit 1 fi + +# The image is firmware: .text executes from flash (IROM), .rodata is read from +# flash (DROM), and neither may land in the SRAM code region the JIT owns. A +# link.ld edit that put them back would otherwise surface as a loader error far +# from here, so check the segment addresses at the source. +SEGS="$("$READELF" -lW "$OUT" | awk '$1 == "LOAD" { print $3 }')" +for VADDR in $SEGS; do + case "$VADDR" in + 0x42*|0x3c0*|0x3C0*|0x3fca*|0x3FCA*|0x3fcb*|0x3FCB*) ;; + *) + echo "error: $OUT has a PT_LOAD segment at $VADDR, outside the modeled" >&2 + echo " flash windows (IROM 0x42000000, DROM 0x3c000000) and image" >&2 + echo " DRAM (0x3fca8000). Check lp-xt/lps-builtins-xt-app/link.ld." >&2 + exit 1 + ;; + esac +done SIZE="$(wc -c < "$OUT" | xargs)" echo "lps-builtins-xt-app: $COUNT builtins ($F32_COUNT native-f32), ${SIZE} B -> ${OUT#"$ROOT"/}"