refactor(xt): model flash — the builtins image is firmware, not JIT output - #247
Merged
Conversation
…pool The esp Xtensa backend cannot select `XtensaISD::PCREL_WRAPPER` over a `TargetConstantPool`, so any `[N x float]` constant LLVM materialises as a constant-pool entry fails to compile for `xtensa-esp32s3-none-elf` at every optimisation level. `lps-builtins/float-f32` tripped it in `rgb2hsv`, whose `p.zw` constant lane pair LLVM emits as `[2 x float] [0.0, -1.0]`. Select the integer bit patterns and bitcast after the fact, so the constants never reach the float constant pool. `f32::to_bits` is const-evaluated, so the values are the same floats the literals were — pinned lane-by-lane against the literal form it replaced by a new test, since these builtins are shared with rv32 and wasm where the original compiles fine. The defect's guess that snoise2's gradient LUT was the trigger was wrong; it and every other suspect (`worley2/3`, `gnoise3_tile`, `psrdnoise2/3`, `snoise3`) compile unchanged. One site was enough. Defect: docs/defects/2026-08-01-xtensa-backend-cannot-select-float-constant-pool.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With the constant-pool workaround in place the image builds, so drop the LP_XT_BUILTINS_F32 opt-in P4 added as a stopgap and assert the f32 symbol count unconditionally, matching scripts/build-builtins.sh (rv32). Un-ignore xt_pipeline_f32's builtin-routing case. It needed more than the ignore attribute removed: the file's link_blob resolves only the module's own functions and loads at the code region's base, where the builtins image already lives, so a sym_call to __lp_lpir_ffloor_f32 could never resolve. Added a builtins-linked run path with rt_emu::xt_image's layout — image .text first, shader 4-aligned after it — hand-rolled rather than calling build_xt_image, which sits behind the `emu` feature the file's other 15 tests do not need. 16/16 green, ffloor(3.75) == 3.0 from the real builtin. Record what the defect actually was: one site (rgb2hsv), not the LUT family, and the reported constant belonged to a different function than the one that failed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…raint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…onger fits Reverts the gate flip from the previous commit, on measurement. Making --features float-f32 the default took the xtn.q32 filetest suite from 849/849 files to 522/849, and the failures are link failures, not wrong answers. Cause is capacity, not codegen. link.ld gives .text 112 KiB (114,688 B) of the emulator's 128 KiB code region. Q32-only .text is 66,300 B; with float-f32 it is 113,757 B — the image links with 931 bytes to spare, and rt_emu::xt_image places compiled shader code in exactly that gap. Small shaders fit, everything else fails to link. That is a second, independent defect from the constant-pool one, and fixing the first is what exposed it: filed as docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md with the measurements and three candidate resolutions. So the family stays behind LP_XT_BUILTINS_F32=1, and the un-ignored builtin-routing test skips loudly when the resident image lacks the f32 symbols rather than failing — one blocked capability should not redden a suite. Verified both ways: skips on the default image, and passes (ffloor(3.75) == 3.0 through the real __lp_lpir_ffloor_f32) on an image built with LP_XT_BUILTINS_F32=1. The constant-pool defect's record is corrected to say the workaround shipped and the build it blocked now succeeds, while noting the default flip is blocked elsewhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On every ESP32 the application's code executes from flash through the
cache (XIP); only JIT-produced code lives in SRAM. The emulator collapsed
both into one SRAM code region, so a resident image competed with JIT'd
code for the same bytes and a shader->builtin call looked local when on
silicon it spans tens of megabytes.
BoardProfile gains an executable flash window (Identity alias, read-only),
a data-only flash window, and the internal-SRAM region an image's
.data/.bss belongs in. Bases are hardware, sourced from esp-hal 1.1.1's
MIT/Apache-2.0 ld/esp32{,s3}/memory.x and corroborated in-repo; lengths
are the model's, since regions are allocated per host call.
Memory gains add_rom (stores fault), a fallible loader write path
(try_load_bytes/try_zero, since flash is not guest-writable), and a
mutual-disjointness assertion on every region rather than only on the
shared window. SHARED_DBUS_BASE moves 0x3F40_0000 -> 0x3000_0000: the old
base IS classic's DROM base, which that assertion caught.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
.text -> IROM (0x42000000), .rodata -> DROM (0x3c000000), .data/.bss -> internal SRAM. LMA = VMA and no boot-time copy machinery: the emulator's loader writes each PT_LOAD straight to its p_vaddr, the way a flasher plus a configured cache MMU would leave memory. rt_emu/xt_image places every segment by classifying its p_vaddr against the board profile, so the SRAM code region is now entirely the shader's (128 KiB S3 / 92 KiB classic, unchanged) and the largest compilable shader no longer depends on the size of the builtins. GuestImage grows a `regions` list for the image's flash/SRAM buffers, each trimmed to the bytes in use since they are reloaded on every host call. float-f32 becomes unconditional for the image: the capacity objection was an artifact of modeling flash-resident firmware as if it shared SRAM with the JIT. The build script now also asserts the segment addresses, so a link.ld regression fails at the source rather than as a loader error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A CALL8 carries an 18-bit signed word displacement (+/-512 KiB), so on the S3 it cannot name a flash builtin from the SRAM code region — 29 MB away — which is why the emitter reaches every target through a literal slot and CALLX8. While both lived in one region that constraint was invisible: an accidentally-direct call would have passed on the host and failed on silicon. Four tests: the displacement arithmetic, an executed direct CALL8 that traps instead of arriving, the JIT's indirect form arriving, and flash being read-only to the guest but writable through the loader path. Recorded rather than assumed: on classic ESP32 flash IROM (0x400d0000) sits ~192 KiB above the SRAM1 I-bus window, so a direct call there IS in range. The emitter must stay indirect because the S3 requires it, not because every Xtensa target does — pinned so it is not rediscovered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…usion The defect presented as capacity and was not: the emulator modeled flash-resident firmware as if it lived in SRAM, so the builtins image and JIT'd shader code contended for bytes they never share on hardware. All three candidate resolutions it proposed (widen the region, split the region, split the feature) would have preserved the wrong map. New defect class `model-conflation` for exactly that shape, plus the two 2026-08-01 entries the index was missing. Corrected classic-ESP32 conclusion, recorded because the old entry invited the wrong one: 92 KiB bounds JIT'd shader code only. Classic f32 viability is a flash-budget plus unprobed-LX6-FPU question, not an SRAM one. Also: lp-xt-emu README gains a flash section, the shared-memory ADR is amended for the moved base (its own text predicted the rot), and the isa-parameterized-engine ADR's "~28 KiB shader budget" follow-up is closed with a note that its proposed fix was the wrong one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
builtins_land_inside_the_modeled_code_region asserted exactly what is now wrong. Inverted: every builtin must be in the modeled IROM window and none may be in the SRAM code region, which belongs to JIT'd shader code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… try_zero assert_free took a formatted String per view, on a path that runs five times per emulator and an emulator is built per host call. The labels are &str now, so nothing is formatted unless the assertion fires. try_zero delegates to try_load_bytes in chunks rather than duplicating its shared-region handling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Records why the builtins image moved out of the SRAM code region and why the three alternatives the defect proposed (widen the region, split the region, split the f32 feature) were all rejected: each preserves a map that does not match the device. Also records the graded provenance rule for the new addresses (bases are hardware from esp-hal's MIT/Apache-2.0 memory.x, lengths are the model's) and the classic-ESP32 finding that a direct CALL8 to flash IS in range there, so the emitter's indirection is an S3 requirement rather than a universal Xtensa one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Yona-Appletree
changed the base branch from
claude/f32-xtensa-fp-constpool-workaround
to
main
August 1, 2026 16:03
Yona-Appletree
marked this pull request as ready for review
August 1, 2026 16:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges after #246 (stacked on
claude/f32-xtensa-fp-constpool-workaround).claude/*-based PRs get no CI until retargeted tomain— only the CLA check runs here, so the gates below were run locally.The change
On real ESP32 silicon, firmware
.textexecutes from flash through the cache (XIP; boot logs showvaddr=0x4200_0020 map). Only JIT-produced code lives in SRAM. The host emulator collapsed both into one modeled SRAM code region, with three consequences:float-f32left 931 bytes for shader code and tookxtn.q32from 849/849 files to 522/849.call8's ±512 KiB. An accidentally-direct call would pass emulation and fail on hardware.So the model now matches the hardware:
BoardProfilegains read-only IROM/DROM windows plus an internal-SRAM region for an image's.data/.bss, the builtins image links as flash-resident firmware, and the whole SRAM code region is the shader's (128 KiB S3 / 92 KiB classic — unchanged in size; the point is that nothing is resident in front of it).--features float-f32is now unconditional for the Xtensa builtins image andLP_XT_BUILTINS_F32is gone.Address map
0x4200_0000ld/esp32s3/memory.xirom_seg; corroborated bylpc-shared's backtrace validator (0x4200_0000..0x4400_0000) and S3 boot logs0x3C00_0000drom_seg0x3FCA_80000x400D_0000ld/esp32/memory.xirom_seg0x3F40_0000drom_seg0x3FFD_0000dram_seg), above the measured stack regionLengths are the model's (256 KiB IROM / 64 KiB DROM / 32 KiB DRAM), not hardware's — regions are allocated per host call. Every flash number is labelled documented/observed in
board.rs, not silicon-probed like the SRAM ones around it. No GPL source consulted.SHARED_DBUS_BASEmoved0x3F40_0000→0x3000_0000: the old base is classic's DROM base, which the existingadd_sharedoverlap assertion caught. The shared-memory ADR is amended accordingly — its own text predicted exactly this rot.Image after the change:
.text113,757 B at0x4200_0000(43% of the modeled IROM),.rodata16,156 B at0x3C00_0000(25% of DROM), no.data/.bss.Oracles (all green, none regenerated)
scripts/filetests.sh -t xtn.q32scripts/filetests.sh -t xtlpn.q32cargo test -p lp-xt-emu(incl.fp_conformance)lpvm-native/tests/xt_pipeline_f32.rsa_builtin_routed_float_op_resolves_and_runsagainst IROM addressesjust test-xt-hostjust checkjust testFidelity test landed
lp-xt/lp-xt-emu/tests/call_range.rs— a directCALL8toward a flash builtin traps instead of arriving, the JIT'sl32r+callx8form arrives, and flash is read-only to the guest but writable through the loader path.It also recorded a finding worth having: on classic ESP32, IROM sits only ~192 KiB above the SRAM1 I-bus window, comfortably inside
CALL8's range — a direct call would work there. The emitter must stay indirect because the S3 requires it, not because every Xtensa target does. Pinned so it is not "optimized away" after testing on one board.Docs
docs/defects/2026-08-01-xt-f32-builtins-exhaust-the-emulator-code-region.md→fixed, with the corrected classic-ESP32 conclusion (92 KiB bounds shader JIT code only; classic f32 viability is a flash-budget + unprobed-LX6-FPU question, not an SRAM one) and a newmodel-conflationdefect class for the shape.docs/adr/2026-08-01-host-emulator-models-flash.md— new, records why the three alternatives the defect proposed all preserve the wrong map.2026-07-30-xtensa-host-shared-memoryamended;2026-07-30-isa-parameterized-host-emu-engine's "~28 KiB shader budget" follow-up closed with a note that its proposed fix was the wrong one.