perf(lpc-engine): persist dataflow resolution across frames

docs/adr/2026-07-28-esp32c6-flash-budget.md

@@ -179,6 +179,18 @@ rather than a cliff discovered by a deploy job.
 - Growth discipline moves to the CI gate. Features that need more than 64 KB
   of headroom must find their own savings or make an explicit budget case.
 
+## Spend ledger (running)
+
+Deliberate spends since this ADR landed, so that future size work finds them
+here — at the entry point the size gate's error message names — rather than
+re-attributing them from a bloat diff. One line per spend: what it bought,
+what (if anything) is clawback-able, and what clawing it back would cost.
+Append; do not editorialize old entries.
+
+| Date | Spend | Bought | Clawback lever |
+|---|---|---|---|
+| 2026-08-01 | **+10,208 B** — resolver persistent resolution (PR #243, `docs/adr/2026-07-31-resolver-persistent-resolution.md`) | −54% engine cycles on the 1-fixture oracle; S3 quad-strips 20→25 fps | **Mostly none** — the spend is the feature; reverting costs the perf win back. The only cheap slice is the intern table's reverse-lookup + error-formatting paths (cycle errors would report ids instead of names): unmeasured, likely single-digit KB flash — its real holding is a few KB of *heap*, not flash. Do not spend an afternoon here expecting 10 KB. |
+
 ## Alternatives Considered
 
 - **Swap ESP-NOW for raw IEEE 802.15.4** (~460 KB). Rejected for now — see

docs/adr/2026-07-31-resolver-persistent-resolution.md

@@ -0,0 +1,218 @@
+# ADR: The dataflow resolver keeps its resolution across frames
+
+- **Status:** Accepted
+- **Date:** 2026-07-31
+- **Deciders:** Photomancer
+- **Supersedes:** None
+- **Superseded by:** None
+
+## Context
+
+Frame cost on the desk ESP32-S3 was measured as a flat ~8.4 ms per
+fixture+output chain — the same whether a fixture rendered 30×1 or 90×90, and
+the same for 10 mapped LEDs or 120. Authors could not buy the cost down, which
+made multi-fixture projects degrade linearly for no reason they could see (4
+fixtures = 20 fps; ~10 would be single digits).
+
+`lp-cli profile` attributed it. The shader — the thing the product is named
+for, JIT-compiled to native code on the device — was **1.1%** of self cycles.
+The frame was dominated by the dataflow resolver re-resolving the binding
+graph from cold every tick: `Resolver::clear_frame_cache` ran each frame, so
+each frame re-walked `resolve → resolve_binding_source → resolve` with
+`SlotPath::parse` per read, `QueryKey` allocation, equality and drop, and the
+allocator/`memcpy` pair at ~44% of self cycles combined.
+
+The resolver already had a cache that outlived nothing. It was cleared per
+tick because one method, `clear_frame_cache()`, served two unrelated callers:
+"a new frame started" and "the graph changed shape". While both said the same
+thing, no caller could persist anything.
+
+## Decision
+
+**Resolution persists across frames and is invalidated by structural change,
+not by the passage of a frame.**
+
+The resolver distinguishes two operations:
+
+- `begin_frame()` — values resolved for the previous frame are stale, because
+  producers tick and time moves. The graph is unchanged.
+- `invalidate_structure()` — bindings, tree topology, node definitions, or
+  project state changed. Everything cached is void.
+
+Three things persist until `invalidate_structure()`:
+
+1. **Routes** (`ResolvedRoute`) — the decision about *how* a query is
+   answered: which binding wins (owner depth for consumed slots, priority for
+   bus providers), the merge policy, and the fully expanded provider list for
+   mergeable receivers. Computing this reads the binding index and introspects
+   authored definitions; none of it can change without a structural change.
+2. **Def-sourced values** — `ProductionSource::Literal` (binding literals) and
+   `::Default` (authored-def reads, including their deep `snapshot_slot_shape`
+   copies). Both are functions of the project, not of the frame.
+3. **Interned queries** — `QueryKey → QueryId`, plus a memo from
+   `(node, constant path)` to the query it names.
+
+Everything else stays per-frame: producer values, merges, and anything reached
+*through* a binding. A cached route still runs the producer behind it, once
+per frame.
+
+Failures are never cached. An ambiguous bus channel or a cyclic binding graph
+keeps reporting itself rather than being remembered as a decision.
+
+## The invalidation contract
+
+**Any engine mutation that can change what a query resolves *through* — as
+opposed to what it resolves *to* this frame — must call
+`Resolver::invalidate_structure()`.**
+
+This is the load-bearing rule, and it does not fail loudly. A missed call
+serves a stale but entirely plausible answer. The current sites:
+
+| Site | Why |
+|---|---|
+| `Engine::apply_project_changes` | Every registry-driven change: deploy, reload, overlay mutation, node add/remove/replace, def changes, asset refresh. Also rebuilds all bindings. |
+| `Engine::remove_runtime_subtree` | Tree topology |
+| `Engine::reattach_runtime_node` | A node's runtime is replaced under its consumers |
+| `Engine::add_binding` | A new binding can win a slot that already resolved |
+| `Engine::clear_bindings` | The binding graph is emptied |
+
+Two of those were found while writing this down. `add_binding` never
+invalidated anything — harmless while every frame cleared regardless, a
+stale-value bug the moment it did not. `clear_bindings` ran on the tree, so
+emptying the binding graph left the resolver's knowledge of it intact and
+depended on whatever ran next to invalidate; it now goes through the engine.
+
+**The rule checks itself.** A contract that lives only in this document will
+not survive the next mutation site, so `Engine::tick_nodes` compares a cheap
+fingerprint of the tree's shape (node count, binding count, newest binding
+revision) against the previous frame's, and asserts that a change was
+accompanied by an epoch bump. Debug builds only — release firmware pays
+nothing, and the guard's whole job is to fire during development and tests,
+long before a device is involved. Verified by reintroducing the bug: removing
+the `invalidate_structure()` from `add_binding` produces
+
+```
+the node tree changed shape ((2, 2, Revision(1)) -> (2, 3, Revision(2)))
+without Resolver::invalidate_structure(); resolution cached against the old
+graph is now being served.
+```
+
+That case is worth noting: the differential test **passed** under that
+sabotage, because the scenario reached `add_binding` through `clear_bindings`,
+which does invalidate. Two independent checks, catching different things.
+
+A playlist entry switch is deliberately **not** structural. It changes which
+child a node demands, not what any query resolves through, and the resolver
+handles it by simply being asked a different question.
+
+## Consequences
+
+### Measured
+
+1-fixture workload (`projects/test/quad-strips-1fix`), steady render, esp32c6
+cycle model:
+
+| | Baseline | After |
+|---|---|---|
+| Total attributed cycles | 2,468,427 | **1,146,110** (−54%) |
+| allocator + memcpy | 44.0% | 34.4% |
+| `[jit] render` | 1.1% | 2.4% (same cycles, less than half the frame) |
+
+`QueryKey::eq`, `merge_policy_for_consumed_slot`,
+`bindings_for_consumed_slot`, `slot_lookup` and `Vec<SlotPathSegment>::clone`
+no longer appear in the top twenty.
+
+The 4-fixture workload moved only −1.9% in the emulator, because
+`HwRegistry::endpoint_status_for` is 46.7% of that profile.
+
+On the **desk ESP32-S3** (2026-08-01, same board and projects as the original
+measurement):
+
+| Config | Before | After | |
+|---|---|---|---|
+| 4 fixtures (`quad-strips`) | 20 fps / 48 ms | **25 fps / 37.5 ms** | +25% fps |
+| 1 fixture (`quad-strips-1fix`) | 50 fps / 19 ms | **67 fps / 13.5 ms** | +34% fps |
+
+Hardware improved *more* than the emulator predicted for the 4-fixture case
+(+25% vs +2%), because the emulator profiles the **virtual** WS281x driver,
+whose endpoint enumeration is more expensive than the real RMT driver's — so
+`endpoint_status_for`'s 46.7% share is inflated there. Treat the emulator as
+an attribution tool, not an fps predictor.
+
+The per-additional-chain cost fell from ~9.7 ms to ~8.0 ms (−17%): most of the
+win is in fixed per-frame overhead, not in the per-chain scaling that made
+this a filed debt. That scaling is still owned by endpoint status. See
+`docs/debt/s3-frame-cost-scales-per-fixture.md`.
+
+### Flash cost
+
+The ESP32-C6 is the tight budget (3 MB app partition; see
+`docs/adr/2026-07-28-esp32c6-flash-budget.md`), so that is where the cost was
+measured — `just fw-esp32c6-size-check`, this branch against `origin/main`:
+
+| | Image | Headroom |
+|---|---|---|
+| Baseline | 2,863,296 B | 282,432 B |
+| After | 2,873,504 B | 272,224 B |
+| | **+10,208 B (+0.36%)** | −10,208 B |
+
+Ten kilobytes for the intern table, the route and structural caches, and the
+static-path memo. Headroom stays four times the 64 KB pre-merge gate.
+
+The self-checking invalidation guard costs **nothing** in shipped firmware:
+it is `#[cfg(debug_assertions)]`, and `release-esp32` inherits `release`.
+Confirmed on the linked ELF rather than by inference — the assertion's message
+string is absent from it, while other resolver strings are present.
+
+The spend and its (mostly absent) clawback lever are registered in the flash
+budget's running ledger — `docs/adr/2026-07-28-esp32c6-flash-budget.md` — which
+is where size work is directed when the gate trips.
+
+### Revision stamps age
+
+A cached literal or def read keeps the revision it was first stamped with,
+rather than being restamped each frame. This is the more truthful reading —
+the value genuinely has not changed since then — and an audit of every
+`changed_at` consumer on the resolve path found none that compares a
+production's revision against the current frame. Playlist triggers compare
+message sequence numbers; wire delta-sync treats an aged stamp as "unchanged,
+do not resend", which is correct for a value cached *because* it is unchanged.
+
+### Ids are epoch-scoped
+
+`QueryId`s are indices into a table that is cleared on every structural
+change. They must never be stored anywhere that outlives the resolver's
+epoch — which is why routes, themselves epoch-scoped, are the one place that
+holds them.
+
+## Alternatives considered
+
+**A compiled resolution plan** — build an explicit schedule of producer ticks
+and routes at load time and execute it per frame. Higher ceiling: no
+per-query overhead at all. Rejected because nodes issue queries that are not
+knowable ahead of the tick (a fixture reads `power.some` only if the def has
+it; a shader reads one query per declared input; a playlist demands whichever
+child is current), so the plan would need a discovery pass per frame anyway —
+which is lazy memoization with extra steps. This decision does not foreclose
+it: routes are already the per-query half of such a plan.
+
+**Leaving the cache per-frame and optimizing the constants** — cheaper key
+comparison, fewer clones. Rejected on the measurement: the work itself was
+redundant, not merely expensive.
+
+## How this is kept
+
+Two mechanisms, because the failure mode is silence:
+
+- **Counters** (`ResolveFrameCounters`) make "did this frame re-derive the
+  graph?" a testable question. `steady_state_frame_does_zero_structural_work`
+  asserts a steady frame performs no binding lookups, no merge-policy reads
+  and no def reads.
+- **A differential test** runs the same scene twice — once caching, once with
+  `set_force_invalidate_per_frame(true)`, which reproduces the old
+  clear-everything-per-tick behaviour — through binding replacement, a
+  producer switch and a node reattach, and demands the two agree frame for
+  frame. This is the check that does not require someone to have first
+  imagined the particular way a cache could go stale, which matters because a
+  wrong cached answer is a plausible answer and survives assertions written by
+  whoever wrote the cache.

docs/debt/s3-frame-cost-scales-per-fixture.md

@@ -11,9 +11,15 @@ related:
 ---
 # Frame cost is per-frame resolution machinery, not the shader: ~8.4 ms/fixture flat on the S3
 
+> **2026-08-01 status:** both mechanisms named below are fixed (PRs #243,
+> #244). The per-chain cost they were blamed for moved only 9.7 → 8.0 ms, so
+> the linear degradation this entry exists for **remains open** and is now
+> unattributed. See the 2026-08-01 incident-log entry.
+
 **Shape** — Measured on the desk ESP32-S3 (2026-07-31, all eight node gates,
 quad-strips variants, 30-LED strips), then attributed with `lp-cli profile`
-(emulator, esp32-c6 cycle model):
+(emulator, esp32-c6 cycle model). The table below is the ORIGINAL measurement;
+see the 2026-08-01 incident-log entry for the post-resolver numbers:
 
 | Config | fps | tick |
 |---|---|---|
@@ -51,11 +57,13 @@ frame; per-clock comparison with the C6 is resolver-bound on both chips.
 Profiles: `profiles/2026-07-31T18-02-07--…-1fix--steady-render--s3-gate-perf-1fix/`
 and `…18-02-44--…quad-strips--steady-render--s3-gate-perf-4fix/` (report.txt).
 
-**Carrying cost** — Multi-fixture projects degrade linearly (~8.4 ms per
-fixture+output chain): 4 fixtures = 20 fps today; ~10 fixtures ≈ single-digit
-fps, regardless of how small the fixtures are. Authors cannot buy the cost
-down with lower resolution or fewer LEDs, which makes the scaling feel
-arbitrary from the outside.
+**Carrying cost** — Multi-fixture projects degrade linearly. As filed:
+~8.4 ms per fixture+output chain, 4 fixtures = 20 fps, ~10 fixtures ≈
+single-digit fps. **After both fixes (2026-08-01): ~8.0 ms per chain, 4
+fixtures = 25 fps, ~10 fixtures ≈ 12 fps.** The linear term is essentially
+unchanged; what improved was fixed per-frame overhead. Authors still cannot
+buy the cost down with lower resolution or fewer LEDs, which is what makes
+the scaling feel arbitrary from the outside.
 
 **Workarounds** — Fewer fixture+output chains (one fixture spanning strips);
 nothing else helps, by measurement.
@@ -122,10 +130,112 @@ nothing else helps, by measurement.
   `QueryKey::eq` 5.7%, `EngineSession::resolve` 2.7%, `SlotPath::parse`. That
   is the dataflow resolver re-resolving from cold each tick, owned by plan
   `2026-07-31-2225-persist-dataflow-resolution`; `…23-33-40` is its baseline.
+- **2026-07-31** — **Resolver half addressed** (PR #243,
+  `docs/adr/2026-07-31-resolver-persistent-resolution.md`). Routes, binding
+  literals and authored-def reads now persist across frames and are dropped
+  on structural change instead of per tick. Measured on the 1-fixture
+  workload (`projects/test/quad-strips-1fix`, committed as the reproducible
+  oracle), steady render, esp32c6 cycle model:
+
+  | | Before | After |
+  |---|---|---|
+  | Total attributed cycles | 2,468,427 | 1,146,110 (−54%) |
+  | allocator + memcpy | 44.0% | 34.4% |
+  | `[jit] render` | 1.1% | 2.4% (same cycles) |
+
+  `QueryKey::eq`, `merge_policy_for_consumed_slot`,
+  `bindings_for_consumed_slot`, `slot_lookup` and `Vec<SlotPathSegment>::clone`
+  are gone from the top twenty.
+
+  The **4-fixture workload moved only −1.9%** (65,811,630 → 64,531,359),
+  because `HwRegistry::endpoint_status_for` is 46.7% of that profile and is
+  untouched by this work. The endpoint-status half is what now caps
+  multi-fixture fps.
+
+  Two smaller per-frame re-derivations surfaced while measuring, both the same
+  shape and neither addressed: the shader node re-reads its consumed-slot
+  *definitions* every frame through `format!`-built paths (now the largest
+  single allocation source in the 1-fixture profile), and a resolver cache hit
+  still clones a `ProductionSource` carrying a `SlotPath`.
+
+  **Desk-S3 re-measured 2026-08-01** (same board `d8:3b:da:47:29:70`, same
+  projects):
+
+  | Config | Before | After | |
+  |---|---|---|---|
+  | 4 fixtures | 20 fps / 48 ms | **25 fps / 37.5 ms** | +25% fps |
+  | 1 fixture | 50 fps / 19 ms | **67 fps / 13.5 ms** | +34% fps |
+
+  Hardware beat the emulator's prediction for 4 fixtures (+25% vs +2%): the
+  profile uses the **virtual** WS281x driver, whose endpoint enumeration is
+  dearer than the real RMT driver's, so `endpoint_status_for`'s 46.7% share is
+  inflated there. The emulator attributes cost well; it does not predict fps.
+
+  Flash cost on the C6 (the tight budget): **+10,208 B (+0.36%)**, headroom
+  282,432 → 272,224 B, still 4× the 64 KB CI gate. The debug-only invalidation
+  guard is absent from release firmware (verified on the linked ELF).
+
+  **The filed shape is only partly fixed.** Per-additional-chain cost went
+  ~9.7 ms → ~8.0 ms (−17%) — most of the win is fixed per-frame overhead, not
+  the per-chain scaling this entry is named for. A 10-fixture project would
+  still be in the low teens. Endpoint status owns that scaling.
+
+- **2026-08-01** — **Both halves on one image, measured together.** Emulator
+  (`quad-strips`, steady render): **65,811,630 → 3,011,467 cycles, 21.9×**, and
+  `[jit] render` is now the largest engine entry (3.6%). Desk S3
+  (d8:3b:da:47:29:70, both fixes flashed, project confirmed from the device
+  heartbeat as `/projects/Quad strips`):
+
+  | Project | Original | Resolver only | Both halves |
+  |---|---|---|---|
+  | 4 fixtures | 20 fps / 48 ms | 25 fps / 37.5 ms | **25 fps / 37 ms** |
+  | 1 fixture | 50 fps / 19 ms | 67 fps / 13.5 ms | **68 fps / 13 ms** |
+
+  The endpoint-status half contributes **nothing on silicon**, exactly as its
+  own ADR predicted: the S3 opens all four channels on frame one and never
+  entered the retry storm. The 21.9× is real but is an artifact of the
+  emulator's virtual board declaring one WS281x channel — worth remembering
+  before quoting emulator ratios as device wins.
+
+  **⚠️ The shape this entry is named for is NOT fixed.** Decomposing the two
+  measurements into fixed overhead plus per-chain cost:
+
+  | | per fixture+output chain | fixed per-frame |
+  |---|---|---|
+  | Original | 9.7 ms | 9.3 ms |
+  | Both halves | **8.0 ms** | 5.0 ms |
+
+  Per-chain cost fell only **−17%**; the fixed overhead nearly halved. That is
+  why 1 fixture improved 36% and 4 fixtures only 25%. Projected 10 fixtures:
+  **9 fps → 12 fps** — still unusable, still linear. Authors still cannot buy
+  the cost down with resolution or LED count.
+
+  So both *named mechanisms* are fixed and the carrying cost below is not.
+  Whatever owns the remaining ~8 ms per chain has not been attributed: the
+  1-fixture profile's top entries are memcpy and the allocator (30% combined),
+  with measured candidates being the shader node's per-frame re-read of its
+  consumed-slot definitions via `format!`-built paths, per-hit
+  `ProductionSource` clones, `FixtureNode::render_control`, and the 1.3 ms/
+  channel blocking RMT send. **This entry stays open on that basis** — a third
+  attribution pass on the 8 ms, not a third guess.
 
 **Exit criteria** — A profiled optimization pass that makes resolved bindings
 and endpoint status persist across frames (invalidate on tree/binding/
 hardware change, not per tick), after which frame cost is dominated by actual
 rendering work and the profile's top self-cycle entries are no longer
 allocator/memcpy/string machinery. Re-measure the same quad-strips matrix on
 the desk S3 as the oracle.
+
+- [x] **Dataflow resolver** — done 2026-07-31, see the incident log above.
+- [x] **`HwRegistry::endpoint_status_for`** — done 2026-07-31 (PR #244): a
+      failed-open retry storm, fixed by parking sinks on
+      `HwRegistry::generation()` rather than by caching status. Emulator-only
+      in steady state; the S3 never paid it.
+- [x] **Desk-S3 re-measurement** — done 2026-08-01, separately and jointly.
+      Both halves on one image: 25 fps at 4 fixtures, 68 fps at 1.
+- [ ] **The linear term itself** — the two named mechanisms are fixed and
+      per-chain cost still sits at ~8.0 ms (was ~9.7). This entry stays open
+      on that number, not on either mechanism. Next step is attribution of
+      that 8 ms, not another guess; `projects/test/quad-strips-1fix` vs
+      `quad-strips` under `lp-cli profile` is the differential that isolates
+      it.