Prompt-cache health: stable prefix, watcher-default polls, read folding - #130
Merged
Merged
Conversation
…lls, superseded-read folding Driven by real transcripts: a nanogpt session spent 8.09M input tokens on a ~75K-token conversation at 3.6% cache reads, because the system message was rebuilt every model round (memory ranking + task index churn), which invalidated the entire prefix for implicit and explicit prompt caches alike. A sol-view session showed background polls still costing one API round-trip per instant peek whenever the model didn't pass wait_secs. Stable request prefix: - Split the system prompt: message[0] now carries only identity, rules, working directory, and durable project guides, and is replaced only when that content actually changes (hash-gated replace_system). - Task-ranked memory, the task context index, session goal, long-horizon goal state, and the decision log move into a marked [hi:context] block attached to each turn's user message; the previous turn's block is stripped when the new one attaches, so exactly one copy exists and the per-turn prefix cost is one late re-anchor instead of a per-round message[0] invalidation. Mid-turn rounds are append-only. - Durable state still survives compaction — it lives in agent state and is re-injected each turn, no longer parked in the system message. Cache visibility: - PrefixStability tracker hashes each outgoing request's message list and classifies every round append-only vs prefix-breaking. New telemetry: prefix_stable_rounds, prefix_break_rounds, earliest_prefix_break — wired through --report so metrics can watch cache health directly. (OpenAI-compat stream_options/include_usage was already requested.) Watcher-default background polls: - bash_output with no wait_secs now parks on the registry's change notification whenever the process is running with no new output, with escalating patience (15s→240s, reset by fresh output). wait_secs:0 stays an instant peek; explicit N parks up to N (600 cap); HI_BG_POLL_WAIT_BASE_SECS=0 restores instant polls. Instant-poll API loops are structurally impossible regardless of model behavior. - Live verification (sol-view, pipenetwork): four polls parked ~6s each and returned with fresh output; 58% cache reads on the same provider that previously reported zero. Transcript hygiene: - Superseded read results fold to a one-line digest per (path, offset, limit) shape, mirroring the bash_output poll folding (one session had re-read src/model.rs 21×; only the newest copy reflects the file). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
[hi:context]block on each turn's user message, exactly one copy in the transcript. Mid-turn rounds are append-only, so provider prompt caches (implicit and explicit) actually hit.prefix_stable_rounds/prefix_break_rounds/earliest_prefix_break) in TurnTelemetry and--report.bash_outputwithoutwait_secsparks on the registry's change notification with escalating patience (15s→240s) instead of returning instantly — instant-poll API loops become structurally impossible.wait_secs: 0= explicit peek;HI_BG_POLL_WAIT_BASE_SECS=0opt-out.readresults of the same (path, offset, limit) shape collapse to one-line digests, like the poll folding.Evidence: a nanogpt session burned 8.09M input tokens on a ~75K-token conversation at 3.6% cache reads purely from per-round system-message churn; sol-view sessions burned an API round-trip per instant poll.
Testing
prefix_stable_rounds=3/0 breaks) and sol-view (four polls parked ~6s each on the watcher, turn ends with an honest status; 58% cache reads on a provider that previously reported zero).🤖 Generated with Claude Code