feat(providers): report which speed tier actually served a request - #409
Merged
Conversation
Requesting a premium tier does not guarantee getting one: Anthropic serves
claude-opus-4-6 at standard speed without erroring, and OpenAI downgrades
priority requests under a sharp traffic ramp. Both bill the tier they
actually ran and both report it, so read it back:
- Usage gains `served_speed`, parsed from Anthropic's `usage.speed` and
from OpenAI's echoed `service_tier` on both the chat-completions and
Responses surfaces, streaming and not.
- The observed side is its own type rather than Option<SpeedTier>, because
a Usage is not always one response — the agent loop folds each call's
reading into a running total. ServedSpeed::{Uniform, Mixed} makes that
fold honest: a turn mixing an expedited call with a downgraded one
reports Mixed instead of whichever call landed last. Collapsing it to
None would make a real downgrade indistinguishable from a provider that
never reported a tier.
- All three usage accumulators (agent loop, root turn, stream splice) now
merge the tier instead of dropping it. usage_increment keeps the current
call's tier: it is a delta between restatements of one call, not a fold.
- Usage derives Default, which the sweep below leans on and which keeps a
future field from being another 160-site change.
Mechanical: the new field broke 163 struct literals. They were swept with
a compiler-position-driven script rather than a regex so it could not touch
a TokenUsage/ApiUsage literal that merely looks similar; two destructuring
patterns and one doctest were fixed by hand.
Anthropic's SSE parser was at clippy's 7-argument ceiling, so the four
loose &mut counters became SseUsageState. Its message_start/message_delta
arms moved into helpers to stay under the line ceiling — refactored, not
silenced with #[allow]. Anthropic documents usage.speed on the
non-streaming response only, so the streaming path reads it from both
terminal-usage frames and takes whichever reports it.
Tests: cargo test --workspace (exit 0); clippy --all-targets
--all-features -D warnings clean.
luizParreira
force-pushed
the
usage-served-speed
branch
from
July 28, 2026 19:19
c71c0a2 to
617d2b4
Compare
This was referenced Jul 28, 2026
Merged
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the observability half of speed tiers:
Usage::served_speedreports which tier a provideractually used. Requesting a premium tier does not guarantee getting one — Anthropic serves
claude-opus-4-6at standard speed without erroring, and OpenAI downgrades priority requestsunder a sharp traffic ramp. Both bill the tier they really ran, and both report it, so this reads it
back from Anthropic's
usage.speedand OpenAI's echoedservice_tieracross every surface(chat completions and Responses, streaming and non-streaming).
Stacked on #408 — the first commit here is that PR's; review only
c71c0a2.Key decisions
Option<SpeedTier>. AUsageis not always oneresponse: the agent loop, the root-turn worker, and the stream-splice wrapper each fold per-call
readings into a running total. A turn that mixes an expedited call with a downgraded one has no
single tier, and collapsing that to
Nonewould make a real downgrade indistinguishable from aprovider that never reported a tier.
ServedSpeed::{Uniform, Mixed}keeps the fold honest, andMixedis exactly the signal worth alerting on.served_speed: Nonewould have done) silently loses it; taking the last writer reports whichevercall happened to land last.
usage_incrementis the one exception and keeps the current tier —it is a delta between two restatements of one call, not a fold across calls.
ServedSpeed::mergeisconstso the usage accumulators it is called from stayconst,avoiding an unrelated de-const ripple through their callers. That forced a
const-callableSpeedTier::same, sincePartialEq::eqis notconst.Usagenow derivesDefault. It makes the sweep below tractable and keeps the next fieldfrom being another 160-site change.
TokenUsage/ApiUsage/ApiUsageMetadataliteral that merely looks similar. Two destructuringpatterns and one doctest fell outside
--all-targetsand were fixed by hand.&mutcountersbecame
SseUsageStateand the two usage-folding match arms moved into helpers to stay under theline ceiling. Refactored per the repo guideline, not silenced with
#[allow].Review guide
Part 1 — the fold semantics (review first; everything else is plumbing). Start at
ServedSpeed::mergeincrates/agent-sdk-foundation/src/llm.rs, reached from all threeaccumulators:
add_usageinagent-sdk/src/agent_loop/llm.rs,add_attempt_usageinagent-server/src/worker/root_turn.rs, andadd_usageinagent-sdk-providers/src/streaming.rs.It stops at
Usage— no provider, wire format, or persisted schema depends on the merge rule.State machine: the usage-accumulation fold, which gains a lattice
(
None → Uniform(t) → Mixed, absorbing atMixed). This is an improvement/evolution (vi) ofan existing fold, so the thing to check is the algebra:
mergemust be associative andorder-independent, since callers fold in arbitrary order.
Nonemust be an identity — a single callthat reports nothing must not erase a known tier —which is what
served_speed_merge_reports_disagreement_instead_of_hiding_itpins down.Part 2 — provider parsing. Entry points are
ApiUsage::served_speedinimpls/anthropic/data.rsandserved_speed_from_service_tierinimpls/openai_reasoning.rs, eachreached from its provider's response mapping. Reach: Anthropic's non-streaming
chatplus the SSEpath via the new
SseUsageState; OpenAI's chat-completions response and stream chunks, and theResponses API body and its three terminal stream frames (
response.completed,response.incomplete,response.failed). It stops atUsage— no request-side behaviorchanged. State machine: none — this is decode-and-map, so review it as data correctness. Two things
to check: an unrecognised
service_tier(e.g."flex") reads as "not reported" rather than beingforced into a known tier, and an absent field stays
Nonerather than defaulting toStandard.Part 3 — the mechanical sweep (skim). 163 literals gained
served_speed: None, plus theSseUsageStaterefactor and its call sites. Purely compile-driven; the risk is a literal thatshould have been given a real tier instead of
None, which Part 2 covers by construction —every production mapping site is listed there.
Test plan
cargo test --workspace— passes (exit 0).cargo clippy --workspace --all-targets --all-features -- -D warnings— clean.used_premium,Usage::default, Anthropic non-streamingspeedpresent/absent/standard, the streaming tiersurviving into the emitted
Usagedelta, OpenAI's echoed tier including the unknown-value case,and both accumulator behaviors (uniform kept, disagreement →
Mixed).documents
usage.speedon the non-streaming response only — the streaming shape is notspecified, so the SSE path reads it from both
message_startandmessage_deltaand takeswhichever reports it. If Anthropic puts it somewhere else on the stream, streaming reports
Nonerather than a wrong tier.
--all-features,agent-service-host'ssqlite_atomic_creation_paths_write_one_outbox_eachfails fromprocess-global failpoints tripping each other in parallel; it passes with
--test-threads=1andfails on
maintoo. This PR touches no code in that crate's failpoint paths.