Skip to content

feat(providers): report which speed tier actually served a request - #409

Merged
luizParreira merged 1 commit into
mainfrom
usage-served-speed
Jul 28, 2026
Merged

feat(providers): report which speed tier actually served a request#409
luizParreira merged 1 commit into
mainfrom
usage-served-speed

Conversation

@luizParreira

Copy link
Copy Markdown
Member

Summary

Closes the observability half of speed tiers: Usage::served_speed reports which tier a provider
actually used. Requesting a premium tier does not guarantee getting one — Anthropic serves
claude-opus-4-6 at standard speed without erroring, and OpenAI downgrades priority requests
under a sharp traffic ramp. Both bill the tier they really ran, and both report it, so this reads it
back from Anthropic's usage.speed and OpenAI's echoed service_tier across every surface
(chat completions and Responses, streaming and non-streaming).

Stacked on #408 — the first commit here is that PR's; review only c71c0a2.

Key decisions

  • The observed side is its own type, not Option<SpeedTier>. A Usage is not always one
    response: the agent loop, the root-turn worker, and the stream-splice wrapper each fold per-call
    readings into a running total. A turn that mixes an expedited call with a downgraded one has no
    single tier, and collapsing that to None would make a real downgrade indistinguishable from a
    provider that never reported a tier. ServedSpeed::{Uniform, Mixed} keeps the fold honest, and
    Mixed is exactly the signal worth alerting on.
  • All three accumulators merge rather than overwrite. Dropping the tier on fold (what a naive
    served_speed: None would have done) silently loses it; taking the last writer reports whichever
    call happened to land last. usage_increment is the one exception and keeps the current tier —
    it is a delta between two restatements of one call, not a fold across calls.
  • ServedSpeed::merge is const so the usage accumulators it is called from stay const,
    avoiding an unrelated de-const ripple through their callers. That forced a const-callable
    SpeedTier::same, since PartialEq::eq is not const.
  • Usage now derives Default. It makes the sweep below tractable and keeps the next field
    from being another 160-site change.
  • The 163-literal sweep was driven off compiler positions, not a regex, so it could not touch a
    TokenUsage/ApiUsage/ApiUsageMetadata literal that merely looks similar. Two destructuring
    patterns and one doctest fell outside --all-targets and were fixed by hand.
  • Anthropic's SSE parser hit clippy's 7-argument ceiling, so the four loose &mut counters
    became SseUsageState and the two usage-folding match arms moved into helpers to stay under the
    line ceiling. Refactored per the repo guideline, not silenced with #[allow].

Review guide

Part 1 — the fold semantics (review first; everything else is plumbing). Start at
ServedSpeed::merge in crates/agent-sdk-foundation/src/llm.rs, reached from all three
accumulators: add_usage in agent-sdk/src/agent_loop/llm.rs, add_attempt_usage in
agent-server/src/worker/root_turn.rs, and add_usage in agent-sdk-providers/src/streaming.rs.
It stops at Usage — no provider, wire format, or persisted schema depends on the merge rule.
State machine: the usage-accumulation fold, which gains a lattice
(None → Uniform(t) → Mixed, absorbing at Mixed). This is an improvement/evolution (vi) of
an existing fold, so the thing to check is the algebra: merge must be associative and
order-independent, since callers fold in arbitrary order. None must be an identity — a single call
that reports nothing must not erase a known tier —which is what
served_speed_merge_reports_disagreement_instead_of_hiding_it pins down.

Part 2 — provider parsing. Entry points are ApiUsage::served_speed in
impls/anthropic/data.rs and served_speed_from_service_tier in impls/openai_reasoning.rs, each
reached from its provider's response mapping. Reach: Anthropic's non-streaming chat plus the SSE
path via the new SseUsageState; OpenAI's chat-completions response and stream chunks, and the
Responses API body and its three terminal stream frames (response.completed,
response.incomplete, response.failed). It stops at Usage — no request-side behavior
changed. State machine: none — this is decode-and-map, so review it as data correctness. Two things
to check: an unrecognised service_tier (e.g. "flex") reads as "not reported" rather than being
forced into a known tier, and an absent field stays None rather than defaulting to Standard.

Part 3 — the mechanical sweep (skim). 163 literals gained served_speed: None, plus the
SseUsageState refactor and its call sites. Purely compile-driven; the risk is a literal that
should have been given a real tier instead of None, which Part 2 covers by construction —
every production mapping site is listed there.

Test plan

  • cargo test --workspace — passes (exit 0).
  • cargo clippy --workspace --all-targets --all-features -- -D warnings — clean.
  • New coverage: the merge lattice (identity, agreement, disagreement, absorption), used_premium,
    Usage::default, Anthropic non-streaming speed present/absent/standard, the streaming tier
    surviving into the emitted Usage delta, OpenAI's echoed tier including the unknown-value case,
    and both accumulator behaviors (uniform kept, disagreement → Mixed).
  • Not covered: no live premium request was made, so the parsed shapes are doc-verified. Anthropic
    documents usage.speed on the non-streaming response only — the streaming shape is not
    specified, so the SSE path reads it from both message_start and message_delta and takes
    whichever reports it. If Anthropic puts it somewhere else on the stream, streaming reports None
    rather than a wrong tier.
  • Unrelated pre-existing flake: under --all-features,
    agent-service-host's sqlite_atomic_creation_paths_write_one_outbox_each fails from
    process-global failpoints tripping each other in parallel; it passes with --test-threads=1 and
    fails on main too. This PR touches no code in that crate's failpoint paths.

@luizParreira luizParreira self-assigned this Jul 25, 2026
Base automatically changed from fast-mode to main July 28, 2026 19:06
Requesting a premium tier does not guarantee getting one: Anthropic serves
claude-opus-4-6 at standard speed without erroring, and OpenAI downgrades
priority requests under a sharp traffic ramp. Both bill the tier they
actually ran and both report it, so read it back:

- Usage gains `served_speed`, parsed from Anthropic's `usage.speed` and
  from OpenAI's echoed `service_tier` on both the chat-completions and
  Responses surfaces, streaming and not.
- The observed side is its own type rather than Option<SpeedTier>, because
  a Usage is not always one response — the agent loop folds each call's
  reading into a running total. ServedSpeed::{Uniform, Mixed} makes that
  fold honest: a turn mixing an expedited call with a downgraded one
  reports Mixed instead of whichever call landed last. Collapsing it to
  None would make a real downgrade indistinguishable from a provider that
  never reported a tier.
- All three usage accumulators (agent loop, root turn, stream splice) now
  merge the tier instead of dropping it. usage_increment keeps the current
  call's tier: it is a delta between restatements of one call, not a fold.
- Usage derives Default, which the sweep below leans on and which keeps a
  future field from being another 160-site change.

Mechanical: the new field broke 163 struct literals. They were swept with
a compiler-position-driven script rather than a regex so it could not touch
a TokenUsage/ApiUsage literal that merely looks similar; two destructuring
patterns and one doctest were fixed by hand.

Anthropic's SSE parser was at clippy's 7-argument ceiling, so the four
loose &mut counters became SseUsageState. Its message_start/message_delta
arms moved into helpers to stay under the line ceiling — refactored, not
silenced with #[allow]. Anthropic documents usage.speed on the
non-streaming response only, so the streaming path reads it from both
terminal-usage frames and takes whichever reports it.

Tests: cargo test --workspace (exit 0); clippy --all-targets
--all-features -D warnings clean.
@luizParreira
luizParreira merged commit f6c27db into main Jul 28, 2026
33 checks passed
@luizParreira
luizParreira deleted the usage-served-speed branch July 28, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant