Per-model token rates, weighted variant routing, and a model availability status API - #6
Merged
Conversation
Cost and quota consumption multiply the served model's configured component weights (cache reads/writes, reasoning); ledger token columns keep the vendor-reported counts. Weights validated finite and non-negative at load.
CallEngine outcomes (upstream 5xx vs success; 4xx and client aborts say nothing about model health) accumulate in an in-process buffer, flushed to minute buckets every 2s — memory ring locally, shared Redis keys fleet-wide — so the claim path never gains a network hop. GET /admin/models/status classifies the configured window into available/unstable/unavailable/no_data; tenant admins see only entitled models.
A public model name may declare a weighted split across other declared models (same protocol, one level, validated at load). Selection hashes the effective user (request id when anonymous) with FNV-1a so every fleet instance maps a user to the same variant with no shared state. The swap reuses the fallback seam: entitlement and the (AK, model) counter judge the public name, billing prices the served variant, the response echoes the requested name, and each variant caches separately.
Round findings applied: realtime turns and the estimate/no-usage branches now weight through the same model_token_rate helper as the pipeline (a cut stream or WS turn must price like a completed one); availability records the client-visible terminal outcome instead of per-attempt (failover recovery is account-health business); availability_window_minutes capped at the store's one-hour retention; duplicate variant targets rejected; status endpoint fans out with join_all; small cleanups from the simplify lenses. total_tokens documented as the weighted platform total.
Regression tests from the verification pass: a failover-recovered request samples exactly one availability success; BadStability rejects window bounds and inverted/out-of-range rates. Plus a WHY note on the e2e wire-vs-ledger total assertion (equal only without token_rate).
External review found two fail-silent gaps: legal variants config on a realtime model routed 100% to the parent (sessions never traverse VariantSelect) — now rejected at load; and availability attributed samples to the served backend name, leaving a tenant's entitled public name at permanent no_data under a variant split — samples now attribute to the requested public name, matching the client-visible-outcome semantic. Realtime models are excluded from /admin/models/status (never sampled; listing them read as a permanent no_data). Docs catch up: the status endpoint in api.md, token_rate/variants/availability in configuration.md. Also deflakes sqlite_erase_roundtrip (5ms marker margin lost races under parallel-suite load).
All avail.record sites sat in CallEngine, so once an account tripped into cooldown every subsequent 503 came from SelectAccount unsampled — a sustained outage held ~failure_threshold samples per window, below min_samples, reading no_data forever (zero-account models: permanently). SelectAccount now records one failure against the requested public name before erroring; cache hits are unaffected (the executor short-circuits them before this node).
Applied: weighted_pair hoisted to gw_models (the dag estimate paths and realtime shared a hand-built TokenInput pattern); platform_total deleted (no production caller survived the BillTokens refactor); served_model/ requested_model narrowed to take the model param so bill() reuses them instead of re-deriving; token_rate validation shares the finite/non-negative predicate with the qps checks; fallback_from doc names both writers. Skipped with reasons: per-request ModelConf memo (tens of ns vs a stale-memo hazard at two rewrite points; rides the perf pass), spawn-loop dedup (touches three pre-existing tasks; loc-justify adjudicates), realtime pre-clamp removal (would desync user-budget consumption from the DAG's clamped totals). Test comments trimmed to the zero-budget, multi-line WHYs compressed.
The retry-success arm recorded availability and account health without the first attempt's aborted-stream guard, so a client cutting the stream during a failover retry read as a model success — against CallEngine's own documented exclusion.
AvailTracker dissolved into the AvailStore impls per the RedisHealth embedded-cache precedent: MemoryAvail records straight into its ring (no buffer between an in-process write and an in-process store), RedisAvail owns the claim-path buffer its flush drains, and the pure-passthrough window wrapper is gone. The buffer/ring bumps reuse slot_mut instead of re-rolling its miss-path idiom; fnv1a folds; the Redis window keys build via flat_map. Adds the GW_TEST_REDIS_URL-gated RedisAvail test matching the health/governance pattern. prompt_includes_cache kept (vendor- inclusive prompt accounting is a plausible near-term need).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three additions to the serving pipeline, converged through two review rounds:
Per-model
token_ratebilling weights — each token component (prompt / cache read / cache write / completion / reasoning) can carry its own weight relative to the model's unit prices (e.g. cache reads at 0.1, cache writes at 1.25). Cost and quota consumption use the weighted counts on every billing path — the request pipeline, aborted-stream and missing-usage estimates, and realtime turns all share onemodel_token_ratehelper, so no surface can price differently. Ledgerprompt_tokens/completion_tokensstay vendor-reported;total_tokensis the weighted platform total that quota metering consumed (documented on the record and pinned by tests). Weights are validated finite and non-negative at load.Weighted variant routing (canary/gray) — a public model name may declare a weighted split across other declared same-protocol models. Selection hashes the effective user with FNV-1a, so every fleet instance maps a user to the same variant with no shared state; anonymous requests spread per request. The swap reuses the fallback seam: entitlement and the per-(AK, model) daily counter judge the public name, billing prices the served variant, the response echoes the requested name, and each variant caches separately. Load-time validation rejects unknown/cross-protocol/nested/duplicate/zero-weight targets — and variants on realtime models, which pin their model at handshake and never traverse the selector.
GET /admin/models/status— per-model availability classified over a configurable window (available/unstable/unavailable/no_data) from minute-bucketed success/error counts. Samples record the client-visible terminal outcome only (a failover-recovered attempt is account-health business, not a model error; pool-exhaustion 503s count, so a sustained outage converges tounavailableinstead of sitting belowmin_samples), attributed to the requested public name so a variant split doesn't scatter a model's health. The claim path never gains a network hop: the in-process store writes its ring directly and the Redis store buffers until a background flush. Realtime models are excluded (never sampled). Tenant-scoped like the other admin reads.Notes
stability.availability_window_minutesis capped at 60 to match the store's one-hour retention.sqlite_erase_roundtripused a 5ms wall-clock margin) and an asymmetry where a stream aborted during a failover retry recorded as a success.docs/api.md,docs/configuration.md, andconf/gateway.yamlcover the new knobs.Testing
cargo fmt --check,cargo clippy --all-targets -- -D warnings, andcargo test --workspacegreen on macOS and in arust:1Linux container. New coverage includes: weighted cost math and ledger columns end-to-end (cached-usage transport), realtime turn weighting, variant stickiness/distribution/attribution, sustained-outage convergence tounavailable, failover-recovery single-sample accounting, and load-time validation of every new config rule. Redis-backed availability has aGW_TEST_REDIS_URL-gated test matching the existing pattern.