Per-operator concentration cap + reputation hardening (URL granularity, rate cooldown, HC dedup, WS rebind) - #523
Merged
Merged
Conversation
…(HTTP + WS) Primary endpoint selection is uniform-random over the valid endpoint pool, so an operator's traffic share tracks its endpoint-count, not its quality. When one registrable domain (eTLD+1) holds most of a service's endpoints, it absorbs an outsized share of the service — and if it degrades, that whole share fails at once. This hits WebSocket hardest: connections are long-lived, there is no per-message content QoS to correct a bad-but-connected supplier, and the session-rebind engine re-homes onto the same lopsided pool. Add an optional per-operator concentration cap applied at the single selection site both HTTP and WebSocket reach (requestContext.Select -> SelectWithMetadata): - qos/selector.SelectWithConcentrationCap: groups the valid pool by eTLD+1 (shared GetEndpointTLDs helper — same granularity as the dashboard domain label and the hedge eTLD+1 exclusion), then water-fills any operator's probability mass above the cap down to it, redistributing the excess proportionally to under-cap operators. Below the cap the distribution is identical to a flat random pick; unresolvable eTLD+1 endpoints stay as singleton operators; the infeasible case (cap < 1/M) falls back to uniform-over-operators. Disabled (maxShare <= 0 or >= 1) is byte-for-byte the prior rand.Intn pick. - Thread the cap through EVMServiceQoSConfig (atomic float64 on the production simpleServiceConfig, mirroring SetSyncAllowance) with a SetMaxOperatorShare setter. - Gate it with the PATH_EVM_MAX_OPERATOR_SHARE env var (unset/0/>=1 = disabled), matching the WebSocket-rebind canary rollout pattern — a redeploy-free A/B off switch. Default off: no behavior change in production until enabled. Covers EVM (the services where WS concentration is observed) for both transports. The helper is shared, so rolling to solana/cosmos/noop later is a one-line swap at each selection site. Per-service config (vs the global env default) is a follow-up. Tested: water-filling distribution (dominant operator converges to the cap, singletons absorb the remainder), no-op below cap, disabled == flat random, feasibility floor, single/empty pool, unresolvable-eTLD isolation, atomic setter round-trip. Race-clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…WS rebind
Follow-up to the per-operator concentration cap. Three changes:
1. Replace the PATH_EVM_MAX_OPERATOR_SHARE env var with a real config field.
- Add `max_operator_traffic_share` to per-service config and to the service
defaults block (gateway.ServiceConfig / ServiceDefaults).
- GetMaxOperatorShareForService resolves per-service -> defaults ->
DefaultMaxOperatorShare, mirroring GetSyncAllowanceForService.
- cmd/qos.go reads it from the unified config instead of the environment.
2. Ship it ON by default. DefaultMaxOperatorShare = 0.75, tuned from production
concentration data: it bounds the clearly-concentrated tail (one operator
holding the large majority of a service's selections) while leaving ordinary
multi-operator distributions unchanged, and redistributes only among
already-reputation-valid operators. A per-service or default value of >= 1
(or <= 0) disables it.
3. Extend the cap to the WebSocket rebind target, not just initial selection.
The tier-2 / stall rebind previously funneled every connection to the single
smallest-address top-scored endpoint; when scores tie (the common case) that
re-homes an outsized share onto one operator. selectReconnectEndpointWithCap
now takes the tied-best band (same capability tier + top score) and spreads it
across operators via the shared concentration helper, seeded by the
connection's original endpoint so the choice stays reproducible across replays
and pods. Disabled cap = the prior deterministic (non-fallback, highest score,
smallest address) pick, byte-for-byte.
The helper gains a deterministic SelectWithConcentrationCapSeeded variant (same
water-filling, seeded RNG) so both random initial selection and reproducible
rebind selection share one implementation. The protocol layer reads the cap
from its existing unifiedServicesConfig reference — no new wiring.
Now covers, for EVM services, both transports' initial selection AND the WS rebind
re-home. Tested: config resolution/override/disable, seeded determinism, seeded
spread respects the cap, rebind disabled==deterministic, rebind spreads across
operators, rebind stays within the top-score band. go build/vet/-race/lint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…a, NoOp
Roll the shared concentration cap out to the remaining QoS types so every
non-fallback endpoint selection is concentration-bounded, not just EVM.
- Cosmos: add getMaxOperatorShare to CosmosSDKServiceQoSConfig (atomic float on
the production simpleCosmosConfig, 0 on the static config) + SetMaxOperatorShare;
swap the rand.Intn pick in serviceState.Select for the shared helper.
- Solana: carry the cap as an atomic float on EndpointStore (+ getter/setter,
promoted onto QoS via embedding); swap the rand.Intn pick in EndpointStore.Select.
- NoOp: carry the cap on NoOpQoS (+ getter/setter); apply it to both picks in
filteringSelector.Select (the valid-pool pick and the all-endpoints fallback pick).
Wire it once in cmd/qos.go: after building each QoS instance, set the cap via an
`interface{ SetMaxOperatorShare(float64) }` assertion (mirroring the SetSyncAllowance
pattern), so EVM/Cosmos/Solana/NoOp are all covered by one block driven by
GetMaxOperatorShareForService. Disabled (>= 1 or <= 0) stays a flat random pick.
The least-stale/empty-pool fallbacks (single-endpoint picks) are intentionally left
alone — the cap has nothing to reshape there.
Tested: SetMaxOperatorShare round-trip + setter-interface satisfaction for each of
Cosmos/Solana/NoOp. go build/vet/-race/lint clean; full qos+cmd+gateway+shannon
unit sweep green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…hot-path allocs Audit follow-up on the concentration cap. Three cleanups, no change to the capping distribution: 1. Drop the seeded/deterministic WS-rebind variant. The rebind target now uses a plain random spread over the tied-best band (SelectWithConcentrationCap), same as initial selection. The determinism it replaced bought nothing useful — a WebSocket connection re-selects fresh at every Shannon session boundary (~20 min), so there is no durable "prefer the same supplier" property to preserve — and it actively hurt: a reconnect *retry* recomputed the same seed and re-picked the endpoint that had just failed to dial, wasting retry attempts. Removes SelectWithConcentrationCapSeeded, the randSource abstraction, and the per-rebind band sort + fnv hash. 2. Cut hot-path allocations. The cap ships on by default, so the grouping path is the common case. Replace the throwaway shannonmetrics.GetEndpointTLDs map (built then read exactly once) with an inline ExtractTLDFromEndpointAddr in the grouping loop, and preallocate operatorOrder. Measured on a 40-endpoint / 5-operator pool at the default 0.75: ~5,600->2,700 ns/op, 5,576->1,984 B/op, 30->16 allocs/op. The disabled path stays 0-alloc. 3. Remove the dead `logger` parameter from SelectWithConcentrationCap (was `_ = logger`); update the EVM/Cosmos/Solana/NoOp call sites. Also harden waterFillToCap with an underSum<=0 guard (the caller's feasibility check makes it unreachable, but it removes the theoretical sub-epsilon mass-drop noted in review) and add a multi-pass cascade case to its unit test. go build/vet/-race/lint clean; qos + protocol/shannon + cmd unit suites green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…t; e2e test
Address the three actionable audit items on the concentration cap.
S1 — observability. The cap ships on but had no runtime signal, so operators
couldn't tell whether or where it was reshaping traffic. Add
path_concentration_cap_reshaped_total{service_id}, incremented once per selection
whose distribution the cap actually altered (an operator exceeded the cap and its
excess was water-filled, or the pool was too concentrated and fell back to
uniform-over-operators). No-op selections stay silent, so a nonzero rate is the
live proof the cap is bounding a real concentration. SelectWithConcentrationCap
now takes the serviceID and emits internally; waterFillToCap reports whether it
clamped.
S3 — de-duplicate the per-QoS atomic float. EVM/Cosmos/Solana/NoOp each carried an
identical maxOperatorShareBits atomic.Uint64 + math.Float64bits/Float64frombits
plumbing (and a math import solely for it). Replace with a shared
selector.AtomicFloat64 (Load/Store); removes the repetition, the per-copy bit
footgun, and four now-unused imports.
S2 — end-to-end coverage. The existing selection tests all run with the static
config (cap == 0). Add an EVM test that drives the real production path
(NewSimpleQoSInstance → SetMaxOperatorShare → SelectWithMetadata) and asserts a
dominant operator's realized share is bounded near the cap and that the reshape
metric increments; plus a disabled-by-default case.
go build/vet/-race/lint clean; full qos + shannon + metrics + gateway + cmd unit
suites green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…ency, guards Clear the remaining low-severity audit items: - Naming: rename the config knob max_operator_traffic_share -> max_operator_share (Go MaxOperatorShare), so the config surface matches the internal getMaxOperatorShare / DefaultMaxOperatorShare / SetMaxOperatorShare naming. The field is new on this branch and undeployed, so no compatibility concern. - Docs: document max_operator_share in config/config.schema.yaml (the service item uses additionalProperties:false, so it needed the entry) and add a commented, explained example to config/examples/config.shannon_example.yaml. Note on the struct field that the cap governs primary single-endpoint selection (HTTP + WS + WS rebind), not multi-endpoint fan-out / hedge. - Rebind band consistency: reconnectTopBand now uses exact score equality to define "tied for best", matching betterReconnectCandidate (the disabled-path picker), so the two paths can't disagree on which endpoints are tied. Removes the epsilon band (and its unused constant) — reputation scores are shared values, so exact equality is correct and avoids a non-transitive comparison. - Silent-assertion guard: cmd/qos.go now warns if a QoS type does not implement SetMaxOperatorShare, so a future type that forgets it can't silently run uncapped under the shipped-on default. go build/vet/-race/lint clean; full qos + shannon + gateway + cmd suites green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…reshape The cap ships on by default, so every selection ran the full per-operator grouping + weight vector + water-fill — even for services whose most-concentrated operator is already under the cap, where the capped weighted pick provably reduces to a flat random pick. Those no-op selections dominate traffic (the highest-volume services sit below the 0.75 default) yet were the most expensive path. Split into two phases. Phase 1 resolves each endpoint's operator (eTLD+1) once, counts endpoints per operator into an int map, and records each endpoint's key. That is enough to decide whether the cap does anything: a single operator, or a dominant share already at/under the cap (and the cap feasible), is a no-op and returns a flat random pick immediately — no per-operator endpoint grouping, no weight vector. Phase 2 (only when an operator is over the cap, or the pool is too concentrated to satisfy it) builds the grouping — reusing the Phase 1 keys, so the eTLD+1 is never resolved twice — and the capped weights. Distribution is unchanged (below-cap selection was already mathematically identical to flat random; the tests assert this). The reshape metric now fires exactly when Phase 2 runs, same semantics as before. Measured (40 endpoints, 5 operators): no-op path 3115->2189 ns/op, 1952->704 B/op, 22->1 allocs/op. Reshape path pays a small counting-pass overhead but its excess bytes drop; it is the lower-traffic path. waterFillToCap no longer needs to report whether it clamped (Phase 1 decides that), so it drops its bool return. go build/vet/-race/lint clean; full qos + protocol/shannon suites green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…_total
A health-check relay was recorded TWICE in path_relays_total: the health-check
executor records each outcome as request_type="health_check", and the Shannon
request context — which had no notion of a health check (only isHedge/isRetry) —
also recorded the same relay as request_type="normal" in handleEndpointSuccess /
handleEndpointError. Net effect: every dashboard panel that filters
request_type="normal" still contained 100% of health-check volume, and on a
service with no user traffic the entire "normal" series was phantom. Verified in
prod: sum by (request_type)(rate(path_relays_total{service_id="robinhood"}[5m]))
returned health_check and normal both 18.15 (bit-identical) while incoming
path_requests_total was 0.
Fix: tag the health-check relay and skip the protocol-layer recording, leaving the
executor as the sole recorder.
- Add isHealthCheck to requestContext + MarkAsHealthCheck() (mirrors isHedge/
isRetry / MarkAsHedge/MarkAsRetry) on the ProtocolRequestContext interface.
- The health-check executor calls MarkAsHealthCheck() before HandleServiceRequest.
- handleEndpointSuccess and handleEndpointError skip their metrics.RecordRelay when
the flag is set.
Chosen over the alternative (make the Shannon layer record request_type=
"health_check" and delete the executor's records): the executor already records
exactly one entry per outcome with richer health-check-specific signals (heuristic
confidence, sync-check, error-detection), AND on the transport-failure path the
Shannon layer's handleEndpointError still fires, so keeping the executor's
pre-relay record would leave a residual double-count. Making the executor the sole
recorder is duplicate-free on every path.
Health checks are still counted — as request_type="health_check", once. They are
paid relays; total paid relays = sum over ALL request_type values, never just
"normal". Documented this on the relays_total metric help + request-type constants.
go build/vet/-race/lint clean; shannon + gateway + metrics suites green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…al failures
The consecutive-strike system decays 3 strikes per success, so it only ever
fires on a burst of clustered critical/fatal errors. A high-volume endpoint
that bleeds a steady fraction of critical errors refills its strike budget with
successful traffic as fast as it spends it and is never cooled — the same
success volume that should disqualify a broken endpoint instead protects it.
Add a volume-independent detector: an EWMA of the per-request critical/fatal
indicator (Score.RecentCriticalRate). When the sustained critical rate stays at
or above CriticalRateThreshold (30%) past a minimum sample, the endpoint is
cooled down regardless of how much successful traffic it also serves.
Escalating backoff mirrors the strike system: the first trip benches for one
Shannon session (20m); a repeat trip shortly after doubles it (40m, ...), capped
at 1h; it resets to a single session once the endpoint runs clean longer than the
cap.
Counts critical+fatal only (unambiguous server faults), not major/minor, to
avoid penalizing hedge losers and transient timeouts. Ships on with tuned
constants — no config change required. RecentCriticalRate and RateCooldownCount
persist to Redis so they survive pod restarts. New metric
path_reputation_rate_cooldown_total{service_id} for canary observability.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
Two suppliers that front the exact same backend URL were scored, penalized, and cooled down independently: the reputation key included the supplier address (per-endpoint granularity), so one broken backend behind N staked supplier addresses had to be proven bad N separate times before all N were filtered. Add a per-URL granularity (URLKeyBuilder) that keys reputation on the endpoint URL alone, and make it the default. An exact URL match is the same physical backend, so its failures are now shared: the backend is penalized (and cooled) once across every supplier fronting it. Unlike per-domain, it does not merge an operator's distinct URLs, so a single bad backend is never diluted by that operator's healthy ones. For URLs served by a single supplier (the common case) this is byte-identical to per-endpoint. The per-endpoint / per-domain / per-supplier options are retained; set key_granularity to override. Changing the key namespace resets accumulated reputation once on rollout (it rebuilds within minutes; the same reset already happens on any granularity change or pod restart). Tests seed scores through the service key builder rather than constructing keys by hand, so they are robust to the configured granularity. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…b-session first offense Change the rate-cooldown backoff from exponential (20m → 40m → …) to linear: the Nth consecutive trip benches for N × 10m — 10m, 20m, 30m, 40m, 50m, capped at 1h. The first offense at 10 minutes sits under one Shannon session (~20m), so a transient spike recovers within the same session instead of being locked out for a whole session; only a persistently broken endpoint ramps toward a full hour. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
A canary session survey (19 services) found the dominant operator holding 58-90% of most services' pools, yet the cap fired on ~0.004% of selections at 0.75: reputation and validation filtering already trims the dominant operator's effective valid-pool share below 0.75 before the cap sees it, leaving it nearly dormant. Lower the default to 0.65 so the cap actually bounds the concentrated tail (12 of the 19 sampled services) at negligible redistribution cost — the excess spreads across a handful of other valid operators, and at observed volumes that is on the order of ~1 extra RPS per receiving endpoint. 0.65 stays above the uniform-over-operators feasibility floor for services with as few as two operators (0.50 would force a 2-operator service to 50/50), so it engages without piling traffic onto thin operator sets. Per-service overrides and the disable path (>= 1 or <= 0) are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
ExtractBlockHeight fell back to the getEpochInfo `absoluteSlot` field when `blockHeight` was absent. absoluteSlot is the SLOT, which runs ~5% higher than block height (skipped slots). Solana consensus is max-based (state.go — perceived only ever rises), so a single slot-valued observation lifts the perceived block height above the true chain tip. Once poisoned, every endpoint reporting the correct (lower) block height looks "behind" the perceived height, and the external getBlockHeight source — which is a floor that only raises perceived — cannot pull it back down. Observed on canary: solana perceived was 434,333,636 (the slot) while every supplier's getEpochInfo.blockHeight and the external publicnode source both reported 412,395,547 (the real block height). Correct endpoints were then failing the external getBlockHeight sync check as "22M behind", generating critical reputation signals that tanked scores to 0 and drove cooldowns — all while users saw 99.8% success, because the endpoints were healthy and only the perceived anchor was wrong. Remove the absoluteSlot fallback: getEpochInfo always includes blockHeight, so a response without a usable numeric blockHeight is skipped rather than silently downgraded to a slot. The stale perceived value self-heals once no further slot values are fed (perceived-block Redis TTL). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…detector The volume-independent rate cooldown counted every critical/fatal signal, including those from active health-check probes. Health checks can be stricter than user impact — a Solana getBlockHeight sync check, for example, fails an endpoint that serves user reads at 99.8% success — so probe failures were hard- benching user-healthy endpoints out of rotation (observed on canary: nodefleet solana endpoints cooled with score 0 while user traffic saw ~0 critical errors). A hard bench must reflect what USERS experience. Tag health-check signals (Signal.IsHealthCheck, set by the health-check executor) and skip the rate-cooldown EWMA update and trip for them. Health-check signals still move the additive score (soft deprioritization) and the strike system as before — only the volume- independent hard-bench detector is now user-traffic-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
… zero endpoints filterByReputation excluded every endpoint that was in cooldown or below the score threshold and could return an empty map. An empty result drops the caller to a reputation-BLIND fallback (random / least-stale) that ignores which endpoints are actually least-bad. On a fully-degraded service this made things worse: canary observed poly-zkevm go from 50% to 100% error once the rate cooldown had cooled every endpoint and selection collapsed to a random fallback over the same bad pool. Add a guard: when reputation would filter out ALL endpoints, keep the least-bad tier (the endpoints at the maximum score) instead of returning empty, so selection stays reputation-aware over the degraded pool. A hard bench should relatively deprioritize an endpoint, never remove the last ones standing when there is no better alternative. New metric path_reputation_pool_collapse_guard_total signals when a service is fully degraded and being kept on its least-bad endpoints. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
The external block-height floor (ConsumeExternalBlockHeight / block consensus) raised the perceived block height to whatever an external RPC source reported, with no sanity bound. The endpoint observation path already rejects an implausible jump (> MaxBlockHeightJump above the current perceived), but the external-floor path did not — so a single mislabeled or misbehaving source could push perceived arbitrarily high and filter out every honest endpoint. This is not theoretical: a configured Solana external source returned the SLOT (~5% above the real block height) for its block-height method, inconsistently across its load-balanced backends. The unguarded floor wrote that slot into perceived on every poll, tracking the live slot and refreshing the Redis TTL, so the poison never healed and re-seeded within one poll of any manual reset. Honest endpoints reporting the real (lower) height then failed the sync check and were penalized/cooled while users were unaffected. Apply the same plausibility guard used on the endpoint path to every external floor: - solana, cosmos, noop: reject an external height > MaxBlockHeightJump above the current perceived (log a warning, do not raise). - evm: guard both the consensus application (block_consensus.go) and the cross-replica Redis write (qos.go), which wrote the raw external height and so bypassed the consensus bound. An external source can still act as a legitimate floor for realistic staleness (< MaxBlockHeightJump); it simply can no longer poison perceived. Adds regression tests for each QoS reproducing the slot-magnitude jump. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…e for Solana
Two changes that let a stuck/poisoned perceived block height be recovered and prevent
it from recurring at the source.
1. POST /admin/chain-state/clear/{serviceId}
Resets a service's perceived block height (in-memory + Redis) so it rebuilds from fresh
endpoint observations. This is the only way to recover a too-high perceived height: the
max-based consensus and the external floor only ever RAISE perceived, so a poisoned value
(e.g. a Solana height that was set to the slot) cannot self-correct, and the max-wins Redis
write cannot lower it. Mirrors the existing circuit-breaker clear — must be called on each
pod, since the perceived floor is per-pod in-memory state.
- Adds DeletePerceivedBlockNumber to the reputation storage + service interfaces (Redis
DEL / memory delete) and all mocks.
- Adds ResetPerceivedBlockHeight to the EVM, Cosmos, Solana, and NoOp QoS (zeroes the
in-memory floor, clears the EVM external-floor, deletes the Redis key).
- Wires a ChainStateAdmin into the router alongside the circuit-breaker admin.
2. getEpochInfo support for the Solana external block source
The external block fetcher's extractBlockHeight now parses a getEpochInfo result by its
explicit blockHeight field. getBlockHeight returns a bare number that some providers
mislabel with the slot (~5% above the real block height); getEpochInfo.blockHeight is
unambiguous and never confused with the slot. absoluteSlot is never used as a fallback.
Recommend pointing Solana's external_block_sources at method: getEpochInfo.
Full unit suite + lint pass; adds tests for getEpochInfo parsing and the reset path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…tion cap is engaged On a Shannon session rollover, getReconnectEndpoint previously reused the original supplier+URL whenever it survived into the new session (tier-1 seamless rebind), pinning every WebSocket connection to whichever supplier was selected first. Because WebSocket endpoints have no active health checks, their reputation scores nearly always tie, so the tied-best band spans the whole valid set — exactly where the per-operator concentration cap can spread load, but the tier-1 shortcut returned before the cap ever ran. This drops the tier-1 shortcut when the per-operator cap is engaged (0 < max_operator_share < 1): every rollover now routes through selectReconnectEndpointWithCap so connections re-spread across the tied-best operator band each session boundary instead of pinning. The original endpoint stays in the candidate set, so a single-endpoint pool never fails and a strictly-best original still wins — only the tied band is reshuffled. Rotating off a still-present supplier is cheap: the bridge already replays subscriptions on any rebind. When the cap is disabled the seamless tier-1 reuse is preserved: with the cap off, selectReconnectEndpointWithCap collapses to a deterministic smallest-address pick that would funnel every connection onto one endpoint — strictly worse than pinning. differentSupplier is now derived from the chosen address rather than hardcoded, and a rebind that lands back on the original endpoint is reported as a seamless (non-different) outcome. A new [WS-REBIND-ROTATE] log distinguishes an intentional rotation off a still-present supplier from the original having rotated out of the session. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…policy
Split the WS rebind selection policy out of getReconnectEndpoint into a pure
helper, chooseRebindEndpoint, that takes the candidate set, preferredAddr,
avoidPreferred flag, maxOperatorShare, and scoreOf directly. getReconnectEndpoint
keeps the session/reputation plumbing (building endpoints, resolving the cap,
constructing the score function) and delegates the decision to the helper, so
the policy is testable without a live session or reputation service.
No behavior change — this is a straight extraction of the code added in the
previous commit.
Adds table-free unit tests covering every branch of the policy:
- cap disabled → seamless tier-1 reuse when the original is present
- cap disabled → best-available fallback when the original rotated out
- cap engaged → repeated rebinds spread across operators (not pinned), with
the original operator staying eligible and differentSupplier tracking
(chosen != original)
- cap engaged, single-endpoint pool → lands back on the original, never errors
- cap engaged, strictly-best original → reused (rotation only reshuffles ties)
- cap engaged → fallback when the original rotated out
- avoidPreferred → excludes the stalling endpoint, forces a different one
- avoidPreferred, sole endpoint → errors with WSRebindFailedNoEndpoints
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
Add an optional `static_routes` config block (per-service and as a global
default under `defaults`) that lets the gateway serve a fixed response for a
given request path directly, without relaying to a backend endpoint. This is
useful for small service-scoped metadata endpoints whose body is a constant
configured value rather than something fetched from a supplier.
Mechanics:
- A StaticRoute configures path, optional methods, status code, content type,
body, and extra headers. Matching is exact on the path (evaluated after the
/v1 and portal-app-id prefixes are stripped) and, when methods are set, on
the request method (case-insensitive).
- Resolution is per-service-first with a global-default fallback: a service's
route overrides a default on the same path; paths only in the defaults still
apply. Content-Type defaults to text/plain and status to 200.
- The router runs a staticResponseMiddleware after the prefix strip and before
the relay handler, short-circuiting a matching request. Because it fires
before RPC-type detection and endpoint selection, it does not interact with a
service's rpc_types — a REST service is unaffected unless it configures the
same path. WebSocket upgrades are never intercepted, and `path: "/"` is
rejected so a route can never shadow all relay traffic.
Static-at-boot: routes are read from config once at load; there is no hot
reload. The resolver is the UnifiedServicesConfig itself, injected into the
router via a small StaticResponseResolver interface (nil = feature off).
Adds validation (well-formed paths, unique path+method coverage, status range),
unit tests for resolution/override/method-filtering/header-copy, and router
middleware tests (served-on-match, relay-on-no-match, websocket bypass).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…nd trigger metric
Two changes to the WebSocket session-rebind path.
1. Operator-uniform rebind strategy (config-driven, default ON)
The rebind added previously re-spread connections via the concentration cap,
which is endpoint-count-weighted: an operator's share tracks how many endpoints
it runs, so a provider that dominates the endpoint pool still receives most of
the rebinds (up to the cap ceiling). This adds an alternative strategy that
picks the target operator uniformly — every provider equally likely — then an
endpoint within it, giving the strongest per-provider spread.
- New selector.SelectOperatorUniform: uniform over operators (eTLD+1), then
uniform within the chosen operator, over the tied-best reputation band so a
degraded endpoint is never a target just because its operator is thin.
- New per-service / global config `websocket_rebind_operator_uniform`
(DefaultWebsocketRebindOperatorUniform = true). When false, the rebind falls
back to the endpoint-count-weighted max_operator_share cap.
- chooseRebindEndpoint takes the strategy and rotates whenever either strategy
is active (operator-uniform on, or the cap engaged), pinning only when both
are off.
Trade-off (documented on the config field): operator-uniform gives a
single-endpoint operator the same share as a large one, so a thin provider
absorbs a full 1/m of the WebSocket load. Disable per-service if that
overloads a small operator.
2. Rebind trigger metric label
path_websocket_rebind_total gains a `trigger` label (rollover/stall) so rebind
outcomes can be read per cause — e.g. whether stalls rotate suppliers more
often than routine rollovers. The trigger is stashed at the start of
ReconnectEndpoint (avoidCurrentSupplier => stall, else rollover) and consumed
by OnReconnectOutcome.
Adds selector tests (equal-per-operator share, within-operator uniformity,
boundaries), chooseRebindEndpoint operator-uniform tests (equalizes operator
share despite 8:1:1 endpoint skew; rotates even with the cap disabled), and a
config-resolution test (per-service > global default > ON).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…edge blind spot)
When a relay exhausts all retry attempts it returns an error to the client (a
500 on the lastErr path), but path_requests_total is recorded only for
completed relays — so these 500s are invisible in PATH's own request metric
while the edge proxy counts them on the wire. That gap can read as "PATH is
clean" during an error spike the edge is alerting on.
Add path_relay_exhausted_total{service_id, rpc_type, category}, incremented
once per give-up in the single-relay retry loop. category classifies the last
failure:
- capability: the backend correctly reported unavailable historical/pruned
state (IsCapabilityLimitationError) — a client asking for data the node
does not retain. This isolates the dominant, benign-but-mis-shaped case
(historical eth_getBalance/eth_getLogs) from genuine failures.
- heuristic: any other heuristic-detected error payload in a 2xx response.
- transport: a connection/protocol error with no heuristic result.
- status: a non-success HTTP status with neither of the above.
Observability only — no behavior change. The retry/response path is untouched;
this only adds a counter and a pure classifier. The batch-item exhaustion path
is not yet instrumented (it does not track the last heuristic result in scope);
the single-relay path covers the observed case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
…sult to siblings
Many staked suppliers front the SAME backend URL (measured live: 2-4x
redundancy per service, up to 10 suppliers behind one URL). Previously every
supplier fired its own health-check relay each cycle, so a shared backend was
probed N times over — dominating health-check relay volume on large operators.
Each cycle now groups a service's endpoints by exact backend URL and fires ONE
relay per URL through a rotating representative supplier. The backend-derived
result — reputation signal, block-height observation, and archival verdict — is
fanned to the other suppliers on that URL without an additional relay.
Correctness:
- Rotation (cycle % groupSize) gives every supplier a directly-probed turn every
N cycles, so its own relay path (signing, session, stake, relay-miner) is
validated rather than permanently assumed from a sibling.
- WebSocket checks are never deduped — connectivity is genuinely per-endpoint;
siblings still probe their own WebSocket.
- The block-height observation is replayed to siblings so their stored height
stays fresh and they are not falsely filtered as out-of-sync.
- Over-serviced (stake-exhausted) results fan the same no-penalty rule.
- The relay-volume metric stays with the representative that made the relay, so
path_relays_total{request_type="health_check"} drops by the dedup factor;
siblings still emit their per-supplier signal metric for reputation visibility.
Gated by active_health_checks.backend_dedup (default on). New counter
path_health_check_deduped_total{service_id} tracks skipped relays.
The representative path is unchanged (capture is a pure addition), so the change
is additive and reversible via config.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
Replace incidental real-domain and app-specific example values in the static-route docs/tests and reputation key tests/comments with generic placeholders (example.com, /example, example-body) for a cleaner, implementation-agnostic public diff. No functional change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFKwFyDKAtnDYtpmhqsG8E
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates the operator-concentration + reputation-hardening work that has been running and validated in production (both environments) into
main. 24 commits, primarily inqos/,reputation/,router/, andgateway/.Recommend squash-merge.
What's included
Per-operator concentration cap
max_operator_share: 0.65); extended to EVM, Cosmos, Solana, and NoOp; two-phase (skips grouping when it can't reshape) with reshape observability.Reputation
path_relays_total.Health checks
WebSocket
QoS / Solana
getEpochInfoexternal source for Solana.Router / ops / metrics
Testing
go build ./...clean.🤖 Generated with Claude Code