feat: v0.10.0 — resilience & dynamic ops (Phase 1) - #29
Merged
Conversation
- per-backend circuit breaker: open after N consecutive failures, 503 meridian_circuit_open pre-flight, half-open probe after open_seconds; state in /meridian/status + meridian_backend_circuit_open gauge - retry with exponential backoff on transport errors (max_retries, retry_backoff_base; default 0 = off); meridian_upstream_retries_total - configurable timeouts: global timeouts block + per-backend backends[].timeout merge; stream_read for SSE - graceful stream read timeout: SSE error event + [DONE], accounted 504 - full dynamic config reload via POST /meridian/reload + SIGHUP: strategy, backends, tiering, health, resilience, affinity, rate limits, telemetry; atomic (bad config → 400, running state kept); keys-only fallback when no config file backs the process - 28 new tests (resilience state machine, proxy gates, retry/backoff, timeout merge, stream timeout e2e, reload atomicity); 406 total, 87% cov Validated live: 502→circuit-open 503 on dead backend, full-scope reload swaps strategy via endpoint + SIGHUP, invalid config rejected atomically.
- forward_stream accepts resilience config: retries transport errors ONLY in the open phase (pre-first-byte) with the same backoff; counting in meridian_upstream_retries_total - a started stream still never retries (delivered tokens can't be un-sent) - any upstream transport failure on a stream — open or mid-stream — now ends well-formed: SSE error event + data: [DONE], accounted via StreamUpstreamError (504 stream_read_timeout / 502 <exc name>) - replaces StreamReadTimeout with the general StreamUpstreamError - +4 tests: open-phase retry-to-success, retry-exhausted graceful end, no-retry default single attempt, mid-stream protocol error 410 tests pass; forward.py 95% covered; ruff + mypy clean
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What / why
Phase 1 of the complete-product track (
docs/FULL.md) — closes the remaining P0 items. v1.0 ships feature-complete, so resilience + dynamic ops land under an honest0.10.0tag first.Deliverables (all FULL.md Phase 1 tracks)
failure_thresholdconsecutive failures; open →503 meridian_circuit_openpre-flight (no upstream call); half-open probe afteropen_seconds; success closes / failure re-opens; state in/meridian/status+meridian_backend_circuit_open{backend}max_retries(default 0 = off) +retry_backoff_baseonhttpx.RequestError. Safe windows only: non-stream requests + the stream open phase (pre-first-byte); a started stream never retries. Counted inmeridian_upstream_retries_total{backend}[DONE]; stalled streams accounted 504stream_read_timeout, other transport errors 502timeouts.{connect,read,write,pool}+ per-backendbackends[].timeoutmerge;stream_readfor SSE; defaults preserve current behaviorPOST /meridian/reload+ SIGHUP apply strategy/backends/weights/tiering/health/resilience/affinity/rate-limits/telemetry without restart; atomic (bad config → 400, state kept); keys-only fallback when no config fileTests
test_resilience.py,test_config_reload.py): breaker state machine incl. half-open transitions, proxy gate, retry counting/backoff (non-stream + stream-open), timeout merge, stream timeout + mid-stream protocol error end-to-end, reload atomicity + rollback.meridian/(resilience.py100%,forward.py95%); ruff + mypy clean.Manual validation (live process)
502, req2 →503 meridian_circuit_open;/meridian/statusshows{"state":"open"}; gauge = 1.POST /meridian/reload→{"reloaded":true,"scope":"full"}, strategy swapped toewma_latencywithout restart.400, running strategy unchanged.kill -HUP→SIGHUP: reload complete ({'scope': 'full' ...}).Docs
CHANGELOG [0.10.0], CONFIGURATION reference (
timeouts,resilience), DEPLOY (Config reload (no restart)), ROADMAP/ship/FULL status flips (P0-1…P0-5 ✅).Known limits (documented)