Phase 3 — platform depth & multi-tenancy (v0.12.0) - #32
Merged
Conversation
In dedicated mode, orgs listed in isolation.pools are pinned to backend tag pools (subset semantics, same as tiering). Pinned orgs never fall back outside their pool — empty pool is a 503, not a silent land on a neighbor's capacity. Unlisted orgs are excluded from every reserved pool, so noise can't starve a dedicated pool in either direction. - IsolationConfig (mode, pools) + validation - meridian/router/isolation.py: pure visibility predicates - routing: org_confined selection path, tier∩pool preference with in-pool-only fallback, pool-confined idle wake - session affinity: pins revalidated against org visibility; out-of-pool pins are remapped, not honored - 21 new tests (21 pass, 446 total)
…ack 3)
Keys are now administrable without config edits while auth.keys_file
stays the durable source of truth: every mutation is written back
atomically (temp+rename, fsync, other YAML sections preserved) and
hot-swapped into the index via the existing reload path.
- key_id: stable non-secret id (mrdn_ + first 8) on KeyConfig +
IdentityContext — flows to ops views now, usage metering next
- authz: can_manage_keys (ops_admin or role admin) + require_key_admin
- GET /meridian/keys: redacted listing, source-tagged (inline vs file)
- POST /meridian/keys: generated or explicit key; full value returned
exactly once in the 201 response; never listed raw afterward
- DELETE /meridian/keys/{key_id}: file keys only — inline keys refused
with a clear message, unknown id 404, ambiguous id 409
- Requires auth.enabled + auth.keys_file; clean 400/401/403 matrix
- 22 new tests (468 total)
Budget meter gains a 'key' scope (budgets.keys[key_id]) capped per API key id, cascading with org/team/user; cost ledger rows now carry the presenting key's non-secret id, so per-key accounting is queryable. - BudgetConfig.keys + build_meter_keys key scope - CostRow/CostLedger key_id dimension (record + query filter) - sqlite: pre-0.12 DBs auto-migrate via table rebuild; old rows carry key_id='' so historical sums are preserved - /meridian/usage?key= filter + key_id column on JSON and CSV; org/team authz clamps unchanged (non-admin can't cross orgs) - 17 new tests (478 total)
rate_limit.global and rate_limit.models.<model> buckets join the existing
per-org/ip scope. All configured scopes are checked first, any failure
rejects with 429 before any bucket consumes — a hot tenant can't burn
the fleet-wide bucket with requests its own bucket would reject.
- global_limit (alias for the Python-keyword YAML key 'global') +
models: Dict[model_id, override]
- meridian_ratelimit_rejections_total{scope=global|model|org|ip}
- Retry-After format unchanged (existing test pinned)
- Reload picks scope changes up via the existing rate_limit diff → store
rebuild (all buckets reset, warned)
- 12 new tests (486 total)
Weight-split traffic between stable and canary tag pools: per-request roll, time-based step promotion, error-based auto-rollback to weight 0 over a rolling window (min-sample guarded). Empty pool spills to the other side; untagged deployments fall back to default routing instead of 503-ing. - CanaryController: thread-safe, injectable clock, HealthChecker-style start/stop, /meridian/status block, CANARY_WEIGHT + CANARY_ROLLBACKS metrics - Routing: _canary_select with tier-pool preference and org visibility at every lookup; dedicated-mode pinned orgs bypass; session pins onto a rolled-back pool are invalidated (remap), weight tuning alone keeps them - Reload: controller rebuilt on section change; auto-rollback state carries over via merge_from when pool tag identity is unchanged - Config: non-empty, disjoint, extra-forbid validated when enabled - finalize_request feeds canary-pool outcomes into the rollback window
…s, cost) Isolation (H3, M8, M9): - Reject empty pool lists, blank org keys/tags, and unknown section keys — an org pinned to an empty tag list could see every backend (empty-set ⊆) - Session pins namespaced by org: cross-tenant session-id reuse can no longer read or overwrite another org's pin - warn_pool_tag_coverage at startup and reload: loud operator signal when isolation/canary pools match zero backends or reserved pools cover all Keys (H4, M1, M2, M3): - keys_file rewrite preserves existing mode, fresh files default 0600 (previously a rewrite widened 0600 → umask-default, e.g. 0664) - create-side key_id conflict check mirrors delete/identity resolution (explicit inline key_id respected), preventing undeletable keys - Self-delete and last-key-admin deletes are refused (409) — the lifecycle API can no longer brick itself - reload_config takes the lifecycle lock, so a config swap can't strand a just-written key in a stale keys_file path Rate limits (M4): - RateLimitStore.check_multi: all scopes checked then all consumed inside one store-locked window — no over-admission under concurrency, no discarded consume results Cost ledger (H5): - Pre-0.12 sqlite migration runs as one transaction (BEGIN IMMEDIATE) and resumes idempotently (INSERT OR IGNORE) from a leftover staging table - Corrupt double-legacy state (legacy table + staging table) fails loudly instead of silently dropping history
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 3 from
docs/FULL.md— five tracks + a full-branch code review with hardening fixes. 7 commits, 537 tests green (+51).Tracks
ed9ca8f):isolation.mode: shared | dedicated; pinned orgs confined to tag pools (no cross-pool fallback), unlisted orgs excluded from every reserved pool; session affinity remaps out-of-pool pins.dcb9f01): weighted split betweenstable_tags/canary_tagspools; time-based step promotion; error-rate auto-rollback over a rolling window;meridian_canary_weightgauge +meridian_canary_rollbacks_total;/meridian/statusblock; hot-reload rebuild that carries rollback state (no silent re-arm).a7fdae1):GET/POST/DELETE /meridian/keys, admin-only; atomic write-back toauth.keys_file; hot index swap; raw key returned exactly once.240efd5):budgets.keys[{key_id}]caps; cost ledgerkey_idcolumn;/meridian/usage?key=filter; pre-0.12 sqlite migration.fd4fc23):global_limit+models: {}scopes checked global → model → org/ip before consuming.2f0082b): 5 highs + 9 mediums, independently reproduced then fixed with regression tests — empty-pool isolation escape, keys_file permission loss on rewrite, non-atomic sqlite migration, session-pin org namespacing, atomic multi-scope rate limiting, self-delete/last-admin guards, reload mid-mutation safety, and more (see commit body).c48cd2f): CHANGELOG, CONFIGURATION (isolation/canary/keys/per-key budgets/rate-limit scopes), OPS_RUNBOOK, ship.md, CLAUDE.md module map,0.12.0bump.Validation
ruff check .— cleanmypy meridian— clean (67 files)pytest -q— 537 passed (486 at branch start)Compatibility notes
/meridian/usage.csvgains akey_idcolumn (positional consumers should read headers or the release notes).isolation:andcanary:config sections areextra="forbid"— unknown keys (typos) now fail load/reload.sharedisolation, canary off, no new budgets/rate-limit scopes).