Gateway Health data for MCP + LLM Gateway#28
Open
aryasuneesh-quilr wants to merge 8 commits into
Open
Conversation
Replaces the fully-mocked LLM Gateway section (fake per-region breakdown) with real synthetic canary data - chat completions, streaming, and model listing - fetched from the same rollup server used for MCP Gateway. Every LLM probe is routed to a zero-cost fake provider (health-check/synthetic_llm_provider.py) via a synthetic API key, so this never touches or bills a real model. Also fixes two bugs in the probe worker that were silently failing every LLM probe cycle since it started: the key-name constant pointed at a stale key created before selected_models was required, and the key-reuse path read the wrong response field (api_key instead of quilr_api_key), so even the correct key was never actually returned.
Adds a plain HTTPS reachability probe (health-check/worker.py's _probe_region) against the real production LLM Gateway regions - guardrails-usa-1/usa-2/india-1.quilr.ai - matching the same host set and the same "skip the global auto-router, it's not meaningful to probe directly" choice already used by the QuilrAI Infrastructure tab's browser-side ping. Unlike that tab, this runs server-side on a schedule and persists results, so Gateway Health gets real hourly history instead of an ephemeral per-visit check. No API key or synthetic provider needed for this - it's an outbound HTTPS request to a public host, same as any browser hitting the site, so no production writes and nothing new to expose publicly. The rollup server and frontend now surface these three regions under Gateway Health's LLM Gateway section instead of the chat/streaming/ models scenario rows (worker still records those separately).
Adds chat completions, streaming, and model listing back to the public rollup allowlist so they render in the Gateway Health tab alongside the region reachability rows, instead of only recording silently. These are a genuinely different, deeper signal than the region pings: confirmed the synthetic key carries the default enabled risk categories, so every chat/streaming probe actually exercises the real guardrails/DLP detection pass, not a bypass - all through a zero-cost synthetic provider, never a real, billable model.
The MCP/LLM Gateway row descriptions and group labels referenced internal implementation details (synthetic canary, zero-cost, fake provider) that don't belong on a public-facing status page. Reworded every user-visible note to describe what each check verifies without naming the mechanism behind it.
Fake LLM/MCP test doubles and the probe worker/rollup server never belonged in this public, backend-less static site. They now live at OTHER_APPS/gateway_health/ in macrodata-refinement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the mocked Gateway Health tab with live zero-cost synthetic checks against the actual MCP Gateway and LLM Gateway plus real reachability against production LLM Gateway regions.
Gateway Health: split into two grouped sections, MCP Gateway and LLM Gateway, each row backed by one specific check:
The backend that produces this data (synthetic LLM/MCP test doubles, probe worker, public rollup API) does not live in this repo -
docsis a public, static site with no backend, so it now lives inmacrodata-refinementunderOTHER_APPS/gateway_health/. See macrodata-refinement#218 for that implementation, including systemd units for production, dynamic port allocation for the two synthetic services, and pinned dependencies.Files changed:
src/components/GatewayHealth/GatewayTimeline.js:
rollup_server.py's endpoint once and partitions the response into MCP Gateway / LLM Gateway sections using agroupfield the server provides.src/components/GatewayHealth/timeline.module.css:
.groupHead/.groupLabel/.groupNote/.groupErrorstyles to visually separate the MCP Gateway and LLM Gateway sections within the same tab.Setting this up on production:
src/components/GatewayHealth/GatewayTimeline.js:GATEWAY_ROLLUP_URLneeds to change from127.0.0.1:8099to wherever the rollup server ends up publicly hosted, over HTTPS (browsers block plain HTTP fetches from an HTTPS page). See macrodata-refinement#218 for the open decision on where that public route lives and how it gets there.A cross-repo dependency: the
/internal/gateway-health/bootstrapand/internal/gateway-health/probe-resultsroutes the worker depends on live inquilr-llm-gateway, on thehealth-check-dashboardbranch. There's a parallel PR againstquilr-llm-gateway'shealth-check-dashboardbranch to land that side - it also carries two fixes found while diffing that repo's worker against this one line by line:_ensure_llm_key()read an already-existing key back askey.get("api_key"), butGET /api-keys/listreturns it underquilr_api_key(confirmed againsthandlers/api_key_handler.py). The very first run worked, because the create response really does useapi_key- but every run after that found the key already existed, read backNone, and sentAuthorization: Bearer None, failing every chat/streaming/models probe from the second cycle onward. Fixed to readquilr_api_key._probe_region/LLM_REGIONS) never existed in quilr-llm-gateway's copy of the worker at all. Without them, the check itself is a harmless plain GET, but nothing ever POSTs its result to/internal/gateway-health/probe-results, so the Region rows would have sat atnodataforever regardless of anything in this repo. Ported over.Network reachability, for both synthetic services: if you want the deep completion checks (Chat Completions/Streaming/Model Listing) and the direct/OneMCP checks running against real production, the real LLM Gateway needs to reach the synthetic LLM provider and the real MCP Gateway needs to reach the synthetic MCP backend, both over the network from wherever they actually run (now
macrodata-refinement, see macrodata-refinement#218). Practically that means both fake services need to sit inside the same private network/VPC as production or be given public addresses (with the exposure tradeoff that implies) - there's no way around this being a real network dependency in both directions.