Week 5 Day 2: add provider aggregate metrics

README.md

@@ -70,6 +70,7 @@ Useful local URLs:
 - Demo metrics: `http://localhost:8000/metrics`
 - AI SRE Assistant: `http://localhost:8001`
 - AI SRE health: `http://localhost:8001/health`
+- AI SRE provider metrics: `http://localhost:8001/metrics`
 
 For the kind-based Kubernetes walkthrough, see `infra/k8s/README.md`.
 
@@ -99,7 +100,7 @@ For the Week 4 release gates, readiness verdict, and next-stage priorities, see
 
 For the phased commercialization roadmap with milestones and success metrics, see `docs/21-commercialization-roadmap.md`.
 
-For the Week 5 provider telemetry contract and privacy boundary, see `docs/22-provider-telemetry.md`.
+For the Week 5 provider telemetry contract, aggregate metrics, and privacy boundary, see `docs/22-provider-telemetry.md`.
 
 Ask the assistant directly:
 

apps/ai-sre-assistant/README.md

@@ -7,6 +7,7 @@ It can use an OpenAI-compatible LLM provider for richer analysis. If no provider
 ## API
 
 - `GET /health`
+- `GET /metrics`
 - `POST /analyze/logs`
 - `POST /analyze/metrics`
 - `POST /ask`
@@ -84,7 +85,9 @@ See `../../docs/16-cost-optimization.md` for the Day 3 cost optimization guide.
 
 When `use_llm=true`, LLM-enabled API responses include `llm_telemetry` with bounded provider and model labels, configuration and attempt state, success or fallback outcome, request latency, and normalized token usage when the provider reports it.
 
-The telemetry object never includes prompts, incident evidence, credentials, provider base URLs, generated output, or user and incident identifiers. It is per-request metadata, not yet a persistent usage ledger, billing record, cost estimate, or Prometheus metrics implementation.
+The telemetry object never includes prompts, incident evidence, credentials, provider base URLs, generated output, or user and incident identifiers. It is per-request metadata, not a persistent usage ledger, billing record, or cost estimate.
+
+`GET /metrics` exposes privacy-safe in-memory aggregates for analysis outcomes, provider request latency, deterministic fallbacks, and provider-reported input/output tokens. Labels are limited to the configured provider/model and fixed outcome, reason, and direction enums. These process-local counters reset when the assistant restarts.
 
 See `../../docs/22-provider-telemetry.md` for field semantics, privacy rules, failure behavior, and the remaining Week 5 sequence.
 

apps/ai-sre-assistant/app/llm.py

@@ -7,6 +7,7 @@
 import httpx
 
 from app.prompts import ANALYSIS_INSTRUCTIONS, SYSTEM_PROMPT
+from app.provider_metrics import PROVIDER_METRICS
 from app.redaction import redact_data, redact_text
 
 
@@ -215,7 +216,7 @@ def _telemetry(
     request_latency_ms: float | None = None,
     usage: Any = None,
 ) -> dict[str, Any]:
-    return {
+    telemetry = {
         "provider": _bounded_label(config.provider or "none"),
         "model": _bounded_label(config.model) if configured else None,
         "configured": configured,
@@ -226,6 +227,8 @@ def _telemetry(
         "fallback_reason": fallback_reason,
         "usage": _normalize_usage(usage),
     }
+    PROVIDER_METRICS.observe(telemetry)
+    return telemetry
 
 
 def _normalize_usage(usage: Any) -> dict[str, int | bool | None]:

apps/ai-sre-assistant/app/main.py

@@ -1,13 +1,14 @@
 from typing import Any
 
-from fastapi import FastAPI
+from fastapi import FastAPI, Response
 from pydantic import BaseModel, Field
 
 from app.analyzer import analyze_logs
 from app.llm import analyze_with_llm, cost_controls_summary, load_config
 from app.log_reader import get_log_path, read_recent_logs
 from app.metrics_analyzer import analyze_metrics, combined_incident_analysis
 from app.metrics_reader import fetch_metrics_text, get_metrics_url, parse_prometheus_text
+from app.provider_metrics import PROVIDER_METRICS
 from app.redaction import redact_data, redact_text
 
 
@@ -47,6 +48,11 @@ def health() -> dict[str, str]:
     return {"status": "healthy", "service": "ai-sre-assistant"}
 
 
+@app.get("/metrics")
+def metrics() -> Response:
+    return Response(content=PROVIDER_METRICS.render_prometheus(), media_type="text/plain")
+
+
 @app.post("/analyze/logs")
 def analyze_recent_logs(request: AnalyzeLogsRequest) -> dict[str, Any]:
     return _analyze(question="What happened recently in demo-service logs?", max_lines=request.max_lines, use_llm=request.use_llm)

apps/ai-sre-assistant/app/provider_metrics.py

@@ -0,0 +1,189 @@
+from collections import Counter, defaultdict
+from math import isfinite
+from threading import Lock
+from typing import Any
+
+from app.redaction import redact_text
+
+
+PROVIDER_LATENCY_BUCKETS_SECONDS = (
+    0.05,
+    0.1,
+    0.25,
+    0.5,
+    1.0,
+    2.5,
+    5.0,
+    10.0,
+    30.0,
+    float("inf"),
+)
+MAX_METRIC_LABEL_CHARS = 100
+VALID_OUTCOMES = {"success", "failure", "not_configured"}
+VALID_FALLBACK_REASONS = {"provider_not_configured", "provider_request_failed"}
+
+
+class ProviderMetricsRegistry:
+    """Small in-memory provider registry with a fixed, privacy-safe label contract."""
+
+    def __init__(self) -> None:
+        self._lock = Lock()
+        self._analyses: Counter[tuple[str, str, str]] = Counter()
+        self._latency_buckets: Counter[tuple[str, str, str, float]] = Counter()
+        self._latency_count: Counter[tuple[str, str, str]] = Counter()
+        self._latency_sum: defaultdict[tuple[str, str, str], float] = defaultdict(float)
+        self._fallbacks: Counter[tuple[str, str, str]] = Counter()
+        self._tokens: Counter[tuple[str, str, str]] = Counter()
+
+    def reset(self) -> None:
+        with self._lock:
+            self._analyses.clear()
+            self._latency_buckets.clear()
+            self._latency_count.clear()
+            self._latency_sum.clear()
+            self._fallbacks.clear()
+            self._tokens.clear()
+
+    def observe(self, telemetry: dict[str, Any]) -> None:
+        provider = _bounded_label(telemetry.get("provider"), default="unknown")
+        model = _bounded_label(telemetry.get("model"), default="none")
+        outcome = telemetry.get("outcome")
+        outcome = outcome if outcome in VALID_OUTCOMES else "unknown"
+
+        with self._lock:
+            self._analyses[(provider, model, outcome)] += 1
+
+            latency_seconds = _latency_seconds(telemetry)
+            if latency_seconds is not None:
+                latency_key = (provider, model, outcome)
+                self._latency_count[latency_key] += 1
+                self._latency_sum[latency_key] += latency_seconds
+                for bucket in PROVIDER_LATENCY_BUCKETS_SECONDS:
+                    if latency_seconds <= bucket:
+                        self._latency_buckets[(*latency_key, bucket)] += 1
+
+            fallback_reason = telemetry.get("fallback_reason")
+            if telemetry.get("fallback_used") is True:
+                reason = (
+                    fallback_reason
+                    if fallback_reason in VALID_FALLBACK_REASONS
+                    else "other"
+                )
+                self._fallbacks[(provider, model, reason)] += 1
+
+            usage = telemetry.get("usage")
+            if isinstance(usage, dict):
+                for direction, field in (
+                    ("input", "input_tokens"),
+                    ("output", "output_tokens"),
+                ):
+                    value = usage.get(field)
+                    if (
+                        isinstance(value, int)
+                        and not isinstance(value, bool)
+                        and value >= 0
+                    ):
+                        self._tokens[(provider, model, direction)] += value
+
+    def render_prometheus(self) -> str:
+        with self._lock:
+            analyses = self._analyses.copy()
+            latency_buckets = self._latency_buckets.copy()
+            latency_count = self._latency_count.copy()
+            latency_sum = dict(self._latency_sum)
+            fallbacks = self._fallbacks.copy()
+            tokens = self._tokens.copy()
+
+        lines = [
+            "# HELP ai_sre_provider_analyses_total LLM enrichment selections by provider, model, and outcome.",
+            "# TYPE ai_sre_provider_analyses_total counter",
+        ]
+        for (provider, model, outcome), count in sorted(analyses.items()):
+            labels = _labels(provider=provider, model=model, outcome=outcome)
+            lines.append(f"ai_sre_provider_analyses_total{{{labels}}} {count}")
+
+        lines.extend(
+            [
+                "# HELP ai_sre_provider_request_duration_seconds Attempted provider request duration in seconds.",
+                "# TYPE ai_sre_provider_request_duration_seconds histogram",
+            ]
+        )
+        for provider, model, outcome in sorted(latency_count):
+            for bucket in PROVIDER_LATENCY_BUCKETS_SECONDS:
+                labels = _labels(
+                    provider=provider,
+                    model=model,
+                    outcome=outcome,
+                    le=_format_bucket(bucket),
+                )
+                count = latency_buckets[(provider, model, outcome, bucket)]
+                lines.append(
+                    f"ai_sre_provider_request_duration_seconds_bucket{{{labels}}} {count}"
+                )
+
+            base_labels = _labels(provider=provider, model=model, outcome=outcome)
+            lines.append(
+                f"ai_sre_provider_request_duration_seconds_sum{{{base_labels}}} "
+                f"{latency_sum[(provider, model, outcome)]:.6f}"
+            )
+            lines.append(
+                f"ai_sre_provider_request_duration_seconds_count{{{base_labels}}} "
+                f"{latency_count[(provider, model, outcome)]}"
+            )
+
+        lines.extend(
+            [
+                "# HELP ai_sre_provider_fallbacks_total Deterministic provider fallbacks by bounded reason.",
+                "# TYPE ai_sre_provider_fallbacks_total counter",
+            ]
+        )
+        for (provider, model, reason), count in sorted(fallbacks.items()):
+            labels = _labels(provider=provider, model=model, reason=reason)
+            lines.append(f"ai_sre_provider_fallbacks_total{{{labels}}} {count}")
+
+        lines.extend(
+            [
+                "# HELP ai_sre_provider_tokens_total Provider-reported tokens by input or output direction.",
+                "# TYPE ai_sre_provider_tokens_total counter",
+            ]
+        )
+        for (provider, model, direction), count in sorted(tokens.items()):
+            labels = _labels(provider=provider, model=model, direction=direction)
+            lines.append(f"ai_sre_provider_tokens_total{{{labels}}} {count}")
+
+        return "\n".join(lines) + "\n"
+
+
+def _latency_seconds(telemetry: dict[str, Any]) -> float | None:
+    if telemetry.get("attempted") is not True:
+        return None
+    latency_ms = telemetry.get("request_latency_ms")
+    if not isinstance(latency_ms, (int, float)) or isinstance(latency_ms, bool):
+        return None
+    if not isfinite(latency_ms) or latency_ms < 0:
+        return None
+    return latency_ms / 1000
+
+
+def _bounded_label(value: Any, *, default: str) -> str:
+    if value is None:
+        return default
+    label = redact_text(str(value).strip())[:MAX_METRIC_LABEL_CHARS]
+    return label or default
+
+
+def _format_bucket(bucket: float) -> str:
+    if bucket == float("inf"):
+        return "+Inf"
+    return f"{bucket:g}"
+
+
+def _labels(**labels: str) -> str:
+    return ",".join(f'{key}="{_escape_label(value)}"' for key, value in labels.items())
+
+
+def _escape_label(value: str) -> str:
+    return str(value).replace("\\", "\\\\").replace("\n", "\\n").replace('"', '\\"')
+
+
+PROVIDER_METRICS = ProviderMetricsRegistry()

apps/ai-sre-assistant/tests/test_provider_metrics.py

@@ -0,0 +1,100 @@
+from fastapi.testclient import TestClient
+
+from app.main import app
+from app.provider_metrics import ProviderMetricsRegistry
+
+
+def test_provider_metrics_aggregate_outcomes_latency_fallbacks_and_tokens():
+    metrics = ProviderMetricsRegistry()
+
+    metrics.observe(
+        {
+            "provider": "openai",
+            "model": "approved-model",
+            "attempted": True,
+            "outcome": "success",
+            "request_latency_ms": 125.0,
+            "fallback_used": False,
+            "fallback_reason": None,
+            "usage": {"input_tokens": 120, "output_tokens": 30},
+        }
+    )
+    metrics.observe(
+        {
+            "provider": "openai",
+            "model": "approved-model",
+            "attempted": True,
+            "outcome": "failure",
+            "request_latency_ms": 2500.0,
+            "fallback_used": True,
+            "fallback_reason": "provider_request_failed",
+            "usage": {},
+        }
+    )
+
+    rendered = metrics.render_prometheus()
+
+    assert (
+        'ai_sre_provider_analyses_total{provider="openai",model="approved-model",outcome="success"} 1'
+        in rendered
+    )
+    assert (
+        'ai_sre_provider_analyses_total{provider="openai",model="approved-model",outcome="failure"} 1'
+        in rendered
+    )
+    assert (
+        'ai_sre_provider_request_duration_seconds_bucket{provider="openai",model="approved-model",'
+        'outcome="success",le="0.25"} 1'
+    ) in rendered
+    assert (
+        'ai_sre_provider_request_duration_seconds_count{provider="openai",model="approved-model",'
+        'outcome="failure"} 1'
+    ) in rendered
+    assert (
+        'ai_sre_provider_fallbacks_total{provider="openai",model="approved-model",'
+        'reason="provider_request_failed"} 1'
+    ) in rendered
+    assert (
+        'ai_sre_provider_tokens_total{provider="openai",model="approved-model",direction="input"} 120'
+        in rendered
+    )
+    assert (
+        'ai_sre_provider_tokens_total{provider="openai",model="approved-model",direction="output"} 30'
+        in rendered
+    )
+
+
+def test_provider_metrics_keep_label_values_bounded_and_ignore_sensitive_payload_fields():
+    metrics = ProviderMetricsRegistry()
+
+    metrics.observe(
+        {
+            "provider": "Bearer secret-provider-token",
+            "model": "model-name",
+            "attempted": False,
+            "outcome": "unexpected-outcome",
+            "request_latency_ms": 123.0,
+            "fallback_used": True,
+            "fallback_reason": "arbitrary-error-message",
+            "prompt": "private incident evidence",
+            "usage": {"input_tokens": -1, "output_tokens": True},
+        }
+    )
+
+    rendered = metrics.render_prometheus()
+
+    assert "secret-provider-token" not in rendered
+    assert "private incident evidence" not in rendered
+    assert 'outcome="unknown"' in rendered
+    assert 'reason="other"' in rendered
+    assert "ai_sre_provider_request_duration_seconds_count" not in rendered
+    assert "ai_sre_provider_tokens_total{" not in rendered
+
+
+def test_provider_metrics_endpoint_exposes_prometheus_contract():
+    response = TestClient(app).get("/metrics")
+
+    assert response.status_code == 200
+    assert response.headers["content-type"].startswith("text/plain")
+    assert "# TYPE ai_sre_provider_analyses_total counter" in response.text
+    assert "# TYPE ai_sre_provider_request_duration_seconds histogram" in response.text

docs/09-roadmap.md

@@ -43,13 +43,13 @@ The [commercialization roadmap](21-commercialization-roadmap.md) runs alongside
 
 ## Week 5 - Provider Telemetry
 
-- Day 1: expose a bounded per-call contract for provider/model identity, request latency, token usage, outcome, and deterministic fallback without storing sensitive content.
-- Add aggregate counters and latency distributions with bounded labels.
+- Day 1 - complete: expose a bounded per-call contract for provider/model identity, request latency, token usage, outcome, and deterministic fallback without storing sensitive content.
+- Day 2 - complete: add aggregate counters and latency distributions with bounded labels.
 - Define explicit pricing inputs and estimated cost metadata without hard-coding unstable provider prices.
 - Join provider usage and cost with evaluation outcomes.
 - Calculate cost per successful evaluated analysis.
 
-See [Provider Telemetry Contract](22-provider-telemetry.md) for the Day 1 API contract, privacy boundary, and remaining-week sequence.
+See [Provider Telemetry Contract](22-provider-telemetry.md) for the Day 1 per-request contract, Day 2 aggregate metrics, privacy boundary, and remaining-week sequence.
 
 **Exit gate:** provider usage, reliability, and cost can be compared with evaluation outcomes without storing prompts, incident evidence, credentials, endpoints, or generated content.