Week 5 Day 4: provider evaluation cost report

.env.example

@@ -10,7 +10,7 @@ MODEL_NAME=gpt-4o-mini
 LLM_MAX_LOG_ENTRIES=50
 LLM_MAX_PROMPT_CHARS=12000
 
-# Optional operator-supplied pricing for per-call estimates. Leave blank when unknown.
+# Optional operator-supplied cost inputs for per-call estimates. Leave blank when unknown.
 # Values are USD per one million provider-reported input/output tokens.
 LLM_INPUT_USD_PER_MILLION_TOKENS=
 LLM_OUTPUT_USD_PER_MILLION_TOKENS=

README.md

@@ -94,11 +94,10 @@ For the assistant evaluation corpus and quality rubric, see `docs/17-assistant-e
 
 For the production observability architecture and staged migration path, see `docs/18-production-observability.md`.
 
-For the advanced model serving decision framework and commercialization path, see `docs/19-advanced-model-serving-roadmap.md`.
+For the advanced model serving decision framework, see `docs/19-advanced-model-serving-roadmap.md`.
 
 For the Week 4 release gates, readiness verdict, and next-stage priorities, see `docs/20-production-readiness-review.md`.
 
-For the phased commercialization roadmap with milestones and success metrics, see `docs/21-commercialization-roadmap.md`.
 
 For the Week 5 provider telemetry contract, aggregate metrics, and privacy boundary, see `docs/22-provider-telemetry.md`.
 
@@ -146,7 +145,6 @@ Reliability-Lab/
     18-production-observability.md
     19-advanced-model-serving-roadmap.md
     20-production-readiness-review.md
-    21-commercialization-roadmap.md
     22-provider-telemetry.md
   infra/                 # Docker, Kubernetes, and Terraform starter notes
     k8s/                 # kind-first Kubernetes manifests and walkthrough
@@ -183,7 +181,7 @@ You should see the assistant report intentionally generated 500s, latency spikes
 - Week 7: an optional OpenTelemetry signal path plus one owned alert-to-runbook exercise.
 - Week 8: a provider-versus-private-endpoint benchmark and an evidence-backed build-versus-buy decision.
 
-The technical weeks run alongside the commercialization phases; they do not bypass audience, design-partner, or willingness-to-pay gates. See `docs/09-roadmap.md` and `docs/21-commercialization-roadmap.md`.
+The technical weeks add measurable capabilities while preserving the small, local-first learning path. See `docs/09-roadmap.md`.
 
 ## Build In Public
 
@@ -197,14 +195,6 @@ This repo is designed to be built in public one small step at a time. Good updat
 
 Start with `docs/build-log.md`.
 
-## Product Direction
-
-The open-source project makes the core learning path, safety controls, and evaluation approach transparent. That foundation should remain useful on its own.
-
-The future paid opportunity is the recurring team workflow around those foundations: hosted evaluation history, private incident datasets, release gates, model and provider comparisons, regression alerts, collaboration, and audit-ready exports. The product value is not simply generating an incident summary. It is helping teams prove that their operational assistant stays useful, safe, private, and cost-aware as their systems change.
-
-The sequenced plan — audience first, wedge product with design partners second, paid team tier third — lives in `docs/21-commercialization-roadmap.md` with per-phase milestones, success metrics, and exit criteria.
-
 ## Lessons Learned
 
 Day 1 lessons are intentionally simple:

SECURITY.md

@@ -24,7 +24,7 @@ Include:
 - expected impact.
 - whether any secret, token, or sensitive log content is involved.
 
-Do not include real API keys, tokens, credentials, private logs, or customer data in a report.
+Do not include real API keys, tokens, credentials, private logs, or sensitive data in a report.
 
 ## Public Issues And Pull Requests
 

apps/ai-sre-assistant/README.md

@@ -87,7 +87,7 @@ See `../../docs/16-cost-optimization.md` for the Day 3 cost optimization guide.
 
 When `use_llm=true`, LLM-enabled API responses include `llm_telemetry` with bounded provider and model labels, configuration and attempt state, success or fallback outcome, request latency, and normalized token usage when the provider reports it.
 
-The telemetry object never includes prompts, incident evidence, credentials, provider base URLs, generated output, or user and incident identifiers. It is per-request metadata, not a persistent usage ledger or billing record. When both deployment-owned input/output prices and provider-reported token directions are available, responses also include an `llm_cost_estimate`; otherwise its cost fields remain explicitly unknown.
+The telemetry object never includes prompts, incident evidence, credentials, provider base URLs, generated output, or user and incident identifiers. It is per-request metadata, not a persistent usage ledger. When both deployment-owned input/output prices and provider-reported token directions are available, responses also include an `llm_cost_estimate`; otherwise its cost fields remain explicitly unknown.
 
 `GET /metrics` exposes privacy-safe in-memory aggregates for analysis outcomes, provider request latency, deterministic fallbacks, and provider-reported input/output tokens. Labels are limited to the configured provider/model and fixed outcome, reason, and direction enums. These process-local counters reset when the assistant restarts.
 
@@ -124,3 +124,13 @@ Redaction covers:
 - final API responses.
 
 This is a pattern-based backup control, not a complete data loss prevention system. Keep secrets out of logs and review operational data before sharing it or enabling an external provider. See `../../docs/15-secret-handling-and-redaction.md`.
+
+## Provider Evaluation Cost Report
+
+To join the deterministic rubric with real optional-provider outcomes and estimated cost, configure the provider and prices, then run:
+
+```bash
+python -m evals.run_evals --provider-report
+```
+
+This explicit command makes one provider call per evaluation case and prints a bounded JSON report, including estimated cost per successful evaluated analysis when complete token and price data is available. It remains outside CI so the normal release gate stays offline and cost-free.

apps/ai-sre-assistant/evals/run_evals.py

@@ -1,9 +1,24 @@
-from evals.runner import RUBRIC_DIMENSIONS, run_suite
+import argparse
+import json
+
+from evals.runner import RUBRIC_DIMENSIONS, run_provider_suite, run_suite
 
 
 def main() -> int:
-    suite = run_suite()
+    parser = argparse.ArgumentParser(description="Run AI SRE Assistant evaluations.")
+    parser.add_argument(
+        "--provider-report",
+        action="store_true",
+        help="Run the corpus with optional provider enrichment and print a bounded JSON cost report.",
+    )
+    args = parser.parse_args()
 
+    if args.provider_report:
+        suite = run_provider_suite()
+        print(json.dumps(suite, indent=2, sort_keys=True))
+        return 0 if suite["passed"] else 1
+
+    suite = run_suite()
     print("AI SRE Assistant evaluation")
     print("=" * 35)
     for result in suite["results"]:

apps/ai-sre-assistant/evals/runner.py

@@ -1,8 +1,10 @@
 import json
+from decimal import Decimal, InvalidOperation
 from pathlib import Path
 from typing import Any
 
 from app.analyzer import analyze_logs
+from app.llm import LLMConfig, analyze_with_llm, estimate_cost, load_config
 from app.log_reader import read_recent_logs
 from app.redaction import REDACTED
 
@@ -26,10 +28,35 @@ def load_cases(cases_path: Path = CASES_PATH) -> list[dict[str, Any]]:
 
 
 def analyze_case(case: dict[str, Any]) -> dict[str, Any]:
+    return _analyze_case_with_logs(case)[1]
+
+
+def evaluate_provider_case(case: dict[str, Any], config: LLMConfig) -> dict[str, Any]:
+    """Evaluate one fixture and attach bounded optional-provider metadata.
+
+    The deterministic rubric remains the quality gate. Provider output is free-form
+    text, so it stays separate until the corpus can score it directly.
+    """
+    logs, output = _analyze_case_with_logs(case)
+    evaluation = evaluate_output(case, output)
+    llm_result = analyze_with_llm(
+        question=case.get("question", ""),
+        logs=logs,
+        rule_based_analysis=output,
+        config=config,
+    )
+    return {
+        **evaluation,
+        "llm_telemetry": llm_result.telemetry,
+        "llm_cost_estimate": estimate_cost(config, llm_result.telemetry),
+    }
+
+
+def _analyze_case_with_logs(case: dict[str, Any]) -> tuple[list[dict[str, Any]], dict[str, Any]]:
     fixture_name = case.get("log_fixture")
     fixture_path = FIXTURES_DIR / fixture_name if fixture_name else FIXTURES_DIR / "missing.log"
     logs = read_recent_logs(log_path=fixture_path, max_lines=100)
-    return analyze_logs(logs, question=case.get("question"))
+    return logs, analyze_logs(logs, question=case.get("question"))
 
 
 def evaluate_output(case: dict[str, Any], output: dict[str, Any]) -> dict[str, Any]:
@@ -93,6 +120,59 @@ def run_suite(cases: list[dict[str, Any]] | None = None) -> dict[str, Any]:
     }
 
 
+def run_provider_suite(
+    cases: list[dict[str, Any]] | None = None,
+    config: LLMConfig | None = None,
+) -> dict[str, Any]:
+    """Join deterministic evaluation results with optional provider outcomes and cost.
+
+    This makes real provider calls when configured. The CLI keeps it opt-in so
+    normal deterministic CI stays offline and has no token cost.
+    """
+    config = config or load_config()
+    results = [evaluate_provider_case(case, config) for case in (cases if cases is not None else load_cases())]
+    checks_passed = sum(result["score"] for result in results)
+    checks_total = sum(result["max_score"] for result in results)
+    successful = [
+        result
+        for result in results
+        if result["passed"] and result["llm_telemetry"]["outcome"] == "success"
+    ]
+    known_costs = [
+        cost
+        for cost in (_cost_decimal(result["llm_cost_estimate"].get("estimated_total_cost_usd")) for result in successful)
+        if cost is not None
+    ]
+    costs_complete = len(known_costs) == len(successful)
+    total_cost = sum(known_costs, Decimal("0")) if costs_complete else None
+    provider_outcomes = _count_provider_outcomes(results)
+
+    if not successful:
+        cost_unavailable_reason = "no_successful_evaluated_analyses"
+    elif not costs_complete:
+        cost_unavailable_reason = "incomplete_cost_data"
+    else:
+        cost_unavailable_reason = None
+
+    return {
+        "passed": all(result["passed"] for result in results) and provider_outcomes["success"] == len(results),
+        "provider": _provider_identity(results),
+        "cases_passed": sum(result["passed"] for result in results),
+        "cases_total": len(results),
+        "checks_passed": checks_passed,
+        "checks_total": checks_total,
+        "provider_outcomes": provider_outcomes,
+        "successful_evaluated_analyses": len(successful),
+        "costed_successful_evaluated_analyses": len(known_costs),
+        "estimated_total_cost_usd": _decimal_string(total_cost),
+        "estimated_cost_per_successful_evaluated_analysis_usd": (
+            _decimal_string(total_cost / len(successful)) if total_cost is not None and successful else None
+        ),
+        "cost_unavailable_reason": cost_unavailable_reason,
+        "results": results,
+    }
+
+
 def _contains_all(values: list[Any], needles: list[str]) -> bool:
     text = " ".join(str(value) for value in values).lower()
     return all(needle.lower() in text for needle in needles)
@@ -103,3 +183,33 @@ def _contains_any(text: str, needles: tuple[str, ...] | list[str], case_sensitiv
         text = text.lower()
         needles = [needle.lower() for needle in needles]
     return any(needle in text for needle in needles)
+
+
+def _count_provider_outcomes(results: list[dict[str, Any]]) -> dict[str, int]:
+    counts = {"success": 0, "failure": 0, "not_configured": 0}
+    for result in results:
+        outcome = result["llm_telemetry"].get("outcome")
+        if outcome in counts:
+            counts[outcome] += 1
+    return counts
+
+
+def _provider_identity(results: list[dict[str, Any]]) -> dict[str, str | None]:
+    if not results:
+        return {"provider": None, "model": None}
+    telemetry = results[0]["llm_telemetry"]
+    return {"provider": telemetry.get("provider"), "model": telemetry.get("model")}
+
+
+def _cost_decimal(value: Any) -> Decimal | None:
+    if not isinstance(value, str):
+        return None
+    try:
+        cost = Decimal(value)
+    except InvalidOperation:
+        return None
+    return cost if cost.is_finite() and cost >= 0 else None
+
+
+def _decimal_string(value: Decimal | None) -> str | None:
+    return format(value.quantize(Decimal("0.00000001")), "f") if value is not None else None

apps/ai-sre-assistant/tests/test_evaluation.py

@@ -45,3 +45,79 @@ def test_evaluation_detects_secret_regression():
 
     assert result["passed"] is False
     assert result["rubric"]["private"] is False
+
+
+def test_provider_report_calculates_cost_per_successful_evaluated_analysis(monkeypatch):
+    from decimal import Decimal
+
+    from app.llm import LLMCallResult, LLMConfig
+    from evals.runner import run_provider_suite
+
+    def successful_enrichment(**_kwargs):
+        return LLMCallResult(
+            analysis="optional provider analysis",
+            notice=None,
+            telemetry={
+                "provider": "openai",
+                "model": "eval-model",
+                "outcome": "success",
+                "usage": {"input_tokens": 100, "output_tokens": 50},
+            },
+        )
+
+    monkeypatch.setattr("evals.runner.analyze_with_llm", successful_enrichment)
+    config = LLMConfig(
+        provider="openai",
+        api_key="test-key",
+        base_url="https://example.invalid/v1",
+        model="eval-model",
+        input_usd_per_million_tokens=Decimal("1"),
+        output_usd_per_million_tokens=Decimal("1"),
+    )
+
+    report = run_provider_suite(CASES[:2], config)
+
+    assert report["passed"] is True
+    assert report["provider_outcomes"] == {"success": 2, "failure": 0, "not_configured": 0}
+    assert report["successful_evaluated_analyses"] == 2
+    assert report["costed_successful_evaluated_analyses"] == 2
+    assert report["estimated_total_cost_usd"] == "0.00030000"
+    assert report["estimated_cost_per_successful_evaluated_analysis_usd"] == "0.00015000"
+    assert report["cost_unavailable_reason"] is None
+
+
+def test_provider_report_keeps_cost_per_success_unknown_with_incomplete_usage(monkeypatch):
+    from decimal import Decimal
+
+    from app.llm import LLMCallResult, LLMConfig
+    from evals.runner import run_provider_suite
+
+    def enrichment_without_usage(**_kwargs):
+        return LLMCallResult(
+            analysis="optional provider analysis",
+            notice=None,
+            telemetry={
+                "provider": "openai",
+                "model": "eval-model",
+                "outcome": "success",
+                "usage": {"input_tokens": 100, "output_tokens": None},
+            },
+        )
+
+    monkeypatch.setattr("evals.runner.analyze_with_llm", enrichment_without_usage)
+    config = LLMConfig(
+        provider="openai",
+        api_key="test-key",
+        base_url="https://example.invalid/v1",
+        model="eval-model",
+        input_usd_per_million_tokens=Decimal("1"),
+        output_usd_per_million_tokens=Decimal("1"),
+    )
+
+    report = run_provider_suite(CASES[:1], config)
+
+    assert report["passed"] is True
+    assert report["costed_successful_evaluated_analyses"] == 0
+    assert report["estimated_total_cost_usd"] is None
+    assert report["estimated_cost_per_successful_evaluated_analysis_usd"] is None
+    assert report["cost_unavailable_reason"] == "incomplete_cost_data"

apps/demo-service/app/routes.py

@@ -12,7 +12,7 @@
 router = APIRouter()
 
 ORDERS = {
-    "ord-1001": {"id": "ord-1001", "item": "notebook", "status": "paid"},
+    "ord-1001": {"id": "ord-1001", "item": "notebook", "status": "completed"},
     "ord-1002": {"id": "ord-1002", "item": "keyboard", "status": "packed"},
     "ord-1003": {"id": "ord-1003", "item": "monitor", "status": "shipped"},
 }

apps/demo-service/requirements.txt

@@ -2,5 +2,5 @@ fastapi>=0.115.0,<1.0
 uvicorn[standard]>=0.30.0,<1.0
 httpx>=0.27.0,<1.0
 pytest>=8.0.0,<9.0
-ruff>=0.6.0,<1.0
+ruff==0.6.9
 

docs/06-production-considerations.md

@@ -97,4 +97,4 @@ They are roadmap items, not Day 1 requirements.
 
 Before adding them, the repo should already explain the operational foundation: logs, metrics, health checks, Kubernetes basics, security, cost, and evaluation.
 
-See [Advanced Model Serving Roadmap](19-advanced-model-serving-roadmap.md) for adoption gates, framework boundaries, benchmark criteria, GPU operating requirements, and potential product tiers.
+See [Advanced Model Serving Roadmap](19-advanced-model-serving-roadmap.md) for adoption gates, framework boundaries, benchmark criteria, GPU operating requirements, and operational tradeoffs.

docs/07-security.md

@@ -4,7 +4,7 @@ Week 4 starts with security hardening basics.
 Day 2 adds focused secret handling rules and an enforced redaction boundary in the AI SRE Assistant.
 
 
-The goal is not to turn the local learning lab into an enterprise platform in one day. The goal is to name the first risks clearly and build safer habits before production.
+The goal is not to turn the local learning lab into an large production platform in one day. The goal is to name the first risks clearly and build safer habits before production.
 
 ## Threat Model
 
@@ -66,15 +66,15 @@ Do not put these values in logs:
 - API keys.
 - bearer tokens.
 - passwords.
-- private customer data.
+- private operational data.
 - cloud credentials.
 - raw authorization headers.
 
 
 Before sharing logs publicly, redact values that identify private systems, accounts, users, or credentials.
 The assistant applies pattern-based redaction when logs are parsed and again before optional LLM requests. Questions, generated LLM text, and final API responses also pass through the same redactor.
 
-Redaction is a backup control, not proof that arbitrary logs are safe. Unknown secret formats, encoded values, and private customer data may still require manual review.
+Redaction is a backup control, not proof that arbitrary logs are safe. Unknown secret formats, encoded values, and private operational data may still require manual review.
 
 See `15-secret-handling-and-redaction.md` for the exact rules, examples, and limitations.