diff --git a/.env.example b/.env.example
index 5ee7af7..df76dc7 100644
--- a/.env.example
+++ b/.env.example
@@ -10,7 +10,7 @@ MODEL_NAME=gpt-4o-mini
LLM_MAX_LOG_ENTRIES=50
LLM_MAX_PROMPT_CHARS=12000
-# Optional operator-supplied pricing for per-call estimates. Leave blank when unknown.
+# Optional operator-supplied cost inputs for per-call estimates. Leave blank when unknown.
# Values are USD per one million provider-reported input/output tokens.
LLM_INPUT_USD_PER_MILLION_TOKENS=
LLM_OUTPUT_USD_PER_MILLION_TOKENS=
diff --git a/README.md b/README.md
index 9cf356b..4bd49d0 100644
--- a/README.md
+++ b/README.md
@@ -94,11 +94,10 @@ For the assistant evaluation corpus and quality rubric, see `docs/17-assistant-e
For the production observability architecture and staged migration path, see `docs/18-production-observability.md`.
-For the advanced model serving decision framework and commercialization path, see `docs/19-advanced-model-serving-roadmap.md`.
+For the advanced model serving decision framework, see `docs/19-advanced-model-serving-roadmap.md`.
For the Week 4 release gates, readiness verdict, and next-stage priorities, see `docs/20-production-readiness-review.md`.
-For the phased commercialization roadmap with milestones and success metrics, see `docs/21-commercialization-roadmap.md`.
For the Week 5 provider telemetry contract, aggregate metrics, and privacy boundary, see `docs/22-provider-telemetry.md`.
@@ -146,7 +145,6 @@ Reliability-Lab/
18-production-observability.md
19-advanced-model-serving-roadmap.md
20-production-readiness-review.md
- 21-commercialization-roadmap.md
22-provider-telemetry.md
infra/ # Docker, Kubernetes, and Terraform starter notes
k8s/ # kind-first Kubernetes manifests and walkthrough
@@ -183,7 +181,7 @@ You should see the assistant report intentionally generated 500s, latency spikes
- Week 7: an optional OpenTelemetry signal path plus one owned alert-to-runbook exercise.
- Week 8: a provider-versus-private-endpoint benchmark and an evidence-backed build-versus-buy decision.
-The technical weeks run alongside the commercialization phases; they do not bypass audience, design-partner, or willingness-to-pay gates. See `docs/09-roadmap.md` and `docs/21-commercialization-roadmap.md`.
+The technical weeks add measurable capabilities while preserving the small, local-first learning path. See `docs/09-roadmap.md`.
## Build In Public
@@ -197,14 +195,6 @@ This repo is designed to be built in public one small step at a time. Good updat
Start with `docs/build-log.md`.
-## Product Direction
-
-The open-source project makes the core learning path, safety controls, and evaluation approach transparent. That foundation should remain useful on its own.
-
-The future paid opportunity is the recurring team workflow around those foundations: hosted evaluation history, private incident datasets, release gates, model and provider comparisons, regression alerts, collaboration, and audit-ready exports. The product value is not simply generating an incident summary. It is helping teams prove that their operational assistant stays useful, safe, private, and cost-aware as their systems change.
-
-The sequenced plan — audience first, wedge product with design partners second, paid team tier third — lives in `docs/21-commercialization-roadmap.md` with per-phase milestones, success metrics, and exit criteria.
-
## Lessons Learned
Day 1 lessons are intentionally simple:
diff --git a/SECURITY.md b/SECURITY.md
index 320e3d2..7fb8c41 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -24,7 +24,7 @@ Include:
- expected impact.
- whether any secret, token, or sensitive log content is involved.
-Do not include real API keys, tokens, credentials, private logs, or customer data in a report.
+Do not include real API keys, tokens, credentials, private logs, or sensitive data in a report.
## Public Issues And Pull Requests
diff --git a/apps/ai-sre-assistant/README.md b/apps/ai-sre-assistant/README.md
index 761c859..b94c4e7 100644
--- a/apps/ai-sre-assistant/README.md
+++ b/apps/ai-sre-assistant/README.md
@@ -87,7 +87,7 @@ See `../../docs/16-cost-optimization.md` for the Day 3 cost optimization guide.
When `use_llm=true`, LLM-enabled API responses include `llm_telemetry` with bounded provider and model labels, configuration and attempt state, success or fallback outcome, request latency, and normalized token usage when the provider reports it.
-The telemetry object never includes prompts, incident evidence, credentials, provider base URLs, generated output, or user and incident identifiers. It is per-request metadata, not a persistent usage ledger or billing record. When both deployment-owned input/output prices and provider-reported token directions are available, responses also include an `llm_cost_estimate`; otherwise its cost fields remain explicitly unknown.
+The telemetry object never includes prompts, incident evidence, credentials, provider base URLs, generated output, or user and incident identifiers. It is per-request metadata, not a persistent usage ledger. When both deployment-owned input/output prices and provider-reported token directions are available, responses also include an `llm_cost_estimate`; otherwise its cost fields remain explicitly unknown.
`GET /metrics` exposes privacy-safe in-memory aggregates for analysis outcomes, provider request latency, deterministic fallbacks, and provider-reported input/output tokens. Labels are limited to the configured provider/model and fixed outcome, reason, and direction enums. These process-local counters reset when the assistant restarts.
@@ -124,3 +124,13 @@ Redaction covers:
- final API responses.
This is a pattern-based backup control, not a complete data loss prevention system. Keep secrets out of logs and review operational data before sharing it or enabling an external provider. See `../../docs/15-secret-handling-and-redaction.md`.
+
+## Provider Evaluation Cost Report
+
+To join the deterministic rubric with real optional-provider outcomes and estimated cost, configure the provider and prices, then run:
+
+```bash
+python -m evals.run_evals --provider-report
+```
+
+This explicit command makes one provider call per evaluation case and prints a bounded JSON report, including estimated cost per successful evaluated analysis when complete token and price data is available. It remains outside CI so the normal release gate stays offline and cost-free.
diff --git a/apps/ai-sre-assistant/evals/run_evals.py b/apps/ai-sre-assistant/evals/run_evals.py
index 00e33f3..f70e244 100644
--- a/apps/ai-sre-assistant/evals/run_evals.py
+++ b/apps/ai-sre-assistant/evals/run_evals.py
@@ -1,9 +1,24 @@
-from evals.runner import RUBRIC_DIMENSIONS, run_suite
+import argparse
+import json
+
+from evals.runner import RUBRIC_DIMENSIONS, run_provider_suite, run_suite
def main() -> int:
- suite = run_suite()
+ parser = argparse.ArgumentParser(description="Run AI SRE Assistant evaluations.")
+ parser.add_argument(
+ "--provider-report",
+ action="store_true",
+ help="Run the corpus with optional provider enrichment and print a bounded JSON cost report.",
+ )
+ args = parser.parse_args()
+ if args.provider_report:
+ suite = run_provider_suite()
+ print(json.dumps(suite, indent=2, sort_keys=True))
+ return 0 if suite["passed"] else 1
+
+ suite = run_suite()
print("AI SRE Assistant evaluation")
print("=" * 35)
for result in suite["results"]:
diff --git a/apps/ai-sre-assistant/evals/runner.py b/apps/ai-sre-assistant/evals/runner.py
index 2b92e5f..f9acd77 100644
--- a/apps/ai-sre-assistant/evals/runner.py
+++ b/apps/ai-sre-assistant/evals/runner.py
@@ -1,8 +1,10 @@
import json
+from decimal import Decimal, InvalidOperation
from pathlib import Path
from typing import Any
from app.analyzer import analyze_logs
+from app.llm import LLMConfig, analyze_with_llm, estimate_cost, load_config
from app.log_reader import read_recent_logs
from app.redaction import REDACTED
@@ -26,10 +28,35 @@ def load_cases(cases_path: Path = CASES_PATH) -> list[dict[str, Any]]:
def analyze_case(case: dict[str, Any]) -> dict[str, Any]:
+ return _analyze_case_with_logs(case)[1]
+
+
+def evaluate_provider_case(case: dict[str, Any], config: LLMConfig) -> dict[str, Any]:
+ """Evaluate one fixture and attach bounded optional-provider metadata.
+
+ The deterministic rubric remains the quality gate. Provider output is free-form
+ text, so it stays separate until the corpus can score it directly.
+ """
+ logs, output = _analyze_case_with_logs(case)
+ evaluation = evaluate_output(case, output)
+ llm_result = analyze_with_llm(
+ question=case.get("question", ""),
+ logs=logs,
+ rule_based_analysis=output,
+ config=config,
+ )
+ return {
+ **evaluation,
+ "llm_telemetry": llm_result.telemetry,
+ "llm_cost_estimate": estimate_cost(config, llm_result.telemetry),
+ }
+
+
+def _analyze_case_with_logs(case: dict[str, Any]) -> tuple[list[dict[str, Any]], dict[str, Any]]:
fixture_name = case.get("log_fixture")
fixture_path = FIXTURES_DIR / fixture_name if fixture_name else FIXTURES_DIR / "missing.log"
logs = read_recent_logs(log_path=fixture_path, max_lines=100)
- return analyze_logs(logs, question=case.get("question"))
+ return logs, analyze_logs(logs, question=case.get("question"))
def evaluate_output(case: dict[str, Any], output: dict[str, Any]) -> dict[str, Any]:
@@ -93,6 +120,59 @@ def run_suite(cases: list[dict[str, Any]] | None = None) -> dict[str, Any]:
}
+def run_provider_suite(
+ cases: list[dict[str, Any]] | None = None,
+ config: LLMConfig | None = None,
+) -> dict[str, Any]:
+ """Join deterministic evaluation results with optional provider outcomes and cost.
+
+ This makes real provider calls when configured. The CLI keeps it opt-in so
+ normal deterministic CI stays offline and has no token cost.
+ """
+ config = config or load_config()
+ results = [evaluate_provider_case(case, config) for case in (cases if cases is not None else load_cases())]
+ checks_passed = sum(result["score"] for result in results)
+ checks_total = sum(result["max_score"] for result in results)
+ successful = [
+ result
+ for result in results
+ if result["passed"] and result["llm_telemetry"]["outcome"] == "success"
+ ]
+ known_costs = [
+ cost
+ for cost in (_cost_decimal(result["llm_cost_estimate"].get("estimated_total_cost_usd")) for result in successful)
+ if cost is not None
+ ]
+ costs_complete = len(known_costs) == len(successful)
+ total_cost = sum(known_costs, Decimal("0")) if costs_complete else None
+ provider_outcomes = _count_provider_outcomes(results)
+
+ if not successful:
+ cost_unavailable_reason = "no_successful_evaluated_analyses"
+ elif not costs_complete:
+ cost_unavailable_reason = "incomplete_cost_data"
+ else:
+ cost_unavailable_reason = None
+
+ return {
+ "passed": all(result["passed"] for result in results) and provider_outcomes["success"] == len(results),
+ "provider": _provider_identity(results),
+ "cases_passed": sum(result["passed"] for result in results),
+ "cases_total": len(results),
+ "checks_passed": checks_passed,
+ "checks_total": checks_total,
+ "provider_outcomes": provider_outcomes,
+ "successful_evaluated_analyses": len(successful),
+ "costed_successful_evaluated_analyses": len(known_costs),
+ "estimated_total_cost_usd": _decimal_string(total_cost),
+ "estimated_cost_per_successful_evaluated_analysis_usd": (
+ _decimal_string(total_cost / len(successful)) if total_cost is not None and successful else None
+ ),
+ "cost_unavailable_reason": cost_unavailable_reason,
+ "results": results,
+ }
+
+
def _contains_all(values: list[Any], needles: list[str]) -> bool:
text = " ".join(str(value) for value in values).lower()
return all(needle.lower() in text for needle in needles)
@@ -103,3 +183,33 @@ def _contains_any(text: str, needles: tuple[str, ...] | list[str], case_sensitiv
text = text.lower()
needles = [needle.lower() for needle in needles]
return any(needle in text for needle in needles)
+
+
+def _count_provider_outcomes(results: list[dict[str, Any]]) -> dict[str, int]:
+ counts = {"success": 0, "failure": 0, "not_configured": 0}
+ for result in results:
+ outcome = result["llm_telemetry"].get("outcome")
+ if outcome in counts:
+ counts[outcome] += 1
+ return counts
+
+
+def _provider_identity(results: list[dict[str, Any]]) -> dict[str, str | None]:
+ if not results:
+ return {"provider": None, "model": None}
+ telemetry = results[0]["llm_telemetry"]
+ return {"provider": telemetry.get("provider"), "model": telemetry.get("model")}
+
+
+def _cost_decimal(value: Any) -> Decimal | None:
+ if not isinstance(value, str):
+ return None
+ try:
+ cost = Decimal(value)
+ except InvalidOperation:
+ return None
+ return cost if cost.is_finite() and cost >= 0 else None
+
+
+def _decimal_string(value: Decimal | None) -> str | None:
+ return format(value.quantize(Decimal("0.00000001")), "f") if value is not None else None
diff --git a/apps/ai-sre-assistant/tests/test_evaluation.py b/apps/ai-sre-assistant/tests/test_evaluation.py
index 3ed4020..f247684 100644
--- a/apps/ai-sre-assistant/tests/test_evaluation.py
+++ b/apps/ai-sre-assistant/tests/test_evaluation.py
@@ -45,3 +45,79 @@ def test_evaluation_detects_secret_regression():
assert result["passed"] is False
assert result["rubric"]["private"] is False
+
+
+def test_provider_report_calculates_cost_per_successful_evaluated_analysis(monkeypatch):
+ from decimal import Decimal
+
+ from app.llm import LLMCallResult, LLMConfig
+ from evals.runner import run_provider_suite
+
+ def successful_enrichment(**_kwargs):
+ return LLMCallResult(
+ analysis="optional provider analysis",
+ notice=None,
+ telemetry={
+ "provider": "openai",
+ "model": "eval-model",
+ "outcome": "success",
+ "usage": {"input_tokens": 100, "output_tokens": 50},
+ },
+ )
+
+ monkeypatch.setattr("evals.runner.analyze_with_llm", successful_enrichment)
+ config = LLMConfig(
+ provider="openai",
+ api_key="test-key",
+ base_url="https://example.invalid/v1",
+ model="eval-model",
+ input_usd_per_million_tokens=Decimal("1"),
+ output_usd_per_million_tokens=Decimal("1"),
+ )
+
+ report = run_provider_suite(CASES[:2], config)
+
+ assert report["passed"] is True
+ assert report["provider_outcomes"] == {"success": 2, "failure": 0, "not_configured": 0}
+ assert report["successful_evaluated_analyses"] == 2
+ assert report["costed_successful_evaluated_analyses"] == 2
+ assert report["estimated_total_cost_usd"] == "0.00030000"
+ assert report["estimated_cost_per_successful_evaluated_analysis_usd"] == "0.00015000"
+ assert report["cost_unavailable_reason"] is None
+
+
+def test_provider_report_keeps_cost_per_success_unknown_with_incomplete_usage(monkeypatch):
+ from decimal import Decimal
+
+ from app.llm import LLMCallResult, LLMConfig
+ from evals.runner import run_provider_suite
+
+ def enrichment_without_usage(**_kwargs):
+ return LLMCallResult(
+ analysis="optional provider analysis",
+ notice=None,
+ telemetry={
+ "provider": "openai",
+ "model": "eval-model",
+ "outcome": "success",
+ "usage": {"input_tokens": 100, "output_tokens": None},
+ },
+ )
+
+ monkeypatch.setattr("evals.runner.analyze_with_llm", enrichment_without_usage)
+ config = LLMConfig(
+ provider="openai",
+ api_key="test-key",
+ base_url="https://example.invalid/v1",
+ model="eval-model",
+ input_usd_per_million_tokens=Decimal("1"),
+ output_usd_per_million_tokens=Decimal("1"),
+ )
+
+ report = run_provider_suite(CASES[:1], config)
+
+ assert report["passed"] is True
+ assert report["costed_successful_evaluated_analyses"] == 0
+ assert report["estimated_total_cost_usd"] is None
+ assert report["estimated_cost_per_successful_evaluated_analysis_usd"] is None
+ assert report["cost_unavailable_reason"] == "incomplete_cost_data"
diff --git a/apps/demo-service/app/routes.py b/apps/demo-service/app/routes.py
index b14fd7c..4dbb121 100644
--- a/apps/demo-service/app/routes.py
+++ b/apps/demo-service/app/routes.py
@@ -12,7 +12,7 @@
router = APIRouter()
ORDERS = {
- "ord-1001": {"id": "ord-1001", "item": "notebook", "status": "paid"},
+ "ord-1001": {"id": "ord-1001", "item": "notebook", "status": "completed"},
"ord-1002": {"id": "ord-1002", "item": "keyboard", "status": "packed"},
"ord-1003": {"id": "ord-1003", "item": "monitor", "status": "shipped"},
}
diff --git a/apps/demo-service/requirements.txt b/apps/demo-service/requirements.txt
index cbc89a3..d0352eb 100644
--- a/apps/demo-service/requirements.txt
+++ b/apps/demo-service/requirements.txt
@@ -2,5 +2,5 @@ fastapi>=0.115.0,<1.0
uvicorn[standard]>=0.30.0,<1.0
httpx>=0.27.0,<1.0
pytest>=8.0.0,<9.0
-ruff>=0.6.0,<1.0
+ruff==0.6.9
diff --git a/docs/06-production-considerations.md b/docs/06-production-considerations.md
index 05ecace..d523ac6 100644
--- a/docs/06-production-considerations.md
+++ b/docs/06-production-considerations.md
@@ -97,4 +97,4 @@ They are roadmap items, not Day 1 requirements.
Before adding them, the repo should already explain the operational foundation: logs, metrics, health checks, Kubernetes basics, security, cost, and evaluation.
-See [Advanced Model Serving Roadmap](19-advanced-model-serving-roadmap.md) for adoption gates, framework boundaries, benchmark criteria, GPU operating requirements, and potential product tiers.
+See [Advanced Model Serving Roadmap](19-advanced-model-serving-roadmap.md) for adoption gates, framework boundaries, benchmark criteria, GPU operating requirements, and operational tradeoffs.
diff --git a/docs/07-security.md b/docs/07-security.md
index dddb95e..e7879c9 100644
--- a/docs/07-security.md
+++ b/docs/07-security.md
@@ -4,7 +4,7 @@ Week 4 starts with security hardening basics.
Day 2 adds focused secret handling rules and an enforced redaction boundary in the AI SRE Assistant.
-The goal is not to turn the local learning lab into an enterprise platform in one day. The goal is to name the first risks clearly and build safer habits before production.
+The goal is not to turn the local learning lab into an large production platform in one day. The goal is to name the first risks clearly and build safer habits before production.
## Threat Model
@@ -66,7 +66,7 @@ Do not put these values in logs:
- API keys.
- bearer tokens.
- passwords.
-- private customer data.
+- private operational data.
- cloud credentials.
- raw authorization headers.
@@ -74,7 +74,7 @@ Do not put these values in logs:
Before sharing logs publicly, redact values that identify private systems, accounts, users, or credentials.
The assistant applies pattern-based redaction when logs are parsed and again before optional LLM requests. Questions, generated LLM text, and final API responses also pass through the same redactor.
-Redaction is a backup control, not proof that arbitrary logs are safe. Unknown secret formats, encoded values, and private customer data may still require manual review.
+Redaction is a backup control, not proof that arbitrary logs are safe. Unknown secret formats, encoded values, and private operational data may still require manual review.
See `15-secret-handling-and-redaction.md` for the exact rules, examples, and limitations.
diff --git a/docs/09-roadmap.md b/docs/09-roadmap.md
index b499d7f..e68afb6 100644
--- a/docs/09-roadmap.md
+++ b/docs/09-roadmap.md
@@ -1,6 +1,6 @@
# Roadmap
-This is the canonical technical execution order for AIOps Lab. Weeks 1-4 are the completed learning foundation, Weeks 5-8 are the next measurement cycle, and later infrastructure remains gated by evidence and named operational ownership.
+This is the canonical technical execution order for Reliability Lab. Each week adds one measurable capability while keeping the default path laptop-friendly and dependency-light.
## Week 1 - Local Learning Lab
@@ -37,19 +37,15 @@ This is the canonical technical execution order for AIOps Lab. Weeks 1-4 are the
- Optional advanced serving roadmap: vLLM, Triton, Ray, KServe, and GPU scheduling.
- Production-readiness review with local and CI release gates.
-## Commercialization Track
-
-The [commercialization roadmap](21-commercialization-roadmap.md) runs alongside the technical weeks: audience first, a design-partner wedge second, and paid tiers third. Completing a local technical week does not bypass its customer-discovery or phase-exit gates.
-
## Week 5 - Provider Telemetry
- Day 1 - complete: expose a bounded per-call contract for provider/model identity, request latency, token usage, outcome, and deterministic fallback without storing sensitive content.
- Day 2 - complete: add aggregate counters and latency distributions with bounded labels.
- Day 3 - complete: accept explicit deployment-owned pricing inputs and return per-call estimated cost metadata only when prices and token directions are known.
-- Join provider usage and cost with evaluation outcomes.
-- Calculate cost per successful evaluated analysis.
+- Day 4 - complete: join provider outcomes and deployment-owned cost estimates with deterministic evaluation results, including cost per successful evaluated analysis when data is complete.
+- Add a local comparison report across deterministic and configured provider paths.
-See [Provider Telemetry Contract](22-provider-telemetry.md) for the Day 1 per-request contract, Day 2 aggregate metrics, privacy boundary, and remaining-week sequence.
+See [Provider Telemetry Contract](22-provider-telemetry.md) for the per-request contract, aggregate metrics, privacy boundary, and remaining-week sequence.
**Exit gate:** provider usage, reliability, and cost can be compared with evaluation outcomes without storing prompts, incident evidence, credentials, endpoints, or generated content.
@@ -76,13 +72,13 @@ See [Provider Telemetry Contract](22-provider-telemetry.md) for the Day 1 per-re
- Run the same evaluation corpus against deterministic, managed-provider, and OpenAI-compatible private endpoints.
- Measure quality, latency, token usage, fallbacks, throughput, and cost per successful evaluated analysis.
- Test representative input sizes, concurrency, and burst behavior.
-- Record a build-versus-buy decision before adding GPU infrastructure.
+- Record an evidence-backed provider-versus-private-endpoint decision before adding GPU infrastructure.
**Exit gate:** evidence supports continuing with a provider or starting one bounded private-model experiment.
-## Internal Pilot Phase
+## Internal Deployment Readiness
-Begin only after the Week 5-8 measurement loop works and the commercialization roadmap has reached the design-partner phase with named owners.
+Begin only after the Week 5-8 measurement loop works and named maintainers own the operational controls.
- Authentication, service identity, and role-based access.
- Managed secrets, rotation, artifact pinning, and supply-chain scanning.
@@ -91,7 +87,7 @@ Begin only after the Week 5-8 measurement loop works and the commercialization r
- Initial SLOs, routed alerts, rollback tests, and recovery exercises.
- Private evaluation datasets and controlled evidence access.
-**Exit gate:** a sanitized internal pilot can operate with explicit ownership, access controls, measurable reliability, and a tested rollback path.
+**Exit gate:** a sanitized internal deployment can operate with explicit ownership, access controls, measurable reliability, and a tested rollback path.
## Advanced Serving Phase - Only If Earned
@@ -99,16 +95,16 @@ Begin only after the Week 5-8 measurement loop works and the commercialization r
- Add an optional single-GPU vLLM example only when the Week 8 benchmark or a named deployment requirement justifies it.
- Add GPU scheduling, quotas, utilization telemetry, queue metrics, and out-of-memory recovery tests for a real workload.
- Introduce Ray Serve, Triton, or KServe only when its specific orchestration problem appears.
-- Consider enterprise VPC, dedicated, or on-premises packaging after customer demand is validated.
+- Consider private VPC, dedicated, or on-premises packaging only for a documented deployment requirement.
The default project remains deterministic, provider-compatible, laptop-friendly, and GPU-free throughout these phases.
-## Longer-Term Product Backlog
+## Longer-Term Community Backlog
-- Hosted evaluation history and regression alerts.
-- Team collaboration and private incident datasets.
+- Versioned evaluation history and regression alerts.
+- Collaboration around private incident datasets.
- Audit-ready exports, policy controls, and usage governance.
-- Privacy-aware product outcome telemetry.
+- Privacy-aware outcome telemetry.
- Provider and model quality/cost comparisons over time.
- Log, metrics, and trace backend examples using Prometheus, Grafana, compatible open-source components, or managed services.
- Multi-environment and cloud deployment examples.
diff --git a/docs/15-secret-handling-and-redaction.md b/docs/15-secret-handling-and-redaction.md
index 84692b3..0e283eb 100644
--- a/docs/15-secret-handling-and-redaction.md
+++ b/docs/15-secret-handling-and-redaction.md
@@ -70,7 +70,7 @@ Before enabling an external provider:
This is a focused pattern-based control, not a complete data loss prevention system.
-It may miss unknown credential formats, secrets split across fields, encoded values, private customer data, or organization-specific identifiers. It may also hide a non-secret value that happens to look like a token.
+It may miss unknown credential formats, secrets split across fields, encoded values, private operational data, or organization-specific identifiers. It may also hide a non-secret value that happens to look like a token.
Redaction does not:
diff --git a/docs/16-cost-optimization.md b/docs/16-cost-optimization.md
index 99ace84..6eecb4d 100644
--- a/docs/16-cost-optimization.md
+++ b/docs/16-cost-optimization.md
@@ -2,7 +2,7 @@
Week 4, Day 3 adds practical cost controls for the AI SRE Assistant.
-The goal is not to build a billing platform. The goal is to make the first cost drivers visible and controllable before the project introduces heavier production tooling.
+The goal is not to build a usage-accounting platform. The goal is to make the first cost drivers visible and controllable before the project introduces heavier production tooling.
## Cost Model
@@ -12,7 +12,7 @@ In this project, the cheapest path is the default path:
logs / metrics -> rule-based analysis -> response
```
-That path runs locally and does not require a paid LLM provider.
+That path runs locally and does not require a external LLM provider.
When an OpenAI-compatible provider is enabled, the flow changes:
@@ -124,7 +124,7 @@ The current implementation is intentionally lightweight:
- prompt limits use characters, not exact model tokenizer counts.
- no provider-specific pricing table is included.
-- no billing dashboard is included.
+- no usage dashboard is included.
- no cache is included yet.
- no per-user quota system is included yet.
diff --git a/docs/17-assistant-evaluation.md b/docs/17-assistant-evaluation.md
index ffbd0d2..f6b1de4 100644
--- a/docs/17-assistant-evaluation.md
+++ b/docs/17-assistant-evaluation.md
@@ -56,24 +56,23 @@ This starter suite is intentionally small. It does not prove that the assistant
As the assistant grows, add real sanitized incidents, adversarial cases, provider/model comparisons, human review, latency measurements, token usage, and versioned acceptance thresholds.
-## From Open Source To A Paid Product
-
-The open-source value is a transparent local assistant, a public evaluation method, and fixtures that anyone can inspect or extend. Keeping that foundation open builds trust and makes the learning lab genuinely useful.
-
-Monetization should attach to the recurring work teams do around evaluation:
-
-- Hosted evaluation runs with history and regression trends.
-- Private, organization-specific incident datasets with access controls.
-- CI release gates and alerts when assistant quality drops.
-- Model and provider comparisons across quality, latency, and cost.
-- Shared review workflows for SRE, platform, security, and compliance teams.
-- Audit-ready reports that show which version passed which safety rules.
-
-That product direction reinforces the user outcome: teams should be able to adopt an operational assistant because they can measure and govern it, not because a demo response looked impressive once.
-
## Next Steps
- Keep these deterministic cases in the normal test suite and CI release gate.
- Add sanitized real-world incidents as the assistant supports more failure modes.
- Record provider usage metadata before comparing quality against cost.
- Version the corpus and thresholds when evaluation results become a release gate.
+
+## Provider Cost Report
+
+With a configured OpenAI-compatible provider and both operator-owned price inputs, run:
+
+```bash
+python -m evals.run_evals --provider-report
+```
+
+This makes one optional provider-enrichment call per fixture and emits a bounded JSON report. It joins the deterministic rubric result, provider outcome, and per-call cost estimate, then calculates `estimated_cost_per_successful_evaluated_analysis_usd`.
+
+A successful evaluated analysis requires a passing deterministic rubric and a successful provider response. The cost-per-success value is emitted only when every successful evaluated analysis has complete price and token data. Otherwise it is `null` with `cost_unavailable_reason`; unknown usage is never treated as zero cost. The report does not write a usage ledger or include fixture evidence, prompts, provider output, endpoints, or credentials.
+
+The normal evaluation command remains deterministic, offline, and cost-free. The provider report is deliberately opt-in and remains outside CI until a controlled provider test environment and explicit budget exist.
diff --git a/docs/18-production-observability.md b/docs/18-production-observability.md
index 31f62b7..832d67c 100644
--- a/docs/18-production-observability.md
+++ b/docs/18-production-observability.md
@@ -29,14 +29,10 @@ flowchart LR
Applications should emit telemetry using stable conventions. A collector handles routing, batching, filtering, enrichment, and export. Backends store and query signals. The assistant receives a bounded, redacted evidence window through controlled APIs rather than direct access to every production record.
-OpenTelemetry is a practical vendor-neutral collection layer. Backends can be open source, managed, or hybrid. The important decision is the interface and operating model, not the logo on the dashboard.
-
-## Observe Three Layers
+## Observe Two Layers
### 1. Service Reliability
-Start with signals that describe user impact:
-
- Request rate by service, route template, method, and status class.
- Error rate and error type.
- Latency distributions, not only averages.
@@ -47,8 +43,6 @@ Keep metric labels bounded. Request IDs belong in logs and traces, not metric la
### 2. Assistant Quality And Safety
-The assistant needs its own operational signals:
-
- Analysis requests by endpoint and analysis mode.
- Rule-based fallback and provider failure counts.
- End-to-end analysis latency.
@@ -58,127 +52,48 @@ The assistant needs its own operational signals:
- Model and provider identity when LLM enrichment is enabled.
- Input and output token counts when the provider returns usage metadata.
-A future instrumentation pass can use a bounded metric contract like this:
-
-| Candidate metric | Bounded labels | Purpose |
-| --- | --- | --- |
-| `ai_sre_analysis_requests_total` | `endpoint`, `analysis_mode`, `outcome` | Request volume and completion outcomes. |
-| `ai_sre_analysis_duration_seconds` | `endpoint`, `analysis_mode` | End-to-end latency distribution. |
-| `ai_sre_provider_fallbacks_total` | `provider`, `model`, `reason` | Deterministic provider fallback behavior. |
-| `ai_sre_prompt_truncations_total` | `reason` | Cost-control activation. |
-| `ai_sre_redactions_total` | `rule_category` | Redaction activity without sensitive values. |
-| `ai_sre_eval_checks_total` | `suite`, `dimension`, `result`, `assistant_version` | Quality and release-gate trends. |
-| `ai_sre_provider_tokens_total` | `provider`, `model`, `direction` | Token usage for quality/cost comparisons. |
-
-Week 5 Days 1 and 2 implement per-request metadata plus process-local Prometheus aggregates for provider outcomes, attempted-request latency, deterministic fallbacks, and reported input/output tokens. Broader end-to-end analysis, truncation, redaction, and evaluation metrics above remain proposed. Request IDs and workspace IDs should not be added as metric labels. Per-customer billing or audit attribution belongs in a controlled event ledger with its own access and retention policy.
-
-Do not place full prompts, log evidence, model responses, API keys, or incident payloads in metrics. Detailed audit records require separate access, retention, and redaction controls.
-
-### 3. Product Value
-
-Eventual monetization requires evidence that users receive repeatable value. Measure product outcomes without turning private incident data into analytics exhaust:
-
-- Time from workspace setup to the first successful analysis.
-- Active workspaces that run analyses or evaluations.
-- Incident summaries reviewed, accepted, or corrected by users.
-- Evaluation regressions caught before a release.
-- Fallback frequency and provider cost per successful analysis.
-- Team adoption of private eval sets, release gates, and audit exports.
-
-Avoid unsupported claims such as "hours saved." Measure user actions and ask for explicit feedback before translating activity into business impact.
+Week 5 implements per-request metadata plus process-local Prometheus aggregates for provider outcomes, attempted-request latency, deterministic fallbacks, and reported input/output tokens. Request IDs and workspace IDs should not be added as metric labels. Durable audit records require separate access, retention, and redaction controls.
## Dashboard Set
-Use a small dashboard set with clear owners:
-
| Dashboard | Primary questions | Owner |
| --- | --- | --- |
| Service health | Are users seeing errors or latency? Which release or dependency changed? | Service or platform team |
-| Assistant operations | Are analyses available, fast, and falling back safely? | AI product team |
-| Quality and safety | Did grounding, privacy, safety, or honesty regress? | AI product and security teams |
-| Cost and capacity | Which provider, model, workspace, or workflow drives usage? | AI product and FinOps teams |
-| Product value | Are teams reaching useful outcomes and adopting governance workflows? | Product team |
+| Assistant operations | Are analyses available, fast, and falling back safely? | Maintainer |
+| Quality and safety | Did grounding, privacy, safety, or honesty regress? | Maintainer and security reviewer |
+| Cost and capacity | Which provider or model drives usage? | Maintainer |
Each chart should answer an operating question. Remove charts that have no owner or decision attached to them.
## SLO And Alert Design
-Define indicators before targets. Measure a baseline, then choose targets that reflect user expectations and operational capacity.
-
-Useful SLI templates include:
-
-- **Service availability:** successful non-simulation requests / valid requests.
-- **Service latency:** requests completed below the chosen threshold / valid requests.
-- **Assistant availability:** completed analyses / accepted analysis requests.
-- **Assistant latency:** analyses completed below the chosen threshold / completed analyses.
-- **Evaluation safety:** required safety and privacy checks passed / required checks executed.
-- **Provider resilience:** analyses completed through the normal or deterministic fallback path / accepted requests.
-
-Safety and privacy evaluation checks should remain release gates, not error-budget averages. One leaked fixture secret is a failed build even if every other case passes.
-
-Alerts should be user-impacting, actionable, and owned. Prefer sustained burn-rate or symptom alerts over one alert per error event. Every page should identify:
-
-- What user impact is likely.
-- Which dashboard and runbook to open.
-- Who owns the response.
-- What safe first checks to run.
-- When to escalate.
-
-## Privacy, Security, And Cost Controls
-
-Production observability can become a second copy of sensitive data. Apply the same discipline used at the assistant boundary:
-
-- Prevent secrets and private payloads from entering telemetry.
-- Redact near ingestion and before assistant access.
-- Use role-based access for logs, traces, dashboards, and evidence APIs.
-- Audit access to sensitive incident evidence.
-- Set different retention periods for metrics, logs, traces, and audit records.
-- Sample high-volume traces and low-value logs intentionally.
-- Enforce metric label allowlists and cardinality budgets.
-- Track ingestion, storage, query, and egress cost by environment and team.
-- Keep development and simulation traffic separate from production SLOs.
-
-Long retention is not automatically better. Retain enough evidence for debugging, reliability trends, security obligations, and customer commitments, then delete it according to policy.
-
-## Deployment Choices
-
-### Open Source Path
-
-Use an OpenTelemetry Collector with Prometheus and Grafana, plus compatible log and trace backends. This maximizes control and learning but adds operating responsibility.
-
-### Managed Path
-
-Send collector output to a managed observability platform. This reduces backend operations but requires deliberate cost, retention, access, and vendor-exit planning.
-
-### Hybrid Path
-
-Keep OpenTelemetry at the collection boundary and choose different backends by environment or signal type. This preserves portability while allowing managed services where they reduce meaningful toil.
+Start with a small set of indicators:
-The default Reliability Lab setup should remain dependency-light. A production stack belongs in an optional deployment path after the signals and ownership model are understood.
+- Service availability: successful requests divided by all requests.
+- Service latency: a chosen percentile for a bounded route group.
+- Assistant availability: completed analyses divided by requested analyses.
+- Assistant latency: time from request receipt to final response.
+- Evaluation safety: passing safety and privacy checks divided by all checks.
+- Provider resilience: provider failures and deterministic fallbacks over provider selections.
-## Staged Migration
+Attach an owner, runbook, and escalation path to every alert. Alert only when an operator can make a different decision from the signal.
-1. **Standardize emission.** Send application logs to stdout, preserve stable structured fields, and keep metric labels bounded.
-2. **Add collection.** Deploy a telemetry collector with batching, resource attributes, redaction, and environment-aware routing.
-3. **Choose storage.** Select metrics, logs, and trace backends with explicit retention and access policies.
-4. **Build core dashboards.** Start with user impact, service health, assistant operations, quality, and cost.
-5. **Define SLOs and alerts.** Baseline first, assign owners, link runbooks, and test alert delivery.
-6. **Control assistant access.** Query bounded evidence through authenticated APIs with redaction and audit logs.
-7. **Measure product value.** Add privacy-aware adoption and outcome signals that support product decisions and future paid tiers.
+## Privacy And Cost Controls
-## Production Readiness Checklist
+- Keep prompts, evidence, model output, credentials, endpoints, and identifiers out of metrics labels.
+- Bound high-cardinality dimensions before emitting telemetry.
+- Use sampling and retention limits for detailed traces and logs.
+- Separate operational telemetry from detailed evidence access.
+- Treat provider usage and cost estimates as operational signals, not durable metering.
-- [ ] Logs, metrics, and traces use stable service and environment attributes.
-- [ ] Metric labels have an allowlist and cardinality budget.
-- [ ] Sensitive fields are blocked or redacted before storage.
-- [ ] Telemetry access uses least privilege and is audited where required.
-- [ ] Retention and sampling policies have named owners.
-- [ ] Dashboards answer user-impact and assistant-quality questions.
-- [ ] SLOs exclude intentional simulation traffic.
-- [ ] Alerts are actionable, routed, tested, and linked to runbooks.
-- [ ] Assistant evidence queries are bounded, redacted, and attributable.
-- [ ] Provider usage can be compared with evaluation quality and user outcomes.
+## Migration Path
-## What Comes Next
+1. Keep the local shared-log path for the quickstart.
+2. Emit structured stdout logs and stable correlation fields.
+3. Introduce a collector once signal contracts are stable.
+4. Route metrics, logs, and traces to a chosen backend.
+5. Build one dashboard and one owned alert.
+6. Exercise one incident end to end: alert, evidence, analysis, runbook action, and recovery review.
+7. Revisit retention, access, and cardinality after real operational use.
-This guide defines the upgrade path; it does not install a production stack into the beginner workflow. The [project roadmap](09-roadmap.md) schedules provider telemetry for Week 5, evaluation maturity for Week 6, and the optional OpenTelemetry signal path plus one alert-to-runbook exercise for Week 7. Backend examples and broader dashboard or alert coverage remain later additions driven by measured operating needs.
+The local setup remains valuable because it teaches the signal path without requiring a full observability stack.
diff --git a/docs/19-advanced-model-serving-roadmap.md b/docs/19-advanced-model-serving-roadmap.md
index c2cb60f..5dd2f58 100644
--- a/docs/19-advanced-model-serving-roadmap.md
+++ b/docs/19-advanced-model-serving-roadmap.md
@@ -1,227 +1,53 @@
# Advanced Model Serving Roadmap
-Week 4, Day 6 defines when Reliability Lab should move beyond external model providers and which serving layer should solve each production problem.
+This guide explains when optional self-hosted inference may be justified for Reliability Lab. The default remains the deterministic analyzer with an optional OpenAI-compatible provider path.
-The project does not need GPUs or a self-hosted model to remain useful. The deterministic analyzer and optional OpenAI-compatible provider path should stay the default until customer requirements or measured workload economics justify more infrastructure.
+## Decision Inputs
-## Day 6 Decision
+Consider a private endpoint or self-hosted model only when measured evidence identifies a concrete need:
-Advanced serving is not one upgrade. It is a sequence of decisions:
+- A privacy or deployment boundary requires controlled infrastructure.
+- An approved model must run in a specific environment.
+- Sustained traffic makes provider latency or capacity unsuitable for the workload.
+- Multiple models need shared routing, scheduling, or evaluation.
+- A benchmark shows a clear improvement for the same quality and safety gates.
-```text
-Need an LLM?
- no -> deterministic analysis
- yes -> external or managed provider
- |
- +-> keep using it while quality, privacy, latency, and unit economics work
- |
- +-> self-host only when measured requirements justify ownership
-```
+Do not add GPUs, Kubernetes operators, or serving frameworks simply because they are common in AI infrastructure diagrams.
-A framework should enter the architecture because it removes a demonstrated constraint. Adding vLLM, Triton, Ray, KServe, and GPU nodes at the same time would multiply failure modes without proving user value.
+## Staged Path
-## Adoption Gates
+### Stage 0: Deterministic And Provider-Compatible
-Consider self-hosted inference only when at least one gate is backed by evidence:
+Keep the rule-based analyzer as the default. Use the same evaluation corpus and privacy controls for every optional provider experiment.
-- **Privacy or deployment boundary:** customers require a private VPC, dedicated environment, regional deployment, or on-premises operation.
-- **Model control:** the product needs a model, adapter, quantization, context configuration, or release schedule that a provider does not support.
-- **Latency:** measured provider latency prevents an agreed user workflow or SLO.
-- **Volume economics:** sustained demand makes self-hosted cost per successful analysis competitive after all operating costs are included.
-- **Availability control:** provider limits or outages materially affect the product and a self-hosted fallback has a justified reliability role.
-- **Multi-model workload:** the product must serve several predictive or generative models with different runtime and scaling needs.
+### Stage 1: Private Endpoint Benchmark
-Do not self-host because GPU infrastructure looks impressive. Low utilization, uncertain demand, and an unmeasured workload usually make managed inference the more rational starting point.
+Test one approved OpenAI-compatible endpoint against the managed-provider and deterministic paths. Measure quality, latency, token usage, fallbacks, throughput, and cost per successful evaluated analysis.
-## Serving Layers
+### Stage 2: Single-GPU Experiment
-```mermaid
-flowchart LR
- product["Reliability Lab API and policy layer"] --> gateway["authenticated model gateway"]
- gateway --> engine["inference engine such as vLLM or Triton"]
- gateway --> appserve["distributed application serving when needed"]
- appserve --> engine
- platform["Kubernetes serving control when needed"] --> gateway
- platform --> appserve
- platform --> engine
- gpu["GPU drivers, device plugins, nodes, and scheduling"] --> engine
- telemetry["quality, latency, tokens, utilization, and cost"] --> product
- engine --> telemetry
- appserve --> telemetry
- platform --> telemetry
-```
+Run one bounded model on one GPU only after the benchmark identifies a specific gap. Record model version, hardware, concurrency, memory behavior, failure handling, and rollback steps.
-The Reliability Lab API remains the product boundary. It owns evidence access, redaction, evaluation, policy, quotas, auditability, and user workflows. Serving frameworks provide model execution and orchestration underneath that boundary.
+### Stage 3: Managed Serving
-## What Each Tool Solves
+Introduce vLLM, Triton, Ray Serve, or KServe only when a measured orchestration problem requires it. Document the exact problem, acceptance criteria, resource budget, and operational owner.
-| Layer | Use it when | It earns its place by | Skip it when |
-| --- | --- | --- | --- |
-| vLLM | The product self-hosts supported language models behind an OpenAI-compatible API. | Improving LLM throughput or memory efficiency and supporting single-node or distributed parallelism. | External providers still satisfy privacy, quality, latency, and cost requirements. |
-| NVIDIA Triton Inference Server | The platform serves multiple model formats, predictive models, ensembles, or workloads needing configurable schedulers and batching. | Standardizing multi-framework inference, model repositories, concurrency, and dynamic batching. | The workload is only a straightforward language model already handled by a simpler engine. |
-| Ray Serve | The inference application has Python-native preprocessing, routing, fan-out, model composition, or components that need independent CPU/GPU scaling. | Scaling a distributed application graph and its components separately. | One model endpoint and normal Kubernetes deployment primitives are enough. |
-| KServe | A Kubernetes platform team needs declarative model lifecycle, standardized runtimes, networking, rollout, and autoscaling patterns across many models or teams. | Providing a shared Kubernetes serving control plane instead of custom manifests per workload. | The team has one deployment, limited Kubernetes operating capacity, or no platform standardization problem. |
-| Kubernetes GPU scheduling | Any containerized inference workload needs specialized hardware. | Making GPUs discoverable and schedulable through vendor drivers, device plugins, resource requests, and node placement controls. | The workload remains provider-hosted or CPU-only. |
+## Operating Requirements
-These tools overlap and can be combined. vLLM can be the LLM engine under a larger Ray Serve or KServe deployment. Triton can serve diverse model types behind a platform layer. The goal is the smallest stack that meets measured requirements, not one stack containing every project.
+Before expanding beyond a bounded experiment, define:
-## Maturity Path
+- Authentication and authorization for the endpoint.
+- Secret management and rotation.
+- Model and container version pinning.
+- Resource requests, limits, and GPU scheduling policy.
+- Queue, latency, error, utilization, and out-of-memory signals.
+- Rollback and recovery procedures.
+- The same privacy, safety, and evaluation release gates used by the default path.
-### Stage 0: Provider-First Product
+## Non-Goals
-Keep deterministic analysis as the free and reliable baseline. Use an external OpenAI-compatible provider only for optional enrichment. Measure quality, token use, latency, fallback frequency, and cost per successful analysis.
+Reliability Lab does not aim to become a general model-serving platform. It keeps optional serving examples small, evidence-driven, and subordinate to the learning path.
-### Stage 1: Local Compatibility Test
+## Exit Criteria
-Point the existing OpenAI-compatible client at a private development endpoint. A vLLM server can fit the existing configuration shape:
-
-```text
-LLM_PROVIDER=openai
-OPENAI_BASE_URL=http://private-model-endpoint:8000/v1
-OPENAI_API_KEY=local-or-gateway-token
-MODEL_NAME=approved-model-name
-```
-
-This is a compatibility experiment, not a production deployment. Keep authentication at the gateway and never commit a real token.
-
-### Stage 2: Single-GPU Benchmark
-
-Serve one approved model on one GPU. Run sanitized incident cases and representative concurrency. Compare it with the current provider path using the same evaluation corpus and acceptance gates.
-
-### Stage 3: Production Inference Service
-
-Add hardened images, model artifact controls, health checks, resource limits, bounded queues, timeouts, rollout and rollback procedures, centralized telemetry, and an on-call owner. Keep one serving engine unless a second one solves a measured workload need.
-
-### Stage 4: Shared Serving Platform
-
-Introduce Ray Serve for distributed application composition or KServe for Kubernetes-wide serving standards only after multiple deployments or teams create repeatable platform work. Triton becomes relevant when multi-framework and model-repository requirements are real.
-
-### Stage 5: Commercial Multi-Tenant Operation
-
-Add tenant isolation, quotas, metering, policy enforcement, regional placement, capacity reservations, chargeback, support processes, and customer-facing reliability commitments. Multi-tenancy must not rely on metric labels or shared prompt logs for billing attribution.
-
-## Benchmark Before Buying GPUs
-
-Use the same sanitized incident workload for providers and self-hosted candidates. Test realistic input sizes, output sizes, concurrency, and burst patterns.
-
-Measure:
-
-- Evaluation pass rate by grounded, useful, safe, private, and honest dimensions.
-- Time to first token for interactive streaming workloads.
-- Time per output token and total response latency.
-- Requests and tokens completed per second.
-- Queue time, rejection rate, timeout rate, and fallback rate.
-- GPU utilization, memory use, out-of-memory events, and model load time.
-- Cost per request and cost per successful evaluated analysis.
-- Operator time, deployment frequency, recovery time, and incident burden.
-
-Quality is a gate before throughput. A faster model that fails required privacy, safety, or grounding checks is not a cheaper product.
-
-## Unit Economics
-
-Provider and self-hosted costs must be compared on the same outcome:
-
-```text
-provider cost per successful analysis
- = provider spend / successful evaluated analyses
-
-self-hosted cost per successful analysis
- = (GPU + nodes + storage + networking + observability + engineering + support)
- / successful evaluated analyses
-```
-
-Include idle capacity, failed requests, cold starts, model downloads, redundancy, and on-call work. GPU hourly price alone is not the self-hosted cost.
-
-Track contribution margin by plan or deployment type only after metering is trustworthy. Do not promise savings before a representative benchmark and sustained utilization profile exist.
-
-## GPU Operating Requirements
-
-A production GPU path needs more than `nvidia.com/gpu: 1`:
-
-- Compatible GPU nodes, drivers, container runtime integration, and vendor device plugins.
-- Dedicated node pools with labels, taints, tolerations, and explicit resource requests.
-- Capacity limits, quotas, priority rules, and protection against one tenant consuming the fleet.
-- Model and image caching strategies with controlled artifact provenance.
-- Memory-aware model placement and tested out-of-memory recovery.
-- Autoscaling signals based on queues, concurrency, token throughput, or utilization rather than CPU alone.
-- Cold-start budgets and a deliberate choice between minimum capacity and scale-to-zero savings.
-- GPU utilization, memory, temperature, errors, queue depth, latency, and cost telemetry.
-- Upgrade testing across drivers, CUDA libraries, serving engines, models, and Kubernetes components.
-
-Autoscaling cannot create GPU capacity that the cluster or cloud account does not have. Capacity planning and reservation strategy remain product and operational decisions.
-
-## Security And Governance
-
-Self-hosting moves responsibility into the product team:
-
-- Verify model licenses and allowed commercial use.
-- Control model sources, hashes, artifacts, container images, and dependencies.
-- Protect model weights, adapters, prompts, retrieved evidence, and generated output.
-- Authenticate every inference endpoint and restrict network paths.
-- Separate tenant evidence, quotas, caches, and audit records.
-- Apply redaction before model access and before durable telemetry.
-- Version prompts, models, serving configuration, and evaluation results together.
-- Require evaluated rollouts, canaries, and rollback paths.
-
-A private deployment is not automatically secure. It only changes who owns the controls.
-
-## Monetization Possibilities
-
-The serving engine should not be the product moat. Open-source inference projects will continue to improve. Reliability Lab can monetize the operational and governance layer that customers repeatedly need.
-
-| Possible tier | User outcome | Potential paid capabilities |
-| --- | --- | --- |
-| Community | Learn and run evidence-grounded incident analysis locally. | Deterministic analysis, provider-compatible configuration, public eval corpus, and deployment guidance remain open. |
-| Team | Adopt the assistant safely across a shared engineering workflow. | Hosted evaluation history, collaboration, private incident sets, quotas, usage and cost dashboards, and CI release gates. |
-| Enterprise cloud | Connect production telemetry with stronger governance. | SSO, RBAC, audit exports, policy controls, private connectors, regional data handling, reliability commitments, and support. |
-| Private deployment | Meet VPC, dedicated, or on-premises requirements. | Customer-controlled model endpoints, approved model catalog, deployment automation, upgrades, capacity planning, and enterprise support. |
-| Platform | Operate models and assistant workflows across teams or clusters. | Fleet policy, routing, model lifecycle, chargeback, benchmark comparisons, capacity governance, and multi-cluster visibility. |
-
-The commercial value is measurable trust and operational control: customers can choose a provider or private model without rebuilding evaluation, evidence policy, observability, quotas, and audit workflows.
-
-## Build Versus Buy Rule
-
-Continue buying inference while it accelerates product learning and meets customer requirements. Build a self-hosted path when it unlocks signed demand, resolves a hard deployment constraint, or improves measured unit economics after operating cost.
-
-Before committing to a serving platform, require:
-
-- A named customer or product requirement.
-- A representative benchmark and quality gate.
-- A 12-month cost model with utilization assumptions.
-- An operating owner and incident response plan.
-- A security and model-license review.
-- A rollback to the provider or deterministic path.
-
-## Recommended Implementation Order
-
-The first four steps are the measurement work scheduled for Weeks 5-8 of the [project roadmap](09-roadmap.md). Serving infrastructure begins only when those steps and the [commercialization gates](21-commercialization-roadmap.md) support it:
-
-1. Capture provider token usage, model identity, latency, outcome, fallback, and cost metadata in Week 5.
-2. Expand and version the evaluation corpus in Week 6 so every candidate uses the same privacy and safety gates.
-3. Stabilize the optional production signal path and operational exercise in Week 7.
-4. Run the provider-versus-private-endpoint benchmark and record the build-versus-buy decision in Week 8.
-5. Test one approved model behind an authenticated OpenAI-compatible endpoint only when the decision supports it.
-6. Add an optional single-GPU deployment example without changing the default quickstart.
-7. Prove quality, reliability, and unit economics under representative load.
-8. Add Kubernetes GPU scheduling and observability for a real deployment requirement.
-9. Introduce Ray Serve, Triton, or KServe only when its specific orchestration problem appears.
-10. Package private deployment, governance, metering, and support around validated customer demand.
-
-## Day 6 Definition Of Done
-
-- The default Reliability Lab path still requires no GPU.
-- Each advanced tool maps to a specific problem and adoption gate.
-- Provider and self-hosted options use the same evaluation standard.
-- GPU cost is compared per successful analysis, including operations.
-- Private deployment and multi-tenancy have explicit security requirements.
-- Monetization focuses on customer outcomes, governance, and support rather than reselling an open-source engine.
-
-## Official References
-
-- [vLLM OpenAI-Compatible Server](https://docs.vllm.ai/en/latest/serving/online_serving/openai_compatible_server/)
-- [vLLM Parallelism and Scaling](https://docs.vllm.ai/en/stable/serving/parallelism_scaling/)
-- [NVIDIA Triton Inference Server](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html)
-- [Ray Serve Model Composition](https://docs.ray.io/en/latest/serve/model_composition.html)
-- [Ray Serve Autoscaling](https://docs.ray.io/en/latest/serve/autoscaling-guide.html)
-- [KServe Documentation](https://kserve.github.io/website/docs/intro)
-- [Kubernetes GPU Scheduling](https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/)
+Advance a serving stage only when the benchmark, evaluation suite, and operational review agree that the next layer solves a documented problem. Otherwise, keep the simpler provider-compatible path.
diff --git a/docs/20-production-readiness-review.md b/docs/20-production-readiness-review.md
index 8bd65b2..00f9f41 100644
--- a/docs/20-production-readiness-review.md
+++ b/docs/20-production-readiness-review.md
@@ -1,83 +1,48 @@
# Production Readiness Review
-Week 4, Day 7 closes the first Reliability Lab learning cycle with an explicit release decision.
+Week 4, Day 7 records what Reliability Lab is ready for today and what remains before a broader deployment.
-The project now demonstrates a complete local path from application signals to evidence-grounded incident analysis. That makes it ready to publish as a learning lab. It does not make the current Docker Compose or kind configuration ready for internet-facing production use.
+## Readiness Verdict
-## Release Decision
-
-| Target | Decision | Reason |
+| Deployment target | Verdict | Evidence and boundary |
| --- | --- | --- |
-| Local learning lab | **Go** | The setup is reproducible, deterministic without an LLM key, documented, tested, and covered by a runnable evaluation corpus. |
-| Public open-source learning release | **Go when `make validate` passes** | Tests, lint, and assistant quality checks are enforced locally and in pull-request CI. |
-| Internal pilot with sanitized data | **Conditional** | Add environment-specific access control, centralized telemetry, secret management, ownership, and a rollback plan first. |
-| Internet-facing or customer production | **No-go today** | The lab does not yet provide production authentication, durable telemetry backends, tenant isolation, managed secrets, SLO operations, or hardened deployment automation. |
-
-This distinction is intentional. A strong readiness review records what the evidence supports instead of relabeling a useful lab as a production platform.
-
-## One Release Gate
-
-Run the full local gate from the repository root:
-
-```bash
-make validate
-```
-
-It performs four checks:
-
-1. Builds both service images.
-2. Runs both pytest suites.
-3. Runs Ruff against both services.
-4. Runs the assistant evaluation corpus and fails on any grounding, usefulness, safety, privacy, or honesty regression.
+| Local learning lab | **Go** | Docker Compose, deterministic fallback, tests, evaluation corpus, and local runbooks are available. |
+| Public learning release | **Go** | Documentation, license, contribution guidance, safety notes, and pull-request validation are present. |
+| Sanitized internal pilot | **Conditional** | Requires named owners, access controls, managed secrets, centralized telemetry, and a tested rollback path. |
+| Internet-facing production | **No-go today** | The lab does not yet provide production authentication, durable telemetry backends, tenant isolation, managed secrets, SLO operations, or hardened deployment automation. |
-The CI workflow runs the same test, lint, and evaluation categories on every pull request and on pushes to `main`. No provider key or network model call is required for the evaluation gate.
+## Current Evidence
-Formatting remains a deliberate author action:
-
-```bash
-make format
-```
-
-## Evidence Review
-
-| Readiness area | Evidence in this repository | Current boundary |
+| Area | Evidence | Remaining boundary |
| --- | --- | --- |
-| Reproducibility | Docker Compose, kind manifests, health checks, tests, and copy-paste workflows. | Local and learning environments only. |
-| Security | Threat model, safe defaults, secret guidance, and input/output redaction tests. | Pattern redaction is not a data-loss-prevention system; production auth and policy enforcement remain future work. |
-| Cost | Deterministic default, bounded evidence windows, prompt limits, and visible cost-control metadata. | Provider token accounting, quotas, budgets, and chargeback are not implemented. |
-| Quality | Seven deterministic incident cases and a five-dimension pass/fail rubric. | The corpus is small and does not replace sanitized real incidents, human review, or provider/model evaluation. |
-| Observability | Structured logs, request correlation, Prometheus-style metrics, incident walkthroughs, and a production signal contract. | The shared file is a teaching mechanism; no collector or durable logs, metrics, and traces backend is installed. |
-| Reliability | Health/readiness endpoints, Kubernetes probes and resources, runbooks, and rollback guidance. | No measured production SLO, alert delivery, autoscaling, multi-zone design, or disaster recovery exercise. |
-| Model serving | Provider-compatible configuration and a staged vLLM, Triton, Ray Serve, KServe, and GPU decision framework. | No GPU path is needed or claimed for the default project. |
-| Product direction | Open learning core plus a governed evaluation, private deployment, and team workflow direction. | Customer discovery and measured demand must precede platform investment. |
+| Security | Redaction tests, secret-handling guidance, bounded prompts, and no-key deterministic default. | Pattern redaction is not complete data-loss prevention; production needs stronger identity and secret controls. |
+| Quality | Deterministic incident corpus, five-dimension rubric, and CI release gate. | Broader sanitized cases, versioned thresholds, and provider comparison reports remain future work. |
+| Reliability | Health checks, Compose health dependencies, Kubernetes probes, resource requests, and operations runbooks. | No multi-replica, rollback, or recovery exercise evidence yet. |
+| Observability | Structured logs, Prometheus text, provider telemetry, and a migration path. | No centralized backend, dashboard, alert ownership, or retention policy is implemented. |
+| Cost | Prompt bounds, deterministic fallback, provider usage metadata, and explicit per-call estimates. | No durable usage history or operational budget policy exists. |
+| Deployment | Docker Compose and local kind manifests. | No ingress, TLS, environment separation, artifact pinning, or deployment automation. |
-## Promotion Gates For A Real Pilot
+## Required Promotion Gates
-Do not promote the current manifests by changing only an image tag. Before an internal pilot, require named owners and evidence for these gates:
+Before a sanitized internal pilot, document and verify:
-- **Identity and access:** authenticate users and services; authorize access to incident evidence; audit sensitive access.
-- **Data boundary:** classify allowed evidence, prevent secrets at the source, test redaction, define retention, and document provider handling.
-- **Secrets and supply chain:** use a managed secret workflow, pin and scan deployable artifacts, and define rotation and patch ownership.
-- **Reliability:** establish baseline SLIs, choose an initial SLO, route actionable alerts, test a runbook, and prove rollback.
-- **Observability:** replace shared files with controlled collection and storage; keep evidence queries bounded and attributable.
-- **Quality:** add sanitized representative incidents, preserve privacy and safety as hard gates, and version model, prompt, corpus, and thresholds together.
-- **Cost:** capture provider identity, latency, token usage, fallback outcome, and cost per successful evaluated analysis; add budgets and quotas.
-- **Deployment:** separate environments, use production storage and networking, define backups where state exists, and test recovery.
+- Identity, authorization, and least-privilege access.
+- Managed secrets, rotation, and auditability.
+- Dependency and container artifact pinning.
+- Environment separation, ingress, and TLS.
+- Centralized logs, metrics, traces, retention, and access controls.
+- SLOs, owned alerts, rollback tests, and recovery exercises.
+- Versioned evaluation corpus, configuration, and acceptance thresholds.
+- Privacy and safety gates that block promotion on failure.
-Any failed privacy or safety gate is a no-go. Other exceptions need a named owner, explicit risk acceptance, a deadline, and a rollback path.
+## Next Technical Priorities
-## First 30 Days After This Milestone
-
-The next four technical weeks deepen measured contracts before adding infrastructure breadth. They match the canonical sequence in the [project roadmap](09-roadmap.md) and do not bypass the audience or design-partner gates in the [commercialization roadmap](21-commercialization-roadmap.md).
-
-| Week | Focus | Required outcome |
-| --- | --- | --- |
-| 5 | Provider telemetry | Start with the privacy-safe per-call contract, then aggregate model, latency, token usage, failure, fallback, and cost metadata and compare it with evaluation outcomes. |
-| 6 | Evaluation maturity | Expand and version sanitized and adversarial cases, produce machine-readable CI results, and preserve privacy and safety as hard gates. |
-| 7 | Production signal path | Add optional OpenTelemetry collection only after signal contracts are stable, then exercise one owned alert-to-runbook recovery flow. |
-| 8 | Provider versus private benchmark | Compare deterministic, provider, and compatible private endpoints with the same corpus and record a build-versus-buy decision. |
+The next four technical weeks deepen measured contracts before adding infrastructure breadth:
-Internal-pilot controls begin only after this measurement loop works and customer discovery supports a design-partner deployment. vLLM, Triton, Ray Serve, KServe, GPU scheduling, multi-tenancy, and paid platform features stay behind demonstrated demand and the adoption gates in the [advanced model serving roadmap](19-advanced-model-serving-roadmap.md).
+1. Provider metadata, usage, latency, and cost estimates.
+2. A larger versioned evaluation corpus with machine-readable reports.
+3. One optional telemetry collector path and an owned alert-to-runbook exercise.
+4. A benchmark across deterministic, managed-provider, and private endpoint paths.
## Day 7 Definition Of Done
@@ -85,5 +50,5 @@ Internal-pilot controls begin only after this measurement loop works and custome
- Pull-request CI blocks an evaluation regression.
- The project has an explicit go/no-go decision for each deployment target.
- Local evidence and production gaps are visible in the same scorecard.
-- Pilot promotion gates have owners and measurable evidence requirements.
+- Promotion gates have owners and measurable evidence requirements.
- The next 30 days prioritize instrumentation, evaluation depth, and operational proof over tool accumulation.
diff --git a/docs/21-commercialization-roadmap.md b/docs/21-commercialization-roadmap.md
deleted file mode 100644
index 04b2b20..0000000
--- a/docs/21-commercialization-roadmap.md
+++ /dev/null
@@ -1,123 +0,0 @@
-# Commercialization Roadmap
-
-This document turns the product direction from `docs/19-advanced-model-serving-roadmap.md` and the next-stage priorities from `docs/20-production-readiness-review.md` into a sequenced plan with milestones and success metrics.
-
-The strategy is deliberate: audience first, wedge product second, paid tiers third. Each phase funds and de-risks the next, and no phase starts before the previous phase's exit criteria are met.
-
-## Strategy Summary
-
-The AI incident-copilot market is crowded with well-funded products. Reliability Lab does not win by out-building them from a standing start. It wins by:
-
-1. Teaching an underserved audience (production engineers new to AI infra) and earning distribution through the open learning lab.
-2. Converting that audience into customer discovery for a narrow wedge: an evidence-grounded incident analyst whose quality is enforced by a CI evaluation gate.
-3. Monetizing the trust and governance layer — evaluation history, private incident sets, redaction policy, audit — not the inference engine.
-
-```mermaid
-flowchart LR
- audience["Phase A
audience and learning lab"] --> wedge["Phase B
wedge product with design partners"]
- wedge --> paid["Phase C
paid team tier"]
- audience -. "customer discovery" .-> wedge
- wedge -. "usage evidence" .-> paid
-```
-
-## Phase A: Public Release and Audience (Months 0-3)
-
-Goal: publish the learning lab, build a distribution channel, and validate that the audience exists and will engage.
-
-### Milestones
-
-| Milestone | Definition of done |
-| --- | --- |
-| Public release | `make validate` passes, naming is consistent, and the repository is public with a clear README, license, and contribution guide. |
-| Launch posts | A "Show HN" or equivalent launch post plus at least one written walkthrough of an incident analysis session. |
-| Content cadence | One build-in-public post per week for eight consecutive weeks, sourced from `docs/build-log.md` and the incident walkthroughs. |
-| Contact channel | An email list or equivalent owned channel that readers can join from the README. |
-| First paid validation | One small paid artifact live: a deep-dive course, ebook, workshop, or sponsorship. Revenue target is trivial; the goal is proof that someone pays. |
-| Customer discovery | Fifteen structured conversations with SRE, DevOps, or platform engineers who used or starred the project, each answering: "What would it take for you to point this at your real logs?" |
-
-### Success metrics
-
-- 1,000+ GitHub stars or 200+ unique cloners per month.
-- 300+ owned-channel subscribers.
-- 15 completed discovery interviews with written notes.
-- At least 3 interviewees who describe a current, painful incident-analysis workflow they would change.
-- First dollar of revenue from any source.
-
-### Exit criteria for Phase B
-
-Proceed only when discovery interviews converge on a repeated, specific pain and at least three teams volunteer to try the assistant against their own sanitized telemetry. If interviews do not converge, stay in Phase A and adjust the audience or the message — do not start building the wedge on guesses.
-
-## Phase B: Wedge Product with Design Partners (Months 3-6)
-
-Goal: make the assistant useful against real telemetry for two or three design-partner teams, free, in exchange for feedback and permission to reference them.
-
-The wedge thesis: teams will adopt an incident analyst they can hold to a measurable quality bar. The evaluation harness is the differentiator; the assistant is the demo.
-
-### Milestones
-
-| Milestone | Definition of done |
-| --- | --- |
-| First real connector | Read-only connector to one real log source (CloudWatch Logs, Loki, or Datadog), honoring the existing bounded-query and redaction policies. The shared-file path remains the learning default. |
-| Provider metering | Build on the Week 5 per-request contract: aggregate model identity, latency, token usage, fallback outcome, and cost per successful evaluated analysis without storing prompt or incident content. The milestone is incomplete until design-partner usage is measurable. |
-| Private incident sets | A team can record sanitized incidents from their own systems and run the evaluation corpus against them in CI, blocking regressions. |
-| Minimum viable trust | OIDC/SSO authentication, per-user audit log of evidence access, and a documented single-tenant deployment using the existing Kubernetes manifests. |
-| Design partners | Two or three teams running the assistant against sanitized real data weekly, with a shared feedback channel and written usage notes. |
-
-### Success metrics
-
-- 2-3 active design partners with weekly usage (not one-time trials).
-- 10+ sanitized real incidents in partner evaluation sets.
-- Assistant passes the five-dimension rubric (grounded, useful, safe, private, honest) on partner incidents, not just the public corpus.
-- Measured cost per successful analysis for each provider configuration.
-- At least one partner quote or case study approved for public use.
-- Zero privacy or safety gate failures on partner data. Any failure stops the phase until resolved.
-
-### Exit criteria for Phase C
-
-Proceed when at least two design partners say they would pay to keep the workflow, and can articulate the budget it would come from. "This is neat" is not an exit signal; "we would lose something we now rely on" is.
-
-## Phase C: Paid Team Tier (Months 6-12)
-
-Goal: convert the design-partner workflow into the Team tier described in `docs/19-advanced-model-serving-roadmap.md`, with pricing validated against real willingness to pay.
-
-### Milestones
-
-| Milestone | Definition of done |
-| --- | --- |
-| Hosted evaluation history | Evaluation runs, thresholds, and pass/fail history stored per team with regression alerts, versioned alongside model, prompt, and corpus (the versioning contract from `docs/20-production-readiness-review.md`). |
-| Usage and cost dashboards | Per-team visibility into analyses run, provider spend, fallback rate, and quality trend. |
-| Quotas and budgets | Per-team token and spend limits with graceful degradation to the deterministic path. |
-| Pricing and packaging | Public pricing page for the Team tier (per-seat) and a contact path for single-tenant private deployment (annual contract). Prices set from design-partner conversations, not guesses. |
-| Billing and terms | Payment, invoicing, terms of service, privacy policy, and a support commitment appropriate to the price point. |
-| First paying customers | Three paying teams, at least one converted from a design partner. |
-
-### Success metrics
-
-- 3+ paying teams; $1,000+ monthly recurring revenue as a floor signal, with growth month over month.
-- Net revenue retention: no paying team churns within the first two quarters.
-- Cost per successful analysis is measured, published internally, and gross-margin positive at the chosen price.
-- Support load is sustainable for a single maintainer (or a hiring/contracting decision is made explicitly).
-- Community tier remains fully useful: deterministic analysis, provider configuration, public corpus, and deployment guidance stay open.
-
-## Explicit Non-Goals Until Gated
-
-These stay behind the adoption gates in `docs/19-advanced-model-serving-roadmap.md` and require a named customer requirement:
-
-- GPU deployment examples, vLLM, Triton, Ray Serve, KServe.
-- Multi-tenant shared infrastructure (single-tenant deployments come first).
-- A general observability platform. The product analyzes evidence; it does not replace Datadog, Grafana, or a log backend.
-- Enterprise features (RBAC, policy controls, regional data handling) before an enterprise buyer exists.
-- Fundraising decisions. This roadmap assumes bootstrap economics; revisit only with Phase C evidence.
-
-## Operating Cadence
-
-- **Weekly:** one build-in-public post; review metrics for the current phase.
-- **Monthly:** compare actuals to the current phase's success metrics; write a one-paragraph verdict in `docs/build-log.md`.
-- **Per phase:** hold the exit-criteria review before starting the next phase. Skipping a gate requires writing down why, what evidence overrides it, and what would trigger a rollback to the prior phase.
-
-## Risks and Honest Caveats
-
-- **Crowded market:** incumbents can ship an adequate incident summary as a feature. The defense is the evaluation-gated trust workflow and the audience, not the summary itself.
-- **Audience does not convert:** educational audiences often read without buying. The Phase A paid artifact exists to test conversion early and cheaply.
-- **Single maintainer:** every phase must remain operable by one person; any milestone that is not gets cut or simplified.
-- **Privacy is existential:** one leaked secret in an analysis output would end trust in the product. Privacy and safety evaluation gates remain hard no-go gates in every phase, exactly as `docs/20-production-readiness-review.md` defines them.
diff --git a/docs/22-provider-telemetry.md b/docs/22-provider-telemetry.md
index f5c7ca8..ac8d2e7 100644
--- a/docs/22-provider-telemetry.md
+++ b/docs/22-provider-telemetry.md
@@ -29,7 +29,7 @@ When `use_llm=true`, `/ask`, `/analyze/logs`, and `/summarize-incident` include
}
```
-This API contract is per request. Day 2 also aggregates its bounded fields as Prometheus text. Day 3 adds a per-call `llm_cost_estimate` only when deployment-owned prices and provider-reported input/output token counts are both available. None of these surfaces is a persistent usage ledger or billing record.
+This API contract is per request. Day 2 also aggregates its bounded fields as Prometheus text. Day 3 adds a per-call `llm_cost_estimate` only when deployment-owned prices and provider-reported input/output token counts are both available. None of these surfaces is a persistent usage ledger.
## Field Semantics
@@ -70,7 +70,7 @@ When `use_llm=true`, the response also includes `llm_cost_estimate`. Pricing is
Set `LLM_INPUT_USD_PER_MILLION_TOKENS` and `LLM_OUTPUT_USD_PER_MILLION_TOKENS` to non-negative USD values per million provider-reported tokens. Both values are optional and default to unknown. The estimate is available only when both prices and both token directions are reported; otherwise every estimated cost remains `null` and `unavailable_reason` is `pricing_not_configured` or `incomplete_token_usage`.
-Cost values are decimal strings to avoid floating-point ambiguity. They are estimates for one provider call, not invoices, budgets, durable metering, or customer-level billing attribution.
+Cost values are decimal strings to avoid floating-point ambiguity. They are estimates for one provider call, not invoices, durable metering, or identity-linked usage attribution.
Unknown, negative, boolean, or malformed token values are treated as unavailable. Providers may omit usage; omission does not turn a successful analysis into a failure.
@@ -87,13 +87,13 @@ Unknown, negative, boolean, or malformed token values are treated as unavailable
The histogram uses fixed buckets from 50 milliseconds through the 30-second request timeout plus `+Inf`. Unconfigured selections increment the analysis and fallback counters but do not create a provider-request latency observation.
-These aggregates are intentionally in memory and reset when the assistant process restarts. Durable metering, billing attribution, retention, multi-process aggregation, and customer-level usage history remain separate production concerns.
+These aggregates are intentionally in memory and reset when the assistant process restarts. Durable metering, retention, multi-process aggregation, and identity-linked usage history remain separate production concerns.
### Label Boundary
Provider and model values come only from deployment configuration, pass through redaction, and are limited to 100 characters. Outcome, fallback reason, and token direction use fixed enums. The registry maps unexpected enum values to `unknown` or `other` rather than accepting arbitrary strings.
-The aggregate contract never labels metrics with prompts, evidence, generated content, provider endpoints, errors, request IDs, users, workspaces, customers, or incidents. Operators should also keep the number of configured provider/model pairs within an explicit cardinality budget.
+The aggregate contract never labels metrics with prompts, evidence, generated content, provider endpoints, errors, request IDs, users, workspaces, or incidents. Operators should also keep the number of configured provider/model pairs within an explicit cardinality budget.
Example:
@@ -135,16 +135,15 @@ Forbidden:
Tests inject secrets into questions, logs, credentials, provider output, and failure messages and verify that those values never enter the telemetry object. The final API response still passes through the existing defense-in-depth redaction boundary.
-## Customer And Product Value
+## Operational Value
-This contract supports the commercialization roadmap without creating a surveillance or billing system prematurely:
+This contract helps maintainers understand provider behavior without storing sensitive content:
-- Design partners can see whether enrichment succeeded or silently fell back.
-- Teams can compare provider latency and token usage against evaluated quality later.
-- Product decisions can use cost per successful analysis instead of token price alone.
-- Private endpoints and managed providers can use the same normalized outcome contract.
+- Operators can distinguish successful enrichment from deterministic fallback.
+- Provider latency and token usage can be compared with evaluated quality later.
+- Managed and private OpenAI-compatible endpoints share the same normalized outcome contract.
-Do not attach customer identifiers to future Prometheus labels. Per-team billing, audit attribution, and durable usage history require an access-controlled event ledger with explicit retention.
+Do not attach user or workspace identifiers to Prometheus labels. Detailed audit records require an access-controlled event store with explicit retention.
## Day 1 Definition Of Done
@@ -175,9 +174,16 @@ Do not attach customer identifiers to future Prometheus labels. Per-team billing
- Decimal-string cost values avoid floating-point ambiguity.
- API, configuration parsing, incomplete usage, and privacy behavior have deterministic tests.
+## Day 4 Definition Of Done
+
+- The deterministic evaluation corpus can be run with explicit optional-provider enrichment.
+- Each result joins the rubric outcome with bounded provider telemetry and the per-call cost estimate.
+- The report calculates cost per successful evaluated analysis only when every successful evaluation has complete cost data.
+- Unknown price or token information remains unknown and includes a reason rather than becoming zero.
+- Normal evaluation remains offline and cost-free; provider calls require an explicit command.
+
## Remaining Week 5 Sequence
-1. Join provider outcomes and cost with evaluation results to calculate cost per successful evaluated analysis.
-2. Add a local comparison report across deterministic and configured provider paths.
-3. Exercise provider failure and fallback behavior end to end.
-4. Close the week with an exit-gate review against the technical and commercialization roadmaps.
+1. Add a local comparison report across deterministic and configured provider paths.
+2. Exercise provider failure and fallback behavior end to end.
+3. Close the week with an exit-gate review against the technical roadmap.
diff --git a/docs/build-log.md b/docs/build-log.md
index d823072..25b2667 100644
--- a/docs/build-log.md
+++ b/docs/build-log.md
@@ -485,7 +485,7 @@ What changed:
Why this matters:
-AI infrastructure cost can grow before a billing dashboard exists. A single incident summary may be cheap, but repeated provider calls, large log windows, large prompts, expensive models, and broad refresh loops can turn the same workflow into a cost problem.
+AI infrastructure cost can grow before a usage dashboard exists. A single incident summary may be cheap, but repeated provider calls, large log windows, large prompts, expensive models, and broad refresh loops can turn the same workflow into a cost problem.
The safest beginner default is still local rule-based analysis. When LLM enrichment is useful, the assistant now sends a bounded amount of evidence and tells callers which limits were active.
@@ -518,13 +518,13 @@ What changed:
- Added tests for every case plus negative tests that prove missing grounding and leaked secrets are detected.
- Added a containerized `make evaluate-assistant` workflow.
- Documented the limits of deterministic checks and the path toward real incident datasets, human review, and model comparisons.
-- Added a product direction that keeps the core evaluation method open while reserving hosted history, private datasets, release gates, alerts, comparisons, and audit exports as potential paid workflows.
+- Documented how the public evaluation method can grow through versioned cases, private fixtures, release gates, alerts, comparisons, and audit exports.
Why this matters:
An AI assistant can sound plausible without being correct. Operational teams need repeatable evidence that a new prompt, model, provider, or code change did not weaken grounding, privacy, or safety.
-Evaluation also creates a credible path to monetization. The durable product is not a one-time incident summary. It is the system that helps teams measure quality over time, use their own private cases, block regressions, compare cost against outcomes, and produce governance evidence.
+Evaluation makes the assistant easier to improve responsibly: maintainers can measure quality over time, use private cases carefully, block regressions, compare cost against outcomes, and preserve governance evidence.
Lessons learned:
@@ -532,7 +532,7 @@ Lessons learned:
- A small known corpus is more useful than an impressive but unrepeatable demo.
- Safety and privacy should be pass/fail gates, not averages that can be hidden by other scores.
- The evaluator itself needs negative tests so teams know it catches real regressions.
-- Open evaluation builds trust; hosted team workflows create a clearer paid value proposition.
+- Open evaluation builds trust and gives contributors a repeatable way to discuss regressions.
What comes next:
@@ -549,30 +549,30 @@ Day 4 made assistant quality testable. Day 5 asks how teams can operate, govern,
What changed:
- Added a production telemetry flow from services through a collector into metrics, logs, traces, dashboards, alerts, and controlled assistant evidence APIs.
-- Separated observability into service reliability, assistant quality and safety, and product-value signals.
+- Separated observability into service reliability and assistant quality-and-safety signals.
- Proposed bounded assistant metric names for requests, latency, fallbacks, truncation, redaction, eval checks, and provider tokens.
-- Added dashboard ownership for platform, AI product, security, FinOps, and product teams.
+- Added dashboard ownership for platform, security, and operations maintainers.
- Added SLI templates for service health, assistant availability, latency, evaluation safety, and provider resilience.
- Documented alert design around user impact, ownership, runbooks, and sustained symptoms.
- Added privacy, security, retention, sampling, cardinality, and telemetry-cost controls.
- Compared open-source, managed, and hybrid deployment paths without changing the default local stack.
- Added a seven-stage migration plan and production-readiness checklist.
-- Connected privacy-aware product telemetry to eventual monetization without treating sensitive incident data as analytics exhaust.
+- Documented privacy-aware telemetry boundaries without treating sensitive incident data as analytics exhaust.
Why this matters:
Production observability is more than installing dashboards. Teams need stable signals, clear ownership, useful alerts, controlled evidence access, and a way to connect assistant quality and provider cost to real user outcomes.
-This also strengthens the product direction. Paid value can grow around governed team workflows, but monetization needs evidence that users reach successful analyses, catch regressions, and adopt private evaluation and release controls. Those signals must be useful without compromising the incident data users trust the platform to protect.
+Those signals must remain useful without compromising the incident data operators trust the lab to protect.
Lessons learned:
- Start with operating questions and owners before choosing a backend.
-- Service health, assistant quality, and product value require different signals.
+- Service health and assistant quality require different signals.
- Safety and privacy checks should remain release gates rather than averages.
- Request IDs belong in logs and traces, not high-cardinality metric labels.
- OpenTelemetry can preserve backend choice, but it does not replace retention, access, or cost decisions.
-- Product telemetry should measure user outcomes without collecting unnecessary incident content.
+- Outcome telemetry should measure useful results without collecting unnecessary incident content.
What comes next:
@@ -582,44 +582,32 @@ What comes next:
## Week 4, Day 6 - Advanced Model Serving Roadmap
-Today I defined when Reliability Lab should move from managed model providers to self-hosted inference and how that decision can support a real product.
+Today I defined when Reliability Lab should move from managed model providers to self-hosted inference.
-Day 5 mapped the production observability path. Day 6 uses those quality, latency, usage, and cost signals to decide whether advanced serving infrastructure is justified.
+Day 5 mapped the production observability path. Day 6 uses quality, latency, usage, and cost signals to decide whether advanced serving infrastructure is justified.
What changed:
-- Added adoption gates for privacy boundaries, model control, latency, sustained volume economics, availability control, and multi-model workloads.
+- Added adoption gates for privacy boundaries, model control, latency, sustained volume, availability control, and multi-model workloads.
- Mapped vLLM, NVIDIA Triton Inference Server, Ray Serve, KServe, and Kubernetes GPU scheduling to the specific problems each layer solves.
-- Added a staged maturity path from deterministic analysis and managed providers to single-GPU benchmarks, production inference, shared platforms, and commercial multi-tenant operation.
+- Added a staged path from deterministic analysis and managed providers to single-GPU benchmarks and bounded production experiments.
- Added a provider-versus-self-host benchmark plan using the same assistant evaluation corpus and safety gates.
-- Defined cost per successful evaluated analysis as the unit-economics comparison instead of GPU price or token price alone.
+- Defined cost per successful evaluated analysis as the comparison instead of GPU price or token price alone.
- Documented GPU operating requirements, including drivers, device plugins, placement, quotas, memory behavior, autoscaling signals, cold starts, telemetry, and upgrades.
-- Added security and governance requirements for model licenses, artifacts, endpoints, tenant separation, redaction, versioning, and rollback.
-- Proposed Community, Team, Enterprise cloud, Private deployment, and Platform product tiers.
+- Added security and governance requirements for model licenses, artifacts, endpoints, isolation, redaction, versioning, and rollback.
- Kept the default project path provider-compatible, deterministic, and GPU-free.
-- Linked the roadmap from the README, production guidance, Kubernetes next steps, and future implementation backlog.
Why this matters:
-Self-hosted model serving can improve control, privacy, and economics for the right workload. It can also create idle GPU cost, upgrade risk, new security ownership, and an on-call burden before customer demand exists.
-
-The product should continue buying inference while that accelerates learning and satisfies users. It should build a private serving path when measured requirements, signed demand, or complete unit economics justify ownership.
-
-Monetization possibilities:
-
-- Team workflows can monetize private evaluation sets, collaboration, quotas, release gates, and quality/cost history.
-- Enterprise cloud can add SSO, RBAC, audit exports, policy controls, private connectors, regional handling, support, and reliability commitments.
-- Private deployments can serve customers that require a dedicated VPC or on-premises model endpoint.
-- A platform tier can add fleet policy, model lifecycle, routing, chargeback, capacity governance, and multi-cluster visibility.
-- The durable product boundary remains evidence governance and measurable trust, not resale of an open-source serving engine.
+Self-hosted model serving can improve control and privacy for the right workload. It can also create idle GPU cost, upgrade risk, new security ownership, and an on-call burden before a measured requirement exists.
Lessons learned:
- Advanced infrastructure should follow a demonstrated constraint.
-- The smallest serving stack is usually the easiest one to operate and sell honestly.
+- The smallest serving stack is usually the easiest one to operate.
- Quality and safety must pass before throughput optimization matters.
- Provider and self-hosted paths need the same benchmark workload and acceptance gates.
-- GPU utilization is not unit economics; engineering, reliability, and idle capacity also count.
+- GPU utilization alone does not capture engineering, reliability, and idle-capacity cost.
- Private deployment changes who owns security controls; it does not make the system secure automatically.
What comes next:
@@ -627,7 +615,7 @@ What comes next:
- Capture provider usage and latency metadata.
- Build a provider-versus-self-host benchmark harness.
- Add one optional single-GPU vLLM example only after the benchmark contract is ready.
-- Validate private deployment demand before building a broad serving platform.
+- Document any future serving layer as a bounded operational experiment.
## Week 4, Day 7 - Production Readiness Review
@@ -639,8 +627,8 @@ What changed:
- Added `make validate` as one local gate for image builds, both test suites, lint, and deterministic assistant evaluations.
- Added the assistant evaluation runner to pull-request CI so quality, safety, and privacy regressions block the build.
-- Added a production-readiness review with separate decisions for a local lab, public learning release, sanitized internal pilot, and customer production.
-- Mapped current evidence and remaining boundaries across security, cost, quality, observability, reliability, model serving, and product direction.
+- Added a production-readiness review with separate decisions for a local lab, public learning release, sanitized internal pilot, and internet-facing production.
+- Mapped current evidence and remaining boundaries across security, cost, quality, observability, reliability, and model serving.
- Defined promotion gates for identity, data handling, secrets, supply chain, reliability, observability, quality, cost, and deployment.
- Prioritized a focused next 30 days around provider metadata, deeper evals, stable telemetry contracts, runbook exercises, and benchmark evidence.
- Updated the roadmap, README, contribution checks, and evaluation guide to match the enforced release workflow.
@@ -655,7 +643,7 @@ Lessons learned:
- Documentation becomes stronger when its claims are enforced in CI.
- Privacy and safety checks should block releases rather than disappear into an average score.
-- A single readiness label is misleading; local, pilot, and customer production targets have different gates.
+- A single readiness label is misleading; local, pilot, and internet-facing targets have different gates.
- Advanced infrastructure should follow measured requirements and operational ownership.
- The strongest closeout is a clear decision, visible evidence, and a small next set of priorities.
@@ -683,22 +671,22 @@ What changed:
- Kept missing or malformed usage honest with explicit `null` values and `reported=false`.
- Exposed the same `llm_telemetry` contract from log analysis, questions, and incident summaries.
- Added tests for successful usage, latency, provider failure, unconfigured fallback, both API paths, and telemetry privacy.
-- Added a provider telemetry guide with field semantics, forbidden data, customer value, and the remaining Week 5 sequence.
-- Restored the canonical Weeks 5-8 technical roadmap while preserving the audience and design-partner gates in the commercialization roadmap.
+- Added a provider telemetry guide with field semantics, forbidden data, operational value, and the remaining Week 5 sequence.
+- Restored the canonical Weeks 5-8 technical roadmap.
Why this matters:
A provider call should not be a black box. Teams need to know whether enrichment ran, whether it failed safely, how long it took, and whether the provider reported usage. Those facts are the foundation for comparing quality and cost later.
-The metadata boundary matters just as much as the fields. Prompt text, incident evidence, credentials, endpoint URLs, generated output, and customer identifiers do not belong in provider telemetry.
+The metadata boundary matters just as much as the fields. Prompt text, incident evidence, credentials, endpoint URLs, generated output, and user identifiers do not belong in provider telemetry.
-Customer and monetization value:
+Operational value:
-- Design partners can distinguish successful enrichment from silent deterministic fallback.
+- Operators can distinguish successful enrichment from silent deterministic fallback.
- Teams can later compare provider latency and token usage with evaluated quality.
- Cost can be measured per successful analysis instead of per token in isolation.
- Managed and private OpenAI-compatible endpoints share the same outcome contract.
-- Completing this local contract does not skip the commercialization roadmap's discovery or design-partner gates.
+- The local contract remains useful without requiring additional platform infrastructure.
Lessons learned:
@@ -706,7 +694,7 @@ Lessons learned:
- Provider failure and provider-not-configured are different operational outcomes.
- A monotonic clock is the right boundary for request duration.
- Useful telemetry can remain small, bounded, and free of incident content.
-- Per-request metadata should come before aggregate metrics, billing ledgers, or dashboards.
+- Per-request metadata should come before aggregate metrics or dashboards.
What comes next:
@@ -719,7 +707,7 @@ What comes next:
Today I turned the per-request provider contract into privacy-safe operational aggregates.
-Day 1 made each provider attempt visible to its caller. Day 2 makes trends scrapeable without retaining prompts, incident evidence, generated content, customer identifiers, or arbitrary error text.
+Day 1 made each provider attempt visible to its caller. Day 2 makes trends scrapeable without retaining prompts, incident evidence, generated content, user identifiers, or arbitrary error text.
What changed:
@@ -738,7 +726,7 @@ Why this matters:
Per-request metadata helps debug one analysis. Aggregate counters and distributions reveal whether provider enrichment is reliable, fast, and consistently reporting usage across many analyses. That is the evidence needed before pricing and quality are joined later in the week.
-The privacy and cardinality boundaries are part of the feature. Prompts, evidence, errors, endpoints, request IDs, users, workspaces, and incidents never become labels. Process-local metrics also remain honest about their limits: they reset on restart and are not a billing ledger.
+The privacy and cardinality boundaries are part of the feature. Prompts, evidence, errors, endpoints, request IDs, users, workspaces, and incidents never become labels. Process-local metrics also remain honest about their limits: they reset on restart and are not durable usage history.
Lessons learned:
@@ -746,7 +734,7 @@ Lessons learned:
- Not configured is an outcome, but it is not an attempted provider request and should not enter request-latency buckets.
- Missing token usage is unknown, not zero.
- Configured provider/model labels are useful only with an explicit cardinality budget.
-- Operational metrics and durable customer metering are different systems with different access and retention requirements.
+- Operational metrics and durable usage history are different systems with different access and retention requirements.
What comes next:
@@ -758,7 +746,7 @@ What comes next:
## Week 5, Day 3 - Provider Cost Estimates
-Today I made the provider telemetry contract cost-aware without turning the lab into a billing system.
+Today I made the provider telemetry contract cost-aware without turning the lab into a usage-accounting system.
Day 1 exposed the outcome and usage of one provider attempt. Day 2 made aggregate reliability and token trends scrapeable. Day 3 turns operator-supplied prices and provider-reported token directions into an honest per-call estimate.
@@ -773,7 +761,7 @@ What changed:
Why it matters:
-Token counts alone do not tell a team what one enriched analysis costs. A small, explicit estimate lets operators compare provider paths later without pretending the project owns a durable billing ledger or that static provider prices are universally correct.
+Token counts alone do not tell an operator what one enriched analysis costs. A small, explicit estimate lets operators compare provider paths later without pretending the project owns durable usage history or that static provider prices are universally correct.
Key lessons:
@@ -781,3 +769,17 @@ Key lessons:
- Unknown usage and unknown prices must remain unknown, never silently become zero.
- Currency values should use decimal representations, not floating-point values.
- Cost per successful evaluated analysis belongs in the next step, where quality outcomes can be joined deliberately.
+
+## Week 5, Day 4 - Evaluation Cost Report
+
+Today I joined deterministic assistant evaluation results to bounded optional-provider outcomes and estimated cost.
+
+What changed:
+
+- Added an explicit provider-report mode that makes one enrichment call per corpus fixture.
+- Joined each deterministic rubric result with the existing privacy-safe provider telemetry and cost-estimate contracts.
+- Calculated estimated total cost and estimated cost per successful evaluated analysis only when all successful evaluations have complete price and token data.
+- Kept incomplete cost data unknown instead of treating it as zero.
+- Kept the default deterministic evaluation command offline and cost-free for CI.
+
+Next: add a local comparison report, exercise fallback end to end, and close the Week 5 exit gate.
diff --git a/infra/k8s/configmap.yaml b/infra/k8s/configmap.yaml
index 88b553d..4497154 100644
--- a/infra/k8s/configmap.yaml
+++ b/infra/k8s/configmap.yaml
@@ -10,7 +10,7 @@ data:
# Safe-to-commit application configuration. Do not put API keys here.
DEMO_SERVICE_LOG_PATH: /shared/logs/demo-service.log
DEMO_SERVICE_METRICS_URL: http://demo-service:8000/metrics
- # Keep the default rule-based path enabled so the lab works without paid API access.
+ # Keep the default rule-based path enabled so the lab works without external API access.
LLM_PROVIDER: none
OPENAI_BASE_URL: https://api.openai.com/v1
MODEL_NAME: gpt-4o-mini