Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Contributing

Thanks for helping improve `ai-infra-starter-kit`.
Thanks for helping improve AIOps Lab.

The main rule is simple: keep the project beginner-friendly without making it shallow. A new feature should teach a real AI infrastructure concept at the right time.

Expand Down
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) 2026 ai-infra-starter-kit contributors
Copyright (c) 2026 AIOps Lab contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
13 changes: 9 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Simple first. Production-minded always.

`ai-infra-starter-kit` is a practical learning lab for AI infrastructure. It starts with a normal web service, adds logs and metrics, then uses an AI SRE Assistant to explain what is happening operationally.
AIOps Lab is a practical learning lab for AI infrastructure. It starts with a normal web service, adds logs and metrics, then uses an AI SRE Assistant to explain what is happening operationally.

The first version runs on a normal laptop. No GPU, Kubernetes, vLLM, Triton, Ray, KServe, or full MLOps platform is required on Day 1.

Expand Down Expand Up @@ -41,8 +41,8 @@ The AI assistant does not need an LLM key to work. If no provider is configured,
## Quickstart

```bash
git clone https://github.com/your-username/ai-infra-starter-kit.git
cd ai-infra-starter-kit
git clone https://github.com/utkarshp845/AIOps-Lab.git
cd AIOps-Lab
cp .env.example .env
make up
make test
Expand Down Expand Up @@ -97,6 +97,8 @@ For the advanced model serving decision framework and commercialization path, se

For the Week 4 release gates, readiness verdict, and next-stage priorities, see `docs/20-production-readiness-review.md`.

For the phased commercialization roadmap with milestones and success metrics, see `docs/21-commercialization-roadmap.md`.

Ask the assistant directly:

```bash
Expand Down Expand Up @@ -124,7 +126,7 @@ curl -s -X POST http://localhost:8001/summarize-incident \
## Repository Structure

```text
ai-infra-starter-kit/
AIOps-Lab/
apps/
demo-service/ # FastAPI app that emits health, failure, latency, logs, and metrics
ai-sre-assistant/ # FastAPI app and CLI that analyze demo-service logs
Expand All @@ -141,6 +143,7 @@ ai-infra-starter-kit/
18-production-observability.md
19-advanced-model-serving-roadmap.md
20-production-readiness-review.md
21-commercialization-roadmap.md
infra/ # Docker, Kubernetes, and Terraform starter notes
k8s/ # kind-first Kubernetes manifests and walkthrough
scripts/ # Local traffic and log helper scripts
Expand Down Expand Up @@ -190,6 +193,8 @@ The open-source project makes the core learning path, safety controls, and evalu

The future paid opportunity is the recurring team workflow around those foundations: hosted evaluation history, private incident datasets, release gates, model and provider comparisons, regression alerts, collaboration, and audit-ready exports. The product value is not simply generating an incident summary. It is helping teams prove that their operational assistant stays useful, safe, private, and cost-aware as their systems change.

The sequenced plan — audience first, wedge product with design partners second, paid team tier third — lives in `docs/21-commercialization-roadmap.md` with per-phase milestones, success metrics, and exit criteria.

## Lessons Learned

Day 1 lessons are intentionally simple:
Expand Down
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Security Policy

`ai-infra-starter-kit` is a learning project, but security issues still matter.
AIOps Lab is a learning project, but security issues still matter.

This repo intentionally starts local and simple. Please do not treat the default Docker Compose or kind setup as production-ready.

Expand Down
10 changes: 5 additions & 5 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
name: ai-infra-starter-kit
name: aiops-lab

services:
demo-service:
build:
context: ./apps/demo-service
container_name: ai-infra-demo-service
container_name: aiops-demo-service
ports:
- "8000:8000"
environment:
Expand All @@ -20,7 +20,7 @@ services:
ai-sre-assistant:
build:
context: ./apps/ai-sre-assistant
container_name: ai-infra-ai-sre-assistant
container_name: aiops-ai-sre-assistant
ports:
- "8001:8001"
environment:
Expand All @@ -46,11 +46,11 @@ services:
# Optional local model provider for later experiments.
# 1. Uncomment this service.
# 2. Set LLM_PROVIDER=ollama and OPENAI_BASE_URL=http://ollama:11434/v1 in .env.
# 3. Pull a model with: docker exec -it ai-infra-ollama ollama pull llama3.1
# 3. Pull a model with: docker exec -it aiops-ollama ollama pull llama3.1
#
# ollama:
# image: ollama/ollama:latest
# container_name: ai-infra-ollama
# container_name: aiops-ollama
# ports:
# - "11434:11434"
# volumes:
Expand Down
2 changes: 1 addition & 1 deletion docs/00-overview.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Overview

`ai-infra-starter-kit` teaches AI infrastructure by starting with the parts every production engineer already understands: services, logs, health checks, metrics, containers, and operational debugging.
AIOps Lab teaches AI infrastructure by starting with the parts every production engineer already understands: services, logs, health checks, metrics, containers, and operational debugging.

The AI part is introduced as an assistant that observes a service, not as a giant platform on Day 1.

Expand Down
4 changes: 4 additions & 0 deletions docs/09-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,10 @@
- Optional advanced serving roadmap: vLLM, Triton, Ray, KServe, and GPU scheduling.
- Production-readiness review with local and CI release gates.

## Commercialization

The phased plan for turning the lab into a monetized product — audience, wedge product, paid team tier — is in `21-commercialization-roadmap.md`.

## Later

- OpenTelemetry collector.
Expand Down
80 changes: 40 additions & 40 deletions docs/10-kubernetes-operations-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,23 +17,23 @@ This runbook assumes you already created the kind cluster, loaded local images,
All Kubernetes resources for this project run in one namespace:

```bash
kubectl get all -n ai-infra-starter-kit
kubectl get all -n aiops-lab
```

For a shorter local session, you can set the namespace on your current context:

```bash
kubectl config set-context --current --namespace=ai-infra-starter-kit
kubectl config set-context --current --namespace=aiops-lab
```

The rest of this guide keeps `-n ai-infra-starter-kit` in commands so each example is explicit and copy-paste friendly.
The rest of this guide keeps `-n aiops-lab` in commands so each example is explicit and copy-paste friendly.

## Quick Status Check

Start with the cluster objects that explain most local issues:

```bash
kubectl get deployments,pods,svc,pvc -n ai-infra-starter-kit
kubectl get deployments,pods,svc,pvc -n aiops-lab
```

What you want to see:
Expand All @@ -47,20 +47,20 @@ What you want to see:
Then confirm rollout status:

```bash
kubectl rollout status deployment/demo-service -n ai-infra-starter-kit
kubectl rollout status deployment/ai-sre-assistant -n ai-infra-starter-kit
kubectl rollout status deployment/demo-service -n aiops-lab
kubectl rollout status deployment/ai-sre-assistant -n aiops-lab
```

## Health, Readiness, And Metrics

Port-forward both services in separate terminals:

```bash
kubectl port-forward svc/demo-service 8000:8000 -n ai-infra-starter-kit
kubectl port-forward svc/demo-service 8000:8000 -n aiops-lab
```

```bash
kubectl port-forward svc/ai-sre-assistant 8001:8001 -n ai-infra-starter-kit
kubectl port-forward svc/ai-sre-assistant 8001:8001 -n aiops-lab
```

Check the app-level signals:
Expand Down Expand Up @@ -117,14 +117,14 @@ For an end-to-end incident walkthrough that uses these signals together, see `13
View recent logs from each Deployment:

```bash
kubectl logs deployment/demo-service -n ai-infra-starter-kit --tail=50
kubectl logs deployment/ai-sre-assistant -n ai-infra-starter-kit --tail=50
kubectl logs deployment/demo-service -n aiops-lab --tail=50
kubectl logs deployment/ai-sre-assistant -n aiops-lab --tail=50
```

Follow logs while generating traffic:

```bash
kubectl logs deployment/demo-service -n ai-infra-starter-kit --follow
kubectl logs deployment/demo-service -n aiops-lab --follow
```

For this learning lab, the assistant also reads the shared file at:
Expand All @@ -142,14 +142,14 @@ Services give pods stable names inside the cluster.
Check service objects:

```bash
kubectl get svc -n ai-infra-starter-kit
kubectl get svc -n aiops-lab
```

Check service endpoints:

```bash
kubectl get endpoints demo-service -n ai-infra-starter-kit
kubectl get endpoints ai-sre-assistant -n ai-infra-starter-kit
kubectl get endpoints demo-service -n aiops-lab
kubectl get endpoints ai-sre-assistant -n aiops-lab
```

If a Service has no endpoints, Kubernetes has not found any ready pods behind it. Check pod labels, readiness probes, and pod status.
Expand All @@ -159,13 +159,13 @@ If a Service has no endpoints, Kubernetes has not found any ready pods behind it
Normal configuration lives in the ConfigMap. For the full config and secrets walkthrough, see `11-kubernetes-config-and-secrets.md`:

```bash
kubectl describe configmap ai-infra-starter-kit-config -n ai-infra-starter-kit
kubectl describe configmap aiops-lab-config -n aiops-lab
```

Sensitive configuration belongs in a Secret:

```bash
kubectl get secret ai-sre-assistant-secrets -n ai-infra-starter-kit
kubectl get secret ai-sre-assistant-secrets -n aiops-lab
```

Do not paste real API keys into screenshots, tweets, issues, or pull requests.
Expand All @@ -177,13 +177,13 @@ The local kind setup uses a shared PersistentVolumeClaim so the assistant can re
Check the claim:

```bash
kubectl get pvc -n ai-infra-starter-kit
kubectl get pvc -n aiops-lab
```

Describe it if it is not `Bound`:

```bash
kubectl describe pvc shared-logs -n ai-infra-starter-kit
kubectl describe pvc shared-logs -n aiops-lab
```

This is a learning bridge from Docker Compose to Kubernetes. It is not the recommended production logging architecture.
Expand All @@ -197,18 +197,18 @@ Likely cause: kind cannot see the local Docker image.
Check:

```bash
kubectl describe pod -l app.kubernetes.io/name=demo-service -n ai-infra-starter-kit
kubectl describe pod -l app.kubernetes.io/name=demo-service -n aiops-lab
```

Fix:

```bash
docker build -t ai-infra-starter-kit/demo-service:local apps/demo-service
docker build -t ai-infra-starter-kit/ai-sre-assistant:local apps/ai-sre-assistant
kind load docker-image ai-infra-starter-kit/demo-service:local --name ai-infra-starter-kit
kind load docker-image ai-infra-starter-kit/ai-sre-assistant:local --name ai-infra-starter-kit
kubectl rollout restart deployment/demo-service -n ai-infra-starter-kit
kubectl rollout restart deployment/ai-sre-assistant -n ai-infra-starter-kit
docker build -t aiops-lab/demo-service:local apps/demo-service
docker build -t aiops-lab/ai-sre-assistant:local apps/ai-sre-assistant
kind load docker-image aiops-lab/demo-service:local --name aiops-lab
kind load docker-image aiops-lab/ai-sre-assistant:local --name aiops-lab
kubectl rollout restart deployment/demo-service -n aiops-lab
kubectl rollout restart deployment/ai-sre-assistant -n aiops-lab
```

### Pods Are Running But Not Ready
Expand All @@ -218,8 +218,8 @@ Likely cause: readiness probes are failing or the app is not listening on the ex
Check:

```bash
kubectl describe pod -l app.kubernetes.io/name=demo-service -n ai-infra-starter-kit
kubectl logs deployment/demo-service -n ai-infra-starter-kit --tail=100
kubectl describe pod -l app.kubernetes.io/name=demo-service -n aiops-lab
kubectl logs deployment/demo-service -n aiops-lab --tail=100
```

Look for failed readiness probe events, startup errors, or port mismatches.
Expand All @@ -236,10 +236,10 @@ Likely causes:
Check:

```bash
kubectl get pvc -n ai-infra-starter-kit
kubectl describe configmap ai-infra-starter-kit-config -n ai-infra-starter-kit
kubectl logs deployment/demo-service -n ai-infra-starter-kit --tail=50
kubectl logs deployment/ai-sre-assistant -n ai-infra-starter-kit --tail=50
kubectl get pvc -n aiops-lab
kubectl describe configmap aiops-lab-config -n aiops-lab
kubectl logs deployment/demo-service -n aiops-lab --tail=50
kubectl logs deployment/ai-sre-assistant -n aiops-lab --tail=50
```

Then generate traffic again:
Expand All @@ -259,8 +259,8 @@ Likely causes:
Check:

```bash
kubectl get endpoints demo-service -n ai-infra-starter-kit
kubectl describe configmap ai-infra-starter-kit-config -n ai-infra-starter-kit
kubectl get endpoints demo-service -n aiops-lab
kubectl describe configmap aiops-lab-config -n aiops-lab
curl http://localhost:8000/metrics
```

Expand All @@ -278,7 +278,7 @@ Check:

```bash
kubectl get pv
kubectl describe pvc shared-logs -n ai-infra-starter-kit
kubectl describe pvc shared-logs -n aiops-lab
```

For this repo, `infra/k8s/storage.yaml` includes a kind-friendly local PersistentVolume. Reapply it if needed:
Expand All @@ -298,14 +298,14 @@ Likely causes:
Check:

```bash
kubectl get svc -n ai-infra-starter-kit
kubectl get svc -n aiops-lab
```

Use different local ports if needed:

```bash
kubectl port-forward svc/demo-service 18000:8000 -n ai-infra-starter-kit
kubectl port-forward svc/ai-sre-assistant 18001:8001 -n ai-infra-starter-kit
kubectl port-forward svc/demo-service 18000:8000 -n aiops-lab
kubectl port-forward svc/ai-sre-assistant 18001:8001 -n aiops-lab
```

Then call:
Expand All @@ -319,7 +319,7 @@ curl http://localhost:18001/health

Use this order when something is wrong:

1. `kubectl get deployments,pods,svc,pvc -n ai-infra-starter-kit`
1. `kubectl get deployments,pods,svc,pvc -n aiops-lab`
2. `kubectl describe pod ...`
3. `kubectl logs deployment/...`
4. app health endpoints through port-forward
Expand All @@ -333,8 +333,8 @@ Start with read-only inspection. Restart or delete resources only after the evid

Useful screenshots for a build-in-public update:

- `kubectl get deployments,pods,svc,pvc -n ai-infra-starter-kit`
- `kubectl rollout status deployment/demo-service -n ai-infra-starter-kit`
- `kubectl get deployments,pods,svc,pvc -n aiops-lab`
- `kubectl rollout status deployment/demo-service -n aiops-lab`
- `curl http://localhost:8000/health` and `curl http://localhost:8000/metrics`
- `POST /analyze/metrics` response from the assistant
- `kubectl logs deployment/demo-service --tail=20`
Expand Down
Loading
Loading