ModelHub — Production-Grade ML Model Serving Platform

A model serving platform that treats the model as the easy part and the serving discipline as the product: versioned model registry, async inference with request micro-batching, canary traffic routing, full prediction logging, and scheduled statistical drift detection (PSI) — with the testing, CI/CD, IaC, and observability any production service gets.

The demo domain is recipe recommendation scoring (given a user profile and a candidate recipe, predict a relevance score) on fully synthetic data. The model is deliberately a small scikit-learn regressor; everything interesting is the infrastructure around it.

No AWS account, cloud credentials, or paid service is ever required to build, run, or test this project. See AWS parity below.

60-second quickstart

Requires Docker only.

docker compose -f docker/docker-compose.yml up -d --build   # api + postgres + redis + prometheus + grafana
./demo.sh                                                   # scripted end-to-end demo

demo.sh runs the whole lifecycle against the actually running services and prints what happened — nothing is asserted blind:

1. trains a real model on synthetic data and registers it (staging)
2. promotes it staging → canary → production
3. trains a candidate model and promotes it to canary (10% traffic)
4. fires 300 concurrent predictions — shows the canary split (e.g. {'production': 277, 'canary': 23}) and the micro-batch sizes actually used (e.g. max=32, mean=20.9)
5. fires 300 deliberately distribution-shifted requests
6. runs the drift check and prints per-feature PSI — the three shifted user features come back DRIFTED (PSI ≈ 0.85–1.06), the untouched recipe features come back ok (PSI ≈ 0.02)
7. prints /health and where to look next

Then explore:

URL	What
http://localhost:8000/docs	OpenAPI docs (interactive)
http://localhost:8000/metrics	Prometheus metrics incl. modelhub_batch_size, modelhub_drift_psi
http://localhost:9090	Prometheus
http://localhost:3000	Grafana (anonymous admin, Prometheus pre-provisioned)

Stop everything: docker compose -f docker/docker-compose.yml down

What's inside

POST /predict                      inference (micro-batched, canary-routed)   public
GET  /models                       registry listing                           public
POST /admin/models/{id}/promote    staging→canary, canary→production          X-API-Key
POST /admin/models/{id}/rollback   restore previous production version        X-API-Key
GET  /drift                        latest per-feature PSI per active model    public
GET  /health                       DB + Redis + production-model check        public
GET  /metrics                      Prometheus                                 public

Every prediction response carries model_version and variant, and every prediction is logged to PostgreSQL (features, output, latency, batch size, variant) — that log is both the audit trail and the drift detector's data source. Errors always come back as {"detail": ..., "error_code": ...}.

Micro-batching benchmark

Requests are queued on an in-process asyncio.Queue and flushed either at max_batch_size (default 32) or after max_wait_ms (default 10 ms), whichever comes first — the same dynamic batching strategy production inference servers (e.g. NVIDIA Triton) use, implemented in plain asyncio to show the mechanism.

Measured with python bench/benchmark_batching.py (2,000 requests, concurrency 200, HistGradientBoostingRegressor, CPU):

Mode	Throughput
Unbatched (one predict() per request)	~760 req/s
Micro-batched (≤32 per predict())	~6,760 req/s
Speedup	~8.9x

Numbers vary by machine; re-run the script to reproduce on yours.

Drift detection

The drift job (APScheduler, default every 15 min) compares each feature's live distribution (recent prediction log) against the reference distribution stored on the model at training time, using the Population Stability Index: PSI < 0.1 no change, 0.1–0.25 moderate shift,

0.25 drift (is_drifted = true + structured warning + Sentry event + Prometheus gauge). Methodology and interpretation: docs/DRIFT.md. The PSI implementation is tested with Hypothesis property-based tests, and an end-to-end test injects a shifted distribution and asserts the monitor actually flags it.

Training a model manually

docker compose -f docker/docker-compose.yml exec api python training/train.py            # register as staging
docker compose -f docker/docker-compose.yml exec api python training/train.py --shift 0.1  # a different candidate

Then promote via the API (X-API-Key: local-dev-admin-key by default in compose):

curl -X POST -H "X-API-Key: local-dev-admin-key" localhost:8000/admin/models/<id>/promote

train.py generates synthetic data, trains, evaluates (RMSE/MAE/R²/Spearman), computes the per-feature reference distribution for drift detection, writes the artifact to the artifact volume, and registers the version — status staging.

Development & tests

python -m venv .venv && . .venv/bin/activate   # Python 3.12+
pip install -e ".[dev]"
pre-commit install

pytest tests/unit -q                 # fast: batcher, PSI (Hypothesis), router, schemas
pytest -q --cov=modelhub             # full suite; integration tests need Docker (testcontainers)
ruff check . && black --check . && mypy

Coverage is gated at 80% (fail_under) and enforced in CI. Integration tests run against real PostgreSQL 16 + Redis 7 containers and apply the actual Alembic migrations (the app never uses create_all()).

CI/CD

• ci.yml (every PR): ruff, black, mypy --strict, pytest with the coverage gate, Docker image build, pip-audit.
• deploy.yml (merge to main): all of the above, push the image to ghcr.io tagged with the git SHA, terraform fmt -check + terraform validate, then stand the full stack up on a throwaway local kind cluster inside the runner (k8s/deploy.sh) and smoke test /health and a real /predict. No cloud account is ever touched.

AWS parity (talking point, not a dependency)

docker compose up is always the primary way to run this project. The Terraform in infra/terraform/ (validated in CI, never applied) documents the target production design — each local service maps 1:1 to its AWS equivalent:

Local (compose / kind)	AWS target (Terraform)
api container	ECS Fargate service behind an ALB (HTTPS via ACM)
postgres container	RDS PostgreSQL 16 (multi-AZ, encrypted)
redis container	ElastiCache Redis 7
artifact volume	S3 (versioned, KMS-encrypted)
image built locally / ghcr.io	ECR
.env / compose environment	Secrets Manager
Prometheus alerts (local)	CloudWatch alarms (5xx, p99 latency, CPU, sustained drift)

Actually deploying to AWS (terraform plan/apply, pushing to ECR) is a documented next step for anyone who wants it — it is intentionally not part of this repository's setup, CI/CD, or definition of done, and nothing here ever asks for AWS credentials.

Documentation

• docs/ARCHITECTURE.md — components, request flow, design decisions
• docs/API.md — endpoint reference
• docs/DRIFT.md — PSI methodology and thresholds
• CONTRIBUTING.md — conventions, pre-commit, how to contribute

License

Released under the MIT License.