Harden outbox, worker leases, and dashboard; publish canonical benchmark evidence

[samarth1412]

Summary

• Harden transactional outbox publication: claim work in short PostgreSQL transactions, publish to Kafka outside row locks, allow observable duplicate delivery over silent loss on uncertain acknowledgement, and preserve per-workflow ordering across concurrent publishers and Redpanda partitions.
• Strengthen worker lease recovery and fencing: active-lease validation, tenant-scoped artifact operations, stale-worker mutation rejection, cancellation fencing, and multi-task crash recovery (expanded from 1 to ~12 simultaneous in-flight tasks in integration tests, and to 10,000 tasks in the benchmark suite).
• Fix heterogeneous Go/Python worker pool routing so capability/handler routing is explicit — intentionally failing validation handlers stay routable while truly unknown handlers are not incorrectly leased.
• Improve dashboard reliability: loading/unauthorized states, dependency and attempt detail views, search, run-detail drawer behavior, and a refresh race that could reopen a closed drawer. Real Playwright coverage (Keycloak auth, live updates, cancellation/dead-letter replay, artifacts, security, tenant isolation) passes against the actual Compose stack.
• Add real dashboard screenshots and a recovery demo video captured from the running system (workers are actually killed mid-lease).
• Add/strengthen CI gates: Go coverage thresholds (overall + workflow/auth/live packages), Python coverage, web branch/line/function coverage, container builds + Trivy scans for all five images, actionlint, Buf validation, Terraform fmt/validate/mock tests, Helm lint, kubeconform, Playwright, Docker-backed integration tests, and a kind rolling-upgrade scenario.
• Fix a dashboard runtime image vulnerability found by Trivy; the image now passes the zero-unfixed-critical gate.
• Fix fresh-stack Kafka/Redpanda topic initialization: rpk topic describe could report success before the topic existed on a brand-new stack, breaking a completely empty-volume docker compose up.
• Repo hygiene: canonical Go module path (github.com/samarth1412/RunMesh), regenerated protobuf metadata, tightened web dependency pinning, MIT LICENSE, SECURITY.md, CONTRIBUTING.md, stronger secret/coverage ignores, and independent-consumer-module validation of the Go SDK.
• Publish the canonical final benchmark run (2026-07-21T003600Z), recorded fresh against the corrected HEAD after the Kafka init fix, with docs updated to report its measured values rather than an earlier, superseded run.

Reliability guarantees tested

Broker outages, duplicate Kafka delivery, worker termination and lease recovery (10,000/10,000 tasks succeeded after killing all 20 workers, 0 permanent loss), stale-worker fencing, cancellation fencing, PostgreSQL restart, Redis outage, artifact/MinIO behavior, tenant isolation, fresh-stack startup from empty volumes, and a rolling Kubernetes (kind) upgrade with an in-flight multi-step workflow.

Benchmark (canonical: tests/benchmarks/results/2026-07-21T003600Z)

Source commit c4b248638f2ae36c4a7bbaf48df37ee59ac1bf74, Apple M4 / 16 GiB, Docker Desktop 4.82.0. All 15 blocking verifier checks passed.

Scenario	Result
Authenticated API reads	100.0 req/s; 1.701/3.307/4.657 ms p50/p95/p99; 0 failures
Workflow submissions	50.028 req/s; 1.587/7.012/220.024 ms p50/p95/p99; 0 failures
Simultaneously active workflows	1,000 created in 18.224s; 0 failures
Queued end-to-end completion	1,000 tasks at 57.608 tasks/s
Scheduler dispatch	407.403 tasks/s across 10,000 tasks
Worker termination and lease recovery	20 workers terminated; 10,000/10,000 succeeded; 7 recovered; 0 permanent loss
Duplicate submission / delivery	Passed

This is local Docker evidence, not a production capacity claim. RunMesh promises at-least-once, not exactly-once, execution, and no AWS deployment has been provisioned; Terraform is validated and mock-tested only.

Validation

No source code changed beyond the prior 647b9c9-c4b2486 hardening commits (already validated); this PR's diff is documentation and benchmark evidence only. Re-verified locally via Docker in this session:

• gofmt -l clean; go vet ./... clean
• go test -race -covermode=atomic ./cmd/... ./internal/... ./sdk/go/... passed; 11.8% overall (>=10% gate)
• Focused package coverage: workflow 73.6% (>=70%), auth 40.6% (>=40%), live 73.5% (>=70%)
• Python ruff, mypy, pytest passed; 40.56% coverage (>=40% gate)
• Web eslint, tsc -b && vite build, vitest --coverage passed; 67.94%/52.14%/58.24%/81.37% stmts/branch/funcs/lines
• docker compose config --quiet valid
• actionlint clean; buf lint clean
• helm lint clean; helm template | kubeconform -strict 24/24 resources valid
• terraform fmt -check -recursive, init -backend=false, validate, test passed (mock provider, no apply)
• gitleaks git --staged no leaks
• Markdown relative-link check across the repo: no broken links
• git diff --check clean

Multi-hour Docker-backed integration/chaos/kind suites were not re-run in this session since no runtime source changed in this commit; GitHub Actions CI will run the full suite on this branch.