self-healing-ops-agent

Autonomous self-healing ops loop for a two-service deployment (a Vercel web app + a Railway backend): collect failure signals, triage with a headless LLM, dispatch fixer sub-agents that ship tested PRs, and auto-merge behind an audited CI gate.

Runs unattended on launchd: a shallow tick every 15 minutes, a deep tick hourly, a changelog digest daily. Healthy ticks cost zero LLM tokens.

How it works

launchd ──▶ bin/monitor.sh
              ├─ collectors/collect-all.sh ──▶ signals/latest.json
              │    (health probes, Vercel + Railway log tails, active JSON-RPC
              │     probe, Sentry issues, failed GH Actions runs, open human PRs)
              │
              ├─ cheap jq gate: snapshot healthy? ──▶ exit 0   (no LLM call)
              │
              └─ claude -p prompts/analyze.md          "the maintainer brain"
                    ├─ triage: real bug vs noise, signature + dedup ledger
                    ├─ fixer sub-agent (isolated git worktree, prompts/fix.md)
                    │     repro ▶ minimal fix ▶ regression test ▶ E2E tests ▶ PR
                    ├─ auditor sub-agent (prompts/audit.md)
                    │     pure diff review ▶ approve / revise / reject
                    └─ merge: bin/ci-gate.sh SUCCESS + audit approve
                              ──▶ gh pr merge --squash        (active mode only)

The gate before the LLM is the economic core: collectors are plain bash + jq, and a tick with no error signals exits before claude is ever invoked. The brain only wakes for unhealthy snapshots, then spends its budget on diagnosis and a scoped fix.

Every fixer runs in an isolated git worktree, opens a fix/auto-* branch, and must pass the repo's own test commands end-to-end before a PR exists. An independent auditor sub-agent reviews the diff (root cause addressed, no weakened tests, no scope creep, no sensitive paths) before any merge.

Quickstart

git clone https://github.com/BaseLayerAI/self-healing-ops-agent.git
cd self-healing-ops-agent
cp .env.example .env              # Sentry + Resend credentials
cp config.example.json config.json # your repos, health URLs, detection regexes
bin/monitor.sh --deep             # run one tick by hand; read logs/run-*.log

Prerequisites: jq, gh (authenticated), the claude CLI, plus vercel / railway CLIs linked in the target repo directories (for log collection). macOS or Linux; gtimeout (coreutils) recommended on macOS.

config.json describes your private topology and stays out of git. The two repo slots are protocol (Vercel web app) and api (Railway backend); collectors are wired to those key names.

Modes

config.json .mode:

• monitor-only (default, safe) — fixers open draft PRs, never merge. Run here first to validate triage quality.
• active — fixers open real PRs and self-merge, but only when local E2E tests passed, the auditor returned approve, and bin/ci-gate.sh reports SUCCESS over blocking CI checks.

Environment variables

Variable	Required	Purpose
SENTRY_AUTH_TOKEN	for Sentry collection	Read-only API token (project:read, event:read, org:read)
SENTRY_ORG	for Sentry collection	Sentry org slug
SENTRY_PROJECT_PROTOCOL	for Sentry collection	Project slug for the web app
SENTRY_PROJECT_API	for Sentry collection	Project slug for the backend
RESEND_API_KEY	for email notify	Resend API key
NOTIFY_EMAIL_TO	for email notify	Alert/digest recipient
NOTIFY_EMAIL_FROM	for email notify	Sender; domain must be verified in Resend
TELEGRAM_CHAT_ID	optional	Telegram alert target (with notify.telegram in config)

Detection knobs live in config.json .detection: rpcErrRe (soft RPC-degradation log markers) and ciNonblockingRe (CI checks that never gate a merge). Tune both to your log formats.

Deployment (launchd)

Three jobs, three plists in ~/Library/LaunchAgents/ (labels are examples):

Label	Schedule	Command
com.baselayer.ops-agent.monitor	every 15 min	bin/monitor.sh
com.baselayer.ops-agent.deep	hourly	bin/monitor.sh --deep (wider Sentry sweep)
com.baselayer.ops-agent.changelog	daily 18:00	bin/changelog.sh (merged-PR digest email)

Minimal plist shape:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
  <key>Label</key><string>com.baselayer.ops-agent.monitor</string>
  <key>ProgramArguments</key>
  <array><string>/bin/bash</string><string>/path/to/self-healing-ops-agent/bin/monitor.sh</string></array>
  <key>StartCalendarInterval</key>
  <array>
    <dict><key>Minute</key><integer>0</integer></dict>
    <dict><key>Minute</key><integer>15</integer></dict>
    <dict><key>Minute</key><integer>30</integer></dict>
    <dict><key>Minute</key><integer>45</integer></dict>
  </array>
</dict></plist>

Load with launchctl load ~/Library/LaunchAgents/com.baselayer.ops-agent.monitor.plist. Plain cron works too (*/15 * * * * /path/to/bin/monitor.sh) — the script sets its own PATH and takes an atomic mkdir lock, so overlapping ticks never collide.

Runtime state lives untracked in the working copy: state/seen.json (dedup ledger), state/incidents.jsonl (append-only audit log), signals/ (snapshots), logs/ (per-tick run logs).

Guardrails

• Never pushes to main. All changes land via gh pr merge --squash after the blocking-CI gate prints SUCCESS. Never --admin, never force-merge.
• Sensitive paths route to a human. Anything touching sensitivePaths / severity.needsHumanPaths (payments, auth, contracts, ...) becomes a draft PR labeled needs-human, regardless of mode.
• Audited merges. An independent auditor sub-agent must return approve on the diff; one revision round max, then reject.
• Dedup + attempt caps. Stable error signatures in state/seen.json prevent duplicate PRs for a recurring error; two failed fix attempts escalate to needs-human.
• Never weakens tests. No skipped tests, removed assertions, or loosened expectations to go green — enforced in both the fixer and auditor prompts.
• One email per (signature, status). Notification dedup is persisted, so a flapping error can't spam.

Operational details: docs/RUNBOOK.md.