Skip to content

feat(dependabot): AI-gated auto-merge for major github-actions bumps (pilot)#5

Merged
bbkrr merged 1 commit into
mainfrom
feature/2229-ai-gated-major-automerge
Jul 10, 2026
Merged

feat(dependabot): AI-gated auto-merge for major github-actions bumps (pilot)#5
bbkrr merged 1 commit into
mainfrom
feature/2229-ai-gated-major-automerge

Conversation

@bbkrr

@bbkrr bbkrr commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Implements the AI risk gate designed in realrate/RealRate-Private#2229.

What

For repos on an allowlist (pilot: realrate/RealRate-Private), a major github-actions Dependabot bump is classified by OpenAI gpt-5-mini and auto-merged only if judged low-risk / non-breaking. Everything else is unchanged:

  • patch/minor → still auto-approve + auto-merge (as today)
  • all other majors (pip/docker) and non-allowlisted repos → comment + needs-manual-review (as today)

Ships in dry-run (AIGATE_DRY_RUN: "true"): the AI posts its verdict as a comment but does not merge, so we can observe verdicts before enforcing.

How / safety

  • AI decides eligibility; CI decides readiness. The verdict step is read-only and never merges; a separate deterministic step approves + gh pr merge --auto. GitHub's existing required checks (verify-output, pre-commit, semgrep, pip-audit, Analyze) still gate the actual merge, so a bump that breaks CI never lands regardless of the verdict.
  • No untrusted code executed. The workflow does not check out the PR head; it only classifies the Dependabot changelog text. The script's system prompt treats that text as untrusted data.
  • Fail-closed to human. On any failure (missing OPENAI_API_KEY, API error, unparseable response) the script emits verdict=human; the step is continue-on-error, so a failed verdict routes to manual review and never blocks the required check or auto-merges.
  • No double-comment. The AI path is mutually exclusive with the generic major-bump comment/label, and runs once on opened.

Files

  • .github/workflows/dependabot-automerge.yaml — the gate (allowlist + dry-run env, verdict step, auto-merge step, route-to-human step).
  • .github/review-tooling/review_dependabot_major.py — OpenAI structured-output verdict ({verdict, risk, reason}); always writes valid JSON + exits 0.
  • .github/review-tooling/requirements.txt — pins openai==2.45.0.

realrate/.github is public, so the injected workflow in a private consumer repo can check this tooling out with the built-in token.

Prerequisite before enforcing (org admin)

Add OPENAI_API_KEY as an organization-level Actions secret (the same OpenAI account already used by RealRate-AIChatbot's gpt-5-mini). Until it exists, the gate simply routes every eligible bump to manual review (fail-closed) — safe to merge this PR first.

Rollout

  1. Dry-run (this PR's default) — observe verdicts on real major github-actions bumps.
  2. Enforce — flip AIGATE_DRY_RUN to "false".
  3. Expand — add repos to AIGATE_ALLOWLIST; later consider pip/docker with stricter prompts + a denylist.

Validated with actionlint (clean) and an offline test of the script's fail-closed path (no key → human).

🤖 Generated with Claude Code

Adds an opt-in AI risk gate to the shared Dependabot auto-merge workflow. For
repos on AIGATE_ALLOWLIST (pilot: RealRate-Private), a MAJOR `github-actions`
bump is classified by OpenAI (gpt-5-mini) and auto-merged only if judged
low-risk / non-breaking. Everything else is unchanged: patch/minor still
auto-merge; all other majors (and non-allowlisted repos) keep comment +
needs-manual-review.

Design:
- The verdict step is read-only and never merges; a separate deterministic step
  approves + enables auto-merge. GitHub's required status checks still gate the
  real merge, so a bump that breaks CI never lands regardless of the verdict.
- No checkout of PR head code; the untrusted changelog is only classified.
- On any failure (missing OPENAI_API_KEY, API error, unparseable response) the
  script emits verdict=human, so we never auto-merge on an errored verdict.
- Ships in DRY_RUN mode: posts the verdict as a comment but does NOT merge, so
  verdicts can be observed before enforcing.

Requires an org-level OPENAI_API_KEY Actions secret. Ref RealRate-Private#2229.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@bbkrr

bbkrr commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator Author

@HolgerBartel — this PR needs one admin action from you to go live: please add an OPENAI_API_KEY as an organization-level GitHub Actions secret (Org → Settings → Secrets and variables → Actions → New organization secret).

  • Why: the injected shared Dependabot workflow uses it to have OpenAI gpt-5-mini risk-classify major github-actions bumps and auto-merge only the low-risk ones (see RealRate-Private#2229).
  • Which key: the same OpenAI account already used by RealRate-AIChatbot (gpt-5-mini) is ideal — no new vendor/billing.
  • Scope: make it visible to the repos that will use the gate (at minimum RealRate-Private; simplest is "all repos" or selected private repos). The consumer repos are private, so the secret must be granted to them.
  • Safe ordering: the secret is not required to merge this PR — the gate is fail-closed, so until the secret exists (and while it ships in dry-run) every eligible bump just routes to manual review. Merge first, add the secret when convenient, then we observe dry-run verdicts before enforcing.

Cost is negligible (~$0.05/month at expected volume). Thanks!

@bbkrr

bbkrr commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator Author

⚙️ Operational note — merging this PR does NOT turn the feature on. It ships inert and fail-closed. Two independent switches must BOTH be set before any major bump auto-merges:

  1. OPENAI_API_KEY org-level Actions secret must exist (granted to the private consumer repos). Until then, the script returns verdict=human and every eligible bump just routes to manual review — no API call, no merge.
  2. AIGATE_DRY_RUN must be flipped to "false" in this workflow. While it stays "true" (the shipped default), the AI posts its verdict as a comment but still does not merge.

Until both are done: patch/minor auto-merge is unchanged, all majors (incl. github-actions in the allowlisted repo) continue to get comment + needs-manual-review, and the required auto-merge check stays green org-wide. So this is safe to merge now; it only becomes active after (1) + (2).

@bbkrr bbkrr merged commit a1d74e5 into main Jul 10, 2026
1 check passed
@bbkrr bbkrr deleted the feature/2229-ai-gated-major-automerge branch July 10, 2026 07:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant