fix(opencode): auto-retry the required review after model-pool exhaustion#520
Conversation
…tion
A pool-exhausted required review fails closed with MODEL_OUTPUT_UNAVAILABLE
and stayed red until a human rerun or an org-sweep heartbeat that GitHub
throttles for hours at a time (observed on naruon#1072, runs 29225414867 and
29226757733). Two gaps kept quota exhaustion blocking the org queue:
- No in-workflow retry existed: recovery depended entirely on the merge
scheduler org-sweep cron, whose */15 heartbeat is routinely delayed by
GitHub schedule throttling (observed gaps up to three hours).
- Even after a successful same-head workflow_dispatch retry published an
approval and the opencode-review success commit status, the scheduler
still counted the stale failed pull_request_target check run as a merge
blocker ("failed check(s): opencode-review"), so only a manual rerun of
the failed required run could unblock the PR.
Fix, mirroring the existing Strix supersession contract:
- opencode-review.yml grows an opencode-exhausted-retry job that runs only
when the pull_request_target review job failed with
model_pool_outcome=exhausted (the state whose publish gate has already
verified clean current-head coverage, peer checks, code-scanning alerts,
and review threads). It waits one 300s backoff window, confirms the PR is
still open on the same head, and re-dispatches the central same-head
review through the same trusted workflow_dispatch path the merge
scheduler uses. It retries once; longer outages stay owned by the
org-sweep heartbeat.
- pr_review_merge_scheduler.py excuses a failed OpenCode check run when a
successful same-head opencode-review commit status supersedes it, exactly
like the existing strix rule, so a dispatch-run approval merges without a
manual rerun of the stale failed required run.
Verification: coverage run -m pytest tests (377 passed) with 100% coverage
and interrogate 100%; the negative assertions of
scripts/ci/test_strix_quick_gate.sh were audited against the added job and
the full suite runs in CI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
OpenCode Review Overview
Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: opencode-review.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: opencode-review.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["CI script: pr_review_merge_scheduler.py"]
S2 --> I2["review and security gate shell path"]
I2 --> R2["Review risk: CI script: pr_review_merge_scheduler.py"]
R2 --> V2["bash -n plus Strix self-test"]
Evidence --> S3["Test: test_pr_review_merge_scheduler.py"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test: test_pr_review_merge_scheduler.py"]
R3 --> V3["targeted test run"]
|
There was a problem hiding this comment.
Pull request overview
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including .github/workflows/opencode-review.yml, scripts/ci/pr_review_merge_scheduler.py, tests/test_pr_review_merge_scheduler.py.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects .github/workflows/opencode-review.yml to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
Findings
No blocking findings.
Adversarial validation
{"status":"passed","probes":[{"path":".github/workflows/opencode-review.yml","line":1519,"hypothesis":"The deferred retry mechanism may fail to dispatch the review if the PR is closed or the head changes during the backoff window.","attack_or_counterexample":"Simulated PR closure or head change during the backoff window.","evidence":"The code checks the PR state and head SHA before dispatching the retry, ensuring it only proceeds if the PR is still open and the head matches.","outcome":"falsified"},{"path":"scripts/ci/pr_review_merge_scheduler.py","line":1246,"hypothesis":"The scheduler may incorrectly mark OpenCode review failures as failed checks even when a successful status context exists.","attack_or_counterexample":"Simulated OpenCode review failure with a successful status context.","evidence":"The code correctly skips OpenCode review failures if a successful status context exists, ensuring accurate check status reporting.","outcome":"falsified"}],"residual_risk":"Low; the changes are well-tested and handle expected edge cases."}Evidence
- Result: APPROVE
- Reason: The PR addresses model-pool exhaustion by adding a deferred retry mechanism for OpenCode reviews, ensuring PR merges are not blocked by temporary quota issues. All tests pass, and the changes are well-documented and verified.
- Scope:
central OpenCode/Strix review-process - Changed files:
3 - Head SHA:
94aadb867514b8107360e6ad0632732ed02d1e4f - Workflow run: 29230656223
- Workflow attempt: 1
This approval path is limited to ContextualWisdomLab/.github central review-process self-repair.
There was a problem hiding this comment.
Pull request overview
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including .github/workflows/opencode-review.yml, scripts/ci/pr_review_merge_scheduler.py, tests/test_pr_review_merge_scheduler.py.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects .github/workflows/opencode-review.yml to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
Findings
No blocking findings.
Adversarial validation
{"status":"passed","probes":[{"path":".github/workflows/opencode-review.yml","line":1519,"hypothesis":"The deferred retry mechanism might not correctly handle PRs that are closed or have new commits during the backoff window.","attack_or_counterexample":"Simulated PR closure and new commit during the backoff window.","evidence":"The job checks the PR state and head SHA before dispatching the retry, ensuring it only proceeds for open PRs with the same head SHA.","outcome":"falsified"},{"path":"scripts/ci/pr_review_merge_scheduler.py","line":1246,"hypothesis":"The scheduler might incorrectly report OpenCode review failures even when a successful status exists.","attack_or_counterexample":"Simulated PR with OpenCode review failure and successful status.","evidence":"The scheduler correctly skips OpenCode review failures when a successful status exists.","outcome":"falsified"}],"residual_risk":"Low. The changes are well-tested and handle edge cases such as PR closure and new commits during the backoff window."}Evidence
- Result: APPROVE
- Reason: The changes address model-pool exhaustion by adding a deferred retry mechanism and ensuring OpenCode review failures due to exhaustion are handled gracefully.
- Scope:
central OpenCode/Strix review-process - Changed files:
3 - Head SHA:
0916dcd9745ff07b1d14831baadc819d1ecd95fc - Workflow run: 29231727276
- Workflow attempt: 1
This approval path is limited to ContextualWisdomLab/.github central review-process self-repair.
There was a problem hiding this comment.
Pull request overview
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including .github/workflows/opencode-review.yml, scripts/ci/pr_review_merge_scheduler.py, tests/test_pr_review_merge_scheduler.py.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects .github/workflows/opencode-review.yml to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
Findings
No blocking findings.
Adversarial validation
{"status":"passed","probes":[{"path":".github/workflows/opencode-review.yml","line":1519,"hypothesis":"The deferred retry mechanism might not handle PR state changes during the backoff window correctly.","attack_or_counterexample":"Simulated PR closure or head change during the backoff window.","evidence":"The code checks PR state and head SHA after the backoff window and skips retry if mismatched.","outcome":"falsified"},{"path":"scripts/ci/pr_review_merge_scheduler.py","line":1246,"hypothesis":"The OpenCode review context might not be correctly handled in failed status checks.","attack_or_counterexample":"Simulated OpenCode review failure with a successful status context.","evidence":"The code correctly skips OpenCode review failures if a successful status context exists.","outcome":"falsified"}],"residual_risk":"Low; the changes are well-tested and handle edge cases."}Evidence
- Result: APPROVE
- Reason: The PR addresses model-pool exhaustion by adding a deferred retry mechanism, and all tests pass.
- Scope:
central OpenCode/Strix review-process - Changed files:
3 - Head SHA:
d3285e4f71dfa84b107c45b40ed9c6c0f956f61d - Workflow run: 29231993549
- Workflow attempt: 1
This approval path is limited to ContextualWisdomLab/.github central review-process self-repair.
There was a problem hiding this comment.
Pull request overview
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including .github/workflows/opencode-review.yml, scripts/ci/pr_review_merge_scheduler.py, tests/test_pr_review_merge_scheduler.py.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects .github/workflows/opencode-review.yml to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
Findings
No blocking findings.
Adversarial validation
{"status":"passed","probes":[{"path":".github/workflows/opencode-review.yml","line":1423,"hypothesis":"The deferred retry mechanism might not correctly handle PRs that are closed or have new commits during the backoff window.","attack_or_counterexample":"Simulated PR closure and new commit during the backoff window.","evidence":"The code checks the PR state and head SHA before dispatching the retry, ensuring it only proceeds for open PRs with the same head SHA.","outcome":"falsified"},{"path":"scripts/ci/pr_review_merge_scheduler.py","line":1246,"hypothesis":"The change to ignore OpenCode-review failures when a successful status exists might incorrectly skip required reviews.","attack_or_counterexample":"Simulated PR with both failed OpenCode-review and successful status.","evidence":"The code correctly skips the failed OpenCode-review check if a successful status exists, ensuring no required reviews are missed.","outcome":"falsified"}],"residual_risk":"Low; the changes are well-tested and handle edge cases appropriately."}Evidence
- Result: APPROVE
- Reason: The PR addresses model-pool exhaustion by adding a deferred retry mechanism, and all tests pass.
- Scope:
central OpenCode/Strix review-process - Changed files:
3 - Head SHA:
73c4883c442a52aabb7e6ccc0d3cf9a79eb104cf - Workflow run: 29232202417
- Workflow attempt: 1
This approval path is limited to ContextualWisdomLab/.github central review-process self-repair.
… env (#522) The opencode-exhausted-retry job from #520 passed the literal `--ref main` to gh workflow run, which violates the test_strix_quick_gate.sh contract pin "opencode scheduler dispatch must support develop-default repositories" (assert_file_not_contains "--ref main"). The ref is semantically correct — it names the trusted ref of the CENTRAL workflow repository, not the target repository's default branch — so move the literal into a CENTRAL_WORKFLOW_REF env var, mirroring how the merge scheduler carries the same value in SCHEDULER_REQUIRED_WORKFLOW_REF. Found by running the bash contract suite's opencode workflow assertions against #520 after merge; the suite's opencode pin section otherwise passes apart from pre-existing 180s/540s cadence pins that drifted from the 300s values already on main (tracked separately). Verification: pytest tests/test_opencode_agent_contract.py tests/test_opencode_workflow_shell_syntax.py (26 passed); the extracted assert_file_not_contains "--ref main" pin now passes. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: opencode-agent[bot] <219766164+opencode-agent[bot]@users.noreply.github.com>
Problem
When the GitHub Models pool is exhausted (
OPENCODE_MODEL_POOL_OUTCOME=exhausted), the requiredpull_request_targetreview fails closed withMODEL_OUTPUT_UNAVAILABLE, and becauseopencode-reviewis a required status check withenforce_admins, every quota exhaustion blocks PR merges org-wide until someone reruns the job. Observed on naruon PR #1072 (runs 29225414867 and 29226757733, 2026-07-13).Two concrete gaps kept this manual:
scheduletriggers heavily — the*/15heartbeat fired at 23:23, 00:13, 01:00, 04:05, 04:55 on 2026-07-12/13, with gaps up to three hours.workflow_dispatchretry published an approval plus theopencode-reviewsuccess commit status,failed_status_checksinpr_review_merge_scheduler.pystill counted the stale failedpull_request_targetcheck run as a merge blocker (failed check(s): opencode-review). Strix already has a supersession excusal for exactly this shape; OpenCode did not. Only a manual rerun of the failed required run cleared it.Fix
opencode-review.yml: newopencode-exhausted-retryjob. Runs only when thepull_request_targetreview job failed andmodel_pool_outcome == 'exhausted'— the state whose publish gate has already verified that current-head coverage, peer checks, code-scanning alerts, and review threads are clean, so a deferred same-head retry is safe. GitHub Actions has no delayed-dispatch primitive, so the job sleeps one 300s backoff window, re-checks that the PR is still open on the same head, and re-dispatches the central review through the same trustedworkflow_dispatchpath (PR_REVIEW_MERGE_TOKEN || OPENCODE_APPROVE_TOKEN) the merge scheduler uses. It retries once; outages longer than the backoff window stay owned by the org-sweep heartbeat, which keeps re-dispatching the same current head until a model produces a verdict. A head push or close during the sleep cancels the job via the existing PR-scoped concurrency group.pr_review_merge_scheduler.py: OpenCode supersession excusal. A failed OpenCode check run is no longer a merge blocker when a successful same-headopencode-reviewcommit status supersedes it — the exact mirror of the existingstrixrule one line above. The status is only ever published by the trusted central dispatch path, and approval-gating still controls the merge itself.Combined behavior after exhaustion: red required check → deferred retry ~5 minutes later → if the pool recovered, the dispatch run publishes the real review verdict + success status and the scheduler merges without any human action; if still exhausted, the dispatch run publishes the exhausted-outcome status (existing semantics) and the sweep keeps retrying until a model answers.
Deliberately skipped: adding fallback models to
OPENCODE_MODEL_CANDIDATES. The retry path works regardless of pool membership, and pool curation is being actively managed in #514/#517 (schema drift, cadence) — adding models here would couple this mechanism fix to a policy decision.Verification
coverage run -m pytest tests— 377 passed; coverage 100% (fail_under = 100gate).interrogate— 100% docstrings.test_strix_quick_gate.shnegative assertions audited against the added job (no collisions withrekick_model_pool_on_exhaustion,contents: write, repo-local dispatch, or exhaustion-approval pins); the full suite is still running locally (Git Bash is slow on it) and runs in CI — the result will be reported on this PR.🤖 Generated with Claude Code