Skip to content

fix(codex): recover ready after stale terminal event#997

Open
swear01 wants to merge 2 commits into
tiann:mainfrom
swear01:fix/issue-996-codex-terminal-recovery
Open

fix(codex): recover ready after stale terminal event#997
swear01 wants to merge 2 commits into
tiann:mainfrom
swear01:fix/issue-996-codex-terminal-recovery

Conversation

@swear01

@swear01 swear01 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #996.

  • allow Codex same-thread terminal recovery during retry/compact recovery when the terminal event carries a stale turn id
  • keep mismatched terminal events ignored unless they are explicitly scoped to the active thread and recovery context
  • add launcher-level regression coverage for retry completion with stale task_complete
  • add guard-level coverage for stale-turn same-thread terminal acceptance/rejection

Root cause

The launcher returned early on a stale/mismatched terminal event before clearing turnInFlight, clearing thinking, waking the loop, or scheduling ready. After same-thread retry, Codex can emit a terminal task_complete scoped to the active thread but with an older turn id; ignoring that event leaves the session permanently shown as thinking.

Validation

  • bun typecheck — pass
  • bunx vitest run --config cli/vitest.issue996.config.ts src/codex/codexRemoteLauncher.test.ts src/codex/utils/terminalEventGuard.test.ts — 66 passed
  • A/B verification: the new stale-terminal regression passes with this fix, fails with the old thread_status-only recovery condition (timeout instead of exit)

Note: local full bun run test was attempted earlier; it failed in cli/src/runner/runner.integration.test.ts runner integration cases in this environment, unrelated to this Codex launcher diff. Hub/web/shared tests passed separately.

@swear01 swear01 marked this pull request as ready for review July 9, 2026 11:20

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings
None.

Summary
Review mode: initial
No issues found in the added or modified lines. Residual risk: tests were not run in this automation; review was limited to diff and surrounding source inspection.

Testing
Not run (automation)

HAPI Bot

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings

  • [Major] Stale retry failures can now finalize the active retry turn — during a same-thread retry, sameThreadRetryAttempt stays > 0 after the replacement turn has started and until a task_complete resets it. With the new matching-thread allowance, any terminal event from the same thread but an old turn id is accepted, including a delayed duplicate task_failed for the already-finalized failed turn. Because the duplicate check is also bypassed for this recovery shape, that stale failure can set pending = activeMessage, clear turnInFlight, wake the loop, and start another retry while the real retry turn is still running. Evidence cli/src/codex/codexRemoteLauncher.ts:2466.
    Suggested fix:
    const allowStaleSameThreadTerminalRecovery = msgType === 'task_complete'
        && (sameThreadRetryAttempt > 0 || sameThreadCompactAttempt > 0);
    const allowSameThreadTerminalRecovery = msg.terminal_source === 'thread_status'
        || allowStaleSameThreadTerminalRecovery;

Summary
Review mode: follow-up after new commits
Found one regression risk in the added stale-terminal recovery path. Residual risk: review was limited to diff and surrounding source inspection.

Testing
Not run (automation). Suggested coverage: add a launcher regression where a stale same-thread task_failed for the previous turn arrives after the retry task_started; assert it is ignored and does not enqueue a second retry or clear thinking before the real retry completion.

HAPI Bot

eventThreadId,
currentThreadId: this.currentThreadId,
allowMatchingThreadIdTerminalEvent: msg.terminal_source === 'thread_status'
allowMatchingThreadIdTerminalEvent: allowSameThreadTerminalRecovery

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Major] This now accepts every stale same-thread terminal event while a retry/compact recovery attempt is active. A delayed duplicate task_failed from the already-finalized old turn can therefore pass the guard after the replacement turn has started, then run the retry branch again and clear turnInFlight even though the real retry is still running. Please scope the stale-turn exception to the recovery success signal this PR is handling.

Suggested fix:

const allowStaleSameThreadTerminalRecovery = msgType === 'task_complete'
    && (sameThreadRetryAttempt > 0 || sameThreadCompactAttempt > 0);
const allowSameThreadTerminalRecovery = msg.terminal_source === 'thread_status'
    || allowStaleSameThreadTerminalRecovery;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(codex): mismatched task_complete can leave runner session permanently thinking

1 participant