Background-job survival, sync circuit breaker, sub-50ms startup - #132
Merged
Conversation
Turn-end pre-verification cleanup killed every background process the turn had started — observed loss: two ~800 GB downloads reaped at turn end hours before completion (a third escaped its group kill and finished despite being marked killed). Background processes now carry their origin. Deliberate work the model started with run_in_background:true (downloads, servers) is spared by kill_started_after — turn end, turn cancel, and retry rewinds all flow through it — and still dies with the session or an explicit bash_kill. Auto-backgrounded foreground overruns remain turn state and are reaped before verification exactly as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, silent best-effort
api.pipenetwork.ai outages surfaced everywhere: one-shots took 36s
(serial 10s registration/flush timeouts against a 4k-record, 63k-event
durable backlog), session switches could burn a ~100s retry ladder and
then fail the switch, TUI startup hung for tens of seconds because the
synced-session host enable awaited portal registration before first
paint, and raw reqwest errors ("operation timed out") leaked into the
transcript and exit output.
Circuit breaker (persisted in the shared sync store): the first
connect/timeout failure trips a cooldown (60s doubling to 15 min,
reset by any success) that every sync path in every hi process
respects — registration, record flush, live-event flush, and
end-session skip the network instantly while open, leaving records
queued durably. Both sync HTTP clients gain a 3s connect timeout;
ensure_registered_now fails fast once the breaker opens.
Sub-50ms startup: the synced TUI's host-mode auto-enable now runs as a
background task whose outcome the event loop applies when it lands —
first frame no longer waits on the portal. HI_STARTUP_TRACE=1 prints
per-milestone stamps plus a TUI first-frame checkpoint so startup
regressions get measured, not guessed.
Sync trouble is never shown and never affects the workflow: exit
flushes, the startup host enable, and session-switch registration are
silent on failure (records stay queued; the session runs unhosted); an
explicit /sessions host command gets a calm availability note instead
of a raw error. Only --daemon, which cannot function without sync,
still announces registration retries.
Measured in the affected workspace: one-shot 36.6s → 1.3s with the
breaker open and zero sync/error lines in the output; TUI first frame
23–24ms (was: tens of seconds).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
run_in_background: trueare no longer reaped by turn-end pre-verification cleanup, turn cancel, or retry rewinds (observed loss: two ~800 GB downloads killed at turn end). Auto-backgrounded foreground overruns are still reaped as before; everything still dies with the session orbash_kill.HI_STARTUP_TRACE=1gives per-milestone timing./sessions hostgets a calm availability note; only--daemonstill announces retries.Testing
kill_started_after(spares deliberate, reaps adopted), turn-leveldeliberate_background_process_survives_turn_end, breaker cooldown doubling/reset, unreachable-endpoint breaker trip with silent follow-up flushes.🤖 Generated with Claude Code