fix(bot): thread-based liveness watchdog for silent long-poll hangs#143
Merged
Conversation
The existing NetworkError/TimedOut self-exit (PR #135) only fires when getUpdates actually raises. It doesn't always: on 2026-07-06 a VPN-pipeline exit rotation (a mandatory background job that swaps the egress VDSina server every few days) tore down the proxy's upstream mid-long-poll and left the read wedged with zero further exceptions or log lines for ~30h on both live instances. Add a cheap async heartbeat plus a watchdog on its own OS thread (independent of the event loop, since a wedged loop can't fire its own asyncio-scheduled timeout) that force-exits after 90s of silence so the supervisor respawns a clean instance. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ccbot,ccbot2on ogame-sina) went alive-but-deaf for ~30h starting 2026-07-06 00:00 UTC after the host's mandatory VDSina VPN-exit rotation swapped the egress proxy's upstream server mid-long-poll. Zero exceptions were ever raised, so the existingNetworkError/TimedOutself-exit (fix(bot): self-exit on sustained getUpdates network outage #135) never tripped — its detector only fires from inside PTB's own error handler, which requires an actual exception._heartbeat_loop) plus a watchdog running on its own OS thread (_liveness_watchdog_loop), independent of the event loop — necessary because if the loop itself is what's wedged, an asyncio-scheduled timeout would never fire either. After 90s of heartbeat silence it force-exits via the existing_terminate_for_sustained_conflict()path so Docker'srestart: unless-stoppedrespawns a clean instance./menumessages) despite the proxy path itself testing healthy again. Redeployed with this fix; both now hold their getUpdates poll again (verified via single-probe 409 Conflict).Test plan
pytest -q -m "not integration"— 754 passed (both locally and in an ephemeral container against this branch's snapshot)tests/test_liveness_watchdog.pycovers: fresh heartbeat tolerated, stale heartbeat triggers exit, just-under-budget tolerated, heartbeat loop actually advances the timestampccbotandccbot2on ogame-sina; verified startup log shows "Liveness watchdog started"; verified both hold their long-poll post-deploy🤖 Generated with Claude Code