fix(websocket): tighten upstream WS dead-peer window under ~30s head silence#9
Closed
snowkide wants to merge 1 commit into
Closed
fix(websocket): tighten upstream WS dead-peer window under ~30s head silence#9snowkide wants to merge 1 commit into
snowkide wants to merge 1 commit into
Conversation
…silence wsPingInterval/wsPongWait were 30s/75s, so a black-holed upstream could silence newHeads on a sticky client WS for longer than typical downstream NoNewHeads thresholds (~30s) before eRPC re-dialed — even while other pods or HTTP failover still tipped. Use 10s/25s (still ≥2 pings per pongWait). Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pongWait > 2× pingInterval(two ping chances before declare-dead).Why (single Chainlink RPC = eRPC)
Chainlink pins one sticky client WS to one eRPC pod. eRPC’s internal HTTP/upstream failover on other pods does not move that socket.
If the head-feeding upstream on that pod blackholes, eRPC previously waited up to 75s before re-dial. Consumers (e.g. Polygon
NoNewHeadsThreshold = 30s) already treat silence as unhealthy by then — so the node looks “eRPC down” while the rest of the eRPC fleet can still tip.This does not refuse client
eth_subscribeor push failure into Chainlink. It only makes upstream self-heal start sooner so the sticky client keeps getting heads.Out of scope
1001 GoingAway(DaemonSet roll) — still closes the client WS; needs roll/drain hygieneTest plan
go test ./clients/ -run 'WsJsonRpc|Pong|Reconnect|Liveness|Dead'go test ./indexer/adapters/wsupstream/ -run 'Reconnect|Healthy|Resub'go test ./erpc/ -run UpstreamDiesUngracefully_SelfHealserpc_upstream_websocket_connecteddrops and re-subscribes within ~25s; Chainlink head sub on that pod does not sit silent past ~30sMade with Cursor