userspace-dp: singleflight-guard + bound the neighbor prewarm scan (#5104)#6147
Open
psaab wants to merge 1 commit into
Open
userspace-dp: singleflight-guard + bound the neighbor prewarm scan (#5104)#6147psaab wants to merge 1 commit into
psaab wants to merge 1 commit into
Conversation
The manager status loop launched a full async neighbor-resolve prewarm
scan on every 1s tick for the first 60s (then every 10s on HA standby)
with no in-flight guard. Each scan did route-indexed NeighList dumps and
spawned one goroutine PER resolve target. Under slow netlink or a large
RIB a scan can outlast its tick, so scans overlapped — multiplying
concurrent scans and probe goroutines (target dedup was per-scan only).
That turned HA readiness warming into self-amplifying control-plane load,
delaying apply/status/recovery.
Bound the cold-path work per tick:
- Singleflight: a lock-free neighborPrewarmInFlight atomic.Bool CAS
guard in proactiveNeighborResolveAsyncLocked coalesces overlapping
ticks onto the running scan; a second scan never starts while one is
in flight. The guard clears when the scan goroutine returns (defer,
so a panic also clears it), matching the existing rgTransitionInFlight
pattern.
- Deadline + cancellation: each scan runs under a 15s
neighborPrewarmDeadline context and checks it between netlink dumps,
so a slow netlink layer cannot pin the singleflight guard
indefinitely; the next tick spawns a fresh scan. (vishvananda/netlink
calls are not context-cancelable mid-syscall, so the deadline bounds
the between-dump loop, not a single wedged syscall.)
- Bounded probe pool: runBoundedNeighborProbes fans probes out through
a fixed 8-worker pool (neighborPrewarmProbeWorkers) instead of one
goroutine per target, so a large resolve set cannot multiply
goroutines every tick.
- One neighbor dump per link per scan: the scan caches
NeighList(FAMILY_ALL) per link index, collapsing the old per-route
re-dump.
An injectable neighborPrewarmScan seam allows deterministic tests without
real netlink. The merge-gate regression
TestNeighborPrewarmSingleflightCoalescesOverlappingTicks5104 fires
overlapping ticks while a scan is artificially in flight and asserts only
one scan runs; reverting the CAS guard makes it RED (peak concurrent
scans = 2, want 1) as an assertion failure. Companion tests cover the
bounded probe pool and context-cancel wiring.
Validation: go test -race ./pkg/dataplane/userspace/... (full package
green; named test x3 for flake), gofmt + go vet clean. Docs:
docs/userspace-cold-start-fix-plan.md records the singleflight + bounded
+ deadline behavior.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0189kE2yTvStUsnFq43brwYR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (#5104, verified on origin/master 865fad6)
The manager status loop launched a full async neighbor-resolve prewarm
scan on every 1s tick for the first 60s (then every 10s on HA standby)
with no in-flight guard. Each scan did route-indexed
NeighListdumpsand spawned one goroutine per resolve target. Under slow netlink or a
large RIB a scan can outlast its tick, so scans overlapped — multiplying
concurrent scans and probe goroutines (target dedup was per-scan only). This
turned HA readiness warming into self-amplifying control-plane load, delaying
apply/status/recovery.
Fix
Bound the cold-path work per status tick:
neighborPrewarmInFlightatomic.BoolCASguard in
proactiveNeighborResolveAsyncLockedcoalesces overlapping ticksonto the running scan; a second scan never starts while one is in flight.
The guard clears when the scan goroutine returns (
defer, so a panic alsoclears it), matching the existing
rgTransitionInFlightpattern.neighborPrewarmDeadlinecontext and checks it between netlink dumps, so aslow netlink layer cannot pin the guard indefinitely; the next tick spawns
a fresh scan. (vishvananda/netlink calls are not context-cancelable
mid-syscall, so the deadline bounds the between-dump loop, not a single
wedged syscall.)
runBoundedNeighborProbesfans probes out througha fixed 8-worker pool (
neighborPrewarmProbeWorkers) instead of onegoroutine per target.
NeighList(FAMILY_ALL)per link index, collapsing the old per-route re-dump.
An injectable
neighborPrewarmScanseam allows deterministic tests withoutreal netlink.
Test (merge gate)
TestNeighborPrewarmSingleflightCoalescesOverlappingTicks5104firesoverlapping ticks while a scan is artificially in flight (blocked in the
injected seam) and asserts only one scan runs (peak concurrent scans == 1,
started == 1). Companion tests assert the bounded probe pool (200 targets →
peak ≤ 4 workers) and the ctx-cancel wiring.
(
concurrent prewarm scans peaked at 2, want 1) as an assertion failure,not a build break. target-count = 1.
Validation
go test -race ./pkg/dataplane/userspace/...— full package green; namedtest x3 for flake.
gofmt+go vetclean.Docs
docs/userspace-cold-start-fix-plan.mdrecords the singleflight + bounded +deadline-bounded prewarm behavior.
HA / smoke assessment
This touches the HA-readiness-warming path, but the guard is unit-provable via
the injected scan seam and the fix only reduces control-plane load without
changing the probe behavior (same
sendICMPProbeFromManagerper target). Noloss-cluster smoke warranted.
Closes #5104