Skip to content

userspace-dp: singleflight-guard + bound the neighbor prewarm scan (#5104)#6147

Open
psaab wants to merge 1 commit into
masterfrom
fix/5104-prewarm-singleflight
Open

userspace-dp: singleflight-guard + bound the neighbor prewarm scan (#5104)#6147
psaab wants to merge 1 commit into
masterfrom
fix/5104-prewarm-singleflight

Conversation

@psaab

@psaab psaab commented Jul 18, 2026

Copy link
Copy Markdown
Owner

Problem (#5104, verified on origin/master 865fad6)

The manager status loop launched a full async neighbor-resolve prewarm
scan on every 1s tick for the first 60s
(then every 10s on HA standby)
with no in-flight guard. Each scan did route-indexed NeighList dumps
and spawned one goroutine per resolve target. Under slow netlink or a
large RIB a scan can outlast its tick, so scans overlapped — multiplying
concurrent scans and probe goroutines (target dedup was per-scan only). This
turned HA readiness warming into self-amplifying control-plane load, delaying
apply/status/recovery.

Fix

Bound the cold-path work per status tick:

  1. Singleflight — a lock-free neighborPrewarmInFlight atomic.Bool CAS
    guard in proactiveNeighborResolveAsyncLocked coalesces overlapping ticks
    onto the running scan; a second scan never starts while one is in flight.
    The guard clears when the scan goroutine returns (defer, so a panic also
    clears it), matching the existing rgTransitionInFlight pattern.
  2. Deadline + cancellation — each scan runs under a 15s
    neighborPrewarmDeadline context and checks it between netlink dumps, so a
    slow netlink layer cannot pin the guard indefinitely; the next tick spawns
    a fresh scan. (vishvananda/netlink calls are not context-cancelable
    mid-syscall, so the deadline bounds the between-dump loop, not a single
    wedged syscall.)
  3. Bounded probe poolrunBoundedNeighborProbes fans probes out through
    a fixed 8-worker pool (neighborPrewarmProbeWorkers) instead of one
    goroutine per target.
  4. One dump per link per scan — the scan caches NeighList(FAMILY_ALL)
    per link index, collapsing the old per-route re-dump.

An injectable neighborPrewarmScan seam allows deterministic tests without
real netlink.

Test (merge gate)

TestNeighborPrewarmSingleflightCoalescesOverlappingTicks5104 fires
overlapping ticks while a scan is artificially in flight (blocked in the
injected seam) and asserts only one scan runs (peak concurrent scans == 1,
started == 1). Companion tests assert the bounded probe pool (200 targets →
peak ≤ 4 workers) and the ctx-cancel wiring.

  • Parent-RED verified: neutralizing the CAS guard makes the named test RED
    (concurrent prewarm scans peaked at 2, want 1) as an assertion failure,
    not a build break. target-count = 1.

Validation

  • go test -race ./pkg/dataplane/userspace/... — full package green; named
    test x3 for flake.
  • gofmt + go vet clean.

Docs

docs/userspace-cold-start-fix-plan.md records the singleflight + bounded +
deadline-bounded prewarm behavior.

HA / smoke assessment

This touches the HA-readiness-warming path, but the guard is unit-provable via
the injected scan seam and the fix only reduces control-plane load without
changing the probe behavior (same sendICMPProbeFromManager per target). No
loss-cluster smoke warranted.

Closes #5104

The manager status loop launched a full async neighbor-resolve prewarm
scan on every 1s tick for the first 60s (then every 10s on HA standby)
with no in-flight guard. Each scan did route-indexed NeighList dumps and
spawned one goroutine PER resolve target. Under slow netlink or a large
RIB a scan can outlast its tick, so scans overlapped — multiplying
concurrent scans and probe goroutines (target dedup was per-scan only).
That turned HA readiness warming into self-amplifying control-plane load,
delaying apply/status/recovery.

Bound the cold-path work per tick:

  - Singleflight: a lock-free neighborPrewarmInFlight atomic.Bool CAS
    guard in proactiveNeighborResolveAsyncLocked coalesces overlapping
    ticks onto the running scan; a second scan never starts while one is
    in flight. The guard clears when the scan goroutine returns (defer,
    so a panic also clears it), matching the existing rgTransitionInFlight
    pattern.

  - Deadline + cancellation: each scan runs under a 15s
    neighborPrewarmDeadline context and checks it between netlink dumps,
    so a slow netlink layer cannot pin the singleflight guard
    indefinitely; the next tick spawns a fresh scan. (vishvananda/netlink
    calls are not context-cancelable mid-syscall, so the deadline bounds
    the between-dump loop, not a single wedged syscall.)

  - Bounded probe pool: runBoundedNeighborProbes fans probes out through
    a fixed 8-worker pool (neighborPrewarmProbeWorkers) instead of one
    goroutine per target, so a large resolve set cannot multiply
    goroutines every tick.

  - One neighbor dump per link per scan: the scan caches
    NeighList(FAMILY_ALL) per link index, collapsing the old per-route
    re-dump.

An injectable neighborPrewarmScan seam allows deterministic tests without
real netlink. The merge-gate regression
TestNeighborPrewarmSingleflightCoalescesOverlappingTicks5104 fires
overlapping ticks while a scan is artificially in flight and asserts only
one scan runs; reverting the CAS guard makes it RED (peak concurrent
scans = 2, want 1) as an assertion failure. Companion tests cover the
bounded probe pool and context-cancel wiring.

Validation: go test -race ./pkg/dataplane/userspace/... (full package
green; named test x3 for flake), gofmt + go vet clean. Docs:
docs/userspace-cold-start-fix-plan.md records the singleflight + bounded
+ deadline behavior.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0189kE2yTvStUsnFq43brwYR
Copilot AI review requested due to automatic review settings July 18, 2026 21:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dataplane/userspace: neighbor prewarm has no in-flight guard — overlapping full scans and unbounded probe goroutines

2 participants