perf: client QPS/Burst, orphan-scan verdicts, persistence ceiling on record - #4
Merged
Conversation
client-go's QPS=5/Burst=10 fallback queued ~400s of client-side throttling when the advertised 2000 pods push status at once, and serialized delete-authorization reads during eviction storms. Fixes #1
…s once A row with a claim_ref this provider does not hold is an externally claimed sandbox, not an orphan candidate; the L3 aggregated apiserver claims directly against sandboxd by design. Remaining verdicts log on first sight instead of every cycle, so genuine orphans stay visible. Audit-only semantics unchanged. Fixes #3
The #3 dedup covered only the unclaimed-row verdicts; a stale claim re-logged every cycle, reproducing the noise the fix removed. All three verdicts now share one record-and-log-on-transition site.
Three identical NewTicker/select loops collapse into runTicker; the zero-ClaimedAt fallback in runningStatus guarded a state no claim path can produce (all three write paths stamp it, pinned by TestEveryClaimPathStampsClaimedAt).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1, closes #3; benchmarks #2 and leaves it deferred per its own assessment.
All numbers: Linux container (golang:1.26) on a bare-metal bench host (AMD Ryzen 7 9700X, 16 threads), commit 6b9f74b, A/B via the benchmarks added in this branch.
#1 — client-go QPS/Burst
kubeConfig()left every client on client-go's QPS=5/Burst=10 fallback. New--kube-api-qps(default 200) /--kube-api-burst(default 400) flags apply to the sharedrest.Configbefore any client is built.Each arm converges on its rate limiter's budget, as expected. A 2000-pod status-push burst queues ~400 s of client-side throttling before, ~10 s after.
#3 — OrphanScan false positives
A row whose
claim_refthis provider does not hold is an apiserver-direct claim (a supported path), not an orphan candidate; it is now excluded from the orphan verdict. Remaining verdicts log on first sight instead of every cycle, so genuine orphans stay visible instead of drowning in repeat noise. Audit-only semantics (never release; failed list skips the cycle) unchanged.A/B: before, every unclaimed row logged every 60 s cycle forever (O(direct claims) lines/min by construction); after, one line per verdict transition.
TestOrphanScanExternalClaimsAndLogDeduppins 3 log lines across 3 cycles for one external + one orphan + one stale row (stale-claim verdicts joined the dedup in review).#2 — persistence ceiling (measured, deferred)
BenchmarkCommitClaimputs a number on the scaling limit: with a full 2000-claim table, one commit costs 1.43 ms (1.9 MB, 12k allocs) — a ~700 claims/s ceiling on this hardware, consistent with the issue's 250–500/s estimate on slower disks. The coalescing redesign stays deferred per the issue's own assessment (it changes the tentative/commit write-ordering semantics and deserves its own round); the benchmark is now in-tree to gate that future change.Gates
make lint(darwin+linux): 0 issues;asl ./...both GOOS: cleango test -race -count=1 ./...: pass