fix: score a released-seat wake once, not twice - #28
Merged
Conversation
The wake-completion arm returns Requeue, and controller-runtime serves the next pass from the informer cache — which can still carry the status this pass just overwrote. The arm then runs a second time, and because it has already cleared the node hint, it scores the same landing as placement=pool. Measured over two full release cycles on a staging CocoonSet: both wakes landed on the hinted node, yet slot_release_wake_total came out hint-node=2 pool=2, with two "restored on" log lines per wake. A 100%-hint-node result reported as a 50/50 split makes the placement ratio — the whole point of the metric — unusable. The hint is the wake's own in-flight marker: set before any pod delete, cleared exactly at completion. Gate the scoring, the log line and the clear on it still being present, so only the pass that owns the completion counts. DeleteManifest stays unconditional (404-tolerant) rather than moving under the guard, so a hint-less completion still drops the tag. Also drop "Every release wake restores via registry pull" from the metric's help: since vk-cocoon v0.3.7 keeps the node-local snapshot across a seat release, a hint-node landing restores locally — measured pull=0 on both cycles above.
…riage, dead guard Fold the 4x firstTransitionAt/observePhaseExit/emitEventf block into announcePhaseExitf; give cocoonset the same emitEventf helper hibernation already had (5 inline Recorder nil-guards collapse); merge triageToolbox's copy-pasted delete arms; inline the single-caller vmNamesForGC; drop the empty-containers guard in resourcesMatch (apiserver validation makes it unreachable and three sibling sites already index Containers[0] bare).
CMGS
force-pushed
the
fix/wake-placement-double-count
branch
from
July 31, 2026 08:18
da491ca to
08bc8e1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while running an e2e of the retain/release hibernate+wake cycle on a staging cluster. Metric-only defect — no state is corrupted, which is presumably why it went unnoticed.
What happens
The wake-completion arm (
reconcileWake,case waking && vmLive(main)) returnsctrl.Result{Requeue: true}, and controller-runtime serves the next pass from the informer cache — which can still carry the status this pass just overwrote. The arm runs a second time. Its own hint-clear is visible by then, sohint == ""and the same landing gets scoredplacement=pool.Measured, not inferred
Two full release cycles on a staging CocoonSet (operator v0.3.2, vk v0.3.7). Both wakes landed on the hinted node:
slot_release_pods_deleted_totalslot_release_wake_total{placement="hint-node"}slot_release_wake_total{placement="pool"}restored on …log lines2/2 reproduction. A 100%-hint-node result reports as a 50/50 split, which makes the placement ratio — the entire point of the metric — unusable.
poolis only ever labelled when the hint is empty or mismatched; the landing node was correct both times, so the empty hint is the only way in, confirming the re-entry.The fix
The hint is the wake's own in-flight marker:
reconcileSuspendReleasesets it before any pod delete (that patch blocks the delete loop, so a released wake always has one) and this arm clears it exactly at completion. Gate the scoring, the log line and the clear on it still being present — only the pass that owns the completion counts.DeleteManifestdeliberately stays outside the guard rather than moving under it: if a hint ever went missing, a guarded version would leave the:hibernatetag behind, and vk's derived restore-from-hibernate path treats a lingering tag as "a wake was lost here". Keeping it unconditional costs one 404-tolerant registry round trip on the re-entry and keeps that safety net.Test
TestWakeScoresPlacementOnceAcrossStaleStatusReentrydrives the completion arm twice, the second time with the hint cleared and the status stillWaking— exactly what the Requeue hits live. Verified it actually catches the defect: with the guard removed it fails withpool delta=1, the same symptom measured in production.go build ./...,go test ./...,make lint(linux + darwin) all green.Also
Drops
Every release wake restores via registry pullfrom the metric's help text. Since vk-cocoon v0.3.7 keeps the node-local snapshot across a seat release, a hint-node landing restores locally —snapshot_pull_duration_seconds_countstayed 0 across both cycles above.