Skip to content

fix: score a released-seat wake once, not twice - #28

Merged
CMGS merged 3 commits into
mainfrom
fix/wake-placement-double-count
Jul 31, 2026
Merged

fix: score a released-seat wake once, not twice#28
CMGS merged 3 commits into
mainfrom
fix/wake-placement-double-count

Conversation

@tonicmuroq

@tonicmuroq tonicmuroq commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Found while running an e2e of the retain/release hibernate+wake cycle on a staging cluster. Metric-only defect — no state is corrupted, which is presumably why it went unnoticed.

What happens

The wake-completion arm (reconcileWake, case waking && vmLive(main)) returns ctrl.Result{Requeue: true}, and controller-runtime serves the next pass from the informer cache — which can still carry the status this pass just overwrote. The arm runs a second time. Its own hint-clear is visible by then, so hint == "" and the same landing gets scored placement=pool.

Measured, not inferred

Two full release cycles on a staging CocoonSet (operator v0.3.2, vk v0.3.7). Both wakes landed on the hinted node:

counter after cycle 1 after cycle 2 per cycle
slot_release_pods_deleted_total 1 2 +1 ✅
slot_release_wake_total{placement="hint-node"} 1 2 +1 ✅
slot_release_wake_total{placement="pool"} 1 2 +1 ❌
restored on … log lines 2 2 2 per wake

2/2 reproduction. A 100%-hint-node result reports as a 50/50 split, which makes the placement ratio — the entire point of the metric — unusable. pool is only ever labelled when the hint is empty or mismatched; the landing node was correct both times, so the empty hint is the only way in, confirming the re-entry.

The fix

The hint is the wake's own in-flight marker: reconcileSuspendRelease sets it before any pod delete (that patch blocks the delete loop, so a released wake always has one) and this arm clears it exactly at completion. Gate the scoring, the log line and the clear on it still being present — only the pass that owns the completion counts.

DeleteManifest deliberately stays outside the guard rather than moving under it: if a hint ever went missing, a guarded version would leave the :hibernate tag behind, and vk's derived restore-from-hibernate path treats a lingering tag as "a wake was lost here". Keeping it unconditional costs one 404-tolerant registry round trip on the re-entry and keeps that safety net.

Test

TestWakeScoresPlacementOnceAcrossStaleStatusReentry drives the completion arm twice, the second time with the hint cleared and the status still Waking — exactly what the Requeue hits live. Verified it actually catches the defect: with the guard removed it fails with pool delta=1, the same symptom measured in production.

go build ./..., go test ./..., make lint (linux + darwin) all green.

Also

Drops Every release wake restores via registry pull from the metric's help text. Since vk-cocoon v0.3.7 keeps the node-local snapshot across a seat release, a hint-node landing restores locally — snapshot_pull_duration_seconds_count stayed 0 across both cycles above.

tonicmuroq and others added 3 commits July 31, 2026 16:17
The wake-completion arm returns Requeue, and controller-runtime serves the
next pass from the informer cache — which can still carry the status this
pass just overwrote. The arm then runs a second time, and because it has
already cleared the node hint, it scores the same landing as placement=pool.

Measured over two full release cycles on a staging CocoonSet: both wakes
landed on the hinted node, yet
slot_release_wake_total came out hint-node=2 pool=2, with two "restored on"
log lines per wake. A 100%-hint-node result reported as a 50/50 split makes
the placement ratio — the whole point of the metric — unusable.

The hint is the wake's own in-flight marker: set before any pod delete,
cleared exactly at completion. Gate the scoring, the log line and the clear
on it still being present, so only the pass that owns the completion counts.
DeleteManifest stays unconditional (404-tolerant) rather than moving under
the guard, so a hint-less completion still drops the tag.

Also drop "Every release wake restores via registry pull" from the metric's
help: since vk-cocoon v0.3.7 keeps the node-local snapshot across a seat
release, a hint-node landing restores locally — measured pull=0 on both
cycles above.
…riage, dead guard

Fold the 4x firstTransitionAt/observePhaseExit/emitEventf block into
announcePhaseExitf; give cocoonset the same emitEventf helper hibernation
already had (5 inline Recorder nil-guards collapse); merge triageToolbox's
copy-pasted delete arms; inline the single-caller vmNamesForGC; drop the
empty-containers guard in resourcesMatch (apiserver validation makes it
unreachable and three sibling sites already index Containers[0] bare).
@CMGS
CMGS force-pushed the fix/wake-placement-double-count branch from da491ca to 08bc8e1 Compare July 31, 2026 08:18
@CMGS
CMGS merged commit f9d1c66 into main Jul 31, 2026
2 checks passed
@CMGS
CMGS deleted the fix/wake-placement-double-count branch July 31, 2026 08:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants