Once vk-cocoon annotates a main pod with vm.cocoonstack.io/lifecycle-state: failed, the reconciler returns before evaluating spec.Suspend (and everything else), so the CocoonSet stops responding to any further intent — permanently, even after the VM recovers and the pod reports Ready. The only escape today is deleting the pod (destructive for a stateful VM) or clearing the annotation by hand.
What happened
internal-cocoon-node-1, 2026-07-25, cocoon-operator v0.3.0-3-gfaaba3d. A wake of an 8 GiB win11 VM missed vk's 45 s fresh-NIC DHCP window (vk-cocoon#47) and vk wrote:
vm.cocoonstack.io/lifecycle-state: failed
vm.cocoonstack.io/lifecycle-state-message: wake vk-staging-cocoonset-vm-a3135912-0: dhcp lease for fresh NIC not observed within 45s
About three minutes later the VM converged on its own. State at that point:
pod vm-a3135912-0 1/1 Running # guest confirmed 172.22.0.205 on its NIC
CR generation 60 == observedGeneration 60
Ready=True reason=AllAgentsReady "1/1 agents ready, 0/0 toolboxes ready"
phase=Failed
status.agents[0].phase=Running, status.agents[0].ip="" # empty
I then set spec.suspend=true and waited 274 s: the phase never left Failed and no hibernate ran at all — cocoon_vk_snapshot_save_duration_seconds_count and ..._push_..._count both stayed 0. Clearing the annotation made the operator pick up the queued intent immediately and the hibernate completed normally in 77 s.
Why
cocoonset/reconciler.go:
if classified.main != nil {
if reason := mainPodFailedReason(classified.main); reason != "" {
return r.handleFailedMainAgent(ctx, &cs, classified, reason) // ← always returns
}
if cs.Status.Phase == cocoonv1.CocoonSetPhaseFailed && meta.IsPodReady(classified.main) && r.Recorder != nil {
r.Recorder.Eventf(&cs, corev1.EventTypeNormal, "RecoveredFromFailure", ...)
}
}
if cs.Spec.Suspend { ... } // ← never reached
mainPodFailedReason (reconciler.go:229) returns "PodLifecycleFailed" whenever meta.ReadLifecycleState(pod) == LifecycleStateFailed, i.e. purely from the annotation.handleFailedMainAgent (reconciler.go:171) only has an escape when the pod spec drifted (delete + requeue); otherwise it records the failure, patches phase Failed, and returns with no requeue.- The
RecoveredFromFailure branch sits after that early return, so it is unreachable for a vk-driven failure. It can only fire for a kubelet-driven failure that has already cleared.
Nothing in the operator or (as far as I can see) in vk clears that annotation on its own, so the CR is wedged until a human intervenes.
Two secondary effects worth noting:
status.agents[0].ip stays empty for the whole time even though the pod and guest have an address. vm-service surfaces that field to users as the connection target, so a VM that actually works looks address-less.- The failure is silent from the CR's perspective:
Ready=True and phase=Failed coexist, with Progressing=False reason=Stable message=Failed.
Suggested behaviour
Some way for a recovered VM to leave the failed state without hand-editing annotations. Options, cheapest first:
- Treat a failed lifecycle state as stale when the pod is Ready: fall through to normal reconciliation (and let the phase be recomputed) instead of returning early.
lifecycle-observed-generation is already on the pod, so "failed, but observed at an older generation than the current spec" is expressible.
- Or keep the early return but make it a bounded state: requeue with backoff and re-evaluate readiness, so the set self-heals when the VM comes good.
- At minimum, do not swallow
spec.Suspend while failed. Hibernate is the one intent that is still meaningful (and safe) for a VM whose wake went wrong — being unable to put it back to sleep is what turned a transient wake timeout into manual work here.
Recovery for the record
Non-destructive unblock (do not delete the pod — that destroys the stateful VM):
kubectl -n <ns> patch pod <set>-0 --type=merge \
-p '{"metadata":{"annotations":{"vm.cocoonstack.io/lifecycle-state":null,"vm.cocoonstack.io/lifecycle-state-message":null}}}'
Note for anyone debugging this: these are annotations, not labels — the pod's labels are only app.kubernetes.io/name and the three cocoonset.cocoonstack.io/* ones. Dumping labels and annotations in one jsonpath and grepping makes it easy to patch the wrong map and see a no-op.
Once vk-cocoon annotates a main pod with
vm.cocoonstack.io/lifecycle-state: failed, the reconciler returns before evaluatingspec.Suspend(and everything else), so the CocoonSet stops responding to any further intent — permanently, even after the VM recovers and the pod reports Ready. The only escape today is deleting the pod (destructive for a stateful VM) or clearing the annotation by hand.What happened
internal-cocoon-node-1, 2026-07-25, cocoon-operator
v0.3.0-3-gfaaba3d. A wake of an 8 GiB win11 VM missed vk's 45 s fresh-NIC DHCP window (vk-cocoon#47) and vk wrote:About three minutes later the VM converged on its own. State at that point:
I then set
spec.suspend=trueand waited 274 s: the phase never leftFailedand no hibernate ran at all —cocoon_vk_snapshot_save_duration_seconds_countand..._push_..._countboth stayed0. Clearing the annotation made the operator pick up the queued intent immediately and the hibernate completed normally in 77 s.Why
cocoonset/reconciler.go:mainPodFailedReason(reconciler.go:229) returns"PodLifecycleFailed"whenevermeta.ReadLifecycleState(pod) == LifecycleStateFailed, i.e. purely from the annotation.handleFailedMainAgent(reconciler.go:171) only has an escape when the pod spec drifted (delete + requeue); otherwise it records the failure, patches phaseFailed, and returns with no requeue.RecoveredFromFailurebranch sits after that early return, so it is unreachable for a vk-driven failure. It can only fire for a kubelet-driven failure that has already cleared.Nothing in the operator or (as far as I can see) in vk clears that annotation on its own, so the CR is wedged until a human intervenes.
Two secondary effects worth noting:
status.agents[0].ipstays empty for the whole time even though the pod and guest have an address. vm-service surfaces that field to users as the connection target, so a VM that actually works looks address-less.Ready=Trueandphase=Failedcoexist, withProgressing=False reason=Stable message=Failed.Suggested behaviour
Some way for a recovered VM to leave the failed state without hand-editing annotations. Options, cheapest first:
lifecycle-observed-generationis already on the pod, so "failed, but observed at an older generation than the current spec" is expressible.spec.Suspendwhile failed. Hibernate is the one intent that is still meaningful (and safe) for a VM whose wake went wrong — being unable to put it back to sleep is what turned a transient wake timeout into manual work here.Recovery for the record
Non-destructive unblock (do not delete the pod — that destroys the stateful VM):
Note for anyone debugging this: these are annotations, not labels — the pod's labels are only
app.kubernetes.io/nameand the threecocoonset.cocoonstack.io/*ones. Dumping labels and annotations in onejsonpathand grepping makes it easy to patch the wrong map and see a no-op.