daemon collectOwnerless field discrepancy + targeted stale-create collect verb for embedders (vk-cocoon#54 PR-0)

[CMGS]

Spun out of cocoonstack/vk-cocoon#54 (vk restart strands or destroys in-flight VM operations). Two things live here: the resolution of a field discrepancy against collectOwnerless, and the scope of the cocoon-side prerequisite (called "PR-0" in the vk issue) that vk's fix depends on.

All references are to master 953457e.

Part 1 — field discrepancy: a creating placeholder survived 14 minutes

In the 2026-07-28 incident (vk-cocoon#54), a clone was SIGKILLed 59ms after ReserveVM wrote the creating placeholder. The placeholder then survived 14 minutes until manual removal. The vk-side reviewer flagged this as contradicting collectOwnerless (daemon/reconcile.go:141), which is expected to reclaim exactly this record shape within seconds, and asked for the discrepancy to be tracked here so nobody later removes vk's own skeleton handling on the strength of that code comment.

Code-side audit: no blocker found in the collect path

If a ≥v0.5.5 daemon had been supervising, the chain is sound end to end:

• 5s ticker (DefaultReconcileInterval, daemon/daemon.go:22) → ScanSupervision returns all records (hypervisor/supervisor.go:53, no state filter) → the creating branch calls collectOwnerless unconditionally (daemon/reconcile.go:64).
• The ops flock was free: it dies with the holder ("a crash never wedges the VM", hypervisor/utils.go:58), and the SIGKILLed clone was the holder.
• No lock starvation from vk's concurrent error loop: the ~2 req/s failing cocoon vm net calls die in plumbingForVM ("no network backend on VM", cmd/vm/netresize.go:50) before any LockVMOps — the netresize lock is taken inside the resizer (hypervisor/cloudhypervisor/netresize.go:31), which is never reached.
• CollectStaleCreate tolerates the placeholder's shape: RunDir is set at reserve time (PrereserveVM, hypervisor/create.go:55), ensureOrphanVMMDead no-ops with no pidfile and no VMM process (hypervisor/gc.go:295), and the CNI teardown protocol handles record-less VMs explicitly (KindOrphan, network/cni/teardown.go:62).

Conclusion: the daemon almost certainly was not running

1. The resident supervisor is optional and six days old at incident time: it landed in 66e4232 (daemon: optional resident supervisor for cocoon-managed VMs (#152 P0+P1) #154, 2026-07-22), first released in v0.5.5.
2. Nothing deploys it: no spawn in vk-cocoon, no systemd unit or service file in os-image/ or packaging/, no manifest anywhere in the workspace. (sandbox's deploy doc references cocoon daemon, but that is the sandbox deployment, not the vk nodes.)
3. The observation that "the daemon respawned as vk's child" at the vk restart matches a different process: vk's StartVMWatcher spawns cocoon vm status --event --format json as a long-lived child (vk-cocoon vm/cocoon_cli.go:305), which respawns with vk and is easy to read as a daemon in ps output. vm status --event is a pure event tail — meta-store subscription plus snapshot diffing (cmd/vm/status.go) — and contains no supervision logic: no reconcile pass, no collectOwnerless.

A 14-minute survival is inconsistent with a running supervisor and fully consistent with no supervisor.

To confirm on the node (needs someone with production access)

• ps ancestry of the cocoon-binary child under vk (expect vm status --event, not daemon).
• systemd unit list / any cocoon daemon process at incident time.
• If a daemon journal exists: it would contain supervising N backend(s) at startup and collected ownerless creating VM <id> (or per-pass collect errors) — absence settles it.

If confirmed, the actionable follow-up is Part 2 plus a deployment decision: either ship cocoon daemon as a supervised unit on vk nodes, or document explicitly that embedders must not rely on daemon-side reclamation. Either way the embedder-facing primitive below is needed — vk adopts at startup, instantly, and cannot wait for any daemon cadence.

Part 2 — PR-0: a safe targeted stale-create collect verb for embedders

vk-cocoon#54 Fix A needs to remove creating skeletons during its startup reconcile, before building its adoption indexes. The two obvious approaches are both wrong:

• vm rm --force is unsafe: the delete path takes the ops lock blocking (hypervisor/stop.go:95 → LockVMOps → l.Lock(ctx), hypervisor/utils.go:60). Against a record whose clone is genuinely in flight (externally driven, concurrent with a vk restart), the rm queues behind the live clone and then deletes the freshly created VM. Raw state == "creating" does not identify a skeleton — an in-flight clone shows creating for its entire pull+clone window.
• Waiting for the daemon / GC is unavailable: the daemon is optional (Part 1) and CreatingStateGCGrace is 24h (hypervisor/backend.go:32).

cocoon already owns the correct predicate — "a free ops lock is the proof, since create and clone hold it from prereserve through the final record commit" (daemon/reconcile.go:141). PR-0 exposes that primitive to embedders:

Deliverable: a CLI verb (naming up to the implementer — e.g. cocoon vm collect-stale-create <id|name>, or a --stale-create-only mode on rm) with exactly collectOwnerless semantics:

1. TryLockVMOps — non-blocking. Busy → exit with a distinct "in-flight operation, refused" outcome (not an error worth retrying blindly).
2. Under the lock, re-validate state == creating (PeekRecord). Anything else → "not a stale create, refused".
3. CollectStaleCreate (hypervisor/supervisor.go:165): ensure no orphan VMM, then run the delete protocol (tombstoned, resumable).
4. JSON output distinguishing collected / busy / not-creating / not-found, so the embedder can branch (vk skips adoption on busy and retries next restart).

Non-goals: no age heuristics, no force fallback — the lock-free proof is the whole point. The daemon's collectOwnerless should ideally share the same code path so the two can never diverge.

Consumer: vk-cocoon's StartupReconcile (vk-cocoon#54 Fix A) calls this per skeleton instead of indexing or Remove-ing it.