perf: remaining performance ledger after the 2026-07 hotspot round

[CMGS]

Status ledger for performance work after the #28 round and the cocoon #123 upgrade verification (bare-metal testbed, 2026-07-12). Numbers of record: warm claim 0.2ms, sequential clone claim 32-37ms, burst16 wall 287ms, cold ~320ms, fork(4) 185ms, exec RTT p50 0.27ms, fs_pull ~602 MiB/s, fs_push steady ~343 MiB/s.

Open — worth doing, needs a product decision

1. Wire protocol v2: multiplexed RPCs + binary bulk frames. The one remaining order-of-magnitude lever in this repo. Today every RPC pays a triple connection setup (SDK TCP dial + HTTP upgrade → relay hijack → fresh vsock dial): the fixed cost is the bulk of the 0.27ms RTT. A pre-dialed spare connection was measured and rejected (p50 0.20/0.48/0.36/0.35/0.39ms across 5 rounds vs a stable 0.27-0.31ms baseline; p90/p99 consistently worse — the background dial contends with the foreground RPC), so the real fix is multiplexing over one long-lived conn. Bulk data rides base64-in-JSON (+33% wire, codec CPU); binary framing should lift fs_pull from ~602 toward 800+ MiB/s. Scope: SDK + relay + silkd (Rust) — cross-repo protocol change, staleness/idempotency semantics needed for reconnects. Do both together or not at all.

2. Refill admission re-eval (refillSem=4 / per-golden). Un-blocked by cocoon #123 (sibling clones no longer serialize on the source flock). Claim-miss provisions are unbounded (verified: ClaimProvision calls provision directly), but refills and forks share the node-wide refillSem=4 — with cocoon now parallelizing, that cap may bound pool-drain recovery. One day of A/B (4 vs 8 vs per-golden admission) once a cocoon release carrying #123 is deployed.

Deferred — reopen triggers documented

item	trigger
claimStore full-map snapshot+rewrite per mutation	>120 live claims per node
idleOnce/archiveOnce O(n) scans under the manager mutex	same
checkpoint TTL sweep re-lists the full store per tick	checkpoint count grows 10x
fastBulk decode buffer per chunk (~600MB/s GC garbage at max pull)	GC pressure shows in sustained-transfer profiles; needs a payload-ownership contract on Recv
resyncEgress serial EnsureLock at restart	many egress claims + restart-time SLA
probeReady vsockOf fallback polls vm list (subprocess) at 100ms	the fallback branch shows up in practice (it is rare by design)

Rejected with data — do not re-litigate without new evidence

• SDK pre-dialed spare connection (H-4): see above.
• fs_push chunk 256K→1M: steady-state identical (~343 MiB/s) across 3 rounds.
• nft Lock netlink-conn merge: conns are lazy in nftables v0.3.0; the two batch round trips cannot merge safely without kernel destroy-table support.
• silkd conn scanner-buffer pooling: unsafe — Close may race a blocked Recv from another goroutine (pty/port semantics).
• encodeTagged rework: already 243ns / 2 allocs per control frame.

Physical boundaries — not this repo's levers

Cold boot (~320ms, guest kernel/init), fork (~185ms, ~512MB memory snapshot I/O), CH burst restore bandwidth, and the cocoon-side 16-way residual serializer (index flock + journal + per-clone CLI process startup; per-clone p50 85ms vs 27ms serial — see cocoon#123's follow-up notes). The CLI-process model itself (one fork+exec of the cocoon binary per engine operation) is the deepest cross-repo lever here (daemon/library mode).

Related: #29 (correctness, found in the same round).

[CMGS]

Ledger update, 2026-07-13:

Item 1 (wire protocol v2) now has a full design: #34 — framing, per-stream credit flow control, negotiation/compat matrix, reconnect semantics, rejected alternatives, acceptance targets (exec RTT p50 ≤0.15ms, fs_pull ≥800 MiB/s). The fastBulk decode-buffer deferred item folds into it (v2 DATA frames carry the payload-ownership contract).

Item 2 (refill admission re-eval) is un-blocked and has its harness + baseline. make bench gained a burst stage (BURST_N concurrent clone claims, runs last) and a warm refill-recovery stage (6615de0). Post-#33 baseline @88fda81, bare metal, two rounds:

• sequential tiers unchanged: warm 0.2–0.3 ms p50, clone 33.2–39.0 ms p50, cold 354–398 ms p50, exec RTT p50 0.18–0.20 ms, fs_pull 594–612 MiB/s — feat(config): state-ownership model — persist API pools, cluster fan-out, config-drift detection #33 confirmed zero hot-path cost on hardware.
• burst 16× concurrent clones: per-claim p50 126.5/149.4 ms (vs ~35 ms sequential — the contention is real and now measured in-repo), wall 331/308 ms.
• warm refill recovery (0→6): 1638 ms in both rounds, identical to the millisecond — recovery is refill-cadence-quantized, not restore-bound. For the refillSem 4-vs-8-vs-per-golden A/B, measure/tune the refill tick first; the semaphore is unlikely to be the binding constraint at this pool depth.

Environment note: host cocoon moved master-be8653e → master-acfc815 between the pre-burst and burst rounds (parallel cocoon work on the bench host); all four rounds' sequential tiers agree, so the comparison stands.

[CMGS]

Precision note on the refill-recovery number, after reading the mechanism (pool.go:35 refillInterval=2s; refillOnce call sites are startup, SetPools, and the 2s ticker only — a claim draining the pool does not kick refill; refillOne self-chains its semaphore slot, refill.go:63-67):

recovery = tick-phase wait (0–2000 ms, dominant) + ~200–400 ms of actual work (6 clones through refillSem=4, self-chained) — and the reported 1638 ms is additionally quantized by the harness's 200 ms poll loop (1638 ≈ 8 × 204.75 ms poll iterations). The round-to-round millisecond identity is the poll quantization plus the deterministic script timeline (same tick phase at drain time), not a system constant. True recovery was in (1433, 1638] both rounds.

Refined implication for item 2: at warm=6, refillSem 4→8 changes only the ~300 ms work term — invisible under the 2 s tick wait, so an A/B at this depth measures nothing. The A/B belongs on a deep pool (target ≳50, work term in seconds); if drain-recovery latency itself ever matters, the lever is an event-driven refill kick (or a shorter tick), not the semaphore. Note the 2 s tick is deliberate: refill is fully off the claim path (a miss provisions directly — the burst row measures exactly that path), so recovery speed only affects warm-hit ratio in the ~2 s after a drain.

[CMGS]

fs_push moved: steady-state 702/741/762 MiB/s (pushbench, 128 MiB ×3, bare metal, new-silkd images @205d3f3) vs the ~343 MiB/s of record. The guest-side change is the inbound base64 decode feeding serde's borrowed slice straight to the decoder (was: an intermediate String per data frame, ~341 KB per 256 KB chunk, on musl); a rustc toolchain bump rides in the same image, so the split between the two contributors is unattributed — codec-level A/B on glibc measured only +6.5%, meaning musl allocation was most of the cost. Full round context in docs/benchmarks.md 2026-07-14 entry (e2e smoke pass, all other tiers in band). Item 1 (protocol v2, #34) remains the lever that deletes base64 entirely; its fs_pull ≥800 MiB/s target may deserve an fs_push target next to it now that push cleared 700 without protocol changes.

[CMGS]

Perf ledger update (bare-metal testbed, 2026-07-20, cocoon master-24bd46b + CH fork dev, 512M none-lane golden):

B=64 concurrent clone, restore-mode A/B (drop_caches before each round; exec spot-checks all passing):

mode	ok	per-clone p50	resident/VM	single-VM smaps
default (eager Copy)	63/64*	2311ms	331MB	Pss_Anon 337MB — full per-VM copy, zero sharing
mmap → CH CopyOnWrite	64/64	1060ms (wall 1915ms)	43MB	Anon 41MB + shared page-cache file pages
ondemand (uffd)	64/64	2083ms	514MB	Anon 525MB — UFFDIO_COPY materializes everything incl. sparse holes

* one slot lost to an instrumentation accident (strace on the CLI wedges Go binaries; don't).

Conclusions for the ledger:

1. CopyOnWrite wins both axes at concurrency: 2.2× latency, 7.7× density. cocoon still defaults to eager copy (restore-mode ""|copy → CH default) — switching the clone default to CoW is the single biggest storm lever currently unwired.
2. ondemand is a density trap for pooled clones (worst of the three at B=64) — keep it for single cold restores only.
3. Caveat to wire alongside the default switch: CH silently falls back CoW→eager on hugepage/shared/NUMA-bound regions, and cocoon auto-enables HugePages whenever host nr_hugepages>0 — the fallback is invisible today (warn in CH stderr only). Needs surfacing as an error or metric.
4. Even on CoW, B=64 per-clone p50 ≈ 1.06s — the residual is CLI-per-op + CH process launch + API sequence, i.e. the daemon/library-mode item already on this ledger.
5. Cross-host note: DirectClone hardlinks the memory file (page-cache sharing intact); the stream-clone path materializes a full per-clone copy — C1M fleet design should land one golden per host and DirectClone-fan-out locally.

Same round: egress-lane concurrent clones now clean 16/16→64/64 after cocoon#142 (tap-attach race), egress ≈ none +10%; FC lane unblocked by FC v1.16.1 (cocoon#140) and upstream main (1.17.0-dev) also passes the vsock repro.

[CMGS]

T1 closed (bare-metal testbed, FC none-lane per lane assignment, cocoon master-24bd46b, FC 1.17.0-dev main build, 512M guests).

Storm scaling (100% success + exec spot-checks at every B):

B	wall	p50	throughput
16	183ms	108ms	87/s
32	509ms	181ms	63/s
64	1063ms	609ms	60/s
128	6033ms	3048ms	21/s
256	18059ms	13413ms	14/s

Density ceiling: 576 × 512M sandboxes on the 60GB host — guard-stopped on MemAvailable (>3GB reserve), zero clone failures, zero OOM, exec verified per wave and at final census. Marginal footprint ~107MB/VM under pressure (84MB/VM at B=64). Nominal RSS sum 80.8GB > physical RAM — MAP_PRIVATE page-cache sharing doing its job. Extrapolation: ~9k+ VMs on a 1TB box → the T2 8-12k/box target is realistic on hardware, and ~110 × 1TB boxes for 1M at 512M guests.

Latency attribution (why p50 grows with B):

• per-clone CPU: 12ms (/usr/bin/time -v across a B=64 wave; 64 clones sum 0.76s user+sys) — clones spend ~98% of wall waiting;
• CLI floors at 64-way concurrency: version p50 12ms, vm list p50 12ms — process startup/config/DB-read are negligible;
• sequential single clone: ~31ms;
• --from-dir (bypasses snapshot store) B=64 p50 473ms vs store-path 609ms — store layer ≈ 20-25% of the wait;
• superlinear wall growth B64→B256 (5.7×, 17× for 2×, 4× work) + wave slowdown at high resident-N ⇒ the dominant wait is the shared-root serialization (2ms-poll flocks + whole-file JSON DB rewrites that grow with resident count) with O(N) per-op character. /proc/locks sampling shows zero waiters — expected, the locks are non-blocking polls, so neither CPU nor lock-wait counters see it.
• isolated-root A/B leg was blocked by the snapshot-metadata path-trust hardening (base-layer path outside --root-dir is rejected — correct behavior, bad for this experiment).

Ledger actions:

1. daemon/library mode: priority UP — the O(N) shared-state cost collapses storm throughput exactly where T2 needs 8-12k sustained; prerequisite measurement = a lock/DB-timing debug build to name the exact serialized sections.
2. journal delta-writes: unchanged MUST (same O(N) family).
3. C1M harness note: per-host golden dir + vm clone --from-dir is both the page-cache-sharing design AND a free −25% storm latency (skips the store layer entirely).
4. FC lane needs no restore-mode wiring (File backend natively MAP_PRIVATE lazy) — only the CH egress lane needs restore_mode: mmap in deployment config (guard: cocoon#144).

[CMGS]

Phase decomposition of the storm superlinearity (B=64 ×3 consecutive, no cleanup between — resident N = 0/64/128; per-clone anchors: exec-stamp → ops.lock ctime → FC /proc starttime → api.sock ctime → vsock.uds ctime → exit-stamp):

Fixed B=64, only resident N grows: wall 1009 → 3653 → 4914ms. The B128/256 superlinearity is the resident-count effect, not concurrency per se.

Round-3 phase split (64/64 complete samples):

phase	p50	p90
P1 cli+locks+db-read→runDir	99ms	168ms
P2 storage-prep + record insert + FC spawn	2347ms	3902ms
P3 FC api up	249ms	252ms
P4 snapshot load (the actual restore)	14ms	22ms
P5 resume + reseed + record finalize	867ms	2013ms

Named culprit: firecracker/db/vms.json whole-file RMW under the polled exclusive flock, plus a full .prev copy on every write (measured 113KB → 226KB → 339KB at ~1.8KB/record, i.e. 3×339KB of IO per write at N=192), at least twice per clone (insert in P2, finalize in P5). Cost per RMW ∝ N ⇒ storm cost O(N·B).

Fix ladder for T2: (1) cocoon#132 — drop the second RMW (halves lock-held IO); (2) delta/segmented index instead of whole-file+.prev rewrites (kills O(N)); (3) daemon/library mode with in-memory index (removes the file RMW entirely). The restore path itself (14ms) and CLI startup (12ms floors) need nothing.

[CMGS]

T1 re-run on the sqlite meta backend (bare-metal testbed; cocoon master-5714e9e with meta P0–P3 landed, CH dev-7901a518, FC v1.16.1; same binary A/B, json round then offline meta convert then sqlite round; all storms 100% success + exec spot-checks):

metric	json (same bin)	sqlite	Δ
3×B64 ladder r1 (N=0)	1128ms	966ms	−14%
ladder r2 (N=64)	3533ms	2238ms	−37%
ladder r3 (N=128)	4032ms	3216ms	−20%
pre-spawn phase p50 @n=128	1795ms	457ms	−75%
B=128 wall	4716ms	2886ms	−39%
B=256 wall	12440ms	9553ms	−23%
CH-egress B64 (mmap)	690ms	717ms	flat (not metadata-bound)

Also validated on hardware: P0 non-regression (json engine behind the meta API reproduces the legacy ladder within noise: 1128/3533/4032 vs legacy 1009–1189/3533–3874/4016–4914) and the conversion procedure behaved per design (both json generations renamed aside with timestamps across all namespaces; records served intact from sqlite).

Design acceptance anchors, honestly scored: r1 ≤1.5s ✅ (966ms); later rounds ≤1.3×r1 ❌ (2.3×/3.3×). Cause is not a metadata shortfall but a bottleneck relocation: the restore(load) phase segment (FC api-up → vsock.uds), historically a constant ~15ms, now carries p50 599ms / p90 1622ms under sqlite. The metadata queue that used to stagger arrivals (pre-spawn 1795ms) is gone, so all 64 clones hit the next serializer nearly simultaneously — the drive-redirect window (per-golden clone-locks + bind-mount setup, kernel-global work). Confirmed no meta-store writes exist in that window (redirect uses dir flocks only).

Next ledger target therefore shifts: the drive-redirect critical section is now the dominant storm serializer at B≥64. Candidates: per-clone mount-ns cost, clone-locks hold time, symlink-fallback retirement (the TODO already in clone.go). Instrumentation of holdRedirectDirs is the next measurement step before any fix.

[CMGS]

Attribution of the post-sqlite restore(load) p90 → it's cocoon, not FC, not a control-plane daemon. Hardware-measured (bare-metal testbed, sqlite backend, FC none-lane 512M hibernate golden):

restore(load) phase (api.sock→vsock.uds) scales purely with concurrency:

B	p50	p90	max
1	5ms	5ms	5ms
16	23ms	41ms	59ms
64	176ms	539ms	904ms

Three independent signals isolate the serializer:

1. Not FC: uncontended (B=1) snapshot/load is 5ms; FC's own restore work is not the cost.
2. Not the kernel mount lock (first suspect): a microbench of 64 concurrent Unshare(CLONE_NEWNS)+Mount(MS_REC|MS_PRIVATE,/) completes in 2ms (0.1ms/op), and all 64 FC processes are in the HOST mount namespace — so the lock-free bind fast path never ran.
3. cocoon's per-golden source-disk flock: flock waiters during the B=64 storm peak at p90 29 / max 51.

Causal chain, verified in source + on disk: a hibernate golden's baked source COW path (…/run/firecracker/<srcVM>/cow.raw) is MISSING once the source VM is removed post-hibernate → bindableRedirects requires a regular file, returns nil → the clone falls out of the lock-free bind path into the symlink fallback, which wraps the whole loadSnapshotFC in withSourceWritableDisksLocked (a flock on the source golden's writable disks). 64 clones of one golden share those paths → one lock → full serialization.

This is exactly what the removal of the O(N) metadata serializer (json→sqlite) exposed: pre-spawn dropped −75%, un-staggering arrivals so they all hit this next lock at once.

Fix is already scoped as a TODO in hypervisor/firecracker/clone.go:150: make dead-source (hibernate golden) clones eligible for the lock-free bind path by binding the clone's fresh disk over a placeholder mountpoint at the missing source path, instead of the locked symlink fallback. That removes the per-golden serializer from the C1M clone-from-hibernate-golden workload — the dominant B≥64 storm cost now that metadata is off the critical path.

[CMGS]

Corrected T1 baseline: both serializers removed (sqlite metadata + keep-source bind path). Same bin (cocoon master-5714e9e, FC v1.16.1), FC none-lane 512M, 100% success + exec-verified. Keep-source = golden's source VM kept stopped (not rm'd), so its baked COW path exists and clones take the lock-free bind redirect path instead of the symlink+flock fallback (attribution proven in the prior comment).

metric	legacy json	sqlite (rm src)	sqlite + keep-src
3×B64 ladder r1 (N=0)	1128ms	966ms	813ms
r2 (N=64)	3533ms	2238ms	1636ms
r3 (N=128)	4032ms	3216ms	2083ms
restore(load) p50 @n=128	15ms	599ms	49ms
restore(load) p90	—	1622ms	74ms
B=128 wall	4716ms	2886ms	2097ms
B=256 wall	12440ms	9553ms	5619ms (14/s→46/s)

The two fixes are additive (B256 12.4→9.6→5.6s). restore(load) collapses 22× (p90 1622→74ms) — confirming both the attribution and that keep-source is a valid, zero-code fix for the C1M clone-from-hibernate-golden workload. Per CMGS: keep-source is the interim answer; the placeholder-mountpoint TODO stays parked until FC upstream lands dynamic drive replacement (PR #5774), at which point the redirect dance retires entirely.

Next bottleneck is now CLI-per-op, not storage: round-3 phase split with both serializers gone shows the two largest segments are pre-spawn (p50 509ms: CLI process start + dir/COW prep) and post-uds (p50 461ms: resume + reseed + finalize), while restore(load) is 49ms. These are the per-process cost of 64 independent cocoon CLI invocations — exactly the daemon/library-mode lever (T0, this ledger). CH-egress mmap flat (~760ms, not storage-bound). The layered plan holds: each serializer removed surfaces the next — metadata → drive-redirect → CLI-per-op.