sandboxd: cross-node snapshot placement — operator-token lifecycle, gossip redirect, peer heal - #52
Merged
Merged
Conversation
Hibernate, fork, promote and checkpoint all resolved the sandbox through its per-sandbox token, so only the client that made the claim could ever use them. An aggregated control plane holds the fleet api_token and nothing else: keeping one secret per sandbox would turn its O(nodes) storage into O(sandboxes), the property the whole design rests on. Add operator variants that resolve by id, authorized by the node's root api_token before the call — the pattern ReleaseOperator already established, so release is no longer the one verb a control plane can reach. A tenant token still takes the per-sandbox path unchanged. Also add an explicit wake verb: waking was previously a side effect of opening an agent connection, which left a control plane no way to resume a sandbox it was not about to talk to. And a single-sandbox read, so a reconcile does not have to scan the whole-node listing.
Checkpoints are node-local, so a branch issued to a node that does not hold the record simply failed — "snapshot on A, branch from B" did not work at all. Gossip the checkpoint ids each node holds, alongside the warm counts and template hashes already published, and answer a branch this node cannot serve with a redirect to one that can: the same 200 + redirect contract a warm-miss claim uses. The record does not move, so the clone still runs on the node whose disk already has the data, on its local reflink fast path — cross-node correctness for one extra round trip and zero bytes transferred. CheckpointClaimRequest gains no_redirect so the retry at the target resolves locally instead of bouncing between two nodes.
…ranch Redirect answers the common cross-node case without moving data, but it cannot answer when the owning node is gone, draining, or full — the choice there is between paying one transfer and failing the request. Add an opt-in peer store backend (checkpoint_peer_heal) that pulls the record from a node that gossiped it, publishes it locally through the same atomic staging path a local capture takes, and serves the branch from there. The cost is paid once: the node then gossips the record itself and serves later branches at local speed. The transport is sandboxd's own, deliberately not cocoon's mover — that one is bound to the engine's VM store layout, and reusing it would tie a checkpoint's mobility to the engine's release cycle. This moves exactly one thing, a store record (export/ plus meta.json), as a tar between two nodes that already authenticate with the fleet token. Entry names are validated against traversal and the stream is size-capped: an authenticated peer is still not allowed to write outside the destination this node chose. Also add a per-sandbox stats read. It is the only per-sandbox usage surface — /metrics is node- and pool-scoped by design — and it reports whether memory was measurable at all, so a hibernated sandbox reads as "not measured" rather than as idle.
Tar had no production caller — TarRecord is the only path the blob handler uses — so the round-trip tests now drive tarInto directly. parseMemSize existed to parse SizeSpec.Memory, whose whole domain is four literals from one table; the table carries MemoryBytes instead. Untar already caps a record by accumulating hdr.Size, and archive/tar bounds each read to the entry size, so writeFile's LimitReader guarded nothing.
Both gates were clean on main and this branch broke them: golangci-lint reported 8 issues and asl 57. Restores zero on both, dual-GOOS. Layout rules applied: one top-level const block per file and the package var block above the types (peer/transport.go had three const blocks and declared ErrNotFound below an interface); the compile-time interface check sits immediately above its type (peer/peer.go); tests precede helpers, which the new tests broke in server_test.go, mesh_test.go and peer_test.go; exported above unexported (pool/operator.go). isRootToken's godoc had been orphaned — rootRequest was inserted between the comment and the function, so go doc rendered it against the wrong symbol. handleSandboxVerb's two inline func types are named (tokenVerb, operatorVerb) so the signature reads on one line. gosec's file-inclusion and SSRF findings are annotated with why the input is already pinned, matching how the dir and pool stores annotate theirs. Comments: 208 added lines against 603 of code was 33%, twice this repo's own 16%. Now 19%, cutting design narrative that duplicated the commit messages and per-field comments that restated the field name.
Checkpoints already skips archive wake images, so CheckpointIDs' own !c.Archive test could never be false — and the godoc claimed that dead condition was what excluded them. forkResolved/promoteResolved opened with `id := sb.ID` only to keep two pre-existing error sites compiling after the claim/resolved split; checkpointResolved, split the same way, reads sb.ID directly. Sandbox rebuilt the SandboxSummary literal Sandboxes already had.
CheckpointIDs reads the store but built its own context.Background while both call sites (the gossip tick and the template notifier) hold main's ctx. TemplateOwners and CheckpointOwners were the same lock/scan/truncate body with only the NodeState field swapped; both now wrap recordOwners. UpdateSelf's godoc still enumerated only pools and templates.
Its two callers hold main's ctx; only merge, behind memberlist's ctx-less delegate interface, still builds its own.
memberlist's delegate interfaces have no context parameter, so merge logged with a fresh Background; New now stores the daemon's ctx for those callbacks.
The peer-heal wrapper was installed as the claim path's own store, so a branch of a record a live owner could serve pulled the whole multi-GB image before the handler's zero-byte redirect branch ever ran — the redirect was dead code exactly when it should win. The healer now lives on its own Manager field and the server sequences the tiers explicitly: local claim, then redirect while a gossiped owner exists (and the request is not already the redirect retry), and only then ClaimCheckpointHeal. The blob endpoint reads the local store alone, so a peer's pull can never recurse into another pull. Gossip no longer lists the store every tick: an in-memory id set mirrors tplSet — seeded from one scan at boot, kept current by publish/delete/heal — so the 1s tick costs a map walk, not N meta reads (measured 15.4ms/tick at 1000 records before). A shared (s3) checkpoint store gossips nothing and installs no healer: every node already resolves every record. Concurrent misses of one id share a single transfer via singleflight, the idiom the s3 backend already uses. peer.Store embeds the local backend instead of forwarding five methods by hand. checkpoint_peer_heal now requires mesh.cluster_key: the heal path presents the fleet api_token to gossip-learned addresses, so gossip must be authenticated before that credential rides on it.
A branch of a checkpoint the node no longer holds is answered with 200 and the owning peers' addresses — the same contract the generic claim path has followed since the mesh landed. The checkpoint path passed that reply straight to handleFrom, so Go built a sandbox with an empty id and token and Python raised on the missing key. Both now retry each candidate with no_redirect set and bind the sandbox to the address that served it, so the redirect tier is reachable from a client at all.
Six verbs each carried a second Operator method that differed only in how it found the sandbox, and the server decided "root api_token or per-sandbox token" in two different shapes across five handlers. A Cred carries the resolved authority instead: the server builds it once per request, resolve looks the claim up by token or by id, and the Manager interface loses six methods. resolve authorizes off the manager mutex, unlike the old Release which held it across the token check and the delete. releaseResolved therefore re-checks under the lock that the claim it was handed is still the live one, so a second release racing on the same id is a miss rather than a second teardown — TestReleaseAfterResolveTearsDownOnce fails without that check. An empty token is now rejected explicitly rather than compared, so a caller presenting nothing can never match a claim.
Warm counts are volatile, so they belong in a 1s gossip tick. Checkpoint ownership is stable but tenant-created and unbounded, and it rode the same tick: measured on bare metal, a 20-node view at 2000 records per node cost 4.94ms of serialize plus deserialize and ~5MB of payload every second, per node, growing linearly with a number no operator controls. Ownership is now asked for on the rare cross-node miss instead: a HEAD against each peer's blob endpoint, in parallel, capped at three answers. The answer is live, so a redirect can no longer point at a node that lost the record between two gossip rounds — which also retires the in-memory id set, its boot seeding, and the truncation that could hide a real owner behind two stale ones. The probe carries no credential. The checkpoint id is itself the unguessable capability to branch, so an unauthenticated HEAD tells a caller only what it already knows; the GET that streams the record keeps its token.
…lica Six defects in the heal path, found once the tiers were sequenced properly. Quota was checked before the local record lookup, so a full or draining node answered 429 instead of "not here" — the handler never reached the redirect or heal tiers, which is exactly the case they exist for. The lookup is a cheap local read, so it goes first now; the heal path keeps quota ahead of the pull, since resolving a record there means moving a guest memory image. GET on the blob endpoint took any tenant token, streaming a raw checkpoint — guest memory and disk — outside the tenant scoping every other checkpoint verb enforces. Its only caller is the peer puller, which holds the fleet token, so it is operator-only now. The HEAD probe stays unauthenticated: the id is the capability, and it answers nothing a caller does not already know. Healing published outside the record lock, against the store's own contract that same-id publish and delete are the caller's to serialize; a delete or TTL sweep racing a heal could resurrect a record or swap a directory under a reader. The heal now runs entirely under the id's write lock, and the peer package stops pretending to be a store: it pulls into a staging dir the manager owns and publishes nothing itself. A pulled record was published unread. A buggy or hostile peer could install an unreadable record — which then suppressed every later heal — or one whose meta named a different checkpoint. The staged copy is now checked for shape and identity before it is trusted, and a rejected copy moves on to the next owner rather than failing the branch. Transfers were bounded per owner, not overall, so two slow owners cost an hour, and nothing capped concurrent heals node-wide. One budget now covers the whole attempt, and a node runs at most four at once. Deleting a checkpoint only removed the local copy, so a healed replica kept serving branches of a record its owner had deleted. A fleet-scoped delete broadcasts to peers; a forwarded one does not, which is what stops the loop. A healed copy carries the source's CreatedAt, so every node's TTL sweep ages it out at the same moment without coordinating.
Added comments were 27% of added code against a repo-wide 16%. The excess was design narrative that repeated the commit messages and justification prose that outlived the decision it argued for; what stays is godoc on the exported API this branch adds and the one-line reasons the code cannot carry itself.
Peer heal makes revocation two-legged: a delete broadcasts to peers, and the TTL sweep ages out any copy the broadcast missed — a healed record carries the source's CreatedAt, so every node expires it at the same moment. Both legs had silent failure modes. checkpoint_ttl_hours = 0 (the default) cuts the second leg entirely: a node offline during the broadcast would keep a deleted checkpoint branchable forever. Heal now requires a nonzero TTL, making the revocation window finite and operator-chosen. A record that should live indefinitely is what promote is for. The TTL also had to match across the fleet for "expires at the same moment" to hold, and nothing checked that: it now rides ClusterDigest, so a node sweeping on a different clock is called out at join time instead of quietly outliving its peers' deletes.
Heal is the first thing that can make a checkpoint id live again, and two places still assumed an id, once gone, stays gone. DeleteCheckpoint decided "not here" before taking the record lock, on the comment's own reasoning that a checkpoint is single-publish and immutable. A delete landing while a heal was mid-transfer therefore saw nothing, returned unknown-checkpoint without ever contending for the lock, and the heal published moments later — a client that deleted a checkpoint could find it back. Only the id's format is checked before the lock now; whether the record exists is decided under the same lock heal holds. The record locks were also evicted from the map while still held, so a waiter kept the old mutex while a newcomer was handed a fresh one and both proceeded to mutate one record. Acquisition now counts references, and a lock is dropped only once none remain — every call site releases the mutex before releasing its reference, which is the ordering that makes the eviction safe. Both fixes are pinned by tests that fail without them: restoring the unlocked pre-check makes the delete return early against an unpublished heal, and evicting without checking the count strands a live holder. The SDK's redirect fallback now reports both halves of a failure — the peers' errors say why the claim left the origin, the origin's says why coming back did not help — instead of dropping the candidates' on the floor.
The assertion checked both halves of the combined error but the fixture gave the candidate an unlabelled 429, so a regression that dropped the candidate's message would still have matched on the origin's alone.
…lish Holding the id's write lock across a multi-GB transfer put every other caller for that id behind up to the whole heal budget, on a plain mutex no cancelled client could leave. The transfer now runs unlocked inside a manager-owned flight: waiters share one pull and leave the moment their own context is done, while the transfer itself is detached and still finishes, so the record is paid for once regardless of who stays. Moving the lock off the transfer reopens resurrection in a subtler form. A delete arriving mid-staging correctly finds nothing and answers not-found, but the heal's locked finish would find nothing either and publish anyway — absence alone cannot say whether nothing happened or whether a delete just answered for this id. So a delete that finds an id absent vetoes any heal pending for it, and the heal reads that veto under the same lock it decides to publish under. Both maps are bounded by the concurrent-heal cap. The healer no longer dedups pulls of its own. Its caller supplies the staging directory, so two concurrent pulls are two destinations and sharing one result leaves a caller publishing a directory nothing ever wrote — dedup belongs to whoever owns the destination, which is now the manager. Each fix is pinned by a test that fails without it: dropping the veto resurrects a deleted checkpoint, a blocking receive strands a cancelled caller, and restoring the healer's own dedup leaves the second caller's staging directory empty.
The redirect target came from a gossiped warm count refreshed once a second, and nothing recorded how many claims this node had already sent against it. Every claim inside one refresh window therefore chased the same stale number: a peer advertising four warm sandboxes received the whole burst, served four instantly, and cold-provisioned the rest one by one — each retry arrives with no_redirect set and cannot bounce again. Observed in production as sandbox allocation slowing roughly tenfold the moment the mesh was enabled. Candidates now debits the winner's count in the local view, so a window's redirects are bounded by what the fleet actually advertised; the peer's next adopted state resets the debit. Only the winner pays — the client walks the list in order and stops at the first success. The debit also rides this node's own outgoing gossip, which is at worst conservative and is corrected by the peer's next epoch. Undebited, the regression test redirects all forty claims of a burst at a peer that advertised four; debited, exactly four leave the node.
The HEAD probe was an open metadata oracle: ids carry 64 bits of randomness, every syntactically valid request cost the target a disk read, and nothing rate-limited it. On an encrypted mesh the probe now carries an HMAC over the id and a coarse time bucket, keyed off a probe-specific derivation of cluster_key, verified before any disk is touched; a keyless mesh keeps the old capability-only posture, since a redirect-only fleet has no shared secret to sign with and losing the probe outright would break it. Peer heal now also requires a nonempty api_token: an open node with heal enabled left the raw blob GET reachable with no credential at all. The fan-out stops as soon as enough owners answered, so one hung peer no longer taxes a redirect that already has candidates. Concurrent probes for one id share a single fan-out, and a short positive cache serves a hot id's repeat redirects without re-asking the fleet — measured before, ten misses of one id against twenty peers cost two hundred HEADs; now the first fan-out answers them all until the entry expires or a local delete evicts it. Misses are never cached, so a record that appears a moment later is found. A redirect wants a couple of live candidates but a heal wants the widest source list a few bad owners cannot hide, so the two callers stop sharing one capped answer.
The eviction landed with the probe boundary; this is the server-level pin that was written for it and lost to a worktree cleanup race.
The claim, blob, probe, and delete surfaces this branch added were undocumented or described mid-flight: the probe is signed on an encrypted mesh (and the "hardening is tracked" placeholder goes away with the gap), peer heal needs a token, a TTL, and an encrypted mesh, the heal-cap 503 carries a Retry-After, and a delete is eventual best-effort cleanup bounded by the checkpoint TTL — not a fleet-wide revocation, and the docs now say so instead of implying otherwise.
This reverts commit 60ee474. Measured on a real two-node mesh, the debit is a regression, and the reasoning behind it was wrong. A burst of forty claims at a node holding a golden but no warm sandboxes, against a peer advertising six warm: no mesh at all 0 redirected, 40 local, p50 1173ms mesh, before the debit 35 redirected, 5 local, p50 601ms mesh, with the debit 6 redirected, 34 local, p50 1210ms So the mesh was not slowing allocation down — it was halving it, by spreading forty clones across two nodes instead of queueing them on one. The debit bounded redirects to the peer's advertised warmth, which is mechanically what it claimed to do, and thereby hoarded every overflow clone on the origin: thirty-four of them serialized behind that node's provision limit, landing back at the no-mesh number. The premise was the error. A redirect is not only a bid for a ready sandbox; it is also how provisioning load reaches a peer that has the golden and idle capacity to serve it. Capping it at the warm count throws that away. Whatever slowdown motivated this needs to be measured on a real fleet first — this branch is not the place to guess at it again.
…delete A second review round found two gaps in the ownership probe. A checkpoint usually lives on one node, so the three-owner cap is rarely reached and the fan-out would block on every missing peer's full timeout before returning the single owner it already had; it now returns on a short grace window once the first owner answers, while a genuine all-miss still waits out the timeout because it cannot conclude nobody holds the record until every peer has answered. A delete's cache eviction also raced an in-flight probe: Forget cleared an entry the probe had not written yet, then the probe cached a result naming the node that was just deleted, leaving a stale positive for the cache TTL. Forget now bumps an epoch under the cache lock, and a probe only caches its result if that epoch still holds — so a delete landing any time during the probe keeps its result out of the cache. Each fix has a test that fails when reverted. The delete-window docs are corrected to match: expiry is per-replica, best-effort, bounded by the TTL plus a sweep interval, not a single fleet-wide wall-clock moment.
…third review found A full-diff review pass turned up three defects the feature work introduced and two gaps in the round-3 probe fixes. Operator reads raced lifecycle writes. Sandbox and Stats took a claim pointer from byID, which releases m.mu, then read HibernateSnap, VMName, ArchiveCk and Deadline — fields hibernate and archive rewrite under m.mu. Both now snapshot under the lock; Stats keeps only the /proc read, which touches no claim state, outside it. A -race test that hammers Sandbox/Stats against Hibernate/Wake trips the detector when the lock is removed. Healed-record validation accepted a shell. It checked the meta parsed, the id matched, the record was not an archive image, and the export directory existed — but not that the key was branchable or the export non-empty. An empty export clones to nothing, and because the healer stops at the first accepted owner and publishes immediately, one such copy suppressed every better owner and poisoned later local claims. Validation now requires a valid key and a non-empty export, so a bad copy is rejected and the next owner tried. The record lock could leak. When a delete's recDoneEvict ran while another holder still referenced the id, it left the entry in place but recorded no pending eviction; the ordinary holder's later recDone then reached zero without removing it. The eviction is now deferred to whichever call drops the last reference. Two probe gaps from the previous round: the grace window that bounds a single-owner redirect was also truncating HealOwners, which must stay exhaustive so a fast owner cannot hide a slower valid source — heal now fans out with no grace. And the delete handler evicted the probe cache before the record was gone, so a probe racing the delete could cache a positive the delete then cleared; eviction now runs after the delete. The TTL-window docs are corrected once more: the checkpoint TTL is the expiry eligibility point, and a node whose hourly sweep keeps failing holds its replica until one succeeds — not a hard ceiling. Each code fix has a test that fails when reverted.
…inish the docs A fourth review pass caught the residue of the last round's validation work. An egress-lane key passes Key.Validate but Capturable is false, so claimLoaded refuses every branch of it; a healed record carrying one would publish, poison the id, and suppress a good owner. Validation now rejects it. An export directory with no regular file — empty, or only empty subdirectories — clones to nothing and is rejected too; the byte content beyond that is cocoon's own format, not sandboxd's to validate, so the check stops at "has content". The checkpoint_ttl_hours line in deploy.md still called the TTL a hard bound; it now matches the other docs — the eligibility point, extended by sweep failure. Test and comment fixes the same pass flagged: the missing-export test built a record with an empty key, so it failed at the key check instead of reaching the export branch; the puller fixture's comment still said it wrote an empty export; and the hibernated-stats test's comment had been left above the new operator-race test. New tests cover the egress-key and contentless-export rejections, both mutation-verified.
A source-side filesystem error between two files leaves TarRecord's deferred Close writing a valid tar footer over a short archive — so the receiver saw a well-formed but incomplete record and published it. On the documented FUSE/NFS checkpoint backends a transient EIO or ESTALE mid-walk is a routine, not adversarial, event, and a published incomplete record then answers probes and serves its own /blob, so the next node to heal from it inherits the same truncated copy — the fault spreads instead of staying put. TarRecord now writes a completion marker as its last entry, after the export has streamed whole; Untar requires it before returning success and never writes it to disk. A transfer cut short lacks the marker and is rejected, so the healer clears its staging and tries the next owner — the transfer converges on a good copy instead of persisting a bad one. The blob handler's comment claimed a mid-stream failure truncates the tar; it does not, and now says what actually guards completeness. This is a transport-completeness signal, not validation of the export's bytes: those are cocoon's snapshot format, and a peer shipping semantically corrupt but well-formed bytes remains out of scope (an authenticated peer would have to be compromised, and the failure is a loud provision error, not silent corruption).
Comment narratives cut back to their load-bearing invariants (checkpoint delete/heal docs, probe fan-out docs, healer history note); the three scheme-default URL builders collapse into one checkpointURL helper; resolveScope reuses isRootToken; a stranded tarDir doc rejoins its func.
…s built
wake, GET /v1/sandboxes/{id}, and GET /v1/sandboxes/{id}/stats existed only
in the route table; hibernate's auth line missed the operator elevation it
shares with release; cluster.md described the HEAD fan-out as capped at 3
peers when the cap is on the answer, not the probe.
Mid-rotation a candidate that has not adopted the new token answers 401 before its handler runs, while the origin proved the token valid by issuing the redirect — so the claim falls back to the origin instead of failing for the whole rotation window with a misleading credential error. 403/409 stay definitive.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes a checkpoint reachable and branchable from any node in a mesh, and lets a
stateless control plane holding only the fleet token drive a sandbox's
lifecycle. Three feature commits set the shape; the rest of the branch is the
review that made them safe.
1. Operator-token lifecycle.
hibernate/wake/fork/promote/checkpointresolved a sandbox through its per-sandbox token, so only theclient that made the claim could reach them. An aggregated control plane holds
the fleet
api_tokenand nothing else — one secret per sandbox would turn itsO(nodes) storage into O(sandboxes). The verbs now take a resolved credential
(
pool.Cred): the server decides root-vs-tenant once per request, so a singlemethod per verb replaces the six paired
*Operatorvariants the first cutadded. An explicit
wakeverb is added — waking was previously only a sideeffect of opening an agent connection — plus a single-sandbox read and a
/statssurface that reads the VMM's RSS by the pid cocoon already reports.2. Redirect — find the node that holds the record. A branch issued to a
node that lacks the checkpoint is answered with the owning peer's address: the
same
200 + redirectcontract a warm-miss claim uses. The record does notmove, so the clone runs on the disk that already holds it, on its local reflink
fast path — cross-node correctness for one round trip and zero bytes
transferred. Ownership is found by a live parallel
HEADprobe of meshpeers, not a gossiped id list: checkpoint ids are tenant-created and unbounded,
and an early version rode them on the 1s full-state gossip (measured 4.9 ms of
serialize/deserialize and ~880 KB of payload per tick at 2000 ids/node × 20
nodes). The probe answers from a short per-id positive cache that a local
delete evicts, so a hot id costs one fan-out, not one per claim.
3. Peer heal (opt-in,
checkpoint_peer_heal). When no owner can beredirected to — gone, draining, or full — the node pulls the record from a peer
as a tar over the control-plane port, validates its shape and identity,
publishes it through the same atomic staging path a local capture takes, and
serves the branch. Paid once: the node then answers probes for it. The transfer
runs outside the record lock in a manager-owned singleflight so a cancelled
caller can leave while one detached pull completes; a delete arriving mid-pull
vetoes the publish under the same lock the publish decision takes. Enabling it
requires an encrypted mesh (
cluster_key), a nonemptyapi_token, and anonzero checkpoint TTL — the credential rides to gossip-learned addresses, and
the TTL is the bound on how long a replica can outlive a delete that missed it.
Delete is eventual cleanup, not fleet revocation
A delete removes the local record and best-effort broadcasts to peers so a
healed copy does not outlive it. A peer offline during the broadcast keeps its
copy until the checkpoint TTL ages it out, so the worst-case window in which a
deleted checkpoint stays branchable by an id-holder is
checkpoint_ttl_hours—which now rides
ClusterDigest, so a node sweeping on a divergent TTL is calledout. This is a deliberate scope choice over a durable tombstone; the API and
cluster docs state it plainly.
The review is most of the branch
Two external review rounds (the same checkpoint design, re-examined against the
implementation) plus a batch-end quality loop drove twenty-one commits beyond
the three features. What they closed, each pinned by a test that fails without
its fix:
so a branch of a record a live owner could serve pulled the whole image
before the redirect branch ran. Heal now lives on its own field and the
server sequences the tiers explicitly.
lock and a heal published after it; the record lock could split identity on a
republished id. Both are now serialized; both have regression tests that
resurrect a deleted checkpoint when reverted.
GET /blobtook any tenant token; theHEADprobe was unauthenticated with 64-bit ids and a disk read per request. GET is
operator-only now; HEAD carries a cluster-key-derived MAC on an encrypted
mesh.
sandbox, Python raised — so the redirect tier was unreachable from any
client. Both now follow it, and fall back once to the origin after every
candidate fails retryably, which is what makes the draining/full case work.
redirect — the common case — waited out every missing peer's timeout before
returning; a delete's cache eviction could race an in-flight probe and leave
a stale positive. Fixed with a grace window and an epoch fence, each with a
test that fails when reverted.
returning 503 +
Retry-After), record validation before publish, and theusual quality pass —
ctxpropagation, one credential type for six verbpairs, comment budget cut from 33% of added code to the repo's own 16%, and
the lint/asl gates restored to zero on both GOOS.
One commit is a revert: a redirect-debit meant to bound a burst was
disproved on a real two-node fleet (mesh spreads clone load and halves burst
latency; the debit hoarded overflow clones on the origin and doubled it) and
backed out, its message carrying the measurements.
Evidence
sandboxdsuite green under-race,go vetclean,GOOS=linuxcross-build ok,make go-lintandaslzero on both GOOS acrossevery module;
sdk/gorace tests andsdk/pythonruff + 135 pytest green.mainbase): pool store benchmarks geomean +0.01%, wire codec −0.18%, all within
±1%. End-to-end on the same host: warm-claim p50 0.3 ms unchanged,
clone-from-golden 0.3 ms, cold boot flat; the multi-step smoke suite (exec,
files, git, pty, hibernate, fork, promote, checkpoint, port, …) passes, and
the reap/restart regression legs pass.
node vs 8-node no-mesh vs 8-node mesh, host-wide provisioning concurrency and
VM count held constant) shows the clone hot path unaffected by gossip
(50–75 ms/VM across all three); the only large costs are cold golden builds
contending on one host's disk — independent of the mesh, and absent on a real
one-node-per-host fleet.