e2b-compatible API + sandbox lifecycle verbs (pause/resume/fork/snapshot) - #1
Merged
Conversation
Adopting cocoon meant rewriting against the Kubernetes agent-sandbox API even for callers who already had working e2b code. Add an opt-in REST surface that speaks the e2b wire contract, so an unmodified e2b SDK (JS or Python) claims from these warm pools by pointing E2B_API_URL at the apiserver. It is a translation layer, not a second control plane: every verb lands on the same scale.SandboxStore the aggregated API uses, so an e2b create is the identical node-local claim and what it creates stays visible to `kubectl get sandboxes`. Nothing is stored here — sandbox identity is the sandboxd claim id the owning node already assigns, so DELETE releases exactly the claimed microVM rather than guessing by name. Covered: create (templateID -> pool template, allow_internet_access -> network lane, 503 on a drained pool so the caller retries), list/get, delete, timeout and refreshes. It runs on its own listener behind X-API-KEY and refuses to start unauthenticated unless anonymous access is passed explicitly, so a misconfigured deployment cannot silently expose an open claim endpoint. envdVersion is always reported: the SDK version-compares it and kills the sandbox it just created when it cannot parse one. Limits — envd host routing, unimplemented pause/fork/snapshot verbs — are documented in docs/e2b-compat.md.
The store could place and release a sandbox but nothing else, so everything a caller might do to one after delivery had no path through the control plane. Add the post-claim verb set to SandboxStore and the sandboxd client behind it. Every verb addresses an already-delivered sandbox on a known node, so there is no pool selection involved — only routing to the owner — and none of them store anything: the control plane stays O(pools+nodes) however many sandboxes are paused, forked or checkpointed. They authenticate with the fleet api_token over sandboxd's operator path, so the control plane needs no per-sandbox secret. Latency is deliberately documented as non-uniform on the interface: resume takes cocoon's mmap restore fast path and a fork's children clone at 28-75ms, but pause and snapshot write the guest's memory out and cost time proportional to its size.
Upstream agent-sandbox has no pause/fork/snapshot concept, and the value of this operator is that an unmodified upstream client keeps working. Adding fields to the standard Sandbox schema would fork it; a subresource is a verb the standard type never has to know about. They are POST-only action verbs (the pods/eviction shape, not pods/status): each is a synchronous node-local transaction, so there is nothing to GET and nothing to reconcile. Every served kind also needs an OpenAPI model, including metav1.TypeMeta under BOTH its Go package path and its canonical name — without them InstallAPIGroup refuses to start the server at all, which is a crash loop on rollout rather than a degraded feature. TestInstallSandboxAPISucceeds exercises that exact startup path so the next missing model fails locally instead of in the cluster.
The compat layer could create, list and delete; every other e2b verb 404'd, so
an SDK could start a sandbox and then do nothing with it.
Add pause, connect, fork, snapshots, metrics and templates. The status codes
here are the contract, not decoration: the SDK derives pause()'s boolean from a
409 ("was already paused"), and connect returns 201 when it actually restored a
paused sandbox versus 200 when it was already running. Both are asserted.
"Already paused" is answered by the owning node, not by the synthesized read
view. That view is republished on a ~30s cadence, so for up to half a minute
after a pause the listed sandbox still reads Running — deciding from it let a
second pause through with 204, telling the caller it had paused a sandbox that
was already down.
e2b's memory=false (a filesystem-only pause whose resume cold-boots) is
rejected with a 400 rather than silently taking a full memory snapshot: the
caller would get back a different sandbox than it asked for.
Sandbox ids are published DNS-label safe. A sandboxd claim id is "sb_"+hex, and
the SDK builds the in-sandbox envd host as "{port}-{sandboxID}.{domain}" — an
underscore there yields a host that cannot resolve, so the sandbox would be
created but unreachable. Both spellings are accepted on the way back in, and
release always uses the raw claim id, which is the only one the node knows.
…replicas The e2b server needs the fleet's node inventory to answer the surfaces that are fleet-wide rather than sandbox-scoped (template and snapshot listing); without it those would report an error rather than an empty list, so a missing dependency cannot read as "none". Also add a PodDisruptionBudget and a hostname topology spread. An unavailable APIService does not just fail `kubectl get sandboxes` — it degrades discovery for the whole cluster — and the two replicas had nothing stopping a single node drain from evicting both, nor anything stopping the scheduler from placing them on one node to begin with. The spread is DoNotSchedule rather than preferred: both replicas on one node is worse than one replica, because it turns a routine drain into a cluster-wide discovery outage. minAvailable rather than maxUnavailable so the guarantee survives a scale to 1, where maxUnavailable: 1 would still permit evicting the only replica.
Adds docs/snapshot-placement.md: the three placement tiers, why the fast path IS the storage choice (hardlink + reflink + mmap all require the checkpoint and the new VM to share one local filesystem), and why JuiceFS is the wrong answer twice over — it has no reflink, so every clone becomes a network read, and it needs object storage anyway, at which point the existing s3 backend is simpler. States the durability boundary plainly, in the doc, the roadmap and the SSOT: a checkpoint lives on one node and is lost with that node's disk. Peer healing addresses "cannot reach", not "lost". Replication is recorded in ROADMAP.md as explicitly not built — checkpoints are branch points for agent workloads, not backups, and replicating them would cost write amplification on every capture. examples/lifecycle/example.go exercises every operation on both surfaces against a live cluster, so the output doubles as acceptance evidence. It uses a REST client rather than controller-runtime's SubResource helper because the action subresources have different request and response types, which that helper cannot express. It also demonstrates the two behaviours callers must handle: reads are eventually consistent, and the published sandbox id is DNS-safe.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two surfaces over the same warm pools, and the verbs that make a delivered sandbox useful.
1. Lifecycle verbs as Kubernetes subresources.
sandboxes/pause,/resume,/fork,/snapshot. They are subresources rather than fields onSandboxSpecon purpose: upstreamagent-sandboxhas no pause/fork/snapshot concept, and the value of this operator is that an unmodified upstream client keeps working. Adding fields would fork the standard schema; a subresource is a verb the standard type never has to know about. They are POST-only actions (thepods/evictionshape) because each is a synchronous node-local transaction — nothing to GET, nothing to reconcile.2. An e2b-compatible REST surface, so an unmodified e2b SDK claims from these same pools: create, list, get, delete, pause, connect, fork, snapshots, metrics, templates, timeout, refreshes.
3. Store routing for all of it, addressing an already-delivered sandbox on a known node. Nothing is stored: the control plane stays O(pools+nodes) however many sandboxes are paused, forked or checkpointed.
Contracts worth reviewing
pause()'s boolean from a 409 ("was already paused"), andconnectreturns 201 when it actually restored a paused sandbox versus 200 when it was already running. Asserted in tests, and verified live.Running. Deciding from it let a second pause through with 204 — telling the caller it paused a sandbox that was already down.memory=falseis rejected with 400, not silently upgraded to a full memory snapshot. The caller would otherwise get back a different sandbox than it asked for.sb_+hex, and the SDK builds{port}-{sandboxID}.{domain}— an underscore there yields an unresolvable host, so the sandbox would be created but unreachable. Both spellings are accepted inbound; release always uses the raw claim id, the only one the node knows.resumetakes cocoon's mmap restore path and a fork's children clone at 28–75 ms, butpauseandsnapshotwrite the guest's memory out and cost time proportional to its size.Availability fix included
The apiserver had two replicas with nothing stopping a single node drain from evicting both, nor stopping the scheduler from co-locating them. An unavailable APIService degrades discovery for the whole cluster, not just this group. Adds a PDB (
minAvailable: 1, so it survives a scale to 1) and a hostname spread withDoNotSchedule— two replicas on one node is worse than one replica.Verified on a live two-node fleet (bd25 / bd26)
Every verb on both surfaces, run twice for reproducibility via
examples/lifecycle:Deployment also proved a bug unit tests had not: every served kind needs an OpenAPI model, including
metav1.TypeMetaunder both its Go package path and its canonical name — without themInstallAPIGrouprefuses to start the server, which is a crash loop on rollout rather than a degraded feature.TestInstallSandboxAPISucceedsnow exercises that exact startup path.Docs
docs/snapshot-placement.mdcovers where a checkpoint lives and what happens when its node cannot serve a branch, including why JuiceFS is the wrong answer twice over (no reflink, so every clone becomes a network read; and it needs object storage anyway, at which point the existings3backend is simpler). The durability boundary is stated plainly in the doc, the README andROADMAP.md: a checkpoint lives on one node and is lost with that node's disk — replication is explicitly not built.Requires the matching sandboxd change: cocoonstack/sandbox#52.