Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 91 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,94 @@ spec:
For low-latency acquisition, define a `SandboxTemplate` + `SandboxWarmPool` and
create `SandboxClaim`s — see [examples/](examples/).

### Lifecycle: pause, resume, fork, snapshot

Beyond create/delete, a delivered sandbox supports four action verbs, served as
**subresources** so the standard `agents.x-k8s.io` schema stays untouched (an
unmodified upstream client keeps working):

| subresource | body | effect |
|---|---|---|
| `sandboxes/pause` | `SandboxPauseOptions` | snapshots guest memory and stops the VM |
| `sandboxes/resume` | `SandboxResumeOptions` | restores it via cocoon's mmap fast path |
| `sandboxes/fork` | `SandboxForkOptions{count,ttlSeconds}` | branches N children; the source keeps running |
| `sandboxes/snapshot` | `SandboxSnapshotOptions{name}` | captures a checkpoint later sandboxes branch from |

```bash
kubectl create --raw \
/apis/agents.x-k8s.io/v1beta1/namespaces/default/sandboxes/my-sandbox/snapshot \
-f - <<<'{"apiVersion":"agents.x-k8s.io/v1beta1","kind":"SandboxSnapshotOptions","name":"before-migration"}'
```

**These verbs are not uniformly fast.** `resume` takes cocoon's mmap restore
path and a fork's children clone node-locally, but `pause` and `snapshot` write
the guest's memory out, so they cost time proportional to its size. Where a
checkpoint lives, and what happens when its node cannot serve a branch, is in
[docs/snapshot-placement.md](docs/snapshot-placement.md).

### Runnable walk-through

[`examples/lifecycle/example.go`](examples/lifecycle/example.go) exercises
**every** operation on **both** surfaces against a live cluster — create, get,
list, snapshot, fork, pause, resume, delete over the Kubernetes API, then the
same lifecycle plus templates, metrics, snapshot listing, timeout and keepalive
over the e2b REST API:

```bash
go run ./examples/lifecycle \
-kubeconfig ~/.kube/config -namespace default \
-e2b-url http://localhost:8080 -e2b-key "$E2B_API_KEY"
```

It discovers a template from the fleet's advertised warm pools, so no image
argument is needed. Omit `-e2b-url` to run only the Kubernetes half. Each step
prints what it did, so the output doubles as acceptance evidence:

```
=== Kubernetes API ===
create Sandbox default/example-219526000
get node=cocoon-bd25-sandboxd claimID=sb_77ace349e7cc1db6
snapshot snapshotID=ck_92799687f14e8fea on node=cocoon-bd25-sandboxd
fork child[0] sandboxID=sb_06d1b2438dcc1d1b node=cocoon-bd25-sandboxd
pause took 315ms (proportional to guest memory)
resume took 109ms (mmap restore fast path)

=== e2b-compatible REST API ===
create sandboxID=sb-175124667cfa1281 envdVersion=0.4.0
metrics cpuCount=1 memTotal=5.36870912e+08
pause 409 on repeat — the already-paused contract holds
connect 201 — restored via the mmap fast path
```

Two behaviors it demonstrates deliberately, because callers must handle them:

- **Reads are eventually consistent.** `Create` returns as soon as the
node-local claim completes, but `List`/`Get` are served from `NodeInventory`,
which nodes republish on a ~30s cadence — so a read immediately after a
create legitimately returns `NotFound`. The example polls; so should you.
- **The published sandbox id is DNS-label safe.** The e2b SDK builds the
in-sandbox host as `{port}-{sandboxID}.{domain}`, so the node's raw claim id
(`sb_...`) is rendered as `sb-...` on the e2b surface.

## Use it with the e2b SDK

The aggregated apiserver can also serve an e2b-compatible REST surface, so an
**unmodified e2b SDK** claims from these same warm pools — point `E2B_API_URL`
at it:

```bash
sandbox-apiserver --enable-e2b-api --e2b-api-key-file=/etc/e2b/keys
```

```js
export E2B_API_URL=https://your-apiserver:8080
const sandbox = await Sandbox.create('registry.example.com/rt:24.04')
```

It is a translation layer, not a second control plane: an e2b create is the same
node-local claim, and the sandbox stays visible to `kubectl get sandboxes`. Flags,
endpoint mapping, and limits in [docs/e2b-compat.md](docs/e2b-compat.md).

## Install

Helm:
Expand Down Expand Up @@ -545,7 +633,9 @@ make test-race
make generate # CRDs, RBAC, deepcopy (idempotent)
```

See [docs/](docs/) for API, configuration, and runtime details.
See [docs/](docs/) for API, configuration, and runtime details, including
[snapshot placement](docs/snapshot-placement.md) — where a checkpoint lives,
how a branch reaches it from another node, and what that costs.

## Community

Expand Down
13 changes: 13 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,19 @@ move any of these forward are welcome.
`sandboxes.agents.x-k8s.io` from live per-node inventory streams rather than
inventory re-list.

## Medium term (continued)

- **Checkpoints that survive node loss.** A checkpoint currently lives on the
node that took it: peer healing moves a record to a node that cannot reach
it, but nothing replicates one, so losing that node's disk loses the
checkpoint (see [docs/snapshot-placement.md](docs/snapshot-placement.md)).
Making them durable needs asynchronous replication to N peers plus a
placement policy that tracks replica sets and repairs under-replication.
**Not built, and deliberately so** — checkpoints are branch points for agent
workloads, not backups, and replication would cost write amplification on
every capture. Until this lands, promote anything that must outlive its node
to a template and distribute it as one.

## Longer term

- **Million-sandbox validation.** Exercise the full L0–L3 design at fleet
Expand Down
138 changes: 138 additions & 0 deletions api/v1beta1/sandbox_lifecycle_types.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
// Copyright 2026 The CocoonStack Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package v1beta1

import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
)

// The lifecycle verbs below are served as SUBRESOURCES of sandboxes
// (sandboxes/pause, /resume, /fork, /snapshot), never as fields on SandboxSpec.
// Upstream agent-sandbox has no pause/fork/snapshot concept, and the value of
// this operator is that an unmodified upstream client keeps working: adding
// fields to the standard schema would fork it, whereas a subresource is a verb
// the standard type does not have to know about.
//
// They are actions, not desired state, which is also why they are POSTed rather
// than reconciled — each one is a synchronous node-local transaction, the same
// shape as the claim that delivered the sandbox.

// +kubebuilder:object:root=true

// SandboxPauseOptions is the body of POST sandboxes/{name}/pause. Pausing
// snapshots the guest's memory and stops its VM, so it costs time proportional
// to that memory — unlike resume, which takes the mmap restore fast path.
type SandboxPauseOptions struct {
metav1.TypeMeta `json:",inline"`
}

// +kubebuilder:object:root=true

// SandboxResumeOptions is the body of POST sandboxes/{name}/resume. Resuming a
// paused sandbox restores it through cocoon's mmap fast path and is idempotent
// on one that is already running.
type SandboxResumeOptions struct {
metav1.TypeMeta `json:",inline"`
}

// +kubebuilder:object:root=true

// SandboxForkOptions is the body of POST sandboxes/{name}/fork. The source is
// checkpointed in place and keeps running; each child is a brand-new sandbox
// with its own id and lease, not a replica of the source's identity.
type SandboxForkOptions struct {
metav1.TypeMeta `json:",inline"`

// count is how many children to branch. Defaults to 1, and is bounded by
// the owning node's configured fork limit.
// +optional
Count int32 `json:"count,omitempty"`

// ttlSeconds is each child's lease. Children never inherit the parent's
// remaining lease — a lease is a per-sandbox resource bound. Zero takes the
// node's default.
// +optional
TTLSeconds int32 `json:"ttlSeconds,omitempty"`
}

// +kubebuilder:object:root=true

// SandboxForkResult is the reply to a fork: one entry per child, in request
// order.
type SandboxForkResult struct {
metav1.TypeMeta `json:",inline"`

// children are the branched sandboxes.
Children []ForkedSandbox `json:"children"`
}

// ForkedSandbox identifies one child of a fork.
type ForkedSandbox struct {
// sandboxID is the child's node-local claim id.
SandboxID string `json:"sandboxID"`
// nodeName is the node that owns the child. A fork is node-local, so every
// child lands on the source's node.
NodeName string `json:"nodeName"`
// address is the child's connection address, when the node published one.
// +optional
Address string `json:"address,omitempty"`
}

// +kubebuilder:object:root=true

// SandboxSnapshotOptions is the body of POST sandboxes/{name}/snapshot. The
// source keeps running; the checkpoint is an immutable state later sandboxes
// can branch from.
type SandboxSnapshotOptions struct {
metav1.TypeMeta `json:",inline"`

// name labels the checkpoint. Optional; the node assigns an id regardless.
// +optional
Name string `json:"name,omitempty"`
}

// +kubebuilder:object:root=true

// SandboxSnapshotResult is the reply to a snapshot.
type SandboxSnapshotResult struct {
metav1.TypeMeta `json:",inline"`

// snapshotID is the checkpoint's node-local id.
SnapshotID string `json:"snapshotID"`
// name echoes the requested label, when one was given.
// +optional
Name string `json:"name,omitempty"`
// nodeName is the node holding the checkpoint. Checkpoints are node-local,
// so branching from or deleting one requires knowing its node.
NodeName string `json:"nodeName"`
// creationTimestamp is when the node captured the checkpoint.
// +optional
CreationTimestamp metav1.Time `json:"creationTimestamp,omitempty"`
}

func init() {
SchemeBuilder.Register(func(s *runtime.Scheme) error {
s.AddKnownTypes(GroupVersion,
&SandboxPauseOptions{},
&SandboxResumeOptions{},
&SandboxForkOptions{},
&SandboxForkResult{},
&SandboxSnapshotOptions{},
&SandboxSnapshotResult{},
)
return nil
})
}
Loading
Loading