Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,46 @@ picklab session destroy --all

Every screenshot, log, and action lands in `.picklab/runs/<runId>/` with a manifest, so a run is inspectable and reproducible after the fact.

### Evidence recording

Computer-use tools record one session-scoped evidence run by default. MCP
desktop, Android, and session actions share the same append-only timeline as
browser DevTools actions. Destroying a session, or reaping a dead one, finalizes
the run and writes a static `report.html` filmstrip.

A finalized evidence run under `.picklab/runs/<runId>/` contains:

- `manifest.json` — run identity, status, and evidence metadata
- `actions.jsonl` — authoritative, append-only sanitized action timeline
- `report.html` — escaped, no-script human filmstrip generated at finalization
- `screenshots/` and `logs/` — associated artifacts, when explicitly captured

Typed values are stored only as length and input type. Network failures keep
only allowlisted method, URL origin/path without its query, status, resource
type, timing, and sanitized error metadata; headers and bodies are never kept.
PickLab does not take implicit screenshots for input actions. Explicit
screenshot tools still capture the screen exactly as displayed.

The journal and associated artifacts have a 100 MiB recording threshold per
run. The record that crosses the threshold may exceed it; PickLab then writes a
durable metadata-only truncation marker and stops appending further payloads.
Only the latest 20 finalized evidence runs are retained; active/running and
legacy runs are never pruned.

Evidence recording is enabled by default. Disable the action timeline for a
project in `.picklab/config.json`:

```json
{
"evidence": {
"enabled": false
}
}
```

This does not block an explicitly requested screenshot command. Screenshot
pixels cannot be redacted; see [SECURITY.md](SECURITY.md#recorded-evidence-and-screenshots).

### Concurrent sessions

Each session gets its own isolated display or emulator, so several agents and projects can run labs side by side. When a command or tool is called without an explicit session id, the default resolves per project: only running sessions created for the same project directory are considered. Pass `session` ids (CLI: `--session <id>`) to target a specific lab, including one belonging to another project.
Expand Down Expand Up @@ -180,6 +220,8 @@ Resources, addressable as `picklab://` URIs:
- `picklab://runs/{runId}/manifest` — run manifest
- `picklab://runs/{runId}/screenshots/{name}` — screenshots
- `picklab://runs/{runId}/logs/{name}` — logs
- `picklab://runs/{runId}/actions` — sanitized action timeline JSON
- `picklab://runs/{runId}/report` — static HTML evidence filmstrip
- `picklab://sessions/{sessionId}/status` — session liveness
The status includes a read-only viewer endpoint/readiness report when VNC is
present. MCP never opens a host GUI; only the CLI launches viewer windows.
Expand Down Expand Up @@ -208,6 +250,8 @@ A TypeScript monorepo. `@pickforge/picklab` is the published package; the rest a
- Relay stdout is protocol-only. A pending JSON-RPC record is capped at 16 MiB. Upstream diagnostic lines are capped at 64 KiB, redacted, and forwarded only to stderr; an over-limit line is dropped with a safe notice. Upstream update checks and usage statistics are disabled.
- VNC binds to loopback only by default: `x11vnc` is started with `-localhost`, so the server listens on `127.0.0.1` and is not reachable from the network. Tunnel over SSH for remote access. Normal `--vnc` and `picklab watch` observation is server-enforced read-only (`-viewonly`); viewer exit never stops the session or its Xvfb/VNC processes. `--vnc-control` is an explicit writable escape hatch for human secret entry and does not yet coordinate with agent input.
- Artifacts are redacted by default: logcat output strips tokens and secrets before it is stored or returned. Only `android adb` is raw, and it says so.
- Evidence timelines persist only allowlisted metadata; typed values become length/type metadata, and network headers, bodies, and URL queries are dropped. Static HTML reports escape page-controlled text and use a no-script, no-network CSP.
- Screenshot files contain raw pixels and cannot be redacted. Avoid explicit captures on screens containing secrets, and use `evidence.enabled: false` when an action timeline is not appropriate. See [SECURITY.md](SECURITY.md#recorded-evidence-and-screenshots).
- PickLab provisions a dedicated locked lab user (`picklab-lab`) and a dedicated AVD (`picklab-avd`) so lab workloads do not borrow your personal resources. Running session processes under the lab user is planned post-MVP.
- Agent config edits are atomic, with backups of the previous config.

Expand Down
40 changes: 40 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,46 @@ fuller model.
and it says so.
- Agent config edits are atomic, with a backup of the previous config.

## Recorded evidence and screenshots

Computer-use evidence is stored as local project data under
`.picklab/runs/<runId>/`. `actions.jsonl` is the authoritative sanitized
timeline; finalization produces an escaped, no-script `report.html` with a
restrictive content security policy and no external requests.

The recorder persists only allowlisted metadata. Typed and filled text becomes
length plus input type. Failed network records keep method, URL origin/path
without the query, status, resource type, timing, and a sanitized error
classification. Request/response headers and bodies are dropped. Cookie,
authorization, session, JWT, CSRF, OTP, query-token, and CDP capability values
are redacted or omitted before persistence.

**Screenshots are different: pixels cannot be redacted.** An explicitly
requested desktop, Android, or DevTools screenshot stores the screen exactly as
displayed and may therefore contain passwords, tokens, personal data, or other
secrets. PickLab never takes an implicit screenshot for typing/fill actions.
Do not explicitly capture a sensitive screen. Files are ordinary local files,
not encrypted storage; anyone who can read the project directory can read its
evidence.

Action evidence is enabled by default. Disable it per project when recording is
not appropriate:

```json
{
"evidence": {
"enabled": false
}
}
```

This opt-out disables the action timeline and its screenshot association; it
does not block a screenshot command that was explicitly requested. The journal
and associated artifacts have a 100 MiB recording threshold per run. The
crossing record may exceed it; PickLab then writes one durable metadata-only
truncation marker and stops appending payloads. PickLab retains the latest 20
finalized evidence runs, while active/running and legacy runs are not pruned.

## VNC binds to loopback (SEC-01 — mitigated)

The desktop VNC server (`x11vnc`) is started with `-localhost`, so it listens on
Expand Down
13 changes: 12 additions & 1 deletion docs/releases/UNRELEASED.md
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,17 @@ then reset this file.
diagnostics, malformed/unmatched traffic, pending-call closure, disabled
configuration, evidence-write failure isolation, and bounded diagnostics.
Two independent changed-HEAD reviews are clean.
- Final evidence integration: `bun run typecheck` and `bun run build` pass;
the full and coverage suites pass 77 files / 945 passed / 2 skipped with all
global thresholds met. A failed DevTools relay flow produces an automatically
finalized, useful HTML report without persisted typed text, URL query tokens,
headers, bodies, JWTs, CDP capability values, or OTPs. Focused lifecycle tests
cover report publication on explicit destruction and dead-session reaping,
inclusion of the final `session_destroy` action, rejection of mismatched
manifest/pointer identities, and fresh relay-target resolution after browser
recreation. README and security guidance document recording, resources, the
100 MiB threshold, 20-run retention, opt-out behavior, and unredactable
screenshot pixels. Two independent changed-HEAD reviews are clean.

### Not tested yet

Expand All @@ -317,4 +328,4 @@ then reset this file.

### Release blockers

- None known for the DevTools relay slice.
- None known for the final evidence integration slice.
229 changes: 229 additions & 0 deletions packages/browser/test/evidence-integration.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,229 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { EventEmitter } from "node:events";
import { Readable, Writable } from "node:stream";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import {
createSession,
destroySessionRecord,
listRuns,
readActions,
runsDir,
type EnvLike,
type SessionRecord,
} from "@pickforge/picklab-core";
import {
runDevtoolsMcpRelay,
type DevtoolsMcpExecutable,
type LiveBrowserSession,
type RelaySignalSource,
} from "../src/index.js";
import { createDevtoolsEvidenceRecorder } from "../src/devtools-evidence.js";

const TYPED_PASSWORD = "typed-password-4821";
const QUERY_TOKEN = "query-token-5932";
const JWT =
"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJwaWNrbGFiIn0.signature-value";
const CDP_GUID = "01234567-89ab-cdef-0123-456789abcdef";
const OTP = "739204";

let root: string;
let projectDir: string;

beforeEach(async () => {
root = await fs.promises.mkdtemp(
path.join(os.tmpdir(), "picklab-evidence-integration-"),
);
projectDir = path.join(root, "project");
await fs.promises.mkdir(projectDir);
});

afterEach(async () => {
await fs.promises.rm(root, { recursive: true, force: true });
});

class Signals extends EventEmitter implements RelaySignalSource {
override on(
signal: "SIGINT" | "SIGTERM" | "SIGHUP",
listener: () => void,
): this {
return super.on(signal, listener);
}

override off(
signal: "SIGINT" | "SIGTERM" | "SIGHUP",
listener: () => void,
): this {
return super.off(signal, listener);
}
}

function session(sessionId: string): LiveBrowserSession {
const record: SessionRecord = {
id: sessionId,
type: "browser",
createdAt: "2026-07-13T00:00:00.000Z",
status: "running",
projectDir,
desktop: { display: ":91", xvfbPid: 11 },
browser: {
browserPid: 12,
browserStartTimeTicks: 13,
binaryPath: "/fake/chrome",
profileMode: "ephemeral",
profileDir: "/fake/profile",
cdpPort: 9222,
},
};
return { record, cdpPort: 9222, browserUrl: "http://127.0.0.1:9222" };
}

function executable(script: string): DevtoolsMcpExecutable {
return {
packageJsonPath: path.join(root, "package.json"),
packageRoot: root,
binPath: script,
version: "1.5.0",
};
}

function sink(chunks: Buffer[]): Writable {
return new Writable({
write(chunk: Buffer, _encoding, callback) {
chunks.push(Buffer.from(chunk));
callback();
},
});
}

async function recursiveText(dir: string): Promise<string> {
const chunks: string[] = [];
const walk = async (current: string): Promise<void> => {
for (const entry of await fs.promises.readdir(current, {
withFileTypes: true,
})) {
const full = path.join(current, entry.name);
if (entry.isDirectory()) {
await walk(full);
} else if (entry.isFile()) {
chunks.push(await fs.promises.readFile(full, "utf8"));
}
}
};
await walk(dir);
return chunks.join("\n");
}

describe("browser evidence integration", () => {
it("turns a failed relay flow into a useful secret-free HTML report", async () => {
const script = path.join(root, "failed-flow.mjs");
await fs.promises.writeFile(
script,
[
'let input = "";',
'process.stdin.setEncoding("utf8");',
'process.stdin.on("data", (chunk) => { input += chunk; });',
'process.stdin.on("end", () => {',
' for (const line of input.trim().split("\\n")) {',
" const request = JSON.parse(line);",
" const result = request.id === 1",
" ? { content: [] }",
" : {",
" isError: true,",
` content: [{ type: "text", text: "Authorization: Bearer ${JWT}; otp=${OTP}; /devtools/browser/${CDP_GUID}" }],`,
" structuredContent: {",
" networkRequests: [{",
' method: "GET",',
` url: "https://example.com/fail?token=${QUERY_TOKEN}",`,
' status: 503, resourceType: "fetch", durationMs: 42,',
` error: "token=${QUERY_TOKEN}",`,
` requestHeaders: { authorization: "Bearer ${JWT}" },`,
` responseBody: "otp=${OTP}"`,
" }]",
" }",
" };",
' process.stdout.write(JSON.stringify({ jsonrpc: "2.0", id: request.id, result }) + "\\n");',
" }",
"});",
].join("\n"),
"utf8",
);
const registryEnv: EnvLike = {
PICKLAB_HOME: path.join(root, "picklab-home"),
};
const sessionId = (
await createSession({ type: "browser", projectDir }, registryEnv)
).id;
const evidence = await createDevtoolsEvidenceRecorder({
projectDir,
sessionId,
});
expect(evidence).toBeDefined();
const requests = [
{
jsonrpc: "2.0",
id: 1,
method: "tools/call",
params: {
name: "fill",
arguments: { uid: "1_1", value: TYPED_PASSWORD },
},
},
{
jsonrpc: "2.0",
id: 2,
method: "tools/call",
params: {
name: "navigate_page",
arguments: {
type: "url",
url: `https://example.com/fail?token=${QUERY_TOKEN}`,
},
},
},
];
const output: Buffer[] = [];

await expect(
runDevtoolsMcpRelay({
session: session(sessionId),
executable: executable(script),
input: Readable.from(
requests.map((request) => `${JSON.stringify(request)}\n`),
),
output: sink(output),
diagnostics: sink([]),
hooks: {
beforeForward: evidence!.beforeForward,
afterResponse: evidence!.afterResponse,
},
signalSource: new Signals(),
shutdownTimeoutMs: 100,
}),
).resolves.toEqual({ code: 0, signal: null });
expect(Buffer.concat(output).toString()).toContain(QUERY_TOKEN);

await destroySessionRecord(sessionId, registryEnv, "failed");
const [finalized] = await listRuns(projectDir);
expect(finalized).toBeDefined();
const runDir = path.join(runsDir(projectDir), finalized!.runId);
const reportPath = path.join(runDir, "report.html");
const records = await readActions(runDir);
const html = await fs.promises.readFile(reportPath, "utf8");

expect(records).toHaveLength(3);
expect(html).toContain("chrome_devtools/fill");
expect(html).toContain("chrome_devtools/navigate_page");
expect(html).toContain("network_failure");
expect(html).toContain("https://example.com/fail");
expect(html).toContain("503");
expect(html).toContain("DevTools tool failed");
expect(finalized!.status).toBe("failed");

const stored = await recursiveText(runDir);
for (const secret of [TYPED_PASSWORD, QUERY_TOKEN, JWT, CDP_GUID, OTP]) {
expect(stored).not.toContain(secret);
}
});
});
Loading
Loading