Add first-class browser computer use with live session viewing

[ElbertePlinio]

Goal

Make PickLab a production-ready, provider-neutral computer-use runtime: agents can launch an isolated headed browser, combine visual and semantic control, watch the session live, inspect replayable evidence, and hand control to a human without exposing the real desktop.

Product decisions

• Xvfb remains the only display server in v1. Xephyr is out.
• picklab watch opens a host-side VNC viewer against the isolated display. Observation is server-enforced read-only.
• A browser session is one record owning private Xvfb and Chrome legs.
• Chrome uses an ephemeral profile, scrubbed environment, loopback CDP with port 0, and DevToolsActivePort readback.
• The official Chrome DevTools MCP is reached through a framing-aware PickLab relay, not reimplemented.
• Evidence is append-only and renders as a static HTML filmstrip.
• Human control uses an atomic lease plus short-lived agent permits. No control state is stored authoritatively in session JSON.
• Android viewing/takeover, persistent browser profiles, Firefox/WebKit, uid sandboxing, and remote multi-user viewing are out of scope for v1.
• No feature flag: each PR preserves current defaults and is independently releasable.

Security posture

Browser/CDP processes still run as the invoking user. V1 is for authorized, trusted development and QA — not hostile-page isolation. VNC and CDP bind loopback only, browser profiles are ephemeral, Chrome receives a scrubbed environment, process groups are verified and killed before profile deletion, and secrets/capability URLs are excluded from persisted diagnostics. SECURITY.md must document the residual same-UID/local-process risk before #18 closes.

PR plan

• PR 1 — VNC safety and CI truthfulness
  • Refs Nested-display desktop sessions (Xephyr): watch the agent live with its own cursor #15, Add first-class browser computer use with live session viewing #17
  • -viewonly, x11vnc installed/asserted, focused tests
• PR 2 — Shared browser/session/process contracts
  • Refs Add isolated headed Chrome browser sessions #18, Add first-class browser computer use with live session viewing #17
  • browser record, brow prefix, geometry, capability resolution, verified process-group stop
• PR 3 — Live watch flow
  • Closes Nested-display desktop sessions (Xephyr): watch the agent live with its own cursor #15, Refs Add first-class browser computer use with live session viewing #17
  • picklab watch, manual/auto preference, host-side viewer, zero/multiple/headless behavior
• PR 4 — Complete desktop input
  • Closes Add complete desktop input primitives for computer use #22, Refs Add first-class browser computer use with live session viewing #17
  • move/hover, scroll, drag, double-click
• PR 5 — Isolated headed Chrome lifecycle
  • Closes Add isolated headed Chrome browser sessions #18, Refs Add first-class browser computer use with live session viewing #17
  • browser package, ephemeral profile, dynamic CDP, CLI/MCP session wiring
• PR 6 — DevTools MCP relay
  • Closes Add a session-aware Chrome DevTools MCP bridge #19, Refs Add first-class browser computer use with live session viewing #17
  • pinned framing-aware relay and static agent configuration
• PR 7 — Action evidence and filmstrip
  • Closes Record computer-use actions and diagnostic evidence #20, Refs Add first-class browser computer use with live session viewing #17
  • append-only journal, sanitized diagnostics, fixed bounds/retention, HTML report
• PR 8 — Supervised human control
  • Closes Add supervised pause and human takeover #21, Refs Add first-class browser computer use with live session viewing #17
  • atomic lease/permits, watch --control, temporary writable VNC, fresh-state handoff
• PR 9 — Final integration
  • Closes Add first-class browser computer use with live session viewing #17
  • complete docs, security notes, visual + semantic + evidence + takeover E2E proof

Dependency order

PR 1 -> PR 3 --------------------------┐
PR 2 -> PR 5 -> PR 6 -> PR 7 ---------+-> PR 8 -> PR 9
PR 4 ----------------------------------┘

PR 1 and PR 2 may proceed independently. PR 4 is also independent, but must land before the final computer-use claim. Shared CLI/MCP session wiring is serialized to keep reviews clean.

Production acceptance

• A coding agent configures PickLab once and can recreate browser sessions without MCP config edits.
• Visual desktop tools and semantic DevTools tools operate on the same isolated headed Chrome.
• picklab watch can attach to an already-running session; closing the viewer never stops or corrupts it.
• Normal viewing is read-only. watch --control prevents PickLab-managed agent input after human acquisition completes.
• Sensitive typed values, cookies, headers, bodies, query tokens, JWTs, OTPs, and CDP websocket capabilities never enter persisted evidence.
• Screenshot pixels are explicitly not claimed to be redactable; automatic capture is suppressed around sensitive inputs.
• Two concurrent browser sessions have distinct displays, profiles, and CDP ports.
• Destroy/reap leaves no Chrome process-group members or ephemeral profile data.
• Hosted CI passes typecheck, tests, coverage, and build without silent prerequisite skips.
• Before closing: attach one real watch recording and one pause → control → return → resume recording.

Current status: In progress
Next: PR 1 and PR 2 foundation slices.

[ElbertePlinio]

Baseline observed with PickLab itself (current main):

• Created isolated 1280x800 Xvfb session
• Launched xterm and visually inspected screenshots
• Click → type → key flow worked and produced visible PICKLAB_BASELINE_OK
• Session destroy cleaned the lab record/process
• desktop_launch did not focus the launched window; typing before an explicit click produced no visible input
• session_status exposes display/PID but not width/height
• Local VNC path could not run because x11vnc is not installed; current error is actionable
• Artifact report is an inventory only, not an action timeline/replay

Artifacts: .picklab/runs/20260712-054636-baseline-before-input, 20260712-054651-baseline-after-input, 20260712-054708-baseline-click-focus

These observations align with PR 1/2/7 scope. No real desktop was touched.

[ElbertePlinio]

Continuation kickoff:

• PR 1 fix(desktop): make VNC observation read-only #23: VNC safety/CI — review-clean, CI green, ready to merge
• PR 2 feat(core): add browser session contracts #24: contracts — CI green but superseding after valid final-round teardown-order P2
• PR 3: picklab watch — branch has no implementation; restarting from reviewed VNC foundation
• PR 4 feat(desktop): add complete pointer controls #25: desktop input parity — review-clean, CI green, ready to merge
• PR 5: Chrome lifecycle — local implementation exists; latest run has one static security failure in supervisor packaging
• PRs 6–9: relay, evidence, takeover, final integration

No feature flag: each slice remains independently releasable and current defaults stay unchanged.

Current status: In progress
Next: merge #23/#25, replace #24 with corrected reviewed contracts, then ship watch and Chrome lifecycle.

[ElbertePlinio]

Foundation checkpoint:

• PR 1 fix(desktop): make VNC observation read-only #23 merged — VNC safety/CI
• PR 2 replacement fix(core): order browser teardown safely #26 merged — browser/session/process contracts
• PR 4 feat(desktop): add complete pointer controls #25 merged — pointer controls
• PR 3 watch — implementation/review complete; rebasing and live-validating
• PR 5 Chrome lifecycle — implementation/review complete; rebasing and real-Chrome validating

No feature flag; current defaults remain unchanged.

Current status: In progress
Next: ship PRs 3 and 5.