Deterministic testing of AI-based systems — the standard way to assert "this prompt leads to these tool calls" without executing real tools or paying model nondeterminism. Full design: docs/agent-testing.md.
v1 scope:
- OpenAI-compatible chat-completions proxy (non-streaming): record mode passes through and captures the full trajectory (prompts, tool calls, tool results, replies) into the trace as a cassette (redacted; keys stay
${VAR} refs); replay mode serves the recording — deterministic, offline, zero model cost.
app: agent process driver: launch the system under test with the proxy URL injected via env.tools: spec block: mocked results returned at the boundary so multi-step trajectories proceed without executing anything real.assert_tool_call grammar: ordered-subsequence matching with argument matchers reusing the existing assertion vocabulary; strict: flow flag to forbid unlisted calls.- Trajectory diff on re-record (the heal moment for prompt-template drift).
Out of scope, by decision recorded in the design doc: model-output quality evals (statistical, not deterministic — must not blur the replay engine's promise).
Open questions to settle in design review: cassette matching tolerance (structural envelope + normalized prompt hash vs byte-exact), branching behavior on divergence, per-driver definition of "reply".
🤖 Generated with Claude Code
https://claude.ai/code/session_01QCp4epPiULuqwoeSuZK7bj
Deterministic testing of AI-based systems — the standard way to assert "this prompt leads to these tool calls" without executing real tools or paying model nondeterminism. Full design:
docs/agent-testing.md.v1 scope:
${VAR}refs); replay mode serves the recording — deterministic, offline, zero model cost.app: agentprocess driver: launch the system under test with the proxy URL injected via env.tools:spec block: mocked results returned at the boundary so multi-step trajectories proceed without executing anything real.assert_tool_callgrammar: ordered-subsequence matching with argument matchers reusing the existing assertion vocabulary;strict:flow flag to forbid unlisted calls.Out of scope, by decision recorded in the design doc: model-output quality evals (statistical, not deterministic — must not blur the replay engine's promise).
Open questions to settle in design review: cassette matching tolerance (structural envelope + normalized prompt hash vs byte-exact), branching behavior on divergence, per-driver definition of "reply".
🤖 Generated with Claude Code
https://claude.ai/code/session_01QCp4epPiULuqwoeSuZK7bj