Skip to content

fix(agent): replay agent cassettes in directory mode, and bound the pipe drain - #185

Merged
AminChirazi merged 2 commits into
mainfrom
fix/agent-replay-suite-dispatch-and-pipe-hang
Jul 28, 2026
Merged

fix(agent): replay agent cassettes in directory mode, and bound the pipe drain#185
AminChirazi merged 2 commits into
mainfrom
fix/agent-replay-suite-dispatch-and-pipe-hang

Conversation

@AminChirazi

Copy link
Copy Markdown
Contributor

Fixes #183. Two bugs that together made app: agent flows unusable in CI. Both were hit recording DataMaker's agent suite; neither is visible from the code alone.

1. run_suite had no app: agent branch

Directory mode fell through to the step-replay loader, which parses a trace one JSON object per line. An agent cassette is a single {app, mocks, cassette} document, so it died on line 1 and every agent flow errored with:

invalid trace line: EOF while parsing an object at line 1 column 1

on traces flowproof record had just written, and that flowproof run <spec> replayed green one at a time. Directory mode is what a suite and a CI job invoke, so the whole suite was dead there while single-spec runs looked fine.

The new branch mirrors the one run_one already had, including the containment line and the before/after hooks. RunReport::agent carries the verdict; its trace_id is deliberately neither "skipped" nor "errored", because the suite tallies those two by that field and an agent flow that ran belongs in the ran/passed count.

2. read_pipes waited for EOF, unbounded, after the wait

EOF arrives when the last write end closes, not when the child dies. An agent that spawns its own server hands that grandchild the same stdout, so killing the child at the 300s deadline closed nothing and read_to_string blocked forever. A DataMaker replay sat 26 minutes on a 300-second timeout, produced no output, and had to be killed by hand. That is the exact failure wait_to_deadline's own comment says it prevents:

Kill rather than wait forever: an agent that hangs ... would otherwise take the whole suite down with it.

Draining now starts at spawn on its own thread, bounded by a grace period, keeping whatever partial output arrived - for a run that already went wrong, that text is the only diagnostic there is.

Starting at spawn also fixes a second latent bug in the same lines: draining only after the wait means a child that outwrites the ~64KB pipe buffer blocks in write, never exits, and is killed at the timeout, reported as a hung agent when nothing was wrong with it.

Both regression tests were confirmed to fail without the fix

Not assumed - mutated and observed:

Mutation Result
if false && gated_spec.app.id() == "agent" reproduces invalid trace line: EOF while parsing an object at line 1 column 1, the exact reported error
recv_timeout(grace) back to recv() test hangs past a 120s kill, exactly the reported symptom

Verified against the suite that produced the report

The fixed binary against DataMaker's five real cassettes, offline, no key, no stack:

  • before: 0/5 flows passed, 5 errored, every one invalid trace
  • after: 0/5 flows passed, zero errored, in 3 seconds

The remaining failures are legitimate verdicts (the agent made 0 model calls, the recording has 3) and belong to DataMaker, not here: that suite's agent needs its REST API reachable, and these cassettes are stale against a since-edited tool description. The point is that flowproof now reaches a verdict at all, and terminates.

cargo test green across flowproof-cli, flowproof-adapters, flowproof-replay, flowproof-trace (agent feature on); clippy clean. flowproof-python fails to link in my environment on a clean tree too, so it is untouched and unrelated.

Not merging this myself: it touches replay determinism, where a wrong fix produces a false green.

…ipe drain

Two bugs that together made `app: agent` flows unusable in CI. Both were hit
recording DataMaker's agent suite; neither is visible from the code alone.

1. `run_suite` had no `app: agent` branch.

Directory mode fell through to the step-replay loader, which parses a trace one
JSON object per line. An agent cassette is a single `{app, mocks, cassette}`
document, so it failed on line 1 and every agent flow errored with
"invalid trace line: EOF while parsing an object at line 1 column 1" - traces
`flowproof record` had just written, and that `flowproof run <spec>` replayed
green one at a time. Directory mode is what a suite and a CI job invoke, so the
whole suite was dead there while single-spec runs looked fine.

The branch mirrors the one `run_one` already had, including the containment
line and the before/after hooks. `RunReport::agent` carries the verdict: its
trace_id is neither "skipped" nor "errored", because the suite tallies those by
that field and an agent flow that ran belongs in the ran/passed count.

2. `read_pipes` waited for EOF, unbounded, after the wait.

EOF arrives when the last write end closes, not when the child dies. An agent
that spawns its own server hands that grandchild the same stdout, so killing
the child at the 300s deadline closed nothing and `read_to_string` blocked
forever. A DataMaker replay sat 26 minutes on a 300-second timeout, produced no
output and had to be killed by hand - the exact failure the deadline's own
comment says it prevents.

Draining now starts at spawn on its own thread and is bounded by a grace
period, keeping whatever partial output arrived, since for a run that already
went wrong that text is the only diagnostic there is. Starting at spawn also
fixes a second latent bug in the same lines: draining only after the wait means
a child that outwrites the ~64KB pipe buffer blocks in `write`, never exits,
and is killed at the timeout - reported as a hung agent when nothing was wrong
with it.

Each fix has a regression test, and each was confirmed to fail without it:
mutating out the suite branch reproduces the exact "invalid trace line" error,
and mutating the drain back to an unbounded wait hangs the test past 120s.

Reported-in: #183
Copilot AI review requested due to automatic review settings July 28, 2026 06:17

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

CI runs `cargo fmt --all --check` and `cargo clippy --workspace
--all-targets --all-features -- -D warnings`; I had run neither in that
exact form. The test module already had its own `use super::*` further
down, so the one I added at the top made it redundant.
@AminChirazi
AminChirazi merged commit 700a593 into main Jul 28, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0.7.0: agent cassettes recorded by record cannot be replayed by run (directory mode errors, single-file mode hangs)

2 participants