fix(agent): replay agent cassettes in directory mode, and bound the pipe drain - #185
Merged
Merged
Conversation
…ipe drain
Two bugs that together made `app: agent` flows unusable in CI. Both were hit
recording DataMaker's agent suite; neither is visible from the code alone.
1. `run_suite` had no `app: agent` branch.
Directory mode fell through to the step-replay loader, which parses a trace one
JSON object per line. An agent cassette is a single `{app, mocks, cassette}`
document, so it failed on line 1 and every agent flow errored with
"invalid trace line: EOF while parsing an object at line 1 column 1" - traces
`flowproof record` had just written, and that `flowproof run <spec>` replayed
green one at a time. Directory mode is what a suite and a CI job invoke, so the
whole suite was dead there while single-spec runs looked fine.
The branch mirrors the one `run_one` already had, including the containment
line and the before/after hooks. `RunReport::agent` carries the verdict: its
trace_id is neither "skipped" nor "errored", because the suite tallies those by
that field and an agent flow that ran belongs in the ran/passed count.
2. `read_pipes` waited for EOF, unbounded, after the wait.
EOF arrives when the last write end closes, not when the child dies. An agent
that spawns its own server hands that grandchild the same stdout, so killing
the child at the 300s deadline closed nothing and `read_to_string` blocked
forever. A DataMaker replay sat 26 minutes on a 300-second timeout, produced no
output and had to be killed by hand - the exact failure the deadline's own
comment says it prevents.
Draining now starts at spawn on its own thread and is bounded by a grace
period, keeping whatever partial output arrived, since for a run that already
went wrong that text is the only diagnostic there is. Starting at spawn also
fixes a second latent bug in the same lines: draining only after the wait means
a child that outwrites the ~64KB pipe buffer blocks in `write`, never exits,
and is killed at the timeout - reported as a hung agent when nothing was wrong
with it.
Each fix has a regression test, and each was confirmed to fail without it:
mutating out the suite branch reproduces the exact "invalid trace line" error,
and mutating the drain back to an unbounded wait hangs the test past 120s.
Reported-in: #183
CI runs `cargo fmt --all --check` and `cargo clippy --workspace --all-targets --all-features -- -D warnings`; I had run neither in that exact form. The test module already had its own `use super::*` further down, so the one I added at the top made it redundant.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #183. Two bugs that together made
app: agentflows unusable in CI. Both were hit recording DataMaker's agent suite; neither is visible from the code alone.1.
run_suitehad noapp: agentbranchDirectory mode fell through to the step-replay loader, which parses a trace one JSON object per line. An agent cassette is a single
{app, mocks, cassette}document, so it died on line 1 and every agent flow errored with:on traces
flowproof recordhad just written, and thatflowproof run <spec>replayed green one at a time. Directory mode is what a suite and a CI job invoke, so the whole suite was dead there while single-spec runs looked fine.The new branch mirrors the one
run_onealready had, including the containment line and the before/after hooks.RunReport::agentcarries the verdict; itstrace_idis deliberately neither"skipped"nor"errored", because the suite tallies those two by that field and an agent flow that ran belongs in the ran/passed count.2.
read_pipeswaited for EOF, unbounded, after the waitEOF arrives when the last write end closes, not when the child dies. An agent that spawns its own server hands that grandchild the same stdout, so killing the child at the 300s deadline closed nothing and
read_to_stringblocked forever. A DataMaker replay sat 26 minutes on a 300-second timeout, produced no output, and had to be killed by hand. That is the exact failurewait_to_deadline's own comment says it prevents:Draining now starts at spawn on its own thread, bounded by a grace period, keeping whatever partial output arrived - for a run that already went wrong, that text is the only diagnostic there is.
Starting at spawn also fixes a second latent bug in the same lines: draining only after the wait means a child that outwrites the ~64KB pipe buffer blocks in
write, never exits, and is killed at the timeout, reported as a hung agent when nothing was wrong with it.Both regression tests were confirmed to fail without the fix
Not assumed - mutated and observed:
if false && gated_spec.app.id() == "agent"invalid trace line: EOF while parsing an object at line 1 column 1, the exact reported errorrecv_timeout(grace)back torecv()Verified against the suite that produced the report
The fixed binary against DataMaker's five real cassettes, offline, no key, no stack:
0/5 flows passed, 5 errored, every oneinvalid trace0/5 flows passed, zero errored, in 3 secondsThe remaining failures are legitimate verdicts (
the agent made 0 model calls, the recording has 3) and belong to DataMaker, not here: that suite's agent needs its REST API reachable, and these cassettes are stale against a since-edited tool description. The point is that flowproof now reaches a verdict at all, and terminates.cargo testgreen acrossflowproof-cli,flowproof-adapters,flowproof-replay,flowproof-trace(agent feature on); clippy clean.flowproof-pythonfails to link in my environment on a clean tree too, so it is untouched and unrelated.Not merging this myself: it touches replay determinism, where a wrong fix produces a false green.