Skip to content

feat(sim): two-agent hand-off scenario with interaction event and per-frame possession GT - #76

Merged
bamdadd merged 5 commits into
bamdadd:mainfrom
Nitjsefnie-OSC:feat/63-two-agent-handoff
Jul 30, 2026
Merged

feat(sim): two-agent hand-off scenario with interaction event and per-frame possession GT#76
bamdadd merged 5 commits into
bamdadd:mainfrom
Nitjsefnie-OSC:feat/63-two-agent-handoff

Conversation

@Nitjsefnie

Copy link
Copy Markdown
Contributor

Summary

Stages the two-agent exchange primitive from #63: two agents converge, a carried parcel changes hands at a defined frame, then they diverge — with the interaction event and per-frame possession as opt-in ground truth. Built on the possession sidecar #73 introduced rather than a second representation.

Related Issues and Pull Requests

Fixes #63
Relates to #73

Changes

  • src/multicam_sim/possession.py — new InteractionEvent (frame, time, giver id, receiver id, object id) carried on PossessionTimeline.events, plus holder_at_frame(...) returning the holder entity id or None while the object is unheld. time rejects negative and non-finite values.
  • src/multicam_sim/dsl/builder.pySceneBuilder.handoff(...): two agent entities on converging-then-diverging paths and one object entity, with the holder changing at the requested frame.
  • examples/handoff.py — runnable 31-frame scene emitting the manifest plus the possession sidecar.
  • src/multicam_sim/__init__.py — exports the new model.
  • tests/test_handoff.py — 9 tests covering scene shape, converge/diverge monotonicity, the possession timeline, the interaction event, geometry-vs-possession agreement, JSON round-trip, manifest exclusion, and the example's sidecars.

On your open design note — new sidecar model vs. an opt-in field — I took neither and extended #73's existing possession channel, so the tree keeps one possession representation instead of two. Happy to split single-carry and hand-off into separate models if you would rather keep them distinct.

Testing

  • Gates from .github/workflows/ci.yml: ruff check clean, ruff format --check clean (75 files), mypy src clean (31 files), pytest 284 passed, 4 skipped against an upstream main baseline of 275 passed, 4 skipped — the 9 added tests, nothing else moved. Both figures measured with a plain uv sync; syncing --extra overlay --extra overlay-mp4 un-skips one test on both sides and the delta is unchanged.
  • The new tests are not vacuous. With builder.py reverted to main and the tests kept, all 8 behaviour tests fail on AttributeError: 'SceneBuilder' object has no attribute 'handoff'; they pass once it is restored. The 9th fails with DID NOT RAISE ValidationError before the non-finite guard.
  • Byte-golden holds for every pre-existing scene, not just the golden fixtures. Manifests were rebuilt from a separate upstream/main worktree with its own venv and compared with cmp: smoke, pose_smoke, multi_entity, mtmc, assembly_station, hand_dose_response, rasterize_demo — all 7 byte-identical, and tests/fixtures/manifest_golden/ is untouched.
  • Geometry agrees with the timeline frame by frame. Across all 31 frames of the example the parcel centre is bitwise equal to holder centre + offset while held and to the staging point while unheld, with holder changes exactly [(3, None, giver), (15, giver, receiver)] — one holder-to-holder change, at the hand-off frame.
  • Invalid constructions raise rather than producing a bad scene: giver == receiver, a hand-off frame at or outside the path bounds, an end beyond num_frames, unknown object or holder ids, and an overlapping double hand-off of the same object (overlapping possession segments).
  • Round-trip checked through model_dump(mode="json"), model_dump_json, the sidecar to_json, and write-to-file-and-reload, including awkward finite floats and unicode ids.

Follow-ups / Known Limitations

  • At the hand-off frame both agents are coincident at the exchange point, so geometry alone cannot distinguish holders on that one frame; the timeline is authoritative there. Worth knowing if a consumer tries to infer possession from positions rather than reading the sidecar.
  • attach semantics are unchanged, so with non-default object_point / holder_point the tracked point is not the centre. That is existing opt-in behaviour, not something this PR alters.
  • The scene is a single hand-off between two agents. Chained exchanges (A→B→C) are representable in the timeline but no builder sugar exists for them yet.

Footer

Generated by Claude Opus 5 (brief, review), Kimi K3 (implementation, verification)

Nitjsefnie and others added 5 commits July 30, 2026 02:04
Extend the possession GT sidecar (bamdadd#73) with an InteractionEvent model —
frame, time (frame / fps), giver id, receiver id, object id — carried as a
sorted, opt-in events list on PossessionTimeline. A two-agent hand-off is
the same possession channel with a holder id that changes, plus this
interaction record; the byte-golden analytic manifest is untouched.

Refs bamdadd#63

Co-Authored-By: Kimi K3 <noreply@kimi.com>
One fluent call stages the full hand-off: the giver carries the object over
[start, frame) and the receiver over [frame, end) via the existing attach
machinery (object tracks holder + offset on every held frame), and an
InteractionEvent is baked into the scene's possession sidecar. The hand-off
frame is the single frame where the holder id changes.

Refs bamdadd#63

Co-Authored-By: Kimi K3 <noreply@kimi.com>
examples/handoff.py stages two agents on converging-then-diverging waypoint
paths (both at the exchange point exactly at the hand-off frame) with one
parcel that is unheld, carried by the giver, then carried by the receiver.
It writes manifest.json plus the possession.json sidecar (segments +
interaction event); the manifest stays byte-golden.

Refs bamdadd#63

Co-Authored-By: Kimi K3 <noreply@kimi.com>
Cover the bamdadd#63 acceptance criteria against the constructed scene: two agents
converge to the exchange point at the hand-off frame then diverge; the
per-frame possession timeline is None -> giver -> receiver with the hand-off
frame the single holder-id change; the InteractionEvent records frame, time,
giver, receiver, object; the held object tracks its holder + offset
geometrically; Scene and sidecar round-trip through JSON losslessly; the
analytic manifest never carries the GT; and the example emits valid
sidecars.

Fixes bamdadd#63

Co-Authored-By: Kimi K3 <noreply@kimi.com>
The _time_non_negative validator only checked value < 0.0, so NaN and
+/-inf passed validation (both comparisons are False), serialised to
JSON null via model_dump_json, and then failed to reload — a
model-admissible value did not round-trip. Add a math.isfinite check
(matches the file's explicit-validator conventions; FiniteFloat is not
used anywhere in src) and keep the negative-time rejection. frame needs
no guard: pydantic rejects non-finite input for int fields natively.

Pin it with a test: NaN, +inf and -inf are each rejected at
construction, and a finite time still validates and round-trips.

Co-Authored-By: Kimi K3 <noreply@kimi.com>
@bamdadd
bamdadd merged commit 78523e5 into bamdadd:main Jul 30, 2026
1 check passed
@bamdadd

bamdadd commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Merged — thank you, this is a really clean contribution. The additive events sidecar on PossessionTimeline keeps the manifest byte-identical, the handoff builder reads nicely, and the examples/handoff.py plus tests make the two-agent exchange easy to reason about. Closes #63. Appreciate you spreading these across the repos — keep them coming!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Two-agent interaction / object hand-off scenario with ground-truth possession labels

2 participants