A contract-to-repair diagnostic layer for AG2 multi-agent workflows.
Concord monitors a multi-agent workflow run, detects when agents violate their behavioural contracts, attributes the root cause to the responsible agent, proposes a repair targeting the correct AG2 primitive, and validates the fix with a sandboxed regression test β all automatically, end-to-end.
Current product loop: register a workflow contract, submit a run or trace, inspect deterministic violations, review one repair patch per violation, validate in Daytona, export the report, and return to persisted history.
Live demo URL: https://concord-lite.vercel.app/ Pipeline (CLI):
python run_all.py --fixtureNorth-star scorecard:docs/PLAN_VS_REALITY.mdArchitecture & Q&A doc:docs/ARCHITECTURE.mdDemo script & cue cards:docs/DEMO_SCRIPT.mdDeep Q&A reference (70 questions):docs/QA_DEEP.md
- Code of conduct - expectations for project participation.
- Contributing - local setup, test gates, and contribution rules.
- Security - supported branches and vulnerability reporting.
- License - Apache License 2.0.
curl -X POST $CONCORD_API_BASE/api/runs \
-H "Content-Type: application/json" \
-d '{
"workflow_id": "WF-...",
"task_spec": {
"task": "Survey reliability patterns in multi-agent systems",
"research_question": "What architectural patterns improve MAS reliability?"
}
}'Omitting mode uses the product default: the real AG2 swarm with Tavily and
Daytona regression. Explicit mode=stub remains available for deterministic
internal tests, not as the public product path.
- Register or import an AG2 workflow contract.
- Submit a real task or trace.
- Watch the run status move through queued, analyzing, and completed.
- Open the completed run and follow violation -> evidence -> AG2 primitive -> repair patch -> regression result.
- Export the completed report JSON.
- Reopen the run from persisted history.
For deterministic local verification, run python run_all.py --fixture (no API keys needed). Fixture mode exercises the same Zone B report path while skipping live Zone A execution.
The FastAPI layer supports a dev bootstrap path and tenant-scoped API keys:
uvicorn api.index:app --port 8765
curl -X POST http://localhost:8765/api/api-keys \
-H 'Content-Type: application/json' \
-d '{"tenant_id":"tenant-a","name":"tenant-a primary"}'
curl http://localhost:8765/api/tenant/usage \
-H "Authorization: Bearer <returned-api-key>"When no API keys exist, unauthenticated requests use the local tenant for demo setup. After a key exists, /api/* routes require a bearer key except /api/health; browser SSE uses a short-lived stream token minted from an authenticated run.
Multi-agent systems fail silently. An agent claims it verified sources but sets verified_sources_count=0. Another runs a side-effect action without waiting for human approval. A third records no tool call despite claiming it searched. These are contract violations β the gap between what an agent says it did and what it actually produced in the trace.
Concord is a diagnostic pipeline that sits outside the target workflow, reads its execution trace, and systematically detects, attributes, and repairs those gaps.
Zone A: target workflow (demo fixture can be broken by design)
β execution trace (JSON)
Zone B: Concord diagnostic (detects, attributes, repairs)
β Contract Violation Report
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β python run_all.py (or --fixture to skip Zone A) β
ββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββ
β
βββββββββββββββββββββΌβββββββββββββββββββββββ
β ZONE A β
β Literature Review Assistant β
β (broken by design) β
βββββββββββββββββββββ¬βββββββββββββββββββββββ
β
task.json βββββββββββββββ€
(task, research_question, β
run_id) β
βΌ
ββββββββββββββββββββββββββββββ
β ResearcherAgent [step 1] ββββ Tavily API
β tool_call_id = "tc_001" β 3 sources retrieved
ββββββββββββββββ¬ββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββ
β CriticAgent [step 2] β critique notes
β tool_call_id = null β risk flags
ββββββββββββββββ¬ββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββ
β VerifierAgent [step 3] β β INTENTIONALLY BROKEN
β tool_call_id = null β β β Contract C2 violation
β verified_sources = 0 β β β Contract C1 violation
ββββββββββββββββ¬ββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββ
β ReporterAgent [step 4] β runs despite 0 verified
β produces final_output β sources (cascading failure)
ββββββββββββββββ¬ββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββ
β HumanGate [gate] β β INTENTIONALLY BROKEN
β approval = "pending" β β β never approves
ββββββββββββββββ¬ββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββ
β ActionAgent [step 5] β β INTENTIONALLY BROKEN
β runs without approval β β β Contract C4 violation
ββββββββββββββββ¬ββββββββββββββ
β
βΌ
trace_emitter.py
zone_b/fixtures/sample_trace.json
β
βββββββββββββββββββΌβββββββββββββββββββββββ
β ZONE B β
β Concord Diagnostic β
βββββββββββββββββββ¬βββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββ
β TraceCollector [B1] β JSON β RunTrace
β (no LLM) β + ContextSnapshot
ββββββββββββββββ¬ββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββ
β ContractChecker [B2] β checks 5 rules
β (deterministic + LLM β β 4 Violation objects
β for human text) β 3 HIGH, 1 MEDIUM
ββββββββββββββββ¬ββββββββββββββ
β
4 violations
β
βΌ
ββββββββββββββββββββββββββββββ
β Attribution [B3] ββββ LLM
β β failed_agent = VerifierAgent
ββββββββββββββββ¬ββββββββββββββ failed_step = 3
β
βΌ
ββββββββββββββββββββββββββββββ
β Repair [B4] ββββ LLM
β β primitive = Guardrail
ββββββββββββββββ¬ββββββββββββββ confidence = 0.85
β
βΌ
ββββββββββββββββββββββββββββββ
β RegressionTest [B5] ββββ LLM generates test code
β ββββ Daytona sandbox runs it
ββββββββββββββββ¬ββββββββββββββ status: pass / fail
β
βΌ
ββββββββββββββββββββββββββββββ
β Reporter [B6] ββββ LLM
β β Contract Violation Report
ββββββββββββββββ¬ββββββββββββββ + narrative
β
βΌ
ββββββββββββββββββββββββββββββ
β HumanGate [B7] β auto-approves (demo)
β β approval_status = approved
ββββββββββββββββ¬ββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββ
β CONTRACT VIOLATION REPORT β
β ββββββββββββββββββββββββββββββββββββ£
β run_id : run_041 β
β violations : 3 (all HIGH) β
β failed_agent : VerifierAgent β
β primitive : Guardrail β
β confidence : 0.85 β
β regression : pass / fail β
β approval : approved β
βββββββββββββββββββββββββββββββββββββ
Zone A is deliberately broken in exactly three ways. Zone B must detect all three.
| ID | Contract type | Severity | What breaks | AG2 primitive to fix |
|---|---|---|---|---|
| C1 | evidence |
HIGH | VerifierAgent sets verified_sources_count=0 in context_delta β ReporterAgent runs with no verified evidence |
Guardrail |
| C2 | tool |
HIGH | VerifierAgent has tool_call_id=None despite claiming to verify sources β no tool use recorded |
OnContextCondition |
| C3 | approval |
HIGH | ActionAgent runs with approval_status="pending" β HumanGate never approves |
HumanGate |
These failures cascade: C2 prevents C1 from being fixable at runtime, and C3 is independent. Zone B's ContractChecker checks each rule deterministically against the ContextSnapshot, then uses an LLM to generate human-readable expected / observed text for each violation.
All data flowing between agents β across both zones β is typed via dataclasses in shared/models.py. Nothing passes as raw dicts between pipeline stages.
ToolEvent
tool_name str "tavily_search"
input Any the query sent
output Any "3 results returned"
status str "success" | "failure"
evidence_id str "ev_001"
timestamp float
TraceEvent β one per agent turn in Zone A
step int 1 β 5
agent str "ResearcherAgent" etc.
type str "agent_turn"
content str the agent's text output
tool_call_id str | None non-null only for ResearcherAgent
context_delta dict incremental state update
handoff_to str | None next agent in chain
timestamp float
RunTrace β full Zone A execution record
run_id str "run_041"
workflow_name str
events list[TraceEvent]
final_output Any
ContextSnapshot β folded state after all events
retrieved_sources list from ResearcherAgent
verified_sources_count int 0 (broken by design)
tool_events list[ToolEvent]
approval_status str "pending" | "approved" | "rejected"
failed_agent str | None
failed_step int | None
final_output Any
Violation β one per broken contract
contract_type str "evidence" | "tool" | "approval" etc.
severity str "high" | "medium" | "low"
rule str
expected str LLM-generated human text
observed str LLM-generated human text
failed_agent str
failed_step int
Each TraceEvent carries a context_delta β the incremental state change that agent produced. TraceCollector._build_context_snapshot() folds all deltas left-to-right into a single ContextSnapshot:
- Scalar fields (
verified_sources_count,approval_status,final_output): last write wins tool_events: accumulated (extended, not replaced)- Unknown keys (e.g.
action_event): silently dropped duringContextSnapshotconstruction
This means Zone A agents only need to emit what they changed β not the full state.
Zone A is the target workflow β the system being monitored. It implements a Literature Review Assistant.
task.json
β
βββ ResearcherAgent calls Tavily, returns 3 sources + summary
βββ CriticAgent critiques sources, returns notes + risk flags
βββ VerifierAgent [BROKEN] returns tool_call_id=None, verified_sources_count=0
βββ ReporterAgent assembles final_output dict (runs anyway)
βββ HumanGate [BROKEN] always returns approval_status="pending"
βββ ActionAgent [BROKEN] runs without checking approval_status
Every Zone A agent must return a dict with exactly these keys β run.py converts them to TraceEvent via _to_trace_event():
{
"step": int, # 1β5
"agent": str, # agent class name
"type": "agent_turn",
"content": str, # agent's primary text output
"tool_call_id": str | None, # non-null only when a real tool was called
"context_delta": dict, # incremental state β only what this agent changed
"handoff_to": str | None, # next agent name
"timestamp": float, # time.time()
}| Agent | Keys emitted in context_delta |
|---|---|
| ResearcherAgent | retrieved_sources, tool_events |
| CriticAgent | (empty) |
| VerifierAgent | verified_sources_count (= 0) |
| ReporterAgent | final_output |
| ActionAgent | action_event |
All Zone A agents use:
make_proxy(name)β creates a standardUserProxyAgentwithllm_config=False,human_input_mode="NEVER",is_termination_msg=lambda x: True,max_consecutive_auto_reply=0,code_execution_config=Falsestrip_json_fences(text)β removes```jsonfences from LLM responses beforejson.loads()
Zone B is the diagnostic pipeline β it never runs Zone A's agents, only reads the trace Zone A emitted.
| Stage | Agent | LLM? | Input | Output |
|---|---|---|---|---|
| B1 | TraceCollector | No | raw JSON | RunTrace + ContextSnapshot |
| B2 | ContractChecker | Yes (text only) | RunTrace + ContextSnapshot |
list[Violation] |
| B3 | Attribution | Yes | violations + trace | failed_agent, failed_step, likely_root_cause |
| B4 | Repair | Yes | violations + attribution | patches[], affected_primitive, patch_code, confidence |
| B5 | RegressionTest | Yes + Daytona | repair patch + violations | test_status, per_violation_results, sandbox_id |
| B6 | Reporter | Yes | all upstream outputs | Contract Violation Report dict |
| B7 | HumanGate | No | report | approval_status = "approved" |
repair.py maps violation type to AG2 primitive without an LLM call:
evidence β Guardrail
tool β OnContextCondition
routing β Handoff
approval β HumanGate
schema β Guardrail
The LLM is only used to generate patch_code and expected_impact text for each violation's mapped primitive.
Five rules, all checked deterministically against ContextSnapshot:
# C1 β evidence
lambda trace, snap: snap.verified_sources_count > 0
# C2 β tool
lambda trace, snap: any(
e.agent == "VerifierAgent" and e.tool_call_id
for e in trace.events
)
# C3 β approval
lambda trace, snap: snap.approval_status == "approved"The LLM is called after a check fails β only to produce expected / observed strings for the report.
make_proxy(name)β same contract as Zone A's versionparse_json_body(body)β strips```jsonfences and raisesValueErroron parse failure (not silent)
RegressionTest uses an LLM to generate a self-contained Python script that simulates the post-repair state and asserts each violation is no longer reachable. The script is executed in a fresh Daytona sandbox via daytona.process.code_run(). The sandbox is always deleted, even on error. The stage returns aggregate test_status plus one per_violation_results[] row per violation. If DAYTONA_API_KEY or DAYTONA_API_URL are absent, the stage returns test_status="error" and sandbox_id="no-sandbox" without crashing the pipeline.
.
βββ run_all.py entry point β chains Zone A β Zone B
β
βββ shared/
β βββ models.py all shared dataclasses (ToolEvent, TraceEvent,
β RunTrace, ContextSnapshot, Violation, ...)
β
βββ zone_a/ Literature Review Assistant (target workflow)
β βββ config.py get_llm_config() β Gemini Flash via OpenRouter
β βββ context_variables.py ZoneAContext dataclass
β βββ workflow_contract.py C1βC5 contract definitions (reference)
β βββ trace_emitter.py writes RunTrace β zone_b/fixtures/sample_trace.json
β βββ run.py pipeline orchestrator + _to_trace_event()
β βββ fixtures/
β β βββ task.json task, research_question, run_id
β βββ agents/
β βββ _utils.py make_proxy(), strip_json_fences()
β βββ researcher.py Tavily search β sources + summary
β βββ critic.py critique notes + risk flags
β βββ verifier.py β BROKEN: tool_call_id=None, verified_sources_count=0
β βββ reporter.py final_output dict assembly
β βββ human_gate.py β BROKEN: always returns pending
β βββ action_agent.py β BROKEN: runs without approval
β
βββ zone_b/ Concord diagnostic pipeline
β βββ config.py get_llm_config() β same model, same pattern
β βββ utils.py make_proxy(), parse_json_body()
β βββ orchestrator.py wires B1βB7 sequentially
β βββ run.py standalone Zone B runner (reads fixture)
β βββ sandbox_run.py Daytona demo runner with mock trace
β βββ fixtures/
β β βββ sample_trace.json pre-baked run_041 trace (4 violations)
β βββ contracts/ dataclass registry + YAML contract DSL
β βββ agents/
β βββ trace_collector.py JSON β RunTrace + ContextSnapshot (no LLM)
β βββ contract_checker.py registry-backed deterministic checks + LLM text
β βββ attribution.py LLM β failed_agent + root cause
β βββ repair.py primitive map + LLM patch code
β βββ regression_test.py LLM test gen + Daytona execution
β βββ reporter.py LLM narrative + report assembly
β βββ human_gate.py auto-approves (demo mode)
β
βββ public/ frontend (Vercel-deployed mission-control dashboard)
β βββ index.html self-contained HTML + inline React app + fixture
β βββ styles.css monospace dark-mode UI tokens
β βββ app.jsx 7-screen React component tree (split-file copy)
β βββ data.js extracted CONCORD_DATA fixture (split-file copy)
β
βββ api/ backend HTTP layer (FastAPI; not deployed yet)
β βββ index.py routes: /api/health, /api/runs, /api/runs/{id}.js, approval
β βββ adapter.py Zone B report β CONCORD_DATA shape
β βββ store.py in-memory run store seeded with RUN-041
β
βββ vercel.json static-only deploy config
βββ requirements.txt FastAPI deps (for the api/ layer)
β
βββ tests/
βββ conftest.py shared fixtures (sample_trace_raw, clean_trace_raw, ...)
βββ test_models.py 21 β dataclass field integrity
βββ test_trace_collector.py 30 β parsing, folding, snapshot building
βββ test_contract_checker.py 26 β contract lambdas + step lookup
βββ test_attribution.py 10 β deterministic fallback paths
βββ test_repair.py 20 β per-violation patches, PRIMITIVE_MAP, scalar aliases
βββ test_regression_test.py 20 β _parse_status, fallback test execution
βββ test_reporter.py 15 β report assembly, severity summary, patches
βββ test_human_gate.py 6 β approval output shape
βββ test_zone_a.py 22 β strip_json_fences, _to_trace_event, agent shapes
βββ test_integration.py 20 β Zone AβB schema, 4 violations, clean trace = 0
βββ test_per_violation_repairs.py 3 β per-violation regression statuses
βββ test_rigorous.py 57 β edge cases, error paths, boundary conditions
βββ test_routing_contract.py 3 β routing contract broken + clean trace cases
βββ test_schema_contract.py 6 β schema contract missing-key + fixture cases
βββ test_swarm.py 27 β AG2 swarm tools, handoffs, guardrails, trace extraction
ββββ
288 total (274 non-integration + 14 integration-marked)
git clone https://github.com/d3v07/AG2_Hackathon.git
cd AG2_Hackathon
pip install -e .Create .env in the repo root:
OPENROUTER_API_KEY=your_openrouter_key
TAVILY_API_KEY=your_tavily_key
DAYTONA_API_KEY=your_daytona_key
DAYTONA_API_URL=https://app.daytona.io/apiOnly OPENROUTER_API_KEY is required to run Zone B. TAVILY_API_KEY is required to run Zone A live. DAYTONA_* is required only for the sandboxed regression test stage.
Uses the pre-baked zone_b/fixtures/sample_trace.json (run_041, 4 violations). Runs the full Zone B diagnostic pipeline live with your LLM keys.
python run_all.py --fixtureRuns Zone A end-to-end (Tavily search β agent chain β trace emission), then runs Zone B on the freshly generated trace.
python run_all.pypython zone_b/run.pypython zone_a/run.py============================================================
CONCORD β Full Pipeline Run
============================================================
[Zone A] Skipped β using fixture trace
Trace loaded from zone_b/fixtures/sample_trace.json
[Zone B] Running diagnostic pipeline...
[1/7] TraceCollector β loading zone_b/fixtures/sample_trace.json
run_041: 5 events, 1 tool call(s), handoff path length 5
[2/7] ContractChecker β applying contracts
4 violation(s) found
[3/7] Attribution β identifying failed agent
failed_agent=VerifierAgent step=3
[4/7] Repair β mapping to AG2 primitive
affected_primitive=Guardrail confidence=0.85
[5/7] RegressionTest β running in Daytona
test_status=pass sandbox=<id>
per_violation=4 pass/0 fail/0 error
[6/7] Reporter β assembling Contract Violation Report
[7/7] HumanGate β approval check
Decision: APPROVED
============================================================
CONTRACT VIOLATION REPORT
============================================================
Run ID : run_041
Violations : 4
Severity : {'high': 3, 'medium': 1, 'low': 0}
Failed agent : VerifierAgent (step 3)
Root cause : VerifierAgent failed to use a tool to gather verified sources
Affected primitive: Guardrail
Repair confidence : 0.85
Approval status : approved
============================================================
# Fast β no API calls (~0.5s, 274 tests)
pytest tests/ -m "not integration"
# Full suite including LLM + Daytona integration tests
pytest tests/| File | Count | What it covers |
|---|---|---|
test_models.py |
21 | All shared dataclasses, field types, optional fields |
test_trace_collector.py |
30 | JSON parsing, context_delta folding, edge cases |
test_contract_checker.py |
26 | Contract lambdas, boundary values, step lookup |
test_attribution.py |
10 | Deterministic fallback, empty violations |
test_repair.py |
20 | Per-violation patch cardinality, scalar alias selection, full PRIMITIVE_MAP coverage |
test_regression_test.py |
20 | _parse_status edge cases, fallback test code executes PASS |
test_reporter.py |
15 | Report assembly, severity summary, repair patch passthrough, fallback narrative |
test_human_gate.py |
6 | Auto-approve shape, handles empty report |
test_zone_a.py |
22 | strip_json_fences, _to_trace_event, all 5 agent return shapes |
test_integration.py |
20 | Zone AβB schema compatibility, exactly 4 violations, clean trace = 0 |
test_per_violation_repairs.py |
3 | Per-violation regression status and reporter aggregation |
test_rigorous.py |
57 | Edge cases, error paths, partial violations, data-flow contracts |
test_routing_contract.py |
3 | Routing contract fails fixture, passes clean trace |
test_schema_contract.py |
6 | Schema contract fails missing keys, passes fixture |
test_swarm.py |
27 | Swarm tools, handoffs, guardrails, trace extraction |
All agents in this codebase follow these patterns. PRs that deviate will be rejected.
from autogen import ConversableAgent
from zone_b.config import get_llm_config # or zone_a.config
from zone_b.utils import make_proxy # or zone_a.agents._utils
agent = ConversableAgent(
name="AgentName",
llm_config=get_llm_config(),
system_message="...",
human_input_mode="NEVER",
max_consecutive_auto_reply=1,
code_execution_config=False, # always False
)
proxy = make_proxy("AgentNameProxy") # shared helper, never inline
result = proxy.initiate_chat(agent, message="...", max_turns=1)
output = result.chat_history[-1]["content"]Never trust LLMs to return bare JSON. Always strip fences first:
# Zone A
from zone_a.agents._utils import strip_json_fences
parsed = json.loads(strip_json_fences(result.chat_history[-1]["content"]))
# Zone B
from zone_b.utils import parse_json_body
parsed = parse_json_body(result.chat_history[-1]["content"]) # raises ValueError on failureBoth zones use the same get_llm_config() pattern β Gemini 2.5 Flash via OpenRouter, temperature 0.1:
def get_llm_config(model: str = "google/gemini-2.5-flash") -> dict:
return {
"config_list": [{
"model": model,
"api_key": os.environ["OPENROUTER_API_KEY"],
"base_url": "https://openrouter.ai/api/v1",
"api_type": "openai",
}],
"temperature": 0.1,
}- Never use
autogen.beta.Agent - Never call
get_config()β useget_llm_config()from the zone'sconfig.py - Always set
code_execution_config=Falseon both agent and proxy - Always set
human_input_mode="NEVER"on both max_consecutive_auto_reply=1on agents,0on proxies- Use
make_proxy()β never copy-paste the 8-lineUserProxyAgentblock - Validate JSON parsing at the boundary β never
json.loads()raw LLM output
| Tool | Version | Role |
|---|---|---|
| AG2 | >=0.12 |
Multi-agent framework β ConversableAgent, UserProxyAgent |
| Gemini 2.5 Flash | β | LLM for all agents, via OpenRouter |
| OpenRouter | β | OpenAI-compatible API proxy for Gemini |
| Tavily | β | Web search API for ResearcherAgent |
| Daytona | β | Sandboxed code execution for regression tests |
| Python | >=3.12 |
Required for `str |
| GitHub | Zone | Sprint 1 | Sprint 2 |
|---|---|---|---|
| d3v07 | Zone B | #1 β Scaffold + models + TraceCollector + ContractChecker | #5 β Wire Zone AβB + run_all.py |
| Frex22 | Zone B | #2 β Attribution + Repair + RegressionTest + Reporter + Orchestrator | #6 β Zone B full pytest suite + Daytona tests |
| PruthviVKadam | Zone A | #3 β Scaffold + ContextVariables + ResearcherAgent + CriticAgent | #7 β Zone A live run + trace schema validation |
| niharika2701 | Zone A | #4 β VerifierAgent + ReporterAgent + ActionAgent + HumanGate + trace_emitter | #8 β Contract Violation Report validation + demo run |