The product idea was an AI copilot that listens to a support call and starts solving the problem before the human agent does; a "realtime speculative assistant" The entire idea rests on one assumption: that support calls reveal the specific root cause early enough to act on, so instead of building the copilot, I built a cheap, blind test on that single assumption.
This repo is that test. I built the first version using the transcripts of a support call, and if the results were positive, move into voice.
The answer was no.
The pass bar was simple: predict the specific root cause from the opening of the call, blind, on at least 60% of calls (or the transcripts, in this case). The idea wasn't to get the category the call belongs to, it was to to find the specific cause, because that's what the human agent would need early for the copilot to be worth anything.
I ran it on 48 synthetic B2B support call transcripts and scored every prediction blind. For this, I gave the LLM the first 6 turns of each transcript (the "first 60-90 seconds") and asked it to name the specific root cause.
Result: 2%.
I checked it a few ways and it held up:
- First, I ran it on transcripts written by a weaker model — got 14%. Then I rewrote the transcripts with a stronger model and it dropped to 2%. The early signal was just the weak model making it obvious in the script.
- I swept the window to find where prediction actually gets reliable. It crosses 60% around turn 12-13 — well into the transcript, not the opening.
- The one bright spot (use case with "migration related" looked predictable around turn 10) vanished with the stronger model: 0% early. It was never real.
- I gave it the entire transcript and it got 92% right; so the AI pipeline is fine, just that the information isn't there early enough.
Used a Dual model pipeline for this:
- GPT writes the calls (each with a hidden answer key).
- Claude reads the call and makes the prediction — it never sees the answer.
- GPT compares the prediction to the answer key.
The predictor and the judge are never the same model. A boundary test enforces that the predictor can't read the answer key. There's also a leakage check that flags calls where the customer accidentally says the answer too early, so I can report the clean calls separately.
- Actual (and live) product telemetry into the copilot; I don't have access to that right now.
- Real support calls - transcripts or voice. These are "synthetic" support call transcripts that were generated from a tool I built: support-call-generator. It could work with production grade support calls.
| Module | What it is |
|---|---|
loader.py |
Resolves the generator's export dir; loads transcripts, ground_truth, leakage reports, manifest. Defines the 3 scenario categories. |
windows.py |
The "how much of the call does the model see" abstraction. early_window = first N turns (the gate's 6-turn proxy); full_window = whole call. render_turns formats them. |
annotator.py |
The blind predictor. System prompt = "you read only an early fragment, you have NOT seen the rest, any answer key, or hidden notes." This is Claude. |
scorer.py |
The judge. Strict GPT prompt: "only call it a match if the prediction captures the same operational root cause." Compares prediction vs hidden key. |
metrics.py |
Aggregation + GATE_THRESHOLD = 0.60. Excludes leakage=FAIL calls from the gate set; computes early vs full accuracy. |
report.py |
Writes scores.csv + summary.md artifacts. |
cli.py |
python -m voice_eval run → predict → score → report into runs/latest. |
llmio.py |
Tolerant JSON parser for model output (strips markdown fences etc.). |
tests/test_boundary.py |
The integrity guarantee — proves the predictor can't reach the answer key. |
tests/test_scorer.py |
Tests the judge logic. |
source .venv/bin/activate
export OPENAI_API_KEY="..." # the judge
export ANTHROPIC_API_KEY="..." # the predictor
python -m voice_eval run # predict -> score -> report into runs/latest
pytest -q # boundary + scorer tests
Config is via env vars (VE_EARLY_TURNS, VE_ANNOTATOR_MODEL,
VE_JUDGE_MODEL, VE_EXPORTS_DIR). See .env.example.
voice_eval/ — loader, windows, annotator, scorer, metrics, report, cli
curve.py — sweeps the window size to find where prediction works
escalation_probe.py — can it predict an escalation early? (no)
postcall_probe.py — the post-call summary fallback probe
tests/ — boundary test + scorer test
Calls written by | Predict from first 6 turns | Given the whole call
weaker model (n=51) | 14% | 92%
stronger model (n=48) | 2% | 92%