Skip to content

Make count-question auto-run authoritative and deterministic#8

Merged
Robbie1977 merged 1 commit into
mainfrom
feature/answer-quality-recipes
Jul 15, 2026
Merged

Make count-question auto-run authoritative and deterministic#8
Robbie1977 merged 1 commit into
mainfrom
feature/answer-quality-recipes

Conversation

@Robbie1977

Copy link
Copy Markdown
Contributor

Fixes the medulla image-count question end to end. After v3.3.1 it regressed to "you can run a query" (no number); before that it had answered "28 images" (PartsOf, a class query). Both were the same two root causes.

Root causes

  1. Guard defeated the injector. maybeInjectCountQueryStep skipped whenever the planner had already planned any run_query — so when the planner picked the wrong query (PartsOf), the injector stepped aside and the wrong query ran; and when the planner planned nothing concrete, the injector still didn't get to correct it.
  2. Relied on the weak extractor. Even when the right query ran, the count had to be pulled out of the result table by the ELM extractor, which sometimes failed → the answer fell back to the digest's "run this query" text (the "no number" regression).

Fix

  • Authoritative routing — the injector no longer skips on a generic run_query. If the planner planned a run_query with the wrong query_type, it retargets that step to the semantically-matched query (PartsOf → ImagesNeurons); otherwise it injects one. It still defers only to the specialised connectivity / scRNA-seq / region-neuron-count macros, and only acts on an unambiguous match.
  • Deterministic count — the step is tagged count_query + count_noun + count_term; runStep reads count straight from the run_query result and adds a deterministic evidence row (VFB holds 226,524 images …), so the number and its correct unit reach the answer regardless of the extractor.

Tests

New offline end-to-end test (mocked ELM/MCP) with the user's exact question including the deliberate medualla typo: it runs ImagesNeurons (never PartsOf) and the evidence reads 226,524 images. Suites: orchestrator 23, orchestrator_run 23; whole offline unit suite green; next lint clean.

After merge

Tag v3.3.2. (Small code delta: lib/orchestrator.mjs + two tests.)

Separately — the task-battery CI gate failed PR #7 on a single transient T3.3 timeout (errors > 0 → exit 1); worth making that non-blocking / retry-once so live-server flakes don't red-flag good PRs. Happy to do that too.

The count-question auto-run was defeated by its own guard (it skipped whenever
the planner had already planned any run_query) and, even when it fired, relied
on the weak extractor to pull the count out of the result table — so "how many
images of neurons with a part in the medulla" regressed to "you can run the
query" (no number), and earlier had run PartsOf (28 subparts) reported as "28
images".

- maybeInjectCountQueryStep no longer skips on a generic run_query. If the
  planner planned a run_query with the wrong query_type, it RETARGETS that step
  to the semantically-matched query (e.g. PartsOf -> ImagesNeurons); otherwise
  it injects one. Still skips only when a specialised connectivity / scRNA-seq /
  region-neuron-count macro already covers the count, and only on an unambiguous
  match.
- The step is tagged count_query + count_noun + count_term; runStep reads the
  count straight from the run_query result and adds a deterministic evidence row
  ("VFB holds 226,524 images …"), so the number and its correct unit reach the
  synthesiser regardless of the extractor.

Offline end-to-end test (mocked ELM/MCP) with the user's exact question incl. the
deliberate "medualla" typo: runs ImagesNeurons (never PartsOf) and the evidence
states "226,524 images". Suites: orchestrator 23, orchestrator_run 23; whole
offline unit suite green; next lint clean.
@Robbie1977 Robbie1977 merged commit 4208716 into main Jul 15, 2026
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant