feat: structured response#54
Merged
Merged
Conversation
Return the agent's final answer as a structured object instead of a single markdown blob, so the frontend can build a richer UI. - Add StructuredResponse schema (response, data_sources, temporal_coverage, sql_query, follow_up_questions) in app/agent/schemas.py - Stream the structured payload via a new EventData.structured_response field and persist it on the message row (new nullable JSON column + migration) - Expose dataset_id on table tools so data_sources can carry BD UUIDs - Rework the 'Resposta Final' prompt: prose-only response, with source, period, SQL and follow-ups moved to dedicated structured fields Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ctured schema When no period is given, always default to period_end and never downgrade to an earlier year on a completeness hunch: table metadata (period_start/period_end) is machine-generated and current, so it takes precedence over the hand-written usage guide, which can be stale. Also slim the StructuredResponse field descriptions so period/source policy lives in the system prompt rather than being duplicated (and contradicted) in the schema, and reframe temporal_coverage as 'the interval the SQL filtered' instead of a period_end prohibition. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply the Clarification Protocol as a gate ahead of the query flow: when a question is broad/exploratory or references an unspecified entity, stop and clarify instead of falling through to executing SQL. Broad questions explore datasets/tables but stop before execute_bigquery_sql and then guide the user; unnamed entities (e.g. 'meu município') are asked about rather than assumed. Also make StructuredResponse.follow_up_questions optional (None default). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ask which entity before querying when the user references one without naming it (any type); never assume a value. Drops the 'São Paulo'/city-specific example that anchored the model into assuming a default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…period rules Translate SYSTEM_PROMPT to English (keeping PT identifiers, dataset keywords and example queries) for stronger instruction-following, while responses stay in the user's language. Also: scope the no-tools clarification permission to the unnamed-entity case; on out-of-range periods inform the user and ask rather than silently adjusting the query; drop the redundant geographic-level cue; rename the sql_query field reference to sql_queries; remove the search-funnel section (now in the search_datasets docstring). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-locate decision-proximate rules with the tools that read them: search_datasets gains the hierarchical search funnel (moved from the system prompt); get_table_details drops the 'Next step: execute_bigquery_sql' nudge so a query isn't primed when one isn't warranted; execute_bigquery_sql gains a precondition to not query on broad or unnamed-entity questions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
StructuredResponse.sql_query (str) becomes sql_queries (list[str]) so a multi-query answer lists every result-backing query for reproducibility. Translate the schema field descriptions to English (values stay in the user's language). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
period_start/period_end now document the format matching `granularity` (YYYY / YYYY-MM / YYYY-MM-DD) with examples, and the granularity field restates the mapping — so the model emits e.g. 2026-05 for a monthly period, not just the year. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replays eval_gold.yaml threads turn-by-turn on the real agent and scores four deterministic dimensions (action / source / period / completed) from the structured response, writing a rich JSON for the later judge eval. Period scoring is granularity-aware (matches period_end at YYYY / YYYY-MM / YYYY-MM-DD); records the system prompt and full tracebacks per run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…to the SQL filter data_sources may now list the tables the answer draws on — those queried, or specific tables recommended when clarifying/guiding — not only tables that were queried. temporal_coverage is now explicitly the interval the SQL query filtered, left empty when no query ran (e.g. a clarification turn) or there is no temporal dimension. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds three transcript scorers on top of the deterministic replay harness,
all reading the transcript eval_output.py produces:
- eval_quality: LLM judge — answer correctness/grounding vs gold
- eval_faithfulness: gold-free structural self-consistency of the
structured output
- eval_queries: source/period from the executed SQL (also works on the
free-text baseline)
Renames eval_threads.py -> eval_output.py so the script names describe what
each evaluates. Adds eval_gold.yaml (10 threads / 18 turns) and ignores the
generated transcripts/scores (eval/*.json).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thoughts are already captured in LangSmith traces, so exposing them in
the frontend stream just adds noise; tool_call args and tool_output
already give clients enough to render good UI/UX.
Also fixes get_table_details tests, which broke after TABLE_DETAILS_QUERY
started requesting `dataset { id }` to populate the table's dataset_id.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a structured output contract for the agent's final answer — alongside the prose response, it now returns the datasets/tables used, the temporal coverage of the answer, the SQL queries run, and suggested follow-up questions. Also updates the system prompt (translated to English, gate ambiguous questions into a clarification step before querying, etc.) and adds a standalone eval harness to score output quality against a gold set.