diff --git a/.claude/skills/srd-creator/SKILL.md b/.claude/skills/srd-creator/SKILL.md index a1892b7c3..342cc0676 100644 --- a/.claude/skills/srd-creator/SKILL.md +++ b/.claude/skills/srd-creator/SKILL.md @@ -10,47 +10,166 @@ style used in the existing Evaluation / Fast Evaluation / STT Evaluation SRDs. ## What an SRD is -An SRD is the contract written *before* code. It states what to build, why, -what is in/out of scope, the execution flow, the API surface, and the DB schema — -specific enough that an engineer can build it and a reviewer can test it against -the Functional Requirements table. It is not design prose; it is testable spec. +An SRD is the contract written *before* code: what to build, why, what is in/out +of scope, the execution flow, the API surface, and the DB schema, specific enough +that an engineer can build it and a reviewer can test it against the Functional +Requirements table. It is testable spec, not design prose. + +## A PRD is mandatory + +**The SRD is derived solely from a PRD.** If the user has not supplied a PRD, a +file path or pasted content, stop and ask for one. Do not invent the problem, +scope, endpoints, or schema from the conversation, the codebase, or assumption. +No PRD → no SRD. ## Workflow 1. **Read the reference material first.** - - `reference/srd-guide.md` — what each section means and what good content looks like. - - `reference/srd-template.md` — the skeleton to fill in. - - If unsure of house style, skim a prior SRD; the three originals live as - `*.docx` in the repo root. + - `reference/srd-guide.md`, what each section means and what good content looks like. + - `reference/srd-template.md`, the skeleton to fill in. + - If unsure of house style, skim a prior SRD; the originals live as `*.docx` in the repo root. -2. **Gather the inputs.** Before writing, you need: the feature name, the problem - it solves, who the users are, what is in scope for Phase 1 vs later phases, the - endpoints, and the data model. If the user has not supplied these, ask — do not - invent endpoints or schema. Missing info is the #1 cause of a useless SRD. +2. **Load the Quality Checklist into a `TodoWrite` list before writing.** Hold it + for the whole creation so each rule is checked once and not re-litigated per + section. Tick items as sections land; run a final pass against it before output. -3. **Fill the template** section by section. Drop optional sections that don't - apply (e.g. Resources, Configuration) rather than leaving them empty. +3. **Read the PRD and map it onto the template.** Problem → Introduction, Goals → + Goals, Users → Introduction/personas, scope → Assumptions & phasing, etc. Fill + section by section. Anything the PRD leaves open goes under *Design Decisions / + Known Limitations* as an open question, never fabricated. Drop optional sections + that don't apply (Resources, Configuration) rather than leaving them empty. + - **Resources links are never fabricated or guessed** (no inventing + `Fast Evaluation SRD.md`-style paths or PRD URLs). Ask the user for the actual + Google Docs / PRD / related-SRD links; if they have none, drop the section. 4. **Write the Functional Requirements table** as the testable core. Every row = one user-facing behavior + a concrete acceptance criterion + a Status. If you - can't write an acceptance criterion for a requirement, the requirement is too - vague — sharpen it. + can't write a checkable acceptance criterion, the requirement is too vague; + sharpen it. 5. **Output** as `features//SRD.md`, where `` is a short kebab-case slug for the feature (e.g. `features/llm-judge-correctness/SRD.md`). Create the `features//` directory if it doesn't exist. Every feature - keeps its SRD and PRD together in this one folder — if a `PRD.md` already exists + keeps its SRD and PRD together in this one folder, if a `PRD.md` already exists for the same feature (written by the `start-prd` skill), write the `SRD.md` alongside it in the same folder rather than creating a new location. Reuse the - existing slug exactly; do not invent a parallel folder. + existing slug exactly; do not invent a parallel folder. The H1 inside the file is + the feature's display name (e.g. `# Native LLM as Judge Correctness Score SRD`). + +6. **Generate exactly one image: the primary execution-flow diagram. Render it into + `assets/`, do not embed it inline, and leave a placeholder band naming the file.** + The author exports the `.md` to Google Docs and manually pastes the image at its + band, so the band's job is to say *which* file goes *where*. + - **One image only.** The single SRD image is the main execution flow. Everything + else is words or tables, no second image: a second flow (e.g. plain config CRUD) + is prose plus the Endpoints section, and the **DB schema is always column tables, + never a diagram**. + - **Render it.** Write a mermaid source for the execution flow, then run the + skill's helper script (high-res mermaid render to a crisp png; no global + installs, it uses `npx`): + ```bash + scripts/render-diagram.sh flow-a.mmd features//assets/flow-a.png + ``` + - **Placeholder band in the `.md`.** A horizontal-rule-fenced block (renders as a + distinct band in GitHub *and* after a Google Docs markdown import, which turns + `---` into a real line) naming the asset file: + ```markdown + --- + + **>> PLACE IMAGE HERE: `assets/flow-a.png`, judged fast-eval run.** + System-level sequence: `User`, `Kaapi Backend`, `OpenAI`, `Langfuse`. + + --- + ``` + - **No HTML tags and no inline CSS** (e.g. no `
`), the Google Docs + markdown importer strips them, so colored callouts do not survive. Plain + markdown only. **No emoji.** Do not use `![](...)` inline embeds (a relative + image path does not resolve after the Google Docs import; the band is the cue). + +## Quality Checklist + +Load these as todos at the start; verify each before output: + +- [ ] PRD supplied and read; every SRD claim traces back to the PRD. +- [ ] Output at `features//SRD.md`, alongside the feature's `PRD.md` + (reuse the existing slug; don't create a parallel folder). +- [ ] All required sections present (Introduction, Goals, Assumptions, Detailed + Design, Functional Requirements; Endpoints if the feature has an API; DB + Schema if there's a data model). +- [ ] **No fact stated more than once**, each fact lives in one home section; + elsewhere reference it, don't restate it. +- [ ] No filler, hedging, or restated bullets; every sentence adds new information. +- [ ] Phasing explicit (Phase 1 vs Phase 2+). +- [ ] Every FR row has a concrete, testable acceptance criterion + Status. +- [ ] Every endpoint shows real request/response JSON, not just field lists. +- [ ] DB schema follows Kaapi conventions (`inserted_at`/`updated_at`, + `organization_id` + `project_id`, snake_case, FK indexes). +- [ ] High-level only, no function names, internal variables, or step internals + (DB schema, class/endpoint contracts, error codes, settings are fine). +- [ ] Existing models reused, not duplicated, checked the codebase first; any new + shape is justified by no existing model fitting. +- [ ] Exactly one image (the execution flow) rendered to + `features//assets/flow-a.png`, with a horizontal-rule-fenced band + naming it (no inline `![](...)` embed, no mermaid blocks, no HTML, no emoji). +- [ ] DB schema is column tables, not a diagram; no second image anywhere. +- [ ] Optional empty sections deleted, not left as placeholders. ## Rules +- **No redundancy.** State each fact exactly once, in its home section. Later + sections reference it ("as in Goals"), they don't repeat it. Goals are outcomes, + not echoes of the Introduction. Assumptions are boundaries, not re-listed goals. + FR acceptance criteria are checkable conditions, not restatements of the behavior + column. Repetition across sections is the #1 cause of bloated SRDs. +- **Keep it high-level. No variable/function-level detail.** An SRD describes + behavior and interfaces, not implementation internals. Allowed: DB schema (tables, + columns, constraints), object/class and endpoint contracts, error codes, settings. + Not allowed: specific function names, internal variable/field names, private + helpers, stage/step internals, concurrency-pool names, the reader sees those in + PR review. Name *what* a thing does and *which entity* holds it, not the symbol + that implements it (e.g. "the run's per-row score storage", not `per_item_scores`). +- **Reuse existing models, don't invent new ones.** Before specifying any config, + request body, or pydantic/SQLModel class, search the codebase for an existing one + that fits (e.g. `TextLLMParams` for model + sampling + prompt). Default order: + (1) reuse as-is, (2) extend/compose the existing one, (3) only if nothing fits, + add a new shape, and call out explicitly why no existing model worked. Never + introduce a parallel config that duplicates an existing one. A reviewer asking + "can't we reuse X?" is a failure of this rule. +- **Diagrams over prose for flows, but the SRD only holds a placeholder.** Both the + execution flow and the data model want a diagram, not a long numbered paragraph. + The skill does not draw it; it leaves an author image placeholder note (Workflow + step 6) and keeps the surrounding prose to what a diagram can't carry (failure + isolation, idempotency, resolution rules). Tell the author what the diagram should + depict so they can draw it: + - **Execution flow at system level, not internals.** The diagram should show the + user and the real systems that talk to each other (e.g. `User`, `Kaapi Backend`, + `OpenAI`, `Langfuse`), each with arrows in and out. It should not turn internal + pipeline steps or in-process helpers into separate lanes (e.g. a "Judge" lane + that is really an OpenAI call), which misleads readers into thinking they're + separate services. + - **Data model** should mark reused vs new entities. +- **No em dashes.** Use commas, periods, or parentheses. Keep prose clean and + developer-readable; short sentences over long dash-joined clauses. +- **Match existing system naming.** Column names, table names, request/response field + names, and class names must follow the codebase's conventions and reuse existing + names where one exists (snake_case columns, the same field names as the reused + model, e.g. `llm_params` carrying `TextLLMParams`). The schema must not read like it + came from a different system. Grep for an existing name before coining a new one. - Match Kaapi conventions: `inserted_at`/`updated_at` timestamps (not `created_at`), `organization_id` + `project_id` on every multi-tenant table, snake_case columns. - Phase the scope explicitly. State Phase 1 (build now) vs Phase 2+ (later) so scope creep is visible. +- **Functional Requirements are PR-testable functional behaviors only.** Every FR row + is something a developer or client can verify works when reviewing the PR before it + ships. Drop rows that merely restate the Intro/Goals, describe internal mechanics, + or can't be checked against the running system. Lean beats exhaustive. - Show real request/response JSON for every endpoint, not just field lists. +- **Error responses add value, don't restate the obvious.** Standard codes (409, 422, + 404) are self-explanatory; the value is the readable client-facing message. Show the + actual message string (or a field-specific example for validation), not a paraphrase + of the status code. - DB schema as a table: Column / Type / Nullable / Default / Description, plus a Constraints list (unique keys, FKs, indexes). +- Keep the reference style: consistent table headers, one blank line between + sections. - Don't pad. Each section earns its place; cut what doesn't apply. diff --git a/.claude/skills/srd-creator/reference/srd-guide.md b/.claude/skills/srd-creator/reference/srd-guide.md index ef5545fff..549386b8b 100644 --- a/.claude/skills/srd-creator/reference/srd-guide.md +++ b/.claude/skills/srd-creator/reference/srd-guide.md @@ -1,20 +1,29 @@ -# SRD Guide — what each section is and how to write it +# SRD Guide, what each section is and how to write it -An SRD (Software Requirements Document) is written before implementation. It is the -shared contract between whoever requested the feature and whoever builds it. Good -SRDs are *testable*: a reviewer can take the Functional Requirements table and check -each row against the running system. +An SRD (Software Requirements Document) is the testable contract between whoever +requested the feature and whoever builds it, written before implementation: a +reviewer can take the Functional Requirements table and check each row against the +running system. This guide describes each section, derived from the Kaapi Evaluation, Fast Evaluation, and STT Evaluation SRDs. Use `srd-template.md` as the fill-in skeleton. +**The SRD is derived solely from a PRD**, no PRD, no SRD. Map PRD sections onto the +template; record anything the PRD leaves open as an open question under *Design +Decisions / Known Limitations*, never as invented detail. + +**No redundancy.** State each fact exactly once, in its home section; later sections +reference it instead of repeating it. Goals are outcomes, not echoes of the +Introduction. Assumptions are boundaries, not re-listed goals. FR acceptance criteria +are checkable conditions, not restatements of the behavior column. + --- ## 1. Introduction & Purpose *(required)* The "what and why" in a few short paragraphs. - One sentence: what capability this SRD defines and for which system. -- The problem / motivation — what's painful today, who feels it (name the early +- The problem / motivation, what's painful today, who feels it (name the early users if known, e.g. "early users from Glific"). - What the feature produces at minimum (the concrete outputs). - Explicit phasing: what is Phase 1, what is deferred to Phase 2/3. This is where @@ -23,8 +32,9 @@ The "what and why" in a few short paragraphs. ## 2. Resources *(optional)* -Links to related SRDs, external API docs, research notes, design docs. Drop the -section if there's nothing to link. +Links to related SRDs, external API docs, research notes, design docs. **Ask the +user for the real links** (Google Docs, PRD, related SRDs), never fabricate or +guess paths/URLs. Drop the section if the user has nothing to link. ## 3. Goals *(required)* @@ -32,7 +42,7 @@ A short bulleted list of what success looks like. Each goal is an outcome, not a task. Keep it to the handful that actually define done. Examples: - "Add a `run_mode="fast"` option so users can run a text evaluation synchronously." - "Identical scoring semantics to the batch path." -- "Failure isolation — one item's failure must not fail the whole run." +- "Failure isolation, one item's failure must not fail the whole run." ## 4. Assumptions & Constraints *(required)* @@ -47,16 +57,20 @@ Cover: ## 5. Detailed Design (Execution Flow) *(required)* -The step-by-step of how it actually runs. Number the steps in order. If there are -multiple flows (upload flow, run flow, async/polling flow, feedback flow), give each -its own numbered subsection. For multi-stage async work, describe each stage, its -skip/idempotency marker, and its retry behavior. Mention the sequence diagram if one -exists (the originals reference a "Sequence Flow Diagram" placeholder). +How it runs, with brief supporting text, not a long numbered paragraph. Render each +flow's diagram to a png in `assets/`, but do **not** embed it inline: where each +diagram belongs, leave a horizontal-rule-fenced band naming the asset file the author +pastes there (no mermaid, no HTML, no emoji). **Exactly one image per SRD: the primary +execution flow.** A second flow (e.g. plain config CRUD) is prose plus the Endpoints +section, not another image, and the DB schema is always column tables, never a diagram. +Keep the text to what a diagram can't carry: failure isolation, idempotency, +resolution rules. -Good content here names the actual mechanics: which API is called, what's written to -S3, which Celery task/queue runs it, what marks a stage complete, how retries dedupe. +Stay high-level, actors and behavior ("Pipeline → Judge: question + answer", +"persist both scores"), not internal function or variable names. Pipeline *stages* +named by behavior are fine; private helpers and field names are not. -## 6. Functional Requirements (Testing) *(required — the core)* +## 6. Functional Requirements (Testing) *(required, the core)* A table, one row per user-facing behavior. Columns: @@ -69,7 +83,7 @@ A table, one row per user-facing behavior. Columns: `dataset_too_large_for_fast` error and the actual unique-row count"). - **Status**: `Not Started` / `In Progress` / `Done`. -If you cannot write a crisp acceptance criterion, the requirement is too vague — +If you cannot write a crisp acceptance criterion, the requirement is too vague; split or sharpen it. This table is what QA and review run against. ## 7. Endpoints *(required when the feature has an API)* @@ -88,7 +102,8 @@ field"). ## 8. Database Schema / Tables *(required when there's a data model)* -For each table, a column table: +Always column tables, never a diagram or image (the one SRD image is the execution +flow). For each table, a column table: | Column | Type | Nullable | Default | Description | @@ -98,7 +113,7 @@ Then a **Constraints** list: primary key, unique constraints (name them, e.g. Kaapi conventions to enforce: - `id INTEGER PK` auto-increment (or UUID where the domain calls for it). - `organization_id` + `project_id` on every multi-tenant table. -- `inserted_at` / `updated_at TIMESTAMP NOT NULL DEFAULT now()` — **not** `created_at`. +- `inserted_at` / `updated_at TIMESTAMP NOT NULL DEFAULT now()`, **not** `created_at`. - Filterable data as first-class columns; bag-of-attributes as `JSONB`. - Prefer reusing existing tables; if so, state "No new tables" and list only the added columns / constraints, with the backfill plan for new non-null columns. @@ -121,3 +136,6 @@ so the next reader doesn't re-litigate it. Required: Introduction & Purpose · Goals · Assumptions & Constraints · Detailed Design · Functional Requirements · Endpoints (if API) · DB Schema (if data). Optional: Resources · Configuration · Design Decisions / Known Limitations. + +Before output, verify the full **Quality Checklist** in `SKILL.md`, PRD-traceable, +correctly named, no repeated facts, testable FRs, Kaapi DB conventions. diff --git a/.claude/skills/srd-creator/reference/srd-template.md b/.claude/skills/srd-creator/reference/srd-template.md index 1a7b9b99e..8b3341cf4 100644 --- a/.claude/skills/srd-creator/reference/srd-template.md +++ b/.claude/skills/srd-creator/reference/srd-template.md @@ -1,10 +1,12 @@ + + # SRD ## Introduction & Purpose <1–2 sentences: what capability this SRD defines and for which system (Kaapi).> - @@ -15,10 +17,10 @@ if known.> - **Phase 1:** - **Phase 2+:** - + ## Resources - + - - - @@ -38,20 +40,27 @@ if known.> ## Detailed Design (Execution Flow) - + + +### + +--- -### -1. -2. +**>> PLACE IMAGE HERE: `assets/flow-a.png`, .** -### -1. -2. +--- -### -1. + -> Sequence Flow Diagram: + + +### + + ## Functional Requirements (Testing) @@ -97,6 +106,9 @@ if known.> ## Database Schema + + ### `` @@ -104,26 +116,26 @@ if known.> |--------|------|----------|---------|-------------| | id | INTEGER (PK) | NO | auto-increment | Unique identifier | | | | | | | -| organization_id | INTEGER | NO | — | Reference to the organization | -| project_id | INTEGER | NO | — | Reference to the project | +| organization_id | INTEGER | NO | n/a | Reference to the organization | +| project_id | INTEGER | NO | n/a | Reference to the project | | inserted_at | TIMESTAMP | NO | now() | Created timestamp | | updated_at | TIMESTAMP | NO | now() | Last-updated timestamp | **Constraints:** -- `` — UNIQUE on () +- ``: UNIQUE on () - FK → .- Index on ## Configuration - + | Setting | Type | Default | Description | |---------|------|---------|-------------| | | | | | ## Design Decisions / Known Limitations - + - **:** - **Known limitation:** diff --git a/.claude/skills/srd-creator/scripts/render-diagram.sh b/.claude/skills/srd-creator/scripts/render-diagram.sh new file mode 100755 index 000000000..931ab5619 --- /dev/null +++ b/.claude/skills/srd-creator/scripts/render-diagram.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# Render a mermaid source to a high-quality png for an SRD. +# Usage: render-diagram.sh +# No global installs: uses npx for mermaid-cli. +set -euo pipefail + +if [[ $# -ne 2 ]]; then + echo "usage: $0 " >&2 + exit 1 +fi + +SRC="$1" +OUT="$2" +mkdir -p "$(dirname "$OUT")" + +# High-res render (wide + scaled) keeps sequence-diagram text crisp. +npx -y -p @mermaid-js/mermaid-cli mmdc -i "$SRC" -o "$OUT" -w 1600 -s 3 -b white + +echo "wrote $OUT" diff --git a/features/llm-judge-correctness/SRD.md b/features/llm-judge-correctness/SRD.md new file mode 100644 index 000000000..b349b21e7 --- /dev/null +++ b/features/llm-judge-correctness/SRD.md @@ -0,0 +1,289 @@ +# Native LLM-as-a-Judge Correctness Score SRD + +## Introduction & Purpose + +This SRD defines a native, reference-based LLM-as-a-judge **correctness** score for +Kaapi fast evaluations. Each evaluated row already returns a *cosine similarity* +score (how close the generated answer is to the ground truth). This feature adds a +second, independent **correctness** score (0 to 1) plus a short reasoning string, +produced by an LLM inside Kaapi with no manual Langfuse configuration. + +Today a correctness judgment is only available by hand-configuring a model-based +evaluator inside the third-party Langfuse dashboard, per project, outside the +platform. That setup is easily forgotten, the judging logic lives outside Kaapi +(unversioned and untailorable without engineering), and Kaapi keeps no record of +how a row was judged. Early users are NGO eval teams running fast evaluations on +their bots. + +The feature produces, per evaluated row: a correctness score, a reasoning string, +both persisted by Kaapi and written to the row's Langfuse trace alongside the +existing similarity score, plus a per-project judge configuration the team can +create, view, update, and delete. + +- **Phase 1 (this release):** automatic correctness judging on fast evaluations + only; zero-config default (built-in prompt + fallback model); per-project judge + config CRUD effective on the next run; both scores persisted and synced to + Langfuse; reasoning per row. +- **Phase 2+ (deferred):** batch-mode judging; judge error/retry handling; + confirmed performance budget and row-count guidance; rating-example entry UX and + final score labels. + +Intent: judging is automatic, zero-config, tailorable, explainable, and reversible, +owned inside Kaapi so eval owners never open Langfuse. + +## Goals + +- Correctness judging runs natively inside Kaapi with zero manual Langfuse setup. +- The judge is reference-based: it compares generated answer against ground truth + (and the question) and returns a 0 to 1 score plus reasoning. +- Judging is automatic, with no new trigger, flag, or opt-in on the run path. +- It works out of the box from a built-in default prompt and a fallback model when + no project config exists. +- A project team can tailor the judge (rating examples, model, model settings) and + have it take effect on the next run with no deploy. +- Deleting a project's judge config reverts to default prompt + fallback model. + +## Assumptions & Constraints + +- **Out of scope:** batch-mode judging (fast only); retiring Langfuse (results keep + syncing to it); robust judge error/retry handling. A failed or malformed judge + call must not block the similarity score or the run; the row's correctness is left + unscoreable and the run still completes. +- **Trigger:** judging runs inside the existing fast-eval pipeline. No new endpoint, + request field, or flag on the run path. +- **Limits:** fast eval is capped at `EVAL_FAST_MAX_UNIQUE_ROWS` (10) unique rows; + the judge adds one model call per evaluated row within that cap. +- **Per-row independence:** one row's judge failure must not fail sibling rows. +- **Reuse:** no new per-row table. Correctness rides the existing + `EvaluationRun.score` record (per-trace `scores` list + `summary_scores`) and a new + durable per-row map mirroring `per_item_scores`. One new table, + `evaluation_judge_config`, holds per-project judge configuration. Judge model + + sampling settings reuse `TextLLMParams` (the same shape the eval config uses); the + results read path already aggregates a `Correctness` score, so surfacing needs no + change. +- **Starting provider/model:** OpenAI, matching the fast-eval response/embedding + path. The fallback model is a configurable default; project config may override + the model and its settings. +- **Pricing:** the judge adds one LLM completion per row (paid), tracked under a + `judge` stage in `EvaluationRun.cost`. + +## Detailed Design (Execution Flow) + +The judge slots into the fast-eval pipeline as a scoring step that runs after +similarity is computed, so a judge failure can never block the cosine score. The run +is marked complete with both summary scores; per-row scores are then written to +Langfuse, and a failed write is recoverable from the durable maps on resync. + +### Judged fast-eval run + +--- + +**>> PLACE IMAGE HERE: `assets/flow-a.png`, judged fast-eval run.** +System-level sequence: `User`, `Kaapi Backend`, `OpenAI`, `Langfuse` (arrows in and out). + +--- + +Each judged row's correctness score and reasoning are appended to that trace's +`scores` list (reasoning carried in the score `comment`), and a `Correctness` summary +score is added to `EvaluationRun.score.summary_scores` next to `Cosine Similarity`. +The durable per-row correctness map is the source of truth for Langfuse resync. + +### Tailor the judge (self-service config) + +An admin creates, views, updates, or deletes the project's judge config through the +endpoints below; no diagram is needed for plain CRUD. Config is resolved once per run, at the judge step, so an in-flight run is unaffected +by a config change. An active (`deleted_at IS NULL`) config for the run's (org, +project) supplies its `llm_params`, `rating_examples`, and `prompt_override` (falling +back to the built-in prompt when unset); absence uses the fallback model and built-in +prompt with no rating examples. + +## Functional Requirements (Testing) + +| ID | What (user-facing behavior) | Acceptance criteria | Status | +|----|-----------------------------|---------------------|--------| +| FR-1 | Every evaluated row gets a correctness score | After a fast run completes, each scoreable row's persisted trace `scores` carries both a `Cosine Similarity` and a `Correctness` entry | Not Started | +| FR-2 | Correctness is 0 to 1 with reasoning | Each `Correctness` trace score has a numeric value in [0,1] and a non-empty reasoning `comment` | Not Started | +| FR-3 | Judging is automatic, no new trigger | Running the existing fast-eval endpoint with no new fields produces correctness scores; no request flag toggles it | Not Started | +| FR-4 | Zero-config default works | A project with no judge config still scores every scoreable row using the fallback model + built-in prompt | Not Started | +| FR-5 | Two summary scores persisted | `EvaluationRun.score.summary_scores` contains both a `Cosine Similarity` and a `Correctness` entry after completion | Not Started | +| FR-6 | Both scores on the Langfuse trace | Each evaluated row's Langfuse trace shows two distinctly-named scores | Not Started | +| FR-7 | Create judge config | `POST /evaluations/judge-config` returns 201 with the saved config (model, settings, rating examples) | Not Started | +| FR-8 | View judge config | `GET /evaluations/judge-config` returns the active config, or a default indicator when none exists | Not Started | +| FR-9 | Update takes effect next run | After `PATCH`, the next run's judge uses the updated model/settings/examples; an in-flight run is unaffected | Not Started | +| FR-10 | Delete reverts to default | After `DELETE`, the next run uses the fallback model + built-in prompt with no rating examples | Not Started | +| FR-11 | One active config per project | A second `POST` while an active config exists returns 409; never two active rows | Not Started | +| FR-12 | Per-row judge failure isolation | If a row's judge call fails or returns malformed output, that row's correctness is left unscoreable, its cosine score is unaffected, and the run still completes | Not Started | +| FR-13 | Judge cost tracked | After a run, `EvaluationRun.cost` includes a `judge` stage with token counts and USD | Not Started | +| FR-14 | Tenant isolation | A judge config for (org A, project A) is never resolved for any other (org, project) | Not Started | + +## Endpoints + +All judge-config endpoints are project-scoped under the evaluations router and +require the existing project permission. The fast-eval run trigger +(`POST /evaluations`) is unchanged: no new request fields, correctness surfaces +through the already-present run results. + +### `GET /evaluations/judge-config` +Return the calling project's active judge configuration, or a default indicator. + +**Response (config exists):** + +```json +{ + "data": { + "id": "b3f1c2d4-5e6a-7b8c-9d0e-1f2a3b4c5d6e", + "llm_params": { "model": "gpt-4o", "temperature": 0.0 }, + "rating_examples": [ + { + "question": "What is the capital of France?", + "answer": "Paris", + "ground_truth": "Paris", + "score": 1.0, + "reasoning": "Exact, correct answer." + } + ], + "prompt_override": null, + "organization_id": 12, + "project_id": 34, + "inserted_at": "2026-06-25T10:00:00Z", + "updated_at": "2026-06-25T10:00:00Z" + } +} +``` + +**Response (no config, project on default):** + +```json +{ "data": { "is_default": true, "llm_params": { "model": "gpt-4o-mini" } } } +``` + +### `POST /evaluations/judge-config` +Create the project's judge configuration. + +**Request body:** + +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| llm_params | object (`TextLLMParams`) | No | fallback model | Judge model and sampling settings | +| rating_examples | array | No | `[]` | Example-graded rows guiding the judge | +| prompt_override | string \| null | No | null | Replaces the built-in judge prompt | + +```json +{ + "llm_params": { "model": "gpt-4o", "temperature": 0.0 }, + "rating_examples": [ + { "question": "...", "answer": "...", "ground_truth": "...", "score": 1.0, "reasoning": "..." } + ], + "prompt_override": null +} +``` + +**Response:** `201 Created`, the saved config (same shape as `GET`). + +**Error responses:** + +| Status | Code | Message | +|--------|------|---------| +| 409 | judge_config_exists | "An active judge configuration already exists for this project." | +| 422 | invalid_judge_config | "rating_examples[0].score must be between 0 and 1." | + +### `PATCH /evaluations/judge-config` +Partially update the project's judge configuration. Effective on the next run. + +**Request body:** any subset of the `POST` fields. + +```json +{ "llm_params": { "temperature": 0.2 } } +``` + +**Response:** `200 OK`, the updated config. + +**Error responses:** + +| Status | Code | Message | +|--------|------|---------| +| 404 | judge_config_not_found | "No active judge configuration for this project." | + +### `DELETE /evaluations/judge-config` +Soft-delete the project's judge configuration; reverts the project to default. + +**Response:** `204 No Content`. + +**Error responses:** + +| Status | Code | Message | +|--------|------|---------| +| 404 | judge_config_not_found | "No active judge configuration for this project." | + +## Database Schema + +One new table plus one reused table with a new column, so the schema is presented as +tables below (no diagram needed at this complexity). + +### `evaluation_judge_config` (new) +Per-project judge configuration. At most one active row per (org, project); absence +means the platform default prompt + fallback model. + +| Column | Type | Nullable | Default | Description | +|--------|------|----------|---------|-------------| +| id | UUID (PK) | NO | uuid4 | Unique identifier | +| llm_params | JSONB | NO | `{}` | `TextLLMParams` (judge model + sampling settings) | +| rating_examples | JSONB | NO | `[]` | Example-graded rows guiding the judge | +| prompt_override | TEXT | YES | NULL | Replaces the built-in judge prompt; NULL means default | +| organization_id | INTEGER (FK) | NO | n/a | Reference to the organization | +| project_id | INTEGER (FK) | NO | n/a | Reference to the project | +| deleted_at | TIMESTAMP | YES | NULL | Soft-delete marker; NULL means active | +| inserted_at | TIMESTAMP | NO | now() | Created timestamp | +| updated_at | TIMESTAMP | NO | now() | Last-updated timestamp | + +**Constraints:** +- `uq_evaluation_judge_config_org_project_active`: UNIQUE on (`organization_id`, `project_id`) WHERE `deleted_at IS NULL` +- FK `organization_id` → `organization.id` (ON DELETE CASCADE) +- FK `project_id` → `project.id` (ON DELETE CASCADE) +- Index on (`project_id`) WHERE `deleted_at IS NULL` for run-time resolution + +### `evaluation_run` (existing, reused) +Correctness rides the existing score columns; one new column holds the durable +correctness map. + +| Column | Type | Now carries | +|--------|------|-------------| +| score | JSONB | `summary_scores` gains a `Correctness` entry; each `traces[].scores` gains a `Correctness` score with value + reasoning `comment` | +| per_item_correctness | JSONB (YES, default NULL) | New column mirroring `per_item_scores`: durable `{trace_id: correctness}` map, source of truth for Langfuse resync | +| unscoreable | JSONB | Reused for rows the judge could not score, alongside existing cosine reasons | +| cost | JSONB | Gains a `judge` stage (tokens + USD) | + +**Backfill plan:** `per_item_correctness` is nullable with default NULL; pre-feature +runs need no backfill (they carry no correctness data). + +## Configuration + +| Setting | Type | Default | Description | +|---------|------|---------|-------------| +| EVAL_JUDGE_FALLBACK_MODEL | str | `gpt-4o-mini` | Judge model when a project has no config | +| EVAL_JUDGE_DEFAULT_TEMPERATURE | float | `0.0` | Default judge temperature when not overridden | + +The built-in default judge prompt and the `Correctness` score name (mirroring +`COSINE_SCORE_NAME`) are owned in Kaapi code as constants. + +## Design Decisions / Known Limitations + +- **No new per-row table.** Correctness reuses the per-trace `scores` list and the + `EvaluationRun.score` summary, matching how cosine per-row data is already + persisted to S3 and synced to Langfuse; reasoning is stored in the score `comment`. +- **Dedicated `evaluation_judge_config` table** rather than the versioned + `config`/`config_version` flow. The judge config needs plain CRUD with + revert-on-delete, so a soft-deletable table keeps "next run reads the latest active + row" trivial. +- **Reuses `TextLLMParams`** for the judge model + sampling settings rather than a + parallel config shape, so the judge config validates and maps like the eval config. +- **`per_item_correctness` as a separate column** (vs folding into `per_item_scores`) + keeps the two score families independently resyncable and mirrors the existing + cosine map exactly. +- **Judge runs after similarity** so cosine is already computed and a judge failure + can never block it. +- **Known limitation, error/retry (deferred):** a failed judge row is left + unscoreable with no judge-specific retry; defined retry/backoff is Phase 2. +- **Open (from PRD):** judge error/retry behavior, performance budget and supported + row count, the rating-example entry format, and the final user-facing score labels. +``` diff --git a/features/llm-judge-correctness/assets/flow-a.png b/features/llm-judge-correctness/assets/flow-a.png new file mode 100644 index 000000000..b27132ee1 Binary files /dev/null and b/features/llm-judge-correctness/assets/flow-a.png differ