EvalMint

EvalMint is an evaluation pipeline for judging Mintlify's documentation answers.

It evaluates whether a support/docs chatbot answered user questions correctly, grounded its claims in documentation, and suggested relevant links.

EvalMint does not use Likert Scale for evaluating responses.

See "Why no 1-5 scale?" section for more info.

What It Does

EvalMint takes chatbot answer records and:

1. normalizes documentation source URLs
2. fetches documentation page text
3. builds records, feeds them to LLM Judge for eval
4. generates summary metrics, CSVs, and charts

Core Evaluation Criteria

The judge focuses on:

• factual correctness
• source/documentation grounding
• suggested link relevance
• query satisfaction
• unsupported or unverifiable claims
• missing important documentation links

resolutionStatus is not used by the judge. It is reserved as a downstream validation signal.

Why no Likert Scale?

EvalMint does not use Likert Scale found in many LLM Judges. The goal is to obtain factual correctness and source relevance instead of subjective quality scoring.

Instead, we use metrics such as:

• pass_fail
• claim_support
• citation_alignment
• response_link_source_match
• primary_failure_mode
• binary failure flags

The let's us ask:

• Which claims were supported?
• Which claims were unsupported or unverifiable?
• Did the suggested links match the sources?
• Was an important documentation link missing?
• What was the primary failure mode?

See the llm_judge_metrics.yaml file for a list of metric definitions and their possible values.

Project Structure

.
├── judge.py                         # Runs the LLM judge
├── schemas.py                       # Pydantic input/output schemas
├── llm_judge_metrics.yaml           # Metric definitions
├── prompts/
│   └── judge_system.md              # Judge instructions
├── tools/
│   ├── process_urls.py              # Normalizes source and response URLs
│   ├── enrich_sources.py            # Fetches docs text for source URLs
│   └── analyze_judge_outputs.py     # Generates metrics and charts
├── PreProcessed/                    # Input datasets
└── runs/                            # Eval outputs