Most evaluation tools end with a score and a table of failures. EvalVitals starts there. It can explore raw eval logs, find recurring patterns, propose falsifiable explanations, test them on held-out cases, and validate candidate interventions against the unchanged baseline.

Use the whole investigation loop—or take only the layer you need:

• Explore existing results: point a coding agent at arbitrary JSON/JSONL logs and get observations, charts, tables, and testable hypotheses.
• Investigate a target model: adaptively probe failures, diagnose systematic modes, verify hypotheses, and test interventions.
• Run one focused analyzer: use the same sklearn-like interface for black-box APIs and white-box local models.

flowchart LR
    A[Eval logs or target model] --> B[Probe]
    B --> C[Explore patterns]
    C --> D[Diagnose]
    D --> E{Held-out verification}
    E -->|supported| F[Test interventions]
    E -->|refuted| B
    F --> G[Validated fix or honest inconclusive result]

Install EvalVitals with the report dashboard:

pip install "evalvitals[dashboard]"

Then point it at a file or directory of JSON/JSONL results:

evalvitals explore ./results \
  --backend codex \
  -q "What distinguishes failed cases from successful ones?" \
  --dashboard

codex can be replaced with claude_code, opencode, gemini_cli, kimi_cli, or antigravity. The selected coding-agent CLI must be installed and authenticated separately.

Prefer a browser? evalvitals web serves an upload page instead: drop a .zip of results and each upload becomes one explore run (M2 exploratory analysis + M3 hypotheses) whose report renders in place, with past runs kept in a sidebar.

EvalVitals writes an auditable analysis bundle instead of returning only prose:

evalvitals_explore_output/
├── exploratory_report.json   # observations, candidate signals, hypotheses
├── records.json              # normalized records used by the analysis
├── figures/                  # rendered charts
├── tables/                   # analysis-ready tables
└── analysis.py               # the generated code that was actually run

A real bundled run: on the synthetic-yield example, Explore identified temperature as the strongest observed correlate (r = 0.86), separated that finding from weak pressure evidence (r = -0.21), and proposed mechanism-level hypotheses for a later confirmatory experiment. See the reproducible example →

Already have your own analysis code? Use the analyzer toolkit directly, or feed the resulting cases into the full diagnosis loop. EvalVitals does not require you to replace your existing eval or observability stack.

Statistical gates use paired tests and e-values, including multiplicity control when several hypotheses or fixes are tried. A run may end inconclusive; EvalVitals does not turn weak evidence into a success verdict.

evalvitals explore recursively samples arbitrary JSON/JSONL shapes. The coding agent performs exploratory data analysis; the host records generated code, adjudicates host-checkable statistics, renders figures, and proposes 1–3 falsifiable hypotheses.

Explore guide →

VLDiagnoseLoop chains the full workflow:

M1 targeted probes
 → M2 exploratory and statistical analysis
 → M3 diagnosis hypotheses
 → M5 held-out hypothesis verification
 → M4 surgery and tiered fixes

Interventions can range from prompt changes and scaffolds to read/write access to model internals. Each candidate is evaluated against the unmodified baseline; automatic escalation happens only when explicitly enabled.

Full-loop quickstart → · Intervention guide →

Every registered analyzer follows the same call shape:

from evalvitals import Capability, compose
from evalvitals.analyzers.attention.summary import AttentionAnalyzer

model = compose(
    "qwen2.5-7b-instruct",
    "hf_local",
    want={Capability.ATTENTION},
)

result = AttentionAnalyzer(layer=-1, top_k=5).run(
    model, "The Eiffel Tower is in"
)

print(result.summary())

The analyzer zoo includes attention, uncertainty, hallucination, attribution, logit-lens, representation-geometry, and agent-trajectory analysis.

Browse the Analyzer Zoo →

The core install stays lightweight—no Torch required:

pip install evalvitals

Add only the capabilities you need:

pip install "evalvitals[api]"        # OpenAI-compatible API models
pip install "evalvitals[local]"      # local Hugging Face models + Torch
pip install "evalvitals[interp]"     # interpretability toolchains
pip install "evalvitals[viz]"        # plots
pip install "evalvitals[dashboard]"  # Streamlit reports
pip install "evalvitals[stats]"      # inferential statistics

For development:

git clone https://github.com/evalvitals/evalvitals.git
cd evalvitals
pip install -e ".[dev]"
pytest -m "not gpu"

Model identity is separate from runtime, and analyzers declare the capabilities they need. The same model spec can run through a black-box API or a white-box local backend; only the available capability set changes.

Read the architecture guide →

See all examples →

• Quickstart
• Exploratory Analysis
• Intervention & Verification
• Analyzer Zoo
• Architecture
• Extending EvalVitals
• Roadmap

EvalVitals is an early-stage research toolkit. Interfaces may evolve, and some full-loop examples require model weights, a GPU, or an external coding-agent CLI. Bug reports, reproducible failure cases, analyzer contributions, and evaluation integrations are welcome.

If EvalVitals helps you understand a model failure, consider starring the repo and sharing the smallest reproducible case—it makes the toolkit better for the next investigation.