docs: fix inaccuracies in ADDING_EVALS.md

[sanchitmehtagit]

Summary

Corrects five claims in docs/ADDING_EVALS.md that had drifted from the framework code:

• setup_command — it does support &&-chained sub-commands (run in sequence by runSetupCommand, relied on by the flask/fastapi evals). The doc wrongly said "no operators, one command only."
• notContainsInSource — lists the full set of non-source extensions it actually skips (.env*, .json, .plist, .xml, .yaml, .yml, .toml, .ini, .cfg, .conf, .md) per text-search-utils.ts, instead of just three.
• matches — defaults to case-insensitive (per primitives.ts), and its signature accepts an options? arg; both were misdocumented.
• caseSensitive default — clarified: true for contains/notContains/notContainsInSource, false for matches.
• Stale example model — claude-sonnet-4-6 → claude-sonnet-5 (matches eval.config.js models.known).

Test plan

Docs-only change. Verified each claim against the source:

• packages/evals-core/src/workspace/workspace.ts (runSetupCommand)
• packages/evals-core/src/graders/executors/text-search-utils.ts
• packages/evals-graders/src/primitives.ts
• apps/auth0-evals/eval.config.js

Summary by CodeRabbit

• Documentation
  • Updated the eval creation guide with clearer frontmatter field descriptions and examples.
  • Documented sequential setup_command behavior using &&.
  • Clarified grader semantics, defaults, skipped file types, case sensitivity, and compilation behavior.
  • Refined grader examples and updated the sample model command.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5dc9f490-dc91-4730-a5d9-a3d9ae8cd515

📥 Commits

Reviewing files that changed from the base of the PR and between 63077fb and 6197646.

📒 Files selected for processing (1)

• docs/ADDING_EVALS.md

---

📝 Walkthrough

Walkthrough

Updated docs/ADDING_EVALS.md with revised frontmatter guidance, clearer grader primitive semantics, reformatted examples, and updated model execution instructions.

Changes

Eval authoring documentation

Layer / File(s)	Summary
Frontmatter and prompt examples docs/ADDING_EVALS.md	Documents &&-separated setup_command execution and updates the example prompt task layout.
Grader primitives and examples docs/ADDING_EVALS.md	Clarifies grader behavior and reformats L1–L5 and annotated grader examples.
Run and iterate guidance docs/ADDING_EVALS.md	Uses claude-sonnet-5 in the sample command and re-renders the modes and tools table.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

• auth0/auth0-evals#148: Also clarifies grader primitive documentation, including compiles() semantics and compile_command behavior.

Suggested reviewers: frederikprijck

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the docs-only update to correct inaccuracies in ADDING_EVALS.md.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch docs/fix-adding-evals-accuracy

⚔️ Resolve merge conflicts

• Resolve merge conflict in branch docs/fix-adding-evals-accuracy

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}