feat: adopt claude-opus-5 as judge and known model

[subhankarmaiti]

Replaces the retired claude-opus-4-8 alias with claude-opus-5 throughout the framework — the LLM-as-judge model, the --model all known set, the cost table, the effort-capable model set, and the Bedrock alias map.

[coderabbitai]

📝 Walkthrough

Walkthrough

Claude Opus 4.8 references are replaced with Claude Opus 5 across judge configuration, Bedrock mappings, shared model settings, CLI model selection, tests, and documentation.

Changes

Claude Opus 5 model migration

Layer / File(s)	Summary
Runtime model configuration and validation packages/evals-core/src/config/*, apps/auth0-evals/eval.config.js, packages/evals-core/tests/*, AGENTS.md, docs/ARCHITECTURE.md, packages/evals/tests/recommendations.test.ts	Judge defaults, Bedrock aliases, pricing, effort-model support, model detection tests, documentation, and alias references now use claude-opus-5.
CLI known-model selection packages/evals/src/cli/constants.ts, packages/evals/tests/cli-config.test.ts	The known working model list and CLI configuration fixtures now use Claude Opus 5 for batch and explicit model handling.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• auth0/auth0-evals#110: Updates related Claude Opus model wiring across judge configuration, model lists, Bedrock mappings, and shared evaluation constants.

Suggested reviewers: frederikprijck

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: switching the judge and known model set to claude-opus-5.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/opus-5-model

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/evals-core/src/config/costs.ts`:
- Line 8: Add package-local Vitest tests covering the runtime migration
contracts: in packages/evals-core/src/config/costs.ts:8-8, verify estimateCost
uses the claude-opus-5 pricing; in
packages/evals-core/src/config/settings.ts:10-11, verify Opus 5 is eligible for
effort models; in apps/auth0-evals/eval.config.js:83-83, verify the configured
judge model; and in apps/auth0-evals/eval.config.js:118-128, verify both LiteLLM
passthrough and Bedrock alias mapping branches. Place the tests in each relevant
package-local tests/ directory.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c92af611-fa6f-41c2-8eb0-e57d428f7caa

📥 Commits

Reviewing files that changed from the base of the PR and between 63077fb and 00473a0.

📒 Files selected for processing (10)

• AGENTS.md
• apps/auth0-evals/eval.config.js
• docs/ARCHITECTURE.md
• packages/evals-core/src/config/costs.ts
• packages/evals-core/src/config/settings.ts
• packages/evals-core/tests/graders/engine.test.ts
• packages/evals-core/tests/model-detect.test.ts
• packages/evals/src/cli/constants.ts
• packages/evals/tests/cli-config.test.ts
• packages/evals/tests/recommendations.test.ts

[coderabbitai]

📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Add package-local Vitest coverage for the runtime model migration.

The migration changes pricing lookup, effort-model eligibility, judge selection, and Bedrock alias routing, but the supplied tests do not cover these runtime configuration contracts.

• packages/evals-core/src/config/costs.ts#L8-L8: test estimateCost with claude-opus-5.
• packages/evals-core/src/config/settings.ts#L10-L11: test Opus 5 effort-model membership.
• apps/auth0-evals/eval.config.js#L83-L83: test the configured judge model.
• apps/auth0-evals/eval.config.js#L118-L128: test LiteLLM passthrough and Bedrock mapping branches.

As per coding guidelines, every logic change must have corresponding Vitest tests in the package-local tests/ directory.

📍 Affects 3 files

• packages/evals-core/src/config/costs.ts#L8-L8 (this comment)
• packages/evals-core/src/config/settings.ts#L10-L11
• apps/auth0-evals/eval.config.js#L83-L83
• apps/auth0-evals/eval.config.js#L118-L128

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/evals-core/src/config/costs.ts` at line 8, Add package-local Vitest
tests covering the runtime migration contracts: in
packages/evals-core/src/config/costs.ts:8-8, verify estimateCost uses the
claude-opus-5 pricing; in packages/evals-core/src/config/settings.ts:10-11,
verify Opus 5 is eligible for effort models; in
apps/auth0-evals/eval.config.js:83-83, verify the configured judge model; and in
apps/auth0-evals/eval.config.js:118-128, verify both LiteLLM passthrough and
Bedrock alias mapping branches. Place the tests in each relevant package-local
tests/ directory.

Source: Coding guidelines