Evals: Added prompt iteration & Real-time config display#241
Merged
Conversation
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR adds an “Iterate on Prompt” action on evaluation results, a backend proxy for prompt improvement, prompt-editor version refresh after save, and a reusable ChangesPrompt iteration and prompt-editor refresh
CommitMessage component
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
app/(main)/configurations/prompt-editor/page.tsx (2)
260-280: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winUpdate the selected version after saving an existing config.
saveConfigonly returnsboolean, sohandleSavenever advancesselectedConfigId/currentConfigVersionto the newly created version. That leaves the header on the old version and can make the unsaved-changes diff compare against the pre-save revision again.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app/`(main)/configurations/prompt-editor/page.tsx around lines 260 - 280, Update handleSave in prompt-editor/page.tsx so that when saveConfig succeeds for an existing config, it advances the editor state to the newly saved version instead of only refreshing versions; use currentConfigParentId plus the refreshed version list to set selectedConfigId and currentConfigVersion, and keep resetEditor() only for new configs. This should be applied alongside the existing saveConfig flow so the header and unsaved-changes diff reflect the just-saved revision rather than the pre-save one.
84-88: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winSurface refresh failures.
refreshVersionsForConfigonly logs the error, so a failed post-save refresh leaves the version list stale with no user-visible signal.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app/`(main)/configurations/prompt-editor/page.tsx around lines 84 - 88, The post-save refresh path in refreshVersionsForConfig only logs failures, so the prompt editor can end up with a stale version list and no visible feedback. Update the refresh flow around refreshVersionsForConfig and the related useEffect in prompt-editor/page.tsx to surface the failure to the user state/UI instead of only calling the logger, while keeping the successful hookVersionItemsMap merge behavior unchanged.
🧹 Nitpick comments (1)
app/(main)/configurations/prompt-editor/page.tsx (1)
146-148: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winFailed refresh is silent to the user.
On error, only
console.errorruns; the user gets no indication that the version list failed to refresh after save.💡 Suggested fix
} catch (err) { console.error("Failed to refresh version list:", err); + // surface via toast if a toast hook is available in this component }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app/`(main)/configurations/prompt-editor/page.tsx around lines 146 - 148, The refresh failure handling in prompt-editor is only logging to console and does not inform the user when the version list refresh fails after save. Update the error path in the version refresh logic (the catch block around the refresh call in the prompt-editor page, near the refresh/save flow) to surface a user-visible error state or notification in addition to console.error, using the existing page state/notification pattern if available so the failure is clearly shown to the user.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/`(main)/evaluations/[id]/page.tsx:
- Around line 203-221: The success state in the prompt iteration flow is being
reported before confirming there is a navigable config result. In the handler
that uses data.success, move invalidateConfigCache() and toast.success("Prompt
iteration created") so they only run after newConfigId is resolved and
router.push can happen, or otherwise handle the missing config case with an
error/neutral message. Update the logic around newConfigId, newVersion, and
router.push so the success toast only fires when the new configuration can
actually be opened.
In `@app/components/CommitMessage.tsx`:
- Around line 3-19: The AI-generated commit parser in parseCommitMessage uses
AI_GENERATED_PATTERN with a body capture that stops at newlines, so multi-line
messages fail detection and lose metadata. Update the regex handling so the
trailing body can span multiple lines in CommitMessage.tsx, and keep the
existing extraction logic for fromVersionRaw, evalName, and body intact so
parseCommitMessage still returns the correct ParsedCommit fields.
---
Outside diff comments:
In `@app/`(main)/configurations/prompt-editor/page.tsx:
- Around line 260-280: Update handleSave in prompt-editor/page.tsx so that when
saveConfig succeeds for an existing config, it advances the editor state to the
newly saved version instead of only refreshing versions; use
currentConfigParentId plus the refreshed version list to set selectedConfigId
and currentConfigVersion, and keep resetEditor() only for new configs. This
should be applied alongside the existing saveConfig flow so the header and
unsaved-changes diff reflect the just-saved revision rather than the pre-save
one.
- Around line 84-88: The post-save refresh path in refreshVersionsForConfig only
logs failures, so the prompt editor can end up with a stale version list and no
visible feedback. Update the refresh flow around refreshVersionsForConfig and
the related useEffect in prompt-editor/page.tsx to surface the failure to the
user state/UI instead of only calling the logger, while keeping the successful
hookVersionItemsMap merge behavior unchanged.
---
Nitpick comments:
In `@app/`(main)/configurations/prompt-editor/page.tsx:
- Around line 146-148: The refresh failure handling in prompt-editor is only
logging to console and does not inform the user when the version list refresh
fails after save. Update the error path in the version refresh logic (the catch
block around the refresh call in the prompt-editor page, near the refresh/save
flow) to surface a user-visible error state or notification in addition to
console.error, using the existing page state/notification pattern if available
so the failure is clearly shown to the user.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 1769bd1a-518b-46f6-b7dc-dfe74a4c0fe0
📒 Files selected for processing (9)
app/(main)/configurations/prompt-editor/page.tsxapp/(main)/evaluations/[id]/page.tsxapp/api/evaluations/[id]/improve-prompt/route.tsapp/components/CommitMessage.tsxapp/components/icons/prompt-editor/SpinnerIcon.tsxapp/components/index.tsapp/components/prompt-editor/ConfigEditorPane.tsxapp/components/prompt-editor/HistorySidebar.tsxapp/lib/types/commit.ts
💤 Files with no reviewable changes (1)
- app/components/prompt-editor/ConfigEditorPane.tsx
vprashrex
approved these changes
Jul 1, 2026
|
🎉 This PR is included in version 0.3.0-main.7 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Closes #233 #232
Summary
Checklist
Before submitting a pull request, please ensure that you mark these task.
npm run devandnpm run buildin the repository root and test.