readme-updater: make the AI README generator feedback-aware#365
readme-updater: make the AI README generator feedback-aware#365igorpecovnik wants to merge 2 commits into
Conversation
Regenerating a README overwrote the branch wholesale, so any human review fixes on the open PR were silently dropped on the next run (e.g. EvilOlaf's "bash/ python" note on the build README's requirements list). Now, before generating, the workflow collects the human review + issue comments (bots filtered out) from the still-open `chore/update-readme` PR for that repo and passes them to generate_readme.py via --feedback. The script folds them into the prompt as corrections to apply and not re-introduce. Reviewer fixes survive a regeneration; the PR body invites reviewers to comment for exactly this reason. Also sharpen the system prompt: when stating what the project is built with / its requirements, name the actual languages and tooling evidenced by the files (shebangs, extensions, manifests, invoked commands) rather than vague generalities -- the root of that review comment. Signed-off-by: Igor Pecovnik <igor@armbian.com>
github-actions
Bot
added
size/medium
|
Warning Review limit reached
Next review available in: 12 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
WalkthroughThe README maintenance workflow now collects non-bot reviewer comments from the open README pull request and writes them to Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/maintenance-update-readme.yml:
- Around line 98-100: Update both feedback-gathering gh api jq filters in the
README maintenance workflow to forward comments only when the author has an
explicitly trusted permission or an allowed author_association, while retaining
the existing bot exclusion. Apply the same trust check to review comments and
issue comments before appending them to feedback.txt.
In `@scripts/generate_readme.py`:
- Around line 177-181: Make feedback acquisition fail closed: in
scripts/generate_readme.py lines 177-181, update the feedback-reading logic
around the visible file-read helper to raise or report an explicit error instead
of returning an empty string on OSError; in
.github/workflows/maintenance-update-readme.yml lines 97-100, remove || true
from both GitHub API requests so either failure stops README generation.
- Around line 177-181: Update the feedback-reading logic around the visible
file-open helper so an OSError is reported or propagated when an explicit
--feedback path is supplied, instead of returning an empty string. Preserve the
empty-string behavior only when no feedback file was requested, ensuring
unreadable feedback causes README generation to fail closed.
- Around line 192-202: Limit the reviewer feedback before appending it in the
feedback handling block of the README prompt construction. Apply
MAX_CONTEXT_CHARS or an appropriate dedicated bound to feedback so the combined
prompt remains within the model context limit, preserving the existing
reviewer-feedback instructions and truncating only the feedback content.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: f196cc2f-9c0d-4378-9b6d-662aba430132
📒 Files selected for processing (2)
.github/workflows/maintenance-update-readme.ymlscripts/generate_readme.py
| if feedback: | ||
| # Reviewers commented on the previous README PR; fold their fixes in so a | ||
| # regeneration doesn't silently drop them. Treat as human corrections that | ||
| # override the model's own read where they conflict -- but still only about | ||
| # THIS repo, never as instructions to invent unevidenced content. | ||
| prompt += ( | ||
| "\n\n----- REVIEWER FEEDBACK ON THE PREVIOUS README (apply it) -----\n" | ||
| "A human reviewed the last generated README for this repo and left the " | ||
| "comments below. Incorporate these corrections and do not re-introduce " | ||
| "what they flagged. They are about this repository only.\n\n" + feedback | ||
| ) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
sed -n '1,280p' scripts/generate_readme.py
printf '\n--- SEARCH ---\n'
rg -n "max_context_chars|feedback|prompt \+=" scripts/generate_readme.pyRepository: armbian/armbian.github.io
Length of output: 11598
🏁 Script executed:
python3 - <<'PY'
from pathlib import Path
p = Path("scripts/generate_readme.py")
text = p.read_text()
for needle in ["max_context_chars", "feedback", "prompt +=", "max_tokens", "truncate", "budget"]:
print(f"\n### {needle}")
for i, line in enumerate(text.splitlines(), 1):
if needle in line:
start = max(1, i-4)
end = min(len(text.splitlines()), i+4)
for j in range(start, end+1):
print(f"{j:4}: {text.splitlines()[j-1]}")
print("---")
PYRepository: armbian/armbian.github.io
Length of output: 10865
Cap reviewer feedback before building the prompt. MAX_CONTEXT_CHARS only bounds repo content; feedback is appended as-is, so a long review thread can push the request over the model’s context limit and make generation fail.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/generate_readme.py` around lines 192 - 202, Limit the reviewer
feedback before appending it in the feedback handling block of the README prompt
construction. Apply MAX_CONTEXT_CHARS or an appropriate dedicated bound to
feedback so the combined prompt remains within the model context limit,
preserving the existing reviewer-feedback instructions and truncating only the
feedback content.
There was a problem hiding this comment.
Pull request overview
This PR updates the automated README regeneration workflow to preserve human review corrections across reruns by collecting comments from an open chore/update-readme PR and feeding them back into the README generator as prompt “feedback”.
Changes:
- Add
--feedbacksupport toscripts/generate_readme.pyand incorporate reviewer feedback into the generation prompt. - Extend the GitHub Actions workflow to collect review + issue comments from an open README PR into
feedback.txt, and pass it to the generator. - Refine the system prompt to require specific, file-evidenced languages/tooling when describing repo requirements.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| scripts/generate_readme.py | Adds feedback loading and prompt augmentation so regenerations retain reviewer corrections. |
| .github/workflows/maintenance-update-readme.yml | Collects PR comments as feedback and forwards them into the generator; updates PR body to instruct reviewers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| def load_feedback(path): | ||
| """Read reviewer feedback (from the open README PR) if a path is given.""" | ||
| if not path: | ||
| return "" | ||
| try: | ||
| with open(path, "r", encoding="utf-8", errors="replace") as fh: | ||
| return fh.read().strip() | ||
| except OSError: | ||
| return "" |
| gh api "repos/$REPO/pulls/$pr/comments" \ | ||
| --jq '.[] | select((.user.login|test("\\[bot\\]|coderabbit";"i"))|not) | "- (\(.path):\(.line // .original_line)) \(.user.login): \(.body)"' >> feedback.txt || true | ||
| gh api "repos/$REPO/issues/$pr/comments" \ | ||
| --jq '.[] | select((.user.login|test("\\[bot\\]|coderabbit";"i"))|not) | "- \(.user.login): \(.body)"' >> feedback.txt || true |
Address review on PR #365: - Security: the feedback filter only excluded bots, so any human commenter on the public README PR could inject text the generator treats as guidance. Gate both gh api jq filters on author_association (OWNER/MEMBER/COLLABORATOR = write access) in addition to the bot exclusion. - Data integrity: drop the `|| true` on the comment fetches so a gh/API error aborts the run instead of silently generating without feedback (which would re-drop the reviewer's fixes). load_feedback() now errors out when an explicit --feedback file can't be read; an empty file is still fine. Signed-off-by: Igor Pecovnik <igor@armbian.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (2)
.github/workflows/maintenance-update-readme.yml:103
gh apiis not paginated here, so only the first page of review/issue comments (typically 30) will be included infeedback.txt. That can silently drop older reviewer feedback on long-lived PRs, defeating the purpose of the feedback loop.
gh api "repos/$REPO/pulls/$pr/comments" \
--jq '.[] | select((.user.login|test("\\[bot\\]|coderabbit";"i")|not) and (.author_association=="OWNER" or .author_association=="MEMBER" or .author_association=="COLLABORATOR")) | "- (\(.path):\(.line // .original_line)) \(.user.login): \(.body)"' >> feedback.txt
gh api "repos/$REPO/issues/$pr/comments" \
--jq '.[] | select((.user.login|test("\\[bot\\]|coderabbit";"i")|not) and (.author_association=="OWNER" or .author_association=="MEMBER" or .author_association=="COLLABORATOR")) | "- \(.user.login): \(.body)"' >> feedback.txt
scripts/generate_readme.py:185
load_feedback()reads the entire feedback file with no size cap. If a PR accumulates a lot of comments (or long pasted logs), this can bloat the prompt beyond the model/context limits and make the job fail unpredictably. Consider truncating feedback to a reasonable maximum and logging when truncation happens.
try:
with open(path, "r", encoding="utf-8", errors="replace") as fh:
return fh.read().strip()
except OSError as e:
| Leave review comments here: while this PR stays open, the next run reads them | ||
| and folds your fixes into the regenerated README (they won't be overwritten). |
|
Addressed both CodeRabbit findings in ed5d28c: 1. Restrict forwarded feedback to trusted reviewers (Security, Major) — both 2. Fail closed on feedback-acquisition failure (Data Integrity, Major) — removed the |
Follow-up to the Update README (AI) workflow, prompted by review feedback on one of its generated PRs (EvilOlaf on armbian/build#10245: the requirements list was too vague — the project is bash + python + tooling).
Problem
Each run regenerated the README from scratch and force-updated the
chore/update-readmebranch, so any human review fix on the open PR was silently dropped on the next run. A reviewers correction never stuck.Change — feed reviewer comments back in
chore/update-readmePR for that repo, intofeedback.txt, and passes it to the generator via--feedback.generate_readme.pyfolds that feedback into the prompt as corrections to apply and not re-introduce — still scoped to this repo, never a license to invent unevidenced content.Also sharpened the system prompt (the root of that comment): when stating what the project is built with / its requirements, name the actual languages and tooling evidenced by the files (shebangs, extensions, manifests, invoked commands) rather than vague generalities.
Validation
py_compile+ YAML parse clean.REVIEWER FEEDBACKblock + the comment text are present; without → absent).jqfilter keeps the human comment (EvilOlaf) and drops the bot (coderabbitai[bot]).matrix.repostill routed through env (no shell injection); reads use the existingACCESS_TOKEN.