ci: Readability-score delta Github Action by mdlinville · Pull Request #2884 · wandb/docs

mdlinville · 2026-07-09T20:06:10Z

Description

Posts an informational, non-blocking PR comment describing how the PR affects the readability of the English docs it changes. It reports the delta (before/after) for well-established formulas (Flesch-Kincaid grade, Flesch reading ease, Gunning fog, SMOG), word-weighted across the changed pages, plus an optional AI-agent-comprehension rating from a W&B Inference LLM judge.

Details

Depends on a beta skill in https://github.com/coreweave/docs-skills/
Runs when a PR is opened, updated, or reopened.
Fork PRs have no access to repo secrets, so the first step detects a fork, posts an Actions notice, and makes the whole
job a no-op (still reporting success). Forks are uncommon in wandb/docs and coreweave repos cannot use forks at all.
The AI agent comprehension judge calls W&B Inference with the WANDB_DOCS_INFERENCE_API_KEY secret (a W&B API key whose entity has Inference credits), passed to the scorer as WANDB_API_KEY. When that secret is absent, the deterministic textstat delta still runs.

Fixes DOCS-2626

Testing

Local build succeeds without errors (mint dev)
Local link check succeeds without errors (mint broken-links)
Upstream testing while developing the upstream check in https://github.com/coreweave/docs-skills/pull/42
Minor content change in this PR (safe to merge) detects a small positive delta to show how the check works
PR tests succeed

Depends on a skill in coreweave/docs-skills#42 Adding a submodule reference is a followup action after that PR merges, some time before merging this into wandb/docs The submodule will depend on the DOCENGINE_TOKEN already in wandb/docs for cross-org authentication.

Points the .claude submodule at coreweave/docs-skills main (11d7344), which lands the readability delta analyzer and AI comprehension scorer (#42) that the readability-delta CI check imports from .claude/scripts/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

w-b-hivemind · 2026-07-09T20:06:28Z

HiveMind Sessions

2 sessions · 27m · $7.96

Session	Agent	Duration	Tokens	Cost	Lines
Address CoPilot Feedback on PR #2884 `8338f2de-0ba2-4d3b-b811-c4d1d7d16472`	claude	5m	15.9K	$1.51	+28 -2
Update Submodule and CI Readability Check `07dfa6e8-3c0a-4e7d-aa2b-5d656efe54ab`	claude	22m	82.8K	$6.45	+79 -22
Total		27m	98.7K	$7.96	+107 -24

View all sessions in HiveMind →

Run claude --resume 8338f2de-0ba2-4d3b-b811-c4d1d7d16472 to pickup where you left off.

socket-security · 2026-07-09T20:06:44Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/openai@2.45.0
	pypi/weave@0.53.1
	pypi/textstat@0.7.13

View full report

socket-security · 2026-07-09T20:06:45Z

All alerts resolved. Learn more about Socket for GitHub.

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report

mintlify · 2026-07-09T20:08:14Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
wandb	🟢 Ready	View Preview	Jul 9, 2026, 8:13 PM

Copilot

Pull request overview

Adds a new CI workflow that computes and posts an informational readability delta (word-weighted across changed English .mdx files) as an upserted PR comment, backed by a Python reporting script and unit tests. It optionally includes an AI-agent-comprehension judge via W&B Inference when the relevant secret is present.

Changes:

Introduces scripts/readability/pr_report.py to diff base/head, score changed pages via the docs-skills submodule analyzer, aggregate results, and build Markdown.
Adds a new workflow .github/workflows/readability-delta.yml to run on PRs touching .mdx and post/upsert a single PR comment.
Adds supporting docs, requirements, and pytest coverage for parsing/aggregation/Markdown generation.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
`scripts/readability/pr_report.py`	Implements the diff → score → aggregate → Markdown report pipeline, plus optional LLM judge integration.
`scripts/readability/README.md`	Documents the readability-delta check, dependencies, behavior, and local run instructions.
`scripts/readability/requirements.txt`	Declares Python dependencies for deterministic scoring and optional judge.
`scripts/readability/tests/test_pr_report.py`	Unit tests for parsing changed files, aggregation, and Markdown formatting; includes an optional integration test.
`scripts/readability/tests/conftest.py`	Registers the `integration` pytest marker to avoid unknown-mark warnings.
`scripts/readability/tests/__init__.py`	Marks the tests directory as a package for test discovery/import behavior.
`.github/workflows/readability-delta.yml`	New workflow to run the script, write a fallback notice on failure, and upsert a single PR comment.
`.github/workflows/README.md`	Adds a section describing the new readability-delta workflow and its configuration/auth.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-07-09T20:13:34Z

📚 Mintlify Preview Links

🔗 View Full Preview

✨ Added (1 total)

⚙️ Other (1)

File
`scripts/readability/README.md`

📝 Changed (2 total)

📄 Pages (1)

File	Preview
`models/artifacts/data-privacy-and-compliance.mdx`	Data Privacy And Compliance

⚙️ Other (1)

File
`.github/workflows/README.md`

🤖 Generated automatically when Mintlify deployment succeeds
📍 Deployment: 515c4c4 at 2026-07-23 19:48:01 UTC

…degradation - Preserve the old path for renamed/copied .mdx so the base version is read from the old path; a renamed+edited page is now scored as a delta instead of being misreported as brand-new. Covers both the deterministic scorer and the comprehension judge. Adds parse + score_entry unit tests. - Pin actions/checkout, actions/setup-python, actions/cache to the commit SHAs already used across this repo's workflows (supply-chain hardening). - Mark the submodule-fetch and dependency-install steps continue-on-error so a transient failure degrades to the informational fallback comment instead of failing the check, matching the workflow's "never blocks a PR" contract. - Pin textstat/openai/weave to the verified-working versions, matching the repo's requirements-pinning convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

mdlinville · 2026-07-09T20:27:29Z

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action Severity Alert (click "▶" to expand/collapse)
Warn
License policy violation: pypi pyphen
Location: Package overview

From: ? → pypi/textstat@0.7.13 → pypi/pyphen@0.17.2

ℹ Read more on: This package | This alert | What is a license policy violation?
Next steps: Take a moment to review the security alert above. Review
the linked package source code to understand the potential risk. Ensure the
package is not malicious before proceeding. If you're unsure how to proceed,
reach out to your security team or ask the Socket team for help at
support@socket.dev.

Suggestion: Find a package that does not violate your license policy or adjust your policy to allow this package's license.
_Mark the package as acceptable risk_. To ignore this alert only
in this pull request, reply with the comment
`@SocketSecurity ignore pypi/pyphen@0.17.2`. You can
also ignore all packages with `@SocketSecurity ignore-all`.
To ignore an alert for all future pull requests, use Socket's Dashboard to
change the [triage state of this alert](https://socket.dev/dashboard/org/wandb/diff-scan/4c859df7-c56d-4b2a-99d5-2ae502e10b84/alert/Q6bjQxgIYySKBM7ScspgLycCHAbnYgYhHVRZ-2F2E_UE).
Warn
License policy violation: pypi pyphen
Location: Package overview

From: ? → pypi/textstat@0.7.13 → pypi/pyphen@0.17.2

ℹ Read more on: This package | This alert | What is a license policy violation?
Next steps: Take a moment to review the security alert above. Review
the linked package source code to understand the potential risk. Ensure the
package is not malicious before proceeding. If you're unsure how to proceed,
reach out to your security team or ask the Socket team for help at
support@socket.dev.

Suggestion: Find a package that does not violate your license policy or adjust your policy to allow this package's license.
_Mark the package as acceptable risk_. To ignore this alert only
in this pull request, reply with the comment
`@SocketSecurity ignore pypi/pyphen@0.17.2`. You can
also ignore all packages with `@SocketSecurity ignore-all`.
To ignore an alert for all future pull requests, use Socket's Dashboard to
change the [triage state of this alert](https://socket.dev/dashboard/org/wandb/diff-scan/4c859df7-c56d-4b2a-99d5-2ae502e10b84/alert/QXBO5h2PeuSAvOrzccErXrKLgn2o9YPQM0ZVvA3c9j4Q).
Warn
License policy violation: pypi setuptools
Location: Package overview

From: ? → pypi/textstat@0.7.13 → pypi/setuptools@83.0.0

ℹ Read more on: This package | This alert | What is a license policy violation?
Next steps: Take a moment to review the security alert above. Review
the linked package source code to understand the potential risk. Ensure the
package is not malicious before proceeding. If you're unsure how to proceed,
reach out to your security team or ask the Socket team for help at
support@socket.dev.

Suggestion: Find a package that does not violate your license policy or adjust your policy to allow this package's license.
_Mark the package as acceptable risk_. To ignore this alert only
in this pull request, reply with the comment
`@SocketSecurity ignore pypi/setuptools@83.0.0`. You can
also ignore all packages with `@SocketSecurity ignore-all`.
To ignore an alert for all future pull requests, use Socket's Dashboard to
change the [triage state of this alert](https://socket.dev/dashboard/org/wandb/diff-scan/4c859df7-c56d-4b2a-99d5-2ae502e10b84/alert/QhPDKYzPKjgIJtGAJPhxIu2Q6u7HWARu_J_Kx0Ys67dI).
Warn
License policy violation: pypi setuptools
Location: Package overview

From: ? → pypi/textstat@0.7.13 → pypi/setuptools@83.0.0

ℹ Read more on: This package | This alert | What is a license policy violation?
Next steps: Take a moment to review the security alert above. Review
the linked package source code to understand the potential risk. Ensure the
package is not malicious before proceeding. If you're unsure how to proceed,
reach out to your security team or ask the Socket team for help at
support@socket.dev.

Suggestion: Find a package that does not violate your license policy or adjust your policy to allow this package's license.
_Mark the package as acceptable risk_. To ignore this alert only
in this pull request, reply with the comment
`@SocketSecurity ignore pypi/setuptools@83.0.0`. You can
also ignore all packages with `@SocketSecurity ignore-all`.
To ignore an alert for all future pull requests, use Socket's Dashboard to
change the [triage state of this alert](https://socket.dev/dashboard/org/wandb/diff-scan/4c859df7-c56d-4b2a-99d5-2ae502e10b84/alert/QagscmSIaet4RDOP11Jcicryy2k6ncICw4OJif9jKVFA).
View full report

These are noise: We are using these tools to evaluate readability but we are not shipping these packages anywhere.

mdlinville · 2026-07-09T20:27:30Z

Thanks for the review, @copilot — all nine comments had merit and are addressed in 9c13165:

Rename scoring (correctness) — git diff --name-status emits R###/C### rows as status\told\tnew, and the parser kept only the new path, so a renamed page's base version was looked up at the new path (empty) and it was misreported as brand-new. parse_changed_files now retains old_path; both the deterministic scorer and the comprehension judge read the base from old_path. Added a parser test and a score_entry test (mocking git_show) that fails if the base is ever read from the new path again.

Action SHA pinning — actions/checkout, actions/setup-python, and actions/cache are now pinned to the exact SHAs used elsewhere in this repo (checkout v6→v7, cache v5→v6 to match).

"Never blocks a PR" contract — the submodule-fetch and dependency-install steps are now continue-on-error: true; on failure the report step falls back to the informational notice and the check stays green.

Dependency pinning — textstat==0.7.13, openai==2.45.0, weave==0.53.1, matching the repo's == convention (versions verified working in CI and locally).

All 14 unit tests pass and the workflow YAML validates. Ready for another look.

Copilot

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated 1 comment.

aggregate() returns None whenever no page has a scorable before-and-after version — which includes PRs that only add new pages or only delete pages, not just insufficient prose. Generalize the message and point at the per-page table for specifics. Addresses Copilot review follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Temporary prose simplification (FK 10.0 -> 9.5, "easier") to trigger and demonstrate the readability-delta workflow on this PR. REVERT before merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-09T20:39:01Z

🔗 Link Checker Results

⚠️ Some issues were detected

Preview: https://wb-21fd5541-docs-2626.mintlify.site

Full Github Actions output

github-actions · 2026-07-09T20:39:18Z

Readability impact

Word-weighted Flesch-Kincaid grade change across 1 changed page: -0.4 (easier).

Lower Flesch-Kincaid grade and higher reading ease both mean easier to read. This check is informational and never blocks a PR.

Human readability

Page	FK grade before	FK grade after	FK Δ	Reading ease Δ	Direction
`models/artifacts/data-privacy-and-compliance.mdx`	10.0	9.6	-0.4	+2.5	easier

AI agent comprehension

Rated 0-3 (higher is easier for an agent to parse and act on).

Page	Before	After	Δ
`models/artifacts/data-privacy-and-compliance.mdx`	—	—	—

_{Curated-docs baseline median FK grade by type: conceptual 10.5, procedural 8.8, reference 9.4.}

From workflow run 30039170175

Copilot

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 4 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 3 comments.

… steps, -0.0) - Submodule checkout uses a one-shot `git -c url.insteadOf` instead of writing DOCENGINE_TOKEN to the runner's global git config, so later steps can't recover the secret from ~/.gitconfig. - Mark the PR-comment upsert and job-summary write steps continue-on-error so a transient failure keeps the informational check green ("never fails"). - Normalize -0.0 to 0.0 in aggregate() so the headline never shows "-0.0 (unchanged)"; add a regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated 1 comment.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

mdlinville and others added 3 commits June 1, 2026 13:11

Merge remote-tracking branch 'origin/main' into DOCS-2626

2eef737

mdlinville requested a review from a team as a code owner July 9, 2026 20:06

mdlinville requested a review from Copilot July 9, 2026 20:06

Copilot started reviewing on behalf of mdlinville July 9, 2026 20:06 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

mintlify Bot deployed to staging July 9, 2026 20:13 View deployment

mdlinville requested a review from Copilot July 9, 2026 20:27

mintlify Bot deployed to staging July 9, 2026 20:27 View deployment

Copilot started reviewing on behalf of mdlinville July 9, 2026 20:28 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread scripts/readability/pr_report.py Outdated

mdlinville and others added 2 commits July 9, 2026 13:36

mdlinville requested a review from Copilot July 9, 2026 20:36

Copilot started reviewing on behalf of mdlinville July 9, 2026 20:37 View session

mintlify Bot deployed to staging July 9, 2026 20:38 View deployment

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread scripts/readability/pr_report.py

Comment thread .github/workflows/readability-delta.yml Outdated

Comment thread models/artifacts/data-privacy-and-compliance.mdx Outdated

Comment thread .github/workflows/readability-delta.yml Outdated

Merge branch 'main' into DOCS-2626

c476e53

mintlify Bot deployed to staging July 13, 2026 22:21 View deployment

Apply suggestions from code review

76bdc52

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

mintlify Bot deployed to staging July 17, 2026 20:42 View deployment

mdlinville requested a review from Copilot July 17, 2026 20:45

Copilot started reviewing on behalf of mdlinville July 17, 2026 20:46 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread .github/workflows/readability-delta.yml Outdated

Comment thread .github/workflows/readability-delta.yml

Comment thread .github/workflows/readability-delta.yml

Merge remote-tracking branch 'origin/main' into DOCS-2626

6b894b4

mintlify Bot deployed to staging July 21, 2026 23:06 View deployment

mdlinville and others added 2 commits July 21, 2026 16:08

Try to fix submodule

bd53081

mdlinville requested a review from Copilot July 21, 2026 23:42

Copilot started reviewing on behalf of mdlinville July 21, 2026 23:42 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread scripts/readability/pr_report.py

mdlinville and others added 2 commits July 21, 2026 17:18

Potential fix for pull request finding

c3aab7e

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Merge branch 'main' into DOCS-2626

b20bc84

mintlify Bot deployed to staging July 22, 2026 00:24 View deployment

Merge branch 'main' into DOCS-2626

515c4c4

mintlify Bot deployed to staging July 23, 2026 19:47 View deployment

Uh oh!

Conversation

mdlinville commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Details

Testing

Uh oh!

w-b-hivemind Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

HiveMind Sessions

Uh oh!

socket-security Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

socket-security Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mintlify Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📚 Mintlify Preview Links

✨ Added (1 total)

📝 Changed (2 total)

📄 Pages (1)

Uh oh!

mdlinville commented Jul 9, 2026

Uh oh!

mdlinville commented Jul 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Link Checker Results

Preview: https://wb-21fd5541-docs-2626.mintlify.site

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Readability impact

Human readability

AI agent comprehension

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

mdlinville commented Jul 9, 2026 •

edited

Loading

w-b-hivemind Bot commented Jul 9, 2026 •

edited

Loading

socket-security Bot commented Jul 9, 2026 •

edited

Loading

socket-security Bot commented Jul 9, 2026 •

edited

Loading

mintlify Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading