Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 8 additions & 57 deletions .github/workflows/report-embeds.yml
Original file line number Diff line number Diff line change
@@ -1,84 +1,35 @@
name: Report embeds

# Validates the W&B report-embed registry (scripts/report-embeds/registry.yaml)
# and its use in docs pages. Static checks run on PRs; URL liveness runs weekly
# and files an issue on failure (it never blocks PRswandb.ai rate-limits
# crawlers, so network results are flaky by design).
# Weekly liveness check of the live W&B report embeds (<WandbReport>) in docs
# pages, discovered by scanning the .mdx sources. Files an issue if a report has
# gone dead. Deliberately does not run on PRs: wandb.ai rate-limits crawlers, so
# network results are too flaky to gate merges on.

on:
pull_request:
types: [opened, synchronize, reopened]
paths:
- "scripts/report-embeds/**"
- "snippets/WandbReport.jsx"
- "**/*.mdx"
schedule:
- cron: "17 6 * * 1" # Weekly, Monday 06:17 UTC
workflow_dispatch:

concurrency:
group: report-embeds-${{ github.ref }}
cancel-in-progress: true

permissions:
contents: read
issues: write # scheduled liveness files an issue on failure
issues: write # files an issue on failure

jobs:
static:
if: github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
fetch-depth: 0 # needed to diff for the registry-changed gate below

- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.12"

- run: pip install pyyaml

- name: Unit tests
run: python3 -m unittest discover -s scripts/report-embeds/tests -p 'test_*.py' -v

- name: Registry + MDX consistency
run: python3 scripts/report-embeds/check_embeds.py --mode static --github

- name: Did the registry change?
id: registry-changed
if: github.event_name == 'pull_request'
uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6
with:
files: scripts/report-embeds/registry.yaml

- name: Liveness for registry-touching PRs
if: >-
github.event_name == 'pull_request' &&
steps.registry-changed.outputs.any_changed == 'true'
run: python3 scripts/report-embeds/check_embeds.py --mode liveness --github

liveness:
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7

- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.12"

- run: pip install pyyaml

- name: Full check (static + liveness)
- name: Check embeds
id: check
continue-on-error: true
run: python3 scripts/report-embeds/check_embeds.py --mode all --output-md ./embed-report.md

run: python3 scripts/report-embeds/check_embeds.py --output-md ./embed-report.md
- name: Create issue from file
if: steps.check.outcome == 'failure'
uses: peter-evans/create-issue-from-file@fca9117c27cdc29c6c4db3b86c48e4115a786710 # v6
with:
title: Fix broken or unregistered report embeds in docs.wandb.ai
title: Fix broken report embeds in docs.wandb.ai
content-filepath: ./embed-report.md
labels: report, automated issue
15 changes: 7 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ To check or apply style, read the relevant pass file(s) and use them as guidance
response="A concise example ARIA response"
/>
```
- **Live report embeds**: To show a live, interactive W&B report on a page, import `/snippets/WandbReport.jsx` and render it with `src` (the report URL), `title` (an accessible description), and optional `height` (500–800; default 640):
- **Live report embeds**: import `/snippets/WandbReport.jsx` and render it with `src` (report URL), `title` (accessible description), and optional `height` (500–800; default 640). The component renders its own "View Report" button, so no separate link is needed.

```mdx
import { WandbReport } from '/snippets/WandbReport.jsx';
Expand All @@ -111,13 +111,12 @@ To check or apply style, read the relevant pass file(s) and use them as guidance
/>
```

Rules for every embed:
- **Always pair it with prose and a plain Markdown link** to the same report in the surrounding text, stating what the reader should take from it. Agents, the llms.txt export, and the translation pipeline read MDX source, where the iframe is opaque; the link is also the fallback wherever third-party frames are blocked. This is enforced by CI (`scripts/report-embeds/check_embeds.py`).
- **Register the report** in `scripts/report-embeds/registry.yaml` — see [`scripts/report-embeds/README.md`](scripts/report-embeds/README.md) for the fields and workflow.
- **The report must be viewable by anonymous visitors**: a report in a public project, or one shared via a view-only link (`Share` → "anyone with the link can view"; see [View-only report links](/models/reports/cross-project-reports#view-only-report-links)). The URL — including any `?accessToken=` — ships in public source and git history, so treat the report as public forever and never embed sensitive data.
- **Regular reports only, not Fully Connected articles.** FC articles keep their full blog chrome in a frame and look broken. Regular reports render in a slim embed view.
- **Keep it skinny and sparse**: prefer purpose-built reports (ideally a single panel grid), one or two per page maximum. Each iframe boots the full W&B app, is a fixed height with inner scroll (no auto-resize), and renders its own light theme regardless of the docs dark-mode toggle — so article-length reports show as a tall scroll region, not the whole page at once.
- English sources only — never add embeds to `ja/`, `ko/`, or `fr/` copies.
- **Regular reports only, not Fully Connected articles** — FC articles keep their full blog chrome in a frame and look broken.
- **Anonymous-viewable** — a public-project report or a [view-only link](/models/reports/cross-project-reports#view-only-report-links). The URL (with any `?accessToken=`) ships in public source and git history, so treat the report as public forever.
- **One or two per page**, skinny and purpose-built — each iframe boots the full W&B app at a fixed height, in its own light theme.
- **English sources only** — not `ja/`, `ko/`, `fr/`.

A weekly CI job (`scripts/report-embeds/check_embeds.py`) verifies each embedded report still renders and files an issue if one breaks. It does not gate PRs.

## Working with the repository

Expand Down
88 changes: 30 additions & 58 deletions scripts/report-embeds/README.md
Original file line number Diff line number Diff line change
@@ -1,70 +1,42 @@
# Report embeds

Tooling that keeps live W&B report embeds honest. A report embedded with the
[`<WandbReport>`](../../snippets/WandbReport.jsx) component is database content:
it can drift or vanish without a docs PR. This directory tracks every embed in a
registry and validates it in CI.

## Files

- `registry.yaml` — one entry per embedded report (id, URL, owner, purpose,
pages, last-reviewed date). Hand-curated. An empty `reports: []` is valid.
- `check_embeds.py` — the validator (schema + usage consistency + URL liveness).
- `tests/test_check_embeds.py` — unit tests (no network).

CI wiring lives in [`.github/workflows/report-embeds.yml`](../../.github/workflows/report-embeds.yml).

## Add an embedded report

1. Confirm the report is viewable by anonymous visitors — a report in a public
project, or one shared via a view-only ("magic") link
(`Share` → "anyone with the link can view"). The URL, including any
`?accessToken=`, ships in public source and git history, so treat the report
as public forever. Never embed sensitive data.
2. Add an entry to `registry.yaml`. `id` is the trailing `Vmlldzo...` token in
the report URL.
3. On the page, add both the component and a plain Markdown link to the same
report in the surrounding prose:

```mdx
import { WandbReport } from '/snippets/WandbReport.jsx';

The following [sweep report](https://wandb.ai/ENTITY/PROJECT/reports/Slug--VmlldzoXXXXXXX)
shows ... .

<WandbReport
src="https://wandb.ai/ENTITY/PROJECT/reports/Slug--VmlldzoXXXXXXX"
title="Accessible description of the report"
height={700}
/>
```

The prose link is required: agents, the llms.txt export, and the translation
pipeline read MDX source, where the iframe is opaque.

4. Validate locally:

```bash
pip install pyyaml
python3 scripts/report-embeds/check_embeds.py --mode static
```
`check_embeds.py` keeps live W&B report embeds honest. It finds every
[`<WandbReport>`](../../snippets/WandbReport.jsx) by scanning the English `.mdx`
sources (no registry), then checks each one: it sits on a page, has a
recognizable report URL, and still renders anonymously over the network.

## Embed a report

Use a **regular W&B report, not a Fully Connected article** (FC articles keep
their full blog chrome in a frame and look broken), viewable anonymously — in a
public project or shared via a view-only link (`Share` → "anyone with the link
can view"). The URL, including any `?accessToken=`, ships in public source and
git history, so treat the report as public forever. Add the component (it
renders a "View Report" button, so no separate link is needed):

```mdx
import { WandbReport } from '/snippets/WandbReport.jsx';

<WandbReport
src="https://wandb.ai/ENTITY/PROJECT/reports/Slug--VmlldzoXXXXXXX"
title="Accessible description of the report"
height={700}
/>
```

## Run the checks

```bash
python3 -m unittest discover -s scripts/report-embeds/tests -p 'test_*.py' -v
python3 scripts/report-embeds/check_embeds.py --mode static # schema + MDX, no network
python3 scripts/report-embeds/check_embeds.py --mode liveness # anonymous HTTP per URL
python3 scripts/report-embeds/check_embeds.py --mode all # both
python3 scripts/report-embeds/check_embeds.py # scan + liveness
python3 -m unittest discover -s scripts/report-embeds -p 'check_embeds.py' # unit tests
```

Static errors (registry drift, missing prose link, unregistered embed) fail CI on
PRs. Liveness runs on a weekly schedule and files an issue on failure — it does
not block PRs, because `wandb.ai` rate-limits crawlers and network results are
flaky (this is also why `wandb.ai` is excluded from the site-wide lychee check).
CI ([`report-embeds.yml`](../../.github/workflows/report-embeds.yml)) runs this
weekly and files an issue if a report has gone dead. It does not run on PRs —
`wandb.ai` rate-limits crawlers, so network results are too flaky to gate merges.

## Component coupling

`check_embeds.py` keys on the literal tag `WandbReport` and the `src` prop
(`COMPONENT_RE` / `SRC_ATTR_RE` at the top of the file). If
`snippets/WandbReport.jsx` renames either, update those constants.
(`COMPONENT_RE` / `SRC_ATTR_RE`). If `snippets/WandbReport.jsx` renames either,
update those constants.
Loading