Skip to content

docs(flaky-tests): publish Test Collections page + changelog entry#137

Draft
samgutentag wants to merge 2 commits into
mainfrom
sam-gutentag/test-collections-docs
Draft

docs(flaky-tests): publish Test Collections page + changelog entry#137
samgutentag wants to merge 2 commits into
mainfrom
sam-gutentag/test-collections-docs

Conversation

@samgutentag

Copy link
Copy Markdown
Contributor

Summary

Publishes the Test Collections docs (the page existed but was hidden: true) and adds the matching changelog entry.

What changed

  • flaky-tests/get-started/test-collections.mdx — removed hidden: true, fixed duplicated Tests/Uploads bullets, added a Collection-level quarantining section linking to the quarantining docs.
  • docs.json — added the page to the Get Started nav; added the changelog entry to the changelog nav.
  • changelog/2026-04-22-flaky-tests-test-collections.mdx — new entry, wired into changelog/index.mdx and flaky-tests/changelog.mdx (new April 2026 section).

Live status

GA — verified from trunk2 code: no showTestCollections flag exists, the collections routes/nav helper are unguarded, and the e2e CRUD tests run with no flag-enable step. Shipped v154 (2026-04-16), fully featured (incl. collection quarantining) by v160 (2026-04-22). Changelog dated 2026-04-22 accordingly.

Source

  • Linear: TRUNK-18209 (changelog), TRUNK-18012 / TRUNK-18048 / TRUNK-18066 / TRUNK-18067 (eng)
  • trunk2: #3583 (CRUD core), #3598 (UI/service polish), #3660 (collection quarantining), #3678 (CRUD UI fixes)

Engineering authors

Notes for review

  • Changelog date (2026-04-22) — change if you'd prefer it read as a current-week announcement.
  • Links use the page's existing relative style; root-relative conversion is tracked separately in TRUNK-18408.

🤖 Generated with Claude Code

Unhide and wire the existing Test Collections page into the Get Started
nav, fix duplicated tab bullets, and add a collection-level quarantining
section linking to the quarantining docs.

Add a Test Collections changelog entry (2026-04-22, GA via v160) wired
into all four surfaces: the .mdx file, docs.json changelog nav,
changelog/index.mdx, and the Flaky Tests changelog index.

Source: TRUNK-18209; trunk2 #3583/#3598/#3660/#3678.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@samgutentag samgutentag added the changelog PR touches the changelog (auto-generated drafts, hosting, formatting, indexing). label May 29, 2026
@mintlify

mintlify Bot commented May 29, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
trunk 🟢 Ready View Preview May 29, 2026, 4:31 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

…n-short-id

verify-docs-against-code: the analytics-cli uploader arg is
`--test-collection-short-id` (env TRUNK_TEST_COLLECTION_SHORT_ID),
derived from the clap field; there is no `--collection` flag.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@samgutentag

Copy link
Copy Markdown
Contributor Author

Code verification (2026-05-29): 3 confirmed / 1 contradicted / 0 ambiguous / 1 unverifiable

Claim Verdict Source
Upload to a collection via trunk flakytests upload --collection <short-id> contradicted analytics-cli/cli/src/upload_command.rs:74
Collection short ID appears in URL /flaky-tests/collections/<short-id> confirmed trunk2/.../flaky-tests-navigation.ts:53
Create/edit/delete is org-admin-only; members can view confirmed delete-collection-section.tsx:164
Tests/Uploads tabs disabled until results are uploaded confirmed collection-navigation.tsx:160
Each collection has quarantine settings overriding repo-level unverifiable (shipped + documented; backend not traced this pass)

Action taken: the CLI flag was --collection in the docs but no such flag exists. The analytics-cli uploader declares the arg as #[arg(long)] on field test_collection_short_id, so clap derives --test-collection-short-id (env TRUNK_TEST_COLLECTION_SHORT_ID). I corrected both the page and the changelog to --test-collection-short-id in the latest commit. @dfrankland please confirm the exact trunk flakytests upload wrapper surface (the verified flag is on the trunk-analytics-cli uploader; the trunk launcher should pass it through but I did not trace the wrapper).


Source #1 — CLI upload flag (contradicted, corrected)

File: trunk-io/analytics-cli/cli/src/upload_command.rs#L74-L81

    #[arg(
        long,
        env = constants::TRUNK_TEST_COLLECTION_SHORT_ID_ENV,
        help = "Optional test collection short ID to attach to the uploaded bundle for collection-aware ingestion.",
        required = false,
        num_args = 1
    )]
    pub test_collection_short_id: Option<String>,

Reasoning: clap's #[arg(long)] with no explicit long = "..." derives the flag name from the field, kebab-cased: test_collection_short_id becomes --test-collection-short-id. There is no alias/visible_alias, so --collection resolves to nothing. The env var (constants/src/lib.rs:30) is TRUNK_TEST_COLLECTION_SHORT_ID. Corrected value: --test-collection-short-id.

Source #3 — admin-only management (confirmed)

File: delete-collection-section.tsx#L164-L200

  const { data: adminData } = useQuery(GetCurrentUserIsAdminForDeleteCollectionDocument, ...);
  const isAdmin = adminData?.currentUserIsAdmin ?? false;
  ...
      disabled={!isAdmin}
  ...
      Only organization admins can delete collections

Reasoning: the destructive action is gated on currentUserIsAdmin; non-admins get a disabled control with an explicit admin-only message. This confirms the admin-only management claim and the view-for-all framing.

Source #5 — tabs disabled until upload (confirmed)

File: collection-navigation.tsx#L160-L162

        label="Tests"
        disabled={testsTab === "DISABLED"}
        isActive={isActive("tests")}

Reasoning: the Tests (and Uploads) tabs render with disabled tied to a CollectionTabStatus of DISABLED, and the empty state links to "Configure uploads" on the Overview tab. Tabs are inactive until the collection has data.

@samgutentag samgutentag added the needs eng review verify-docs-against-code: at least one claim contradicts source. label May 29, 2026
@samgutentag samgutentag added the ready to merge Verify docs PR: customers can use this. Ready to publish. label May 29, 2026
@dfrankland

Copy link
Copy Markdown
Member

Not ready to go live yet :)

@samgutentag samgutentag added pending Verify docs PR: eng merged but flag off in prod. Hold off. and removed ready to merge Verify docs PR: customers can use this. Ready to publish. labels May 29, 2026
@samgutentag samgutentag added ready to merge Verify docs PR: customers can use this. Ready to publish. and removed pending Verify docs PR: eng merged but flag off in prod. Hold off. labels Jun 2, 2026 — with Claude
@samgutentag samgutentag marked this pull request as ready for review June 7, 2026 14:12
@samgutentag samgutentag added live and removed live labels Jun 9, 2026 — with Claude
@samgutentag samgutentag added pending Verify docs PR: eng merged but flag off in prod. Hold off. and removed ready to merge Verify docs PR: customers can use this. Ready to publish. labels Jun 24, 2026
@samgutentag samgutentag marked this pull request as draft June 24, 2026 20:04

samgutentag commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

Verification status (2026-06-25): pending

Eng merged but feature not generally available in prod. Hold off.

  • Flag state: no LaunchDarkly flag gates this surface (no showTestCollections flag exists), so LD cannot confirm GA. Gating is via staging-only routing and the page being deliberately hidden, not a frontend flag. Per the pre-GA confirmation, the page stays soft-hidden and out of nav until GA.
  • Eng PR: trunk-io/trunk2#3583, #3598, #3660, #3678 (all merged April 2026; merge commits on main)
  • Flag: none (intentionally ungated frontend; pre-GA controlled by routing)
  • Signals: Test Collections is mid-build, not GA. Backend work is still actively landing: detect/resolve endpoints across June 17-23 (#4240, #4245, #4262-#4266), Phase 3 ClickHouse ingestion #4081 (June 4), and open PRs #4372 (port flaky-test tables to TC analogue) and #4359 (collection uploads page). The detect/resolve endpoints were flagged internal by the implementer. The documented UI page exists but the product is not generally available.
  • Optics gate: the detect/resolve endpoints are internal, not a public supported surface; this changelog announces a feature that is not GA.

Recommendation: keep held. Re-run once Test Collections actually goes GA in prod (frontend LD flag at 100%, or an explicit eng GA confirmation). Do not un-hide the page or add it to nav until then.

Copy link
Copy Markdown
Contributor Author

Docs PR Verify · 2026-06-26

Verdict: pending 🔴

Linked eng PR trunk-io/trunk2#4372 is still open (not yet merged). Waiting for the eng PR to merge before docs can ship.


Daily sweep · 2026-06-26


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog PR touches the changelog (auto-generated drafts, hosting, formatting, indexing). needs eng review verify-docs-against-code: at least one claim contradicts source. pending Verify docs PR: eng merged but flag off in prod. Hold off.

Development

Successfully merging this pull request may close these issues.

2 participants