From b7ac65cd899a8415085fc8881df52effcc6b422b Mon Sep 17 00:00:00 2001 From: Ashlen Date: Wed, 15 Jul 2026 19:33:15 -0600 Subject: [PATCH 1/4] docs(plans): add dogfood policy-tests design + plan --- .../plans/2026-07-07-dogfood-policy-tests.md | 383 ++++++++++++++++++ .../2026-07-07-dogfood-policy-tests-design.md | 232 +++++++++++ 2 files changed, 615 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-07-dogfood-policy-tests.md create mode 100644 docs/superpowers/specs/2026-07-07-dogfood-policy-tests-design.md diff --git a/docs/superpowers/plans/2026-07-07-dogfood-policy-tests.md b/docs/superpowers/plans/2026-07-07-dogfood-policy-tests.md new file mode 100644 index 0000000..ead4c1d --- /dev/null +++ b/docs/superpowers/plans/2026-07-07-dogfood-policy-tests.md @@ -0,0 +1,383 @@ +# Dogfood policy-tests (+ sha-pin offline gate) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Dogfood the template's policy-tests layer onto the maintainer repo — a stdlib-only `tests/policy/test_gates.py` asserting the maintainer's own gates cannot be silently weakened — closing gap #4 and (via `test_actions_are_sha_pinned`) the offline half of gap #8. + +**Architecture:** Add `tests/policy/{__init__.py,test_gates.py}` (five stdlib assertions, ROOT = maintainer root); add a `policy` recipe and chain it into `just ci` (`… test policy audit`); document it in AGENTS.md and sweep every `ci` gate-list site. No CI/dep/mise/CHANGELOG/generation-test change — the existing CI `test` job's bare `uv run pytest` already collects `tests/policy`, and `tests/test_generation.py` already asserts the template layer. + +**Tech Stack:** Python 3.11 stdlib (`re`, `tomllib`, `pathlib`, `typing.cast`); pytest; ruff (`select=["ALL"]`); basedpyright (`recommended` + `failOnWarnings`); `just`; `uv`. + +Design: `docs/superpowers/specs/2026-07-07-dogfood-policy-tests-design.md`. +Closes Phase-1 item #5 (gap #4 policy-tests + gap #8 sha-pin) from `docs/superpowers/plans/2026-07-01-dogfood-gap-audit.md` §7. + +## Global Constraints + +- Branch `chore/dogfood-policy-tests`, cut from `main` at `b7dd8b8` (after PR #9 merged). +- Commits: Conventional Commits, GPG-signed, author `Ashlen `, **no** AI-attribution trailer. +- Mirror the template; document forced divergences. **Locked decisions:** fenced-only docs-can't-lie regex; **include** the `failOnWarnings is True` assertion. +- The authoritative `test_gates.py` body is the one inlined in **Task 1 Step 2** below (ruff/basedpyright/pytest all green in the design-review worktree). Use it **verbatim** — in particular the fenced docs-can't-lie test must use `cast("list[str]", …)` + an annotated `referenced: set[str]` accumulator loop, **not** a set-comprehension (which trips basedpyright `reportAny` ×3). +- **AGENTS.md prose invariants — the docs-can't-lie test parses AGENTS.md as raw text.** The extractor pairs triple-backtick runs *sequentially*, so an **odd** number of runs mispairs every fence after the offender and sweeps unrelated prose into a "block". Therefore, in AGENTS.md: **(a)** never write a literal triple-backtick run in prose — write "fenced code block", never the delimiter itself; **(b)** never write a bare `just ` for a non-recipe (a template-only recipe such as `fuzz`/`mutate`) anywhere, prose included — name it without the `just ` prefix. (b) is defence-in-depth: it stops a future (a)-slip from turning into a red gate. This is not hypothetical — the first draft of the Task 3 section violated (a) *while describing this very divergence*, taking AGENTS.md from 16 triple-backtick runs to 19 and failing the suite on `{fuzz, mutate}`. +- No file under `template/`; no `test-template.yml`, `pyproject.toml`, `mise.toml`, `uv.lock`, `tests/test_generation.py`, or CHANGELOG change; no inline `# noqa`; no Action SHA bump. +- Environment: run `just` via `mise exec -- just` (mise is not auto-activated in non-interactive shells); `uv`/`uvx` are on PATH. The `policy` subset (`uv run pytest tests/policy`) is fast and needs no roomy TMPDIR; only the full `just test`/`just ci` matrix does. + +--- + +### Task 0: Commit the design spec + this plan + +**Files:** +- Create: `docs/superpowers/specs/2026-07-07-dogfood-policy-tests-design.md` (already written) +- Create: `docs/superpowers/plans/2026-07-07-dogfood-policy-tests.md` (this file) + +- [ ] **Step 1: Cut the feature branch** + +```bash +git checkout main && git pull +git checkout -b chore/dogfood-policy-tests +``` + +- [ ] **Step 2: Stage and commit the docs pair** + +```bash +git add docs/superpowers/specs/2026-07-07-dogfood-policy-tests-design.md docs/superpowers/plans/2026-07-07-dogfood-policy-tests.md +git commit -S -m "docs(plans): add dogfood policy-tests design + plan" +``` + +Expected: one commit, GPG-signed, author `Ashlen `, no AI trailer. + +--- + +### Task 1: `tests/policy/` — the policy suite (gap #4 + #8) + +**Files:** +- Create: `tests/policy/__init__.py` (empty) +- Create: `tests/policy/test_gates.py` + +**Interfaces:** +- Consumes: the maintainer's own `pyproject.toml`, `justfile`, `AGENTS.md`, `.github/workflows/*.yml` (read at `ROOT = Path(__file__).resolve().parent.parent.parent`). +- Produces: five pytest functions — `test_type_checking_is_recommended`, `test_type_checking_fails_on_warnings`, `test_ruff_select_present`, `test_agents_md_recipes_exist_in_justfile`, `test_actions_are_sha_pinned`. These are collected by `just policy` (Task 2) and by bare `uv run pytest` (the CI `test` job + `just test`). + +- [ ] **Step 1: Create the empty package marker** + +Create `tests/policy/__init__.py` with **no content** (0 bytes; `D104` is ignored under `tests/**`, and end-of-file-fixer leaves an empty file alone). + +- [ ] **Step 2: Create the test suite** + +Create `tests/policy/test_gates.py` with exactly this content (verbatim from the validated file): + +```python +"""Config-literal pins: gates cannot be silently weakened. Stdlib only.""" + +import re +import tomllib +from pathlib import Path +from typing import cast + +ROOT = Path(__file__).resolve().parent.parent.parent +PYPROJECT = tomllib.loads((ROOT / "pyproject.toml").read_text()) + + +def test_type_checking_is_recommended() -> None: + assert PYPROJECT["tool"]["basedpyright"]["typeCheckingMode"] == "recommended" + + +def test_type_checking_fails_on_warnings() -> None: + assert PYPROJECT["tool"]["basedpyright"]["failOnWarnings"] is True + + +def test_ruff_select_present() -> None: + select = cast("list[str]", PYPROJECT["tool"]["ruff"]["lint"]["select"]) + assert "ALL" in select + + +def test_agents_md_recipes_exist_in_justfile() -> None: + """docs-can't-lie: the first `just ` per fenced invocation is a real recipe.""" + justfile = (ROOT / "justfile").read_text() + recipes = set(re.findall(r"^([a-z][a-z-]*):", justfile, re.MULTILINE)) + agents = (ROOT / "AGENTS.md").read_text() + blocks = cast("list[str]", re.findall(r"```.*?```", agents, re.DOTALL)) + referenced: set[str] = set() + for block in blocks: + referenced |= set(re.findall(r"just ([a-z][a-z-]*)", block)) + assert referenced <= recipes, f"AGENTS.md names missing recipes: {referenced - recipes}" + + +def test_actions_are_sha_pinned() -> None: + """Every third-party `uses:` is a 40-char SHA + a v-prefixed version comment.""" + # SHA + a version comment (v[.minor[.patch]]); our pins carry the action's + # exact tag. The maintainer re-verifies the comment matches the SHA at bump time + # via `gh api .../tags`. + pattern = re.compile(r"uses:\s*\S+@([0-9a-f]{40})\s+#\s*v\d+(\.\d+){0,2}\b") + for wf in (ROOT / ".github" / "workflows").glob("*.yml"): + for line in wf.read_text().splitlines(): + stripped = line.strip() + if stripped.startswith(("- uses:", "uses:")): + assert pattern.search(stripped), f"unpinned action in {wf.name}: {stripped}" +``` + +- [ ] **Step 3: Run the suite — expect 5 passed on the current tree** + +Run: `uv run pytest tests/policy -q` +Expected: `.....` → **5 passed**. (All five assertions hold today: `recommended` + `failOnWarnings` at pyproject L35–36; `"ALL"` at L50; every fenced `just ` in AGENTS.md is real; all 12 `uses:` lines are SHA + `# v`.) + +- [ ] **Step 4: Lint the new files — expect clean** + +Run: `uv run ruff check tests/policy` +Expected: `All checks passed!` (the `tests/**` per-file-ignores already cover `S101`/`PLR2004`/`D103`/`D104`; no inline `# noqa`). + +- [ ] **Step 5: Type-check — expect 0 errors, 0 warnings** + +Run: `uv run basedpyright` +Expected: `0 errors, 0 warnings, 0 notes`. (If it reports `reportAny` warnings, the docs-can't-lie test was written as a set-comprehension instead of the annotated-loop form above — fix to match Step 2 exactly.) + +- [ ] **Step 6: Commit** + +```bash +git add tests/policy/__init__.py tests/policy/test_gates.py +git commit -S -m "test(policy): dogfood the gates-can't-be-weakened suite (closes gap #8 sha-pin)" +``` + +--- + +### Task 2: `justfile` — `policy` recipe + `ci` member (gap #4) + +**Files:** +- Modify: `justfile` (add the `policy` recipe after `test`; add `policy` to the `ci` recipe at line 11) + +**Interfaces:** +- Consumes: `tests/policy/` (Task 1). +- Produces: recipe `policy` (so `just policy` and any AGENTS.md fenced `just policy` example resolve to a real recipe); the `ci` recipe now runs `… test policy audit`. + +- [ ] **Step 1: Add `policy` to the `ci` recipe** + +In `justfile`, replace line 11: + +```just +ci: fmt-check lint typecheck test audit +``` + +with: + +```just +ci: fmt-check lint typecheck test policy audit +``` + +- [ ] **Step 2: Update the `ci` recipe comment for accuracy** + +In `justfile`, the `ci` comment currently reads (lines 7–8): + +```just +# Mirrors template/justfile.jinja's `ci` (audit is a member — the template chains it, +# and audit is itself a PR-blocking check here via the CI `scan` job's pip-audit step). +``` + +Replace those two lines with: + +```just +# Mirrors template/justfile.jinja's `ci` (policy + audit are members — the template chains +# both; each is also an independent PR-blocking check here — policy via the CI `test` job's +# pytest collection, audit via the CI `scan` job's pip-audit step). +``` + +(No other comment line changes; there is no enumerated gate list in the comment to sync.) + +- [ ] **Step 3: Add the `policy` recipe immediately after the `test` recipe** + +In `justfile`, the `test` recipe is: + +```just +# Run the template generation + update tests +test: + uv run pytest +``` + +Insert directly **after** it (before `# Lint this repo's own tooling` / `lint:`): + +```just + +# Policy gate: pins the gate config literals (type mode, failOnWarnings, ruff select, Action +# SHAs) so they cannot be silently weakened — not the ignore lists or the recipe bodies. +# Stdlib-only; also auto-collected by `just test`. Mirrors template/justfile.jinja's `policy`. +policy: + uv run pytest tests/policy +``` + +- [ ] **Step 4: Verify `just` parses the recipes** + +Run: `mise exec -- just --show ci` then `mise exec -- just --show policy` +Expected: `ci: fmt-check lint typecheck test policy audit` (+ its `@echo`); `policy` shows `uv run pytest tests/policy`. + +- [ ] **Step 5: Run the recipe — expect 5 passed** + +Run: `mise exec -- just policy` +Expected: `uv run pytest tests/policy` → **5 passed**. + +- [ ] **Step 6: Commit** + +```bash +git add justfile +git commit -S -m "feat(ci): dogfood 'just policy' + chain it into 'just ci'" +``` + +--- + +### Task 3: `AGENTS.md` — new Policy-gate section + `ci` gate-list sweep + +**Files:** +- Modify: `AGENTS.md` (new `## Policy gate` section after `## Dependency audit`; four `ci` gate-list sweep edits at the `## Run every gate` section) + +**Interfaces:** +- Consumes: the `policy` recipe (Task 2) — so the new section's fenced `just policy` example resolves to a real recipe, keeping `test_agents_md_recipes_exist_in_justfile` green. + +- [ ] **Step 1: Sweep site — the `+`-form fenced comment** + +Replace (in the `## Run every gate` fenced block): + +``` +just ci # fmt-check + lint + typecheck + test + audit, then "ci: all gates passed" +``` + +with: + +``` +just ci # fmt-check + lint + typecheck + test + policy + audit, then "ci: all gates passed" +``` + +- [ ] **Step 2: Sweep site — the prose recipe reproduction** + +Replace the sentence fragment: + +``` +`just ci` is the complete local gate — `ci: fmt-check lint typecheck test audit` ending `@echo "ci: all gates passed"`, with `verify` a bare alias (`verify: ci`). It **mirrors** `template/justfile.jinja`'s `ci`: the template chains `audit` when `enable_dependency_audit` is on (it is), so the faithful maintainer recipe includes it. +``` + +with: + +``` +`just ci` is the complete local gate — `ci: fmt-check lint typecheck test policy audit` ending `@echo "ci: all gates passed"`, with `verify` a bare alias (`verify: ci`). It **mirrors** `template/justfile.jinja`'s `ci`: the template chains `policy` and `audit` when `enable_policy_tests`/`enable_dependency_audit` are on (they are), so the faithful maintainer recipe includes both. +``` + +- [ ] **Step 3: Sweep site — the forward-sync note** + +Replace the `**Forward-sync:**` paragraph: + +``` +**Forward-sync:** when the policy-tests (gap #4) and property-tests (gap #9) layers land, add `policy`/`fuzz` to **both** the `ci` recipe **and** this section, to keep mirroring the template's conditional `ci` (`fmt-check lint typecheck test{% if enable_property_tests %} fuzz{% endif %}{% if enable_policy_tests %} policy{% endif %}{% if enable_dependency_audit %} audit{% endif %}`). +``` + +with: + +``` +**Forward-sync:** the policy-tests layer (gap #4) has landed — `policy` is now a `ci` member (see "Policy gate"). When the property-tests layer (gap #9) lands, add `fuzz` to **both** the `ci` recipe **and** this section, to keep mirroring the template's conditional `ci` (`fmt-check lint typecheck test{% if enable_property_tests %} fuzz{% endif %}{% if enable_policy_tests %} policy{% endif %}{% if enable_dependency_audit %} audit{% endif %}`). +``` + +(The quoted template conditional string is left **verbatim** — it is the template's own literal.) + +- [ ] **Step 4: Sweep site — the divergences paragraph** + +Replace: + +``` +Deliberate divergences from `template/justfile.jinja`'s `ci`: **none in the gate list** — `ci: fmt-check lint typecheck test audit`, `verify: ci`, and the `@echo` are exact. +``` + +with: + +``` +Deliberate divergences from `template/justfile.jinja`'s `ci`: **none in the gate list** — `ci: fmt-check lint typecheck test policy audit`, `verify: ci`, and the `@echo` are exact. +``` + +(The "none in the gate list" claim stays true — the template conditionally adds `policy` via `{% if enable_policy_tests %}`, which is on.) + +- [ ] **Step 5: Add the new `## Policy gate` section** + +In `AGENTS.md`, insert this section **immediately after** the `## Dependency audit` section (i.e. after the Dependency-audit "Deliberate divergences … pin-sync note" paragraph, before `## Add a guardrail layer`). + +**Insert the section body only.** The four-backtick fence below is this plan's quoting wrapper (it exists so the inner bash fence can be shown verbatim); it must **not** land in AGENTS.md. After inserting, AGENTS.md must gain exactly **two** triple-backtick runs — the bash fence's open and close — taking it from 16 to **18** (even). Step 6 checks this. + +```` +## Policy gate (`just policy`) + +```bash +just policy # pins the gate config literals so they cannot be silently weakened (stdlib-only) +``` + +`just policy` runs `uv run pytest tests/policy` — a stdlib-only (`re`/`tomllib`/`pathlib`) suite pinning this repo's gate **config literals** so they cannot be silently weakened: basedpyright `typeCheckingMode == "recommended"` **and** `failOnWarnings is True`; `"ALL" in ruff select`; the first `just ` of each **fenced** AGENTS.md example is a real justfile recipe (docs-can't-lie); and every third-party `uses:` is `@<40-hex> # v` (`test_actions_are_sha_pinned` — this **closes the offline half of the SHA-pin policy**, gap #8). It is a `ci` member (`ci: … test policy audit`) and is also auto-collected by bare `just test` (`uv run pytest`, `testpaths=["tests"]`), so CI enforces it inside the existing `test` job with **no** workflow change. + +Changing any of those gates (ruff select, type mode, `failOnWarnings`, an Action pin) requires editing the matching test in `tests/policy/test_gates.py` in the **same commit** — that is the point of the suite. **Scope:** it pins those config *literals* — not the ruff `ignore` / `per-file-ignores` lists (each is separately audited; see "Lint & format this repo"), not the recipe *bodies*, and not the CI workflow that runs the gates (candidate F — see "Run every gate"). + +Deliberate divergences from `template/tests/…/test_gates.py.jinja`: the **coverage-floor** assertion is dropped (the maintainer has no `[tool.coverage]` / no runtime package — nothing to protect; the `policy` recipe likewise drops `--no-cov`, which is invalid with no `pytest-cov`); a **`failOnWarnings is True`** assertion is added (the template ships none — the maintainer pins the type-gate's teeth explicitly, so the suite guards that pin); and the **docs-can't-lie regex is scoped to fenced code blocks** rather than the whole file. The template's naive whole-doc scan still passes here *today*, but it flips to a hard fail the moment this file names a template-only recipe (`fuzz`/`mutate`) in prose — which gap #9 will do. Fenced-only gates the examples you are told to **run** and lets prose name any recipe freely. + +**Two invariants this file must hold** (the test reads AGENTS.md as raw text, and the extractor pairs fences *sequentially*): **(a)** never write a literal triple-backtick run in prose — an odd total mispairs every fence after it and sweeps unrelated prose into a "block"; write "fenced code block", never the delimiter. **(b)** Never write a bare `just ` for a non-recipe anywhere, prose included — name it without the `just ` prefix (as `fuzz`/`mutate` are named above). (b) is defence-in-depth: it stops a future (a)-slip from turning into a red gate. The suite also scans `#` comments *inside* fences, so the same rule holds there. + +The SHA-pin sub-check overlaps the zizmor job (the security control), so its net-new value is annotation hygiene; the config-literals + docs-can't-lie are the substantive net-new gate. `tests/policy/__init__.py` exists because the maintainer's `tests/` is a package (the template ships no such file). **Known limits:** the recipe allowlist regex matches only `[a-z][a-z-]*` names, so a future parametrized or underscored recipe cited in a fenced example would false-positive (broaden the pattern in the same commit); and nothing here asserts that CI still *runs* the gates (candidate F). +```` + +- [ ] **Step 6: Fence-parity guard — AGENTS.md must hold an even number of triple-backtick runs** + +```bash +python3 -c "import pathlib,sys;n=pathlib.Path('AGENTS.md').read_text().count(chr(96)*3);print('fence runs:',n);sys.exit(n%2)" +``` + +Expected: `fence runs: 18`, exit 0 (16 before this section + the bash fence's open/close). An **odd** count means a literal triple-backtick run leaked into prose — the extractor will mispair every fence after it and sweep prose into a "block", harvesting non-recipe `just` tokens. Fix the prose before continuing; do **not** "fix" the test. (This is exactly how the first draft of this section failed: 19 runs → `{fuzz, mutate}` → red.) + +- [ ] **Step 7: Re-run the *whole* policy suite** + +Run: `uv run pytest tests/policy -q` +Expected: **5 passed**. The docs-can't-lie test now scans the new section: its only fenced `just` token is `just policy` (a real recipe as of Task 2), and no bare non-recipe `just` token appears anywhere in the file. Run the whole suite, not the single test — Task 1 Step 3's "5 passed" predates this section and never exercised this path. + +- [ ] **Step 8: Sweep verification — no stale gate-list literal remains** + +Run: `grep -nE 'test ?\+? ?(policy ?\+? ?)?audit' justfile AGENTS.md` +Expected: every hit shows the `… policy … audit` form; **no** bare `test audit` / `test + audit` remains. + +- [ ] **Step 9: Commit** + +```bash +git add AGENTS.md +git commit -S -m "docs(agents): document 'just policy' + sweep the ci gate-list sites" +``` + +--- + +### Finishing: full verification + PR + +- [ ] **Step 1: Full local gate (roomy TMPDIR — the `test` matrix needs it)** + +Run: `TMPDIR=/home/user/.cache/kickstarter-test-tmp mise exec -- just ci` +Expected: fmt-check + lint + typecheck + test + **policy** + audit all pass, ending `ci: all gates passed`. (`audit` reaches the network; `policy` runs twice — once auto-collected by `test`, once as the explicit member — both trivially fast.) + +- [ ] **Step 2: Pre-commit clean over the whole tree** + +Run: `mise exec -- just precommit` +Expected: all hooks pass (new files end in `\n` / are empty; none are `^template/`-excluded; basedpyright on push clean). + +- [ ] **Step 3: Confirm the guard rails held** + +Run: `git diff --stat main` +Expected: only `docs/superpowers/{specs,plans}/2026-07-07-dogfood-policy-tests*`, `tests/policy/__init__.py`, `tests/policy/test_gates.py`, `justfile`, `AGENTS.md`. **No** `template/`, `test-template.yml`, `pyproject.toml`, `mise.toml`, `uv.lock`, `tests/test_generation.py`, or `CHANGELOG.md`. + +- [ ] **Step 4: Clean tree** + +Run: `git status --short` +Expected: empty (no lingering `requirements-audit.txt` — the `audit` step `rm`s it on success). + +- [ ] **Step 5: Push and open the PR** + +```bash +git push -u origin chore/dogfood-policy-tests +``` + +Then open the PR (base `main`) with a body per the pr-descriptions skill. State: zizmor N/A (no workflow touched); CI enforces `tests/policy` inside the existing `test` job (bare `uv run pytest` collects it) with no CI change; closes gap #4 and the offline half of gap #8. **Hand the merge to the operator** (do not merge). + +## Self-review notes (author check against the spec) + +- Five assertions, coverage-floor dropped, `failOnWarnings` added — matches design D1/D2. ✔ +- Fenced-only docs-can't-lie in the annotated-loop (typecheck-passing) form — matches design D-REGEX + the empirical finding. ✔ +- `policy` recipe after `test`; `ci: … test policy audit`; `test` stays bare (double-collect accepted) — design D3. ✔ +- No CI/dep/mise/CHANGELOG/generation-test change — design D4; `tests/test_generation.py` already covers the template layer. ✔ +- `tests/policy/__init__.py` present — design D5. ✔ +- All four `ci` gate-list sweep sites (AGENTS.md `+`-form, prose, forward-sync note, divergences) + the justfile recipe — design D6; the broadened acceptance grep catches the `+`-form. ✔ +- No `template/` file → no generation-test obligation; no inline `# noqa`; no Action SHA bump. ✔ +- **Fence parity**: the inserted section adds exactly two triple-backtick runs (16 → 18, even); no literal delimiter appears in its prose; no bare `just ` token appears anywhere in AGENTS.md. Guarded by Task 3 Step 6. ✔ *(This was the blocker in the first draft: the divergence paragraph wrote the delimiter literally **while describing fenced-only**, taking the count to 19 and failing — on `{fuzz, mutate}` — the very suite it was introducing.)* +- Headline wording scoped to what the suite actually pins (config **literals** — not the ruff ignore lists, recipe bodies, or CI wiring); the docstring says *first* `just ` per fenced invocation. ✔ diff --git a/docs/superpowers/specs/2026-07-07-dogfood-policy-tests-design.md b/docs/superpowers/specs/2026-07-07-dogfood-policy-tests-design.md new file mode 100644 index 0000000..870f8ea --- /dev/null +++ b/docs/superpowers/specs/2026-07-07-dogfood-policy-tests-design.md @@ -0,0 +1,232 @@ +# Dogfood policy-tests (+ sha-pin offline gate) — design (2026-07-07) + +## Goal + +Close two Phase-1 dogfooding gaps from `docs/superpowers/plans/2026-07-01-dogfood-gap-audit.md` +§7 item #5 in one bundle: + +- **Gap #4 — policy-tests.** `template/tests/{% if enable_policy_tests %}policy{% endif %}/test_gates.py.jinja` + ships a stdlib-only suite that reads a downstream's own `pyproject.toml`/`justfile`/`AGENTS.md`/`.github/workflows` + and asserts the gates cannot be silently weakened. The maintainer repo — whose whole pitch is that suite — + does not run it on itself. +- **Gap #8 — sha-pin offline gate.** `test_actions_are_sha_pinned()` (gated on `enable_sha_pin_policy`, inside + the same `test_gates.py`) enforces `uses: …@<40-hex> # v` offline. The maintainer's zizmor job already + enforces the SHA (surface 1, fully dogfooded); the offline comment gate (surface 2) is absent. It **closes for + free with gap #4** — porting the policy suite brings the sha-pin test along (both toggles default on). + +Governing principle (as at every prior gap): **mirror the template; diverge only where the maintainer surface +forces it, and document each forced divergence in AGENTS.md.** + +### Empirical anchors (verified against the tree, not roadmap prose) + +- **The template's assertions** (`template/tests/…/test_gates.py.jinja`): coverage `fail_under >= coverage_floor`; + `typeCheckingMode == "recommended"`; `"ALL" in ruff select` (when `ruff_ruleset == "all"`); docs-can't-lie + (`re.findall(r"just ([a-z][a-z-]*)", AGENTS.md) <= justfile recipes`); and (when `enable_sha_pin_policy`) + `test_actions_are_sha_pinned()`. +- **Maintainer `pyproject.toml` has no `[tool.coverage]`** and no `--cov` in `addopts` (`package = false`, no + `src/`). The coverage-floor assertion has nothing to protect, and `--no-cov` (which the template's `policy` + recipe and the generation test at `tests/test_generation.py:343` pass to the *downstream*) is an **invalid + arg here** — no `pytest-cov` is installed. +- **Maintainer `pyproject.toml` pins the type gate's teeth explicitly**: `typeCheckingMode = "recommended"` + (L35) **and** `failOnWarnings = true` (L36, with a comment: "pin it so a future mode/version drift … cannot + silently un-fail the build"). A policy test asserting `failOnWarnings is True` is the natural enforcement of + that stated intent. +- **Maintainer `ruff.lint.select = ["ALL"]`** (L50). +- **All 12 `uses:` lines** in `.github/workflows/test-template.yml` (4 distinct actions) are already + `@<40-hex> # v` — `test_actions_are_sha_pinned()` passes on the current tree unchanged. +- **`tests/test_generation.py` already asserts the template's policy layer** present-when-on/absent-when-off + (`test_policy_layer` L340, `test_sha_pin_policy` L587, `test_sha_pin_audit_ships_without_policy_tests` L597, + `test_coverage_floor_wiring` L785). This PR adds **maintainer-root** files, not `template/` + files → **no generation-test obligation** and no `template/` change. +- **Maintainer AGENTS.md `just` tokens inside ```-fenced blocks** are all real recipes today (audit, ci, fmt, + fmt-check, lint, precommit, scan, setup, test, typecheck). The maintainer's AGENTS.md **prose** is the one + place that legitimately discusses *template-only* recipes (`fuzz`, `mutate`) — a downstream's AGENTS.md never + does. (See D-REGEX.) + +## Decisions + +- **D1 — Drop the coverage-floor assertion.** No `[tool.coverage]` / no runtime package → nothing to protect + (roadmap §140: "genuinely N/A"). **Surface-forced** divergence; documented. +- **D2 — Keep four assertions, add a fifth (unforced).** Port `test_type_checking_is_recommended`, + `test_ruff_select_present`, the docs-can't-lie test, and `test_actions_are_sha_pinned` (gap #8). **Add** + `test_type_checking_fails_on_warnings` asserting `failOnWarnings is True` — the maintainer pins this teeth + explicitly (pyproject L36) and the policy suite exists precisely to stop such a pin being silently removed + (roadmap §79). This is an **unforced** strengthening (the template ships *no* `failOnWarnings` assertion — the + reviewers confirmed it), but it protects a real maintainer-only pin, so it is documented and **offered for + sign-off** alongside D-REGEX. +- **D-REGEX — Tune docs-can't-lie to fenced-code-block invocations only.** THE key decision, and an **unforced, + preventive** divergence — *not* surface-forced: the template's naive whole-doc regex `re.findall(r"just + ([a-z][a-z-]*)", AGENTS.md)` **passes on the current tree** (`missing=[]`; every prose `just X` token today is a + real recipe). The invariant fenced-only encodes is "every recipe I tell you to **run** exists," not "every + recipe I **mention** exists." Rationale for diverging anyway: the maintainer's AGENTS.md is the one doc that + will document template-only recipes in prose — the naive regex flips to **FAIL** the moment gap-#9's maintainer + prose mirrors `template/AGENTS.md.jinja` L32/L36's verbatim `just fuzz` / `just mutate` wording (both confirmed + present in the template). Fenced-only gates runnable examples and lets prose freely reference template recipes. + **Costs / caveats:** it stops checking *inline-prose* `just ` references against recipe renames; it + still scans `#` comments *inside* fences (so the discipline "no bare `just ` in a fenced block" + remains); and `just --show` / multi-recipe lines fail **open** (only the first token per invocation is read). + Two cases do **not** fail open, and both bit the plan's first draft: + **(a) fence parity** — the extractor pairs triple-backtick runs *sequentially*, so an **odd** number of runs in + AGENTS.md mispairs every fence after the offender and sweeps arbitrary prose into a "block". Writing the + delimiter literally in prose — e.g. to *describe this very divergence* — is enough to trigger it, and the + spurious block then harvests any prose `just ` token into `referenced` → hard **FAIL**. + **(b) recipe-name shape** — the allowlist regex matches only `[a-z][a-z-]*`, so a future parametrized or + underscored recipe cited in a fenced example would be absent from `recipes` → **false-positive**. + Mitigations (now mandated in the plan's Global Constraints and in the shipped AGENTS.md section): never write a + literal triple-backtick run in AGENTS.md prose, and never write a bare `just ` for a non-recipe anywhere in + AGENTS.md — name it without the `just ` prefix. Plan Task 3 Step 6 adds a fence-parity guard. + *Alternative — naive whole-doc mirror (exact template parity, checks all prose) + a standing "never write a bare + `just ` in prose" discipline that breaks the moment gap #9 lands.* **This is the one call worth + explicit sign-off.** +- **D3 — `policy` recipe + `ci` member; `test` stays bare.** Add `policy: uv run pytest tests/policy` (no + `--no-cov` — D1's root cause). Chain it: `ci: fmt-check lint typecheck test policy audit` (mirrors the template's + distinct `policy` member and honors the gap-#3 forward-sync note). Leave `test` as bare `uv run pytest`: it also + auto-collects `tests/policy` (roadmap §81) — a redundant safety net and keeps `just test` ≡ the CI `test` job's + inline command. The resulting stdlib millisecond re-run inside `just ci` is negligible. *Alternative — + `test: … --ignore=tests/policy` for a clean template-style partition: rejected (YAGNI; the double-run is + microseconds, and matching the CI command + the safety net is worth more).* +- **D4 — No CI / dep / mise / CHANGELOG change.** The CI `test` job runs bare `uv run pytest`, which already + collects `tests/policy` → the gate is enforced in CI with **zero** workflow edit (roadmap §127). No new dep + (stdlib only). CHANGELOG's `[Unreleased]` describes the *template product*; this PR dogfoods the maintainer + surface only (consistent with gaps #1/#2/#3/#5). +- **D5 — `tests/policy/__init__.py` (empty).** The maintainer's `tests/` **is** a package (`tests/__init__.py` + exists); the subpackage mirrors that layout (roadmap §79). Divergence from the template (which ships no + `policy/__init__.py`), forced by the maintainer's package layout. Covered by the `tests/**` per-file-ignores + (D104). +- **D6 — Sweep every `ci` gate-list site (reviewer-enumerated).** Adding `policy` lands half of the gap-#3 + forward-sync note's obligation. Same discipline as the gap-#3 "out-of-band" sweep. The **complete** site list + (the reviewers found the design's original `grep 'test audit'` would have **missed** the `+`-form at + AGENTS.md:8): + - `justfile:11` — the recipe: `ci: fmt-check lint typecheck test audit` → `… test policy audit` (**the + functional change**). The `ci` comment (L6–10) has **no** enumerated gate list to sync (prose names only + `test`/`audit`); optionally mention `policy`. + - `AGENTS.md:8` — **`+`-form fenced comment** `# fmt-check + lint + typecheck + test + audit` → insert + `+ policy` before `+ audit`. (Missed by a `grep 'test audit'`; see the broadened acceptance grep.) + - `AGENTS.md:11` — prose recipe reproduction `ci: fmt-check lint typecheck test audit` → `… test policy audit`. + - `AGENTS.md:17` — the **forward-sync note**: reword prose to "when the property-tests (gap #9) layer lands, add + `fuzz` …" (policy now landed). The **quoted template conditional string** (`test{% if enable_property_tests %} + fuzz{% endif %}{% if enable_policy_tests %} policy{% endif %}…`) stays **verbatim** — it is the template's own + literal and is already correct. + - `AGENTS.md:19` — `ci: fmt-check lint typecheck test audit` inside the "none in the gate list" sentence → + `… test policy audit`. The surrounding "**none in the gate list**" claim **stays true** (the template + conditionally adds `policy` via `{% if enable_policy_tests %}`, which is on) — only the literal changes. + +## Wiring + +### `tests/policy/__init__.py` (new, empty) + +Package marker; empty (D104 ignored under `tests/**`). + +### `tests/policy/test_gates.py` (new, gap #4 + #8) + +Stdlib only (`re`, `tomllib`, `pathlib`, `typing.cast`). `ROOT = Path(__file__).resolve().parent.parent.parent` +(→ maintainer root). Five tests: + +1. `test_type_checking_is_recommended` — `PYPROJECT["tool"]["basedpyright"]["typeCheckingMode"] == "recommended"`. +2. `test_type_checking_fails_on_warnings` — `PYPROJECT["tool"]["basedpyright"]["failOnWarnings"] is True`. **(new; D2)** +3. `test_ruff_select_present` — `"ALL" in cast("list[str]", PYPROJECT["tool"]["ruff"]["lint"]["select"])`. +4. `test_agents_md_recipes_exist_in_justfile` — **fenced-only** (D-REGEX): extract ```-fenced blocks, collect + `just ([a-z][a-z-]*)` within them, assert `referenced <= recipes` where `recipes` = `^([a-z][a-z-]*):` in the + justfile. **Typecheck-passing form (mandatory — the reviewers' first draft failed basedpyright with 3 + `reportAny` warnings):** `blocks = cast("list[str]", re.findall(r"```.*?```", agents, re.DOTALL))` then an + **annotated accumulator** loop `referenced: set[str] = set(); for block in blocks: referenced |= set(re.findall(…))` + — **not** a bare set-comprehension (whose `block` is `Any`, tripping `reportAny` under `recommended` + + `failOnWarnings`). +5. `test_actions_are_sha_pinned` — **verbatim** from the template: `uses:\s*\S+@([0-9a-f]{40})\s+#\s*v\d+(\.\d+){0,2}\b` + over each `.github/workflows/*.yml` line starting `- uses:`/`uses:`. **(gap #8)** + +Written to pass ruff `select=["ALL"]` (S101/PLR2004/D103 already ignored under `tests/**`) and basedpyright +`recommended` + `failOnWarnings` (module docstring, `-> None`, `cast` for the ruff list). + +### `justfile` (gap #4) + +Add immediately after the `test` recipe (mirroring the template's `test` → `policy` → `audit` relative order): + +```just +# Policy gate: config-literal pins so the gates above cannot be silently weakened +# (stdlib-only; also auto-collected by `just test`). Mirrors template/justfile.jinja's `policy`. +policy: + uv run pytest tests/policy +``` + +Change the `ci` recipe: `ci: fmt-check lint typecheck test policy audit` (insert `policy` before `audit`, matching +the template's `test{…} policy{…} audit` order). The `ci` comment (L6–10) has **no** enumerated gate list to keep +in sync (it names only `test`/`audit` in prose) — do not chase a nonexistent enumeration; optionally mention +`policy`. + +### `AGENTS.md` (gap #4) + +- New `## Policy gate (`just policy`)` section documenting: the five assertions; the dropped coverage floor and + the added `failOnWarnings` pin (both keyed to the maintainer's real config); the fenced-only docs-can't-lie + tuning **as a documented divergence**, plus the **two prose invariants** it requires (no literal triple-backtick + run in prose; no bare `just ` for a non-recipe) and its **known limits** (recipe-name shape; CI wiring + unguarded — candidate F); that `test_actions_are_sha_pinned` closes gap #8 and overlaps the zizmor + job (net-new value = the config-literals + docs-can't-lie); `policy` as a `ci` member also auto-collected by + `test`; the scope caveat (config *literals*, not the ruff ignore lists or recipe bodies); no CI/dep change. +- Sweep every `ci` gate-list site per **D6** (justfile:11; AGENTS.md:8 `+`-form, :11, :17 note, :19) — the + complete reviewer-enumerated list is in D6. + +## Documented divergences (mirror-except-where-surface-differs) + +**Surface-forced (the maintainer surface makes the template idiom impossible/invalid):** +- **Coverage-floor assertion dropped** (D1) — no `[tool.coverage]`/no package; nothing to protect. +- **`--no-cov` dropped from the `policy` recipe** (D1 root cause) — no `pytest-cov`; the flag is invalid here + (the template keeps it because the downstream has a global `--cov`). +- **`tests/policy/__init__.py` added** (D5) — the maintainer's `tests/` is a package; the template ships none. +- **`test` stays bare / double-collects** (D3) — the maintainer has no `tests/unit`; bare `test` keeps the CI + command match + a safety net. + +**Unforced (taste — *not* surface-forced; offered for explicit sign-off):** +- **`failOnWarnings is True` assertion added** (D2) — the template ships no such assertion; this strengthens a + real maintainer-only pin (pyproject L36). Keep-recommended. +- **Docs-can't-lie tuned to fenced-only** (D-REGEX) — the naive whole-doc mirror *passes today*; fenced-only is + preventive (breaks-proofs against gap-#9 prose) at the cost of not checking inline-prose references. + Keep-recommended; the strict-mirror alternative is a legitimate choice. It also makes AGENTS.md + **fence-parity-sensitive** (an odd number of triple-backtick runs mispairs the extractor), so it ships with two + prose invariants and a parity guard — see D-REGEX (a)/(b). + +## Acceptance + +- `TMPDIR= just ci` runs `… test policy audit` and ends `ci: all gates passed`; `just policy` alone passes + in milliseconds. +- **Self-referential check — all five assertions pass on the *current* maintainer tree** (empirically proven in + the design-review worktree: `uv run ruff check .` EXIT 0, `uv run basedpyright` "0 errors, 0 warnings", + `uv run pytest tests/policy` → 5 passed): recommended ✓, failOnWarnings ✓, ALL ✓, every fenced `just ` + real ✓, all 12 `uses:` SHA+`# v` pinned ✓. +- **That baseline predates the new AGENTS.md section and does *not* prove the post-edit state** — the section's + first draft in fact **failed** it (D-REGEX (a)). The suite must be re-run **after** the AGENTS.md edit lands + (plan Task 3 Step 7), preceded by a **fence-parity check**: AGENTS.md must hold an **even** number of + triple-backtick runs — 16 today, **18** after the section is inserted (plan Task 3 Step 6). +- `just lint` + `just typecheck` clean on the new `tests/policy/*.py`. +- `just precommit` clean (new files end in `\n`; not `^template/`-excluded). +- Every `ci` gate-list site updated (D6 list). Verify with the **broadened** grep that catches the `+`-form: + `grep -nE 'test ?\+? ?(policy ?\+? ?)?audit' justfile AGENTS.md` shows only `… policy … audit` forms, no stale + `test audit` / `test + audit`. +- No `template/`, `test-template.yml`, `pyproject.toml` (deps), `mise.toml`, `uv.lock`, `tests/test_generation.py`, + or CHANGELOG change. +- CI green (no workflow change; the existing `test` job collects `tests/policy`). + +## Out of scope / tracked follow-up + +- **Candidate F** (CI jobs call `just` recipes) — still deferred (gap #3 D5). +- **Property-tests (gap #9)** — will add `fuzz` to the `ci` list + this section (the forward-sync note's remaining + obligation). +- **`_migrations`-required policy check** (roadmap Group F) — separate maintainer-side guard; not here. +- Dated `docs/superpowers/` snapshots — left frozen (pin-sync convention). + +## Explicit no-touch (guard rails) + +- No file under `template/`; no generation-test change (the layer's downstream assertions already exist). +- No `test-template.yml`, `pyproject.toml` deps, `mise.toml`, `uv.lock`, or CHANGELOG change. +- No inline `# noqa`; no Action SHA bump. +- No change to `scan`/`audit`/`test` recipe **bodies** (only `test`'s appearance in the `ci` list is unchanged; + `test` body stays bare). + +## Files touched (maintainer root only) + +- `tests/policy/__init__.py` (new, empty) +- `tests/policy/test_gates.py` (new) +- `justfile` (add `policy`; `ci` gains `policy`; update `ci` comment) +- `AGENTS.md` (new Policy-gate section; sweep the `ci` gate-list sites + forward-sync note) +- `docs/superpowers/specs/2026-07-07-dogfood-policy-tests-design.md` + + `docs/superpowers/plans/2026-07-07-dogfood-policy-tests.md` (this pair) From edc229f3ef2f829beafa26cad127e28517ad7e7f Mon Sep 17 00:00:00 2001 From: Ashlen Date: Wed, 15 Jul 2026 19:36:23 -0600 Subject: [PATCH 2/4] test(policy): dogfood the gates-can't-be-weakened suite (closes gap #8 sha-pin) --- tests/policy/__init__.py | 0 tests/policy/test_gates.py | 47 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+) create mode 100644 tests/policy/__init__.py create mode 100644 tests/policy/test_gates.py diff --git a/tests/policy/__init__.py b/tests/policy/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/policy/test_gates.py b/tests/policy/test_gates.py new file mode 100644 index 0000000..811466d --- /dev/null +++ b/tests/policy/test_gates.py @@ -0,0 +1,47 @@ +"""Config-literal pins: gates cannot be silently weakened. Stdlib only.""" + +import re +import tomllib +from pathlib import Path +from typing import cast + +ROOT = Path(__file__).resolve().parent.parent.parent +PYPROJECT = tomllib.loads((ROOT / "pyproject.toml").read_text()) + + +def test_type_checking_is_recommended() -> None: + assert PYPROJECT["tool"]["basedpyright"]["typeCheckingMode"] == "recommended" + + +def test_type_checking_fails_on_warnings() -> None: + assert PYPROJECT["tool"]["basedpyright"]["failOnWarnings"] is True + + +def test_ruff_select_present() -> None: + select = cast("list[str]", PYPROJECT["tool"]["ruff"]["lint"]["select"]) + assert "ALL" in select + + +def test_agents_md_recipes_exist_in_justfile() -> None: + """docs-can't-lie: the first `just ` per fenced invocation is a real recipe.""" + justfile = (ROOT / "justfile").read_text() + recipes = set(re.findall(r"^([a-z][a-z-]*):", justfile, re.MULTILINE)) + agents = (ROOT / "AGENTS.md").read_text() + blocks = cast("list[str]", re.findall(r"```.*?```", agents, re.DOTALL)) + referenced: set[str] = set() + for block in blocks: + referenced |= set(re.findall(r"just ([a-z][a-z-]*)", block)) + assert referenced <= recipes, f"AGENTS.md names missing recipes: {referenced - recipes}" + + +def test_actions_are_sha_pinned() -> None: + """Every third-party `uses:` is a 40-char SHA + a v-prefixed version comment.""" + # SHA + a version comment (v[.minor[.patch]]); our pins carry the action's + # exact tag. The maintainer re-verifies the comment matches the SHA at bump time + # via `gh api .../tags`. + pattern = re.compile(r"uses:\s*\S+@([0-9a-f]{40})\s+#\s*v\d+(\.\d+){0,2}\b") + for wf in (ROOT / ".github" / "workflows").glob("*.yml"): + for line in wf.read_text().splitlines(): + stripped = line.strip() + if stripped.startswith(("- uses:", "uses:")): + assert pattern.search(stripped), f"unpinned action in {wf.name}: {stripped}" From fe42060b3473f858a7691f9e2a73f1968e68ee5d Mon Sep 17 00:00:00 2001 From: Ashlen Date: Wed, 15 Jul 2026 19:40:41 -0600 Subject: [PATCH 3/4] feat(ci): dogfood 'just policy' + chain it into 'just ci' --- justfile | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/justfile b/justfile index c7a71d5..3d357c8 100644 --- a/justfile +++ b/justfile @@ -4,11 +4,12 @@ default: @just --list # The complete local gate — every PR-blocking check, reproducible locally. -# Mirrors template/justfile.jinja's `ci` (audit is a member — the template chains it, -# and audit is itself a PR-blocking check here via the CI `scan` job's pip-audit step). +# Mirrors template/justfile.jinja's `ci` (policy + audit are members — the template chains +# both; each is also an independent PR-blocking check here — policy via the CI `test` job's +# pytest collection, audit via the CI `scan` job's pip-audit step). # `test` is the full generation matrix: give it a roomy TMPDIR (the default 4G tmpfs # /tmp can overflow) — see "Run every gate" in AGENTS.md. Scanners stay CI-only (off `ci`). -ci: fmt-check lint typecheck test audit +ci: fmt-check lint typecheck test policy audit @echo "ci: all gates passed" verify: ci @@ -17,6 +18,12 @@ verify: ci test: uv run pytest +# Policy gate: pins the gate config literals (type mode, failOnWarnings, ruff select, Action +# SHAs) so they cannot be silently weakened — not the ignore lists or the recipe bodies. +# Stdlib-only; also auto-collected by `just test`. Mirrors template/justfile.jinja's `policy`. +policy: + uv run pytest tests/policy + # Lint this repo's own tooling lint: uv run ruff check . From 5fa218c5035b2c09329db6b023e74a65dcacebba Mon Sep 17 00:00:00 2001 From: Ashlen Date: Wed, 15 Jul 2026 19:46:30 -0600 Subject: [PATCH 4/4] docs(agents): document 'just policy' + sweep the ci gate-list sites --- AGENTS.md | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b4a1d3e..3e90f77 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,18 +5,18 @@ This repo is a Copier template. `template/` holds the generated project as `.jin ## Run every gate (`just ci`) ```bash -just ci # fmt-check + lint + typecheck + test + audit, then "ci: all gates passed" +just ci # fmt-check + lint + typecheck + test + policy + audit, then "ci: all gates passed" ``` -`just ci` is the complete local gate — `ci: fmt-check lint typecheck test audit` ending `@echo "ci: all gates passed"`, with `verify` a bare alias (`verify: ci`). It **mirrors** `template/justfile.jinja`'s `ci`: the template chains `audit` when `enable_dependency_audit` is on (it is), so the faithful maintainer recipe includes it. `test` is the heavy generation matrix, so `just ci` needs a roomy `TMPDIR` — the default 4G tmpfs `/tmp` can overflow (export e.g. `TMPDIR=/path/to/roomy/dir` first). The fast inner loop is `just fmt-check lint typecheck` (one invocation, no matrix, no network); there is deliberately **no** `check:` recipe (the template ships none). `audit` is non-hermetic (it queries the OSV/PyPI advisory DB), so `just ci` reaches the network exactly as `just audit` does; scanners (`just scan`) deliberately stay **off** `ci` (CI-only), matching the template. +`just ci` is the complete local gate — `ci: fmt-check lint typecheck test policy audit` ending `@echo "ci: all gates passed"`, with `verify` a bare alias (`verify: ci`). It **mirrors** `template/justfile.jinja`'s `ci`: the template chains `policy` and `audit` when `enable_policy_tests`/`enable_dependency_audit` are on (they are), so the faithful maintainer recipe includes both. `test` is the heavy generation matrix, so `just ci` needs a roomy `TMPDIR` — the default 4G tmpfs `/tmp` can overflow (export e.g. `TMPDIR=/path/to/roomy/dir` first). The fast inner loop is `just fmt-check lint typecheck` (one invocation, no matrix, no network); there is deliberately **no** `check:` recipe (the template ships none). `audit` is non-hermetic (it queries the OSV/PyPI advisory DB), so `just ci` reaches the network exactly as `just audit` does; scanners (`just scan`) deliberately stay **off** `ci` (CI-only), matching the template. This maintainer `just ci` is distinct from the **downstream's** `just ci` referenced under "Run the tests": `just test` renders each project in the answer matrix and runs *its* `just ci` — a different, generated recipe. CI does **not** run `just ci`. It keeps parallel per-gate jobs (`test`/`typecheck`/`lint`/`scan`), each inlining its command; only `test` fans out over the OS×Python matrix, and the `scan` job runs `uvx`/`mise exec` (not `uv run`); because every gate is enforced independently (audit via the `scan` job's `pip-audit` step), `just ci` here is the local reproduction of the PR gate, not the CI entry point. Pointing the CI jobs at these recipes (the gap-audit "candidate F") is deferred: the `scan` job aggregates pip-audit + semgrep + gitleaks as three steps that don't map onto the `audit`+`scan` recipe split, so it is a real restructure, not a one-line swap, and the follow-up must reconcile the recipe encoding with the parallel jobs. -**Forward-sync:** when the policy-tests (gap #4) and property-tests (gap #9) layers land, add `policy`/`fuzz` to **both** the `ci` recipe **and** this section, to keep mirroring the template's conditional `ci` (`fmt-check lint typecheck test{% if enable_property_tests %} fuzz{% endif %}{% if enable_policy_tests %} policy{% endif %}{% if enable_dependency_audit %} audit{% endif %}`). +**Forward-sync:** the policy-tests layer (gap #4) has landed — `policy` is now a `ci` member (see "Policy gate"). When the property-tests layer (gap #9) lands, add `fuzz` to **both** the `ci` recipe **and** this section, to keep mirroring the template's conditional `ci` (`fmt-check lint typecheck test{% if enable_property_tests %} fuzz{% endif %}{% if enable_policy_tests %} policy{% endif %}{% if enable_dependency_audit %} audit{% endif %}`). -Deliberate divergences from `template/justfile.jinja`'s `ci`: **none in the gate list** — `ci: fmt-check lint typecheck test audit`, `verify: ci`, and the `@echo` are exact. The divergences are structural: the maintainer's CI never runs `just ci` (parallel per-gate jobs — only `test` matrixed — with inline `uv run`/`uvx`/`mise exec` commands, vs the downstream `ci.yml`'s single `just ci`); there is no `check:` fast split (use `just fmt-check lint typecheck`); and candidate F is deferred. (The `audit` recipe's dropped `--no-dev` is a separate divergence documented under "Dependency audit", not a `ci`-list divergence.) +Deliberate divergences from `template/justfile.jinja`'s `ci`: **none in the gate list** — `ci: fmt-check lint typecheck test policy audit`, `verify: ci`, and the `@echo` are exact. The divergences are structural: the maintainer's CI never runs `just ci` (parallel per-gate jobs — only `test` matrixed — with inline `uv run`/`uvx`/`mise exec` commands, vs the downstream `ci.yml`'s single `just ci`); there is no `check:` fast split (use `just fmt-check lint typecheck`); and candidate F is deferred. (The `audit` recipe's dropped `--no-dev` is a separate divergence documented under "Dependency audit", not a `ci`-list divergence.) ## Run the tests @@ -80,6 +80,22 @@ Deliberate divergences from the template's dependency-audit layer: `--no-dev` is Because nothing here re-derives the pin (no Renovate; the generation drift test reads only the *rendered* downstream), **bump every literal by hand, against the template.** pip-audit (`2.10.1`) has three maintainer sites — the `just audit` recipe, the `pip-audit` step in `test-template.yml`, and the prose above — synced to `template/.github/workflows/…scan.yml….jinja` (the only exact-version template site; the template justfile uses unpinned `uv run pip-audit` and template pyproject floors `pip-audit>=2.10`). No `mise.toml` or `pyproject.toml` pip-audit literal exists (uvx-run, unlike gitleaks). **Sync only the pin *value* — never the export flags:** the template's `uv export` keeps `--no-dev`, but the maintainer must not (it exports 0 packages here — see above), so a mechanical sync against the template would silently neuter the gate. (Mirrors the semgrep/gitleaks pin-sync note and the pre-commit "bump both `rev:` pins together" rule.) +## Policy gate (`just policy`) + +```bash +just policy # pins the gate config literals so they cannot be silently weakened (stdlib-only) +``` + +`just policy` runs `uv run pytest tests/policy` — a stdlib-only (`re`/`tomllib`/`pathlib`) suite pinning this repo's gate **config literals** so they cannot be silently weakened: basedpyright `typeCheckingMode == "recommended"` **and** `failOnWarnings is True`; `"ALL" in ruff select`; the first `just ` of each **fenced** AGENTS.md example is a real justfile recipe (docs-can't-lie); and every third-party `uses:` is `@<40-hex> # v` (`test_actions_are_sha_pinned` — this **closes the offline half of the SHA-pin policy**, gap #8). It is a `ci` member (`ci: … test policy audit`) and is also auto-collected by bare `just test` (`uv run pytest`, `testpaths=["tests"]`), so CI enforces it inside the existing `test` job with **no** workflow change. + +Changing any of those gates (ruff select, type mode, `failOnWarnings`, an Action pin) requires editing the matching test in `tests/policy/test_gates.py` in the **same commit** — that is the point of the suite. **Scope:** it pins those config *literals* — not the ruff `ignore` / `per-file-ignores` lists (each is separately audited; see "Lint & format this repo"), not the recipe *bodies*, and not the CI workflow that runs the gates (candidate F — see "Run every gate"). + +Deliberate divergences from `template/tests/…/test_gates.py.jinja`: the **coverage-floor** assertion is dropped (the maintainer has no `[tool.coverage]` / no runtime package — nothing to protect; the `policy` recipe likewise drops `--no-cov`, which is invalid with no `pytest-cov`); a **`failOnWarnings is True`** assertion is added (the template ships none — the maintainer pins the type-gate's teeth explicitly, so the suite guards that pin); and the **docs-can't-lie regex is scoped to fenced code blocks** rather than the whole file. The template's naive whole-doc scan still passes here *today*, but it flips to a hard fail the moment this file names a template-only recipe (`fuzz`/`mutate`) in prose — which gap #9 will do. Fenced-only gates the examples you are told to **run** and lets prose name any recipe freely. + +**Two invariants this file must hold** (the test reads AGENTS.md as raw text, and the extractor pairs fences *sequentially*): **(a)** never write a literal triple-backtick run in prose — an odd total mispairs every fence after it and sweeps unrelated prose into a "block"; write "fenced code block", never the delimiter. **(b)** Never write a bare `just ` for a non-recipe anywhere, prose included — name it without the `just ` prefix (as `fuzz`/`mutate` are named above). (b) is defence-in-depth: it stops a future (a)-slip from turning into a red gate. The suite also scans `#` comments *inside* fences, so the same rule holds there. + +The SHA-pin sub-check overlaps the zizmor job (the security control), so its net-new value is annotation hygiene; the config-literals + docs-can't-lie are the substantive net-new gate. `tests/policy/__init__.py` exists because the maintainer's `tests/` is a package (the template ships no such file). **Known limits:** the recipe allowlist regex matches only `[a-z][a-z-]*` names, so a future parametrized or underscored recipe cited in a fenced example would false-positive (broaden the pattern in the same commit); and nothing here asserts that CI still *runs* the gates (candidate F). + ## Add a guardrail layer 1. Add an `enable_*` toggle to `copier.yml`.