Rebuild the skills catalog from scratch: 53 verified-stale skills become 23 tested ones

[MaxGhenis]

Rebuild the skills catalog from scratch

Why

The command/agent layer (July era: /reform-pipeline, /analyze-policy, /create-dashboard, /implement-structural) is current and actively developed — but the skills layer underneath was written in the March–May era for weaker models, was never executed, and taught a dead stack. A five-way audit of all 53 skills against the live repos found:

• Dead and phantom APIs shipped for months: a policyengine.Simulation(situation=...) API that no longer exists; USHouseholdInput / calculate_household_impact imports that raise ImportError (test-run); the microdf skill's own quickstart crashes with RecursionError, and it documents methods (decile_values(), percentile()) that don't exist. (The audit initially also flagged niceTicks="snap125" as fabricated — that was wrong: the prop is real, landed in recharts 3.8.0 as a PolicyEngine contribution, and the audit had checked a lockfile-pinned 3.7.0 install. A follow-up commit restores the guidance, version-gated.)
• Archived-stack guidance as the canonical recipe: hf://policyengine/policyengine-us-data/states/NY.h5 and districts/*.h5 (repo archived 2026-07-02, artifacts removed) as THE state/district method; enhanced_cps_2024 / enhanced_frs_2023_24 as defaults; Mantine claims about app-v2 (zero @mantine/* deps); black, 79-char, changelog_entry.yaml relics.
• Model-advance obsolescence: whole skills teaching things frontier models know (generic Python, SEO checklists, generic reviewer agents).
• 4-way inconsistent Python API cluster and a 6-way frontend/deploy cluster teaching different, mutually contradictory stacks.

What

53 skills → 23, flat layout (skills/<name>/), every claim verified against live code before being written down (fixes #28, supersedes #17):

• policyengine — the canonical interface: pe.us/uk.calculate_household, population Simulation + ensure_datasets, economic_impact_analysis / calculate_budgetary_impact (including the configure_budgetary_impact_variables ordering gotcha, found by running it), Populace datasets (populace_us_2024, populace_us_2024_acs_local by name, populace_uk_2023 + token), regional analysis (RowFilterStrategy, region_registry, CD/constituency impact functions), the three reform formats, MicroSeries weighting discipline, memory guidance. Absorbs microsimulation / simulation-mechanics / python-client / district-analysis / microdf / analysis.
• policyengine-us / -uk / -canada / -healthcare — domain rewrites on the live API (verified numbers: CTC $2,200 in 2026, the BHC-vs-AHC split household, Scotland's 6 bands vs Wales-on-rUK-schedule, the ACA-PTC-not-in-net-income default).
• policyengine-model-development — one skill + 7 references consolidating the ten pattern skills (the verified gold: bracket paths without .brackets, -.inf first brackets, period ÷12 semantics, adds/add(), defined_for, in-model reform factories with 5-year lookahead, YAML test rules) with 14 source verifications.
• policyengine-data — the Populace stack (architecture, certified releases, the two-defaults nuance, local-area = filter-one-national-dataset, QRF/calibration/L0 concepts, the sparse-untargeted-input gotcha). Absorbs us-data / uk-data / microimpute / microcalibrate / l0 / data-testing.
• policyengine-app / -tools / -design / -api — the real frontend stack (radix + Tailwind v4 + ui-kit 0.4.0 with its actual export surface, verified against node_modules), multizone mechanics, Modal gateway pattern, v1-API-live / v2-alpha status (and a false Supabase claim removed).
• Process skills kept and repaired (writing, standards, research-lookup, user-guide, content, github-agent, plugin-maintenance); Codex workflow trio kept as-is.

Testing, so it can't rot again:

• tests/test_skill_examples.py — CI executes every <!-- verify --> example against the latest released policyengine (new CI job); <!-- verify: slow --> marks population-scale examples (run at authoring time; PE_SKILLS_RUN_SLOW=1 locally). The harness caught a bug in this PR's own flagship example (person-entity default columns) before it shipped.
• tests/test_no_stale_references.py — anti-rot lint banning the exact patterns that killed the old catalog, with <!-- stale-ok --> markers for deliberate history notes.

Wrapper/agents/commands (surgical, not a rewrite):

• Deleted the model-compensation era: generic api/app reviewers, 4 SEO checkers, performance-optimizer, documentation-enricher, integration-agent, dashboard-build-validator, dashboard-overview-updater, design-token-validator (dup), the never-populated lessons stub, the stale agents README, /dashboard-overview.
• Fixed dead refs and stale destinations: implement-structural's nonexistent parameter-architect agent, a contributor-machine-hardcoded lessons path in encode-reform, v1-app post destinations in report-logger/analyze-policy, us-data → populace in calibration agents, changelog_entry.yaml → towncrier in pr-pusher, Cosilico hooks removed from hooks.json.
• Bundle repair: the entire July command family (reform-pipeline, implement-structural, household-calc, prior-scores, prior-analysis, text-to-reform) + model-corroborator/outcome-predictor shipped in no bundle; create-dashboard.yml installed analysis-tools but the command wasn't in it (headless dispatch could not resolve). All wired in; bundles now on flat paths; versions aligned at 4.0.0 with a test enforcing template/bundle alignment.

Website sync: policyengine.org/us/claude-plugin displayed hardcoded stats frozen at 24/21/4/7 (actual before this PR: 53/47/24/9). Paired app-v2 PR makes the stats data-driven from a checked-in pluginStats.json + adds scripts/export_site_stats.py here and a CONTRIBUTING section on keeping them in sync.

Validation

• uv run pytest green (contract tests updated: 8 bundles, flat paths, version alignment; new anti-rot lint; new example harness).
• python3 scripts/build_claude_wrapper.py builds clean; manifest regenerated.
• Fast <!-- verify --> examples all executed locally against policyengine 4.21.0 (pe-us 1.764.6); the slow population example ran end-to-end (CTC $2,200→$3,000: −$31.9B/yr, all federal via calculate_budgetary_impact after economic_impact_analysis).

Follow-ups (filed as issues)

• Collapse the command + Codex-workflow-skill duplication (encode trio) into one source.
• Author a correct pre-obbba baseline preset (README no longer over-claims it exists).
• Upstream: policyengine.py README's UK population quickstart still shows enhanced_frs_2023_24.

[MaxGhenis]

Dual-model consumer trials: Fable 5/5, Sol 5/5.

Beyond the CI example harness (which executes code snippets mechanically), we ran consumer trials: five representative tasks, each given to a fresh Claude Fable subagent and to GPT-5.6 Sol (codex CLI) with the relevant rebuilt skill injected as context, judged against pre-registered criteria written before seeing any output.

Task	Skill under test	Expected behavior	Fable	Sol
Household CTC reform (executed)	policyengine	pe.us.calculate_household, flat reform dict, CTC $2,200→$3,000, net +$800	pass	pass
Population cost methodology	policyengine	ensure_datasets → Simulation(policy=) → economic_impact_analysis before calculate_budgetary_impact, Populace default, sign convention	pass	pass
CA-only SNAP total	policyengine	filter one national dataset (populace_us_2024_acs_local by name + Aggregate filter / RowFilterStrategy), never a per-state file	pass	pass
Bracket parameter path	policyengine-model-development	gov.states.ca.tax.income.rates.single[1].rate, no .brackets	pass	pass
Round chart ticks	policyengine-tools	niceTicks="snap125" (recharts ≥3.8) + domain={["auto","auto"]} pairing	pass	pass

Both models independently reproduced not just the APIs but the gotcha-handling (the configure ordering, the domain pairing, the no-per-state-files rule, the budget sign convention). Both executed runs produced identical, correct numbers.

Scope note: these test content steering (skill in context → correct behavior). Auto-trigger routing from descriptions is exercised separately by the harness's own matching plus the description-limit/token contract tests.

Transcript criteria and outputs preserved in the session workspace; happy to attach if useful.

🤖 Generated with Claude Code