Skip to content

artifact-audit 2026-07-25: monthly drift pass (6 fixes, 12 flags) - #12

Merged
espi merged 1 commit into
mainfrom
claude/artifact-audit-2026-07-25
Aug 1, 2026
Merged

artifact-audit 2026-07-25: monthly drift pass (6 fixes, 12 flags)#12
espi merged 1 commit into
mainfrom
claude/artifact-audit-2026-07-25

Conversation

@espi

@espi espi commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Artifact drift

First scheduled run of the monthly artifact-audit skill. Four parallel audit agents (skills / templates / guardrails / runbooks+docs) checked every practice artifact against the knowledge base (last verified 2026-07-20). No prior artifact-audit PR was open; no update-knowledge PR was in flight. Per the skill: fixes below are backed by High-confidence KB entries or verified internal cross-references; everything touching a hard stop, protected file, or Medium/unverifiable evidence is flagged, not rewritten.

Fixed

File What was stale Evidence
.claude/skills/new-loop/SKILL.md "Use /loop only for recurring/scheduled work" steered scheduled/unattended jobs to /loop, which is session-scoped (no laptop-closed runs; recurring tasks expire after 7 days). Now states the limits and points laptop-closed scheduling at a Routine (/schedule, alias /routines). Primer §4 /loop + Routines entries — High (code.claude.com/docs/en/routines read directly)
.claude/skills/new-loop/SKILL.md "/goal with a Haiku validator" → "separate validator model (defaults to Haiku)" — Haiku is the default, not fixed. Primer §4 — High
guardrails/README.md Tool-agnostic enforcement section predated Claude Code's native runaway-loop caps (v2.1.212, Jul 17: WebSearch cap 200/session, subagent-spawn cap 200/session, MCP >2 min auto-background). Added as complements to — not replacements for — the three hard stops (none is a dollar ceiling; defaults are far above a sane per-loop cap). Primer §4 / sources.md changelog entry — High
guardrails/README.md Receipts list missing the newest named-company hard cap: Tesla $200/week effective Jul 6, 2026 (third after Uber, Microsoft). sources.md — High (multi-outlet)
guardrails/checklist.md Permissions checklist item didn't warn that auto mode is on by default on Bedrock/Vertex/Foundry since v2.1.207 (disableAutoMode to opt out) — exactly the unintended-autonomy case the item exists to catch. Primer §4 — High (changelog)
templates/loop/loop.md Starter invocation /loop 15m /review-pr 1234 calls a skill that doesn't exist → /review (the bundled PR-review skill). Internal cross-reference check
runbooks/first-loop.md "Durable asset" pointer cited primer §5; the skills-as-durable-asset material is §4. Internal cross-reference check
runbooks/ui-audit-overnight.md Garbled sentence ("Rename FINDINGS.md is already in place."). Internal cross-reference check

Flagged for you (not changed — human decision needed)

Hard-stop / protected areas:

  1. templates/ralph/run.sh — the budget ceiling is a disarmed placeholder by default (highest-severity finding). cost_so_far() returns ${LOOP_COST_USD:-0}, which nothing in the loop updates — so hard stop knowledge: 2026-06-22 update pass (7-day delta from Jun 15) #3 evaluates 0 >= 10 forever unless the user wires a meter. Two decisions: (a) the comment's example claude cost command is not documented anywhere in the KB (documented spend surfaces are /usage, the Rate Limits API, the Analytics Admin API); (b) consider making the script refuse to start when the meter is unwired, per the repo's own "never run an uncapped loop." Related non-hard-stop nit: eval "$SUCCESS_CHECK" evals an env-overridable string.
  2. .claude/skills/artifact-audit/SKILL.md (this skill's own guardrails) — "the native CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION cap bounds the ~4-agent fan-out" overstates the default: it's 200 natively (v2.1.212), no practical bound unless set ≈12 in the Routine env as runbooks/staying-current.md instructs. Suggested wording: "…the CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION cap (set ≈12 in the Routine env — native default is 200) bounds the ~4-agent fan-out."
  3. guardrails/README.md hard-stops section never states that max_turns / max_budget_usd (Agent SDK) and --max-iterations (ralph-wiggum) all default to unlimited — KB-High and load-bearing, but inside the protected three-hard-stops text. (Also max_budgets_usd typo → max_budget_usd.)
  4. templates/ui-audit/PROMPT.md references a RUN.md that doesn't exist in the template dir ("stop at the caps set in RUN.md") — the caps actually live in runbooks/ui-audit-overnight.md's invocation. Flag-only because it defines where the loop's caps live.
  5. runbooks/ui-audit-overnight.md — the recommended interactive lane's budget stop is a morning /usage check, i.e. an alert, not enforcement (the runbook is honest about it and offers run.sh as the hard path).

Possibly-dead or KB-unverifiable claims (re-verify against live docs; candidates for the KB re-verify backlog):

  1. runbooks/babysit-prs.md — the laptop-closed path rests on a "cloud auto-fix-PR feature", a "web CI bar", and /autofix-pr, none of which the KB tracks. The KB-verified mechanism is a Routine with a GitHub-event (pull_request) trigger via /schedule. If /autofix-pr isn't real, this runbook's core path is a dead end — likely needs a rewrite around Routines. (High severity if confirmed.)
  2. Stop commandsCtrl+C for /goal and /cancel-ralph appear in guardrails/checklist.md, runbooks/first-loop.md, and runbooks/ui-audit-overnight.md, but neither is in the KB (only Esc for /loop is verified).
  3. runbooks/staying-current.md — four unbacked claims: /schedule disabled in web sessions; a /schedule update subcommand; a "per-account daily routine-run cap" (load-bearing — both maintenance skills cite that "Budget & caps" section as their budget hard stop); Routine network tiers (Trusted/Full/Custom).
  4. templates/goal/recipes.md — operational details not in the KB (alias list stop/off/reset/cancel, ~4,000-char condition limit, trust-dialog/disableAllHooks prerequisites, headless claude -p "/goal …"). Not contradicted, just unverified.

Behavior-shift notes (v2.1.198, subagents background-by-default + auto-commit/push/draft-PR on worktree finish):

  1. templates/loop/loop.md and runbooks/babysit-prs.md both say "use a worktree agent to fix them" with no note that worktree agents now auto-commit/push/open draft PRs without pausing — decide whether to annotate or constrain.

Descriptive-only observations:

  1. guardrails/README.md intro says Anthropic's billing layer "does not auto-disable" — still true per the KB, but now understates the Jul 2 Claude Enterprise spend controls (alerts/entitlements, Medium-High, not confirmed hard-stop). Human to decide whether/how to acknowledge without softening the rule.
  2. CLAUDE.md (protected) — its Routines note omits that Routines run with no permission prompts (KB-High, safety-relevant).

Clean (no drift found)

loop-guardrails and update-knowledge skills, templates/ralph/PROMPT.md, templates/goal/recipes.md (load-bearing claims), the ui-audit PREFLIGHT/PAGES/FINDINGS files, runbooks/overnight-build.md, README.md, budget.env, and both self-edit-guard files (all sentinel regions, paths, and the 10-line cap verified on disk).

No self-edit: commits in this PR; nothing here touches knowledge/. Human reviews and merges — never merged to main by the routine.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KzGwSojW7vAPQiM6pDggD3


Generated by Claude Code

…guardrail docs, runbooks

Monthly artifact-audit pass against the knowledge base (verified 2026-07-20).
All fixes backed by High-confidence KB entries or verified internal
cross-references; hard-stop and protected-area findings are flagged in the
PR body only, not rewritten.

- new-loop skill: /loop guidance now states session scope, 7-day expiry,
  and points laptop-closed scheduling at Routines (/schedule); /goal
  validator described as 'separate validator model (defaults to Haiku)'
- guardrails/README.md: document native Claude Code runaway-loop caps
  (v2.1.212) as complements to (not replacements for) the three hard
  stops; add Tesla $200/week spend-cap receipt
- guardrails/checklist.md: note auto mode is on by default on
  Bedrock/Vertex/Foundry since v2.1.207 (disableAutoMode)
- templates/loop/loop.md: /review-pr -> /review (skill does not exist)
- runbooks/first-loop.md: fix primer cross-reference (§5 -> §4 Skills)
- runbooks/ui-audit-overnight.md: fix garbled sentence

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KzGwSojW7vAPQiM6pDggD3
@espi
espi merged commit e1ec782 into main Aug 1, 2026
1 check passed
@espi
espi deleted the claude/artifact-audit-2026-07-25 branch August 1, 2026 10:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants