diff --git a/ResearchStudio-Idea/skills/idea_spark/SKILL.md b/ResearchStudio-Idea/skills/idea_spark/SKILL.md index 44cd0b9..a1a4531 100755 --- a/ResearchStudio-Idea/skills/idea_spark/SKILL.md +++ b/ResearchStudio-Idea/skills/idea_spark/SKILL.md @@ -5,547 +5,224 @@ description: Generate ONE reviewer-defensible, implementable research idea — w # Idea Spark Skill -Convert an under-specified research direction into ONE reviewer-defensible Oral-level research proposal — grounded in 1947 ICLR/ICML/NeurIPS papers (2021-2025) — by running a 5-phase workflow that retrieves recent literature, diagnoses the bottleneck, selects + generates a candidate using corpus-derived ideation pattern cards, and runs it through a single quality gauntlet. +Convert an under-specified research direction into ONE reviewer-defensible Oral-level research proposal — grounded in 1947 ICLR/ICML/NeurIPS papers (2021-2025) — via a 5-phase workflow: retrieve recent literature, diagnose the bottleneck, select + generate a candidate using corpus-derived ideation pattern cards, run it through a quality gauntlet, expand into an idea card. -## Design principles - -1. **Innovation Patterns are diagnostic vocabulary judged per-gap, not classification labels OR generative templates.** The 15 induced ideation patterns (reframe-as-solvable-object, assumption-audit-and-pivot, algebraic-equivalence-unification, heterogeneous-decomposition, architectural-operator-substitution, structural-prior-encoding, characterize-limit-then-surpass, self-supervised-signal-engineering, targeted-self-supervised-objective, controlled-diagnostic-design, unify-into-shared-representation, adapt-via-conditioning, generative-process-redesign, decompose-and-delegate, relax-discrete-search-to-continuous) are how the corpus describes productive research moves. Phase 2.1 reads each pattern's **Definition + Operational signature + When to apply** panels and judges per-pattern per-gap whether the pattern's move closes the gap. Phase 2.2 picks ONE sub-pattern under each chosen main pattern by reading its `tactical_pattern` + `Step-by-Step` + `when_to_pick_this_one` + `differentiation_within_parent` panels at generation. The sub-pattern's Step-by-Step is 5 abstract steps distilled from the cluster's [Accept] examples — domain-agnostic structural moves with [Reject]-derived boundaries embedded, written WITHOUT paper-ID citations so the candidate-author applies the abstract pattern to their own gap rather than mimicking specific papers. Treating patterns as generative templates (verbatim recipe execution + siblings_considered + lock-in rules) converged generation toward corpus-validated incremental work; the cognitive-tool framing avoids that. - -2. **Novelty comes from gap-coverage + saturation-aware pattern picking, not from pattern aesthetics.** Phase 2.1 picks 2-4 gaps from `phase1.what_phase_0_did_not_address[]` (1 anchor + 1-3 randomly-sampled siblings + coherence filter — siblings that cannot be coherently closed under the anchor's machinery move to deferred_gaps) and matches each sibling gap to one main pattern by judgment ("does this pattern's operational signature close this gap?") — the anchor gap instead carries 1-3 ranked candidates whose binding is deferred to Phase 2.2 — with saturation-aware preference (avoid both saturated and untested patterns; prefer mid-frequency in lit_table; saturated patterns require novel-angle defense at audit). Multi-gap closure with shape-diverse patterns naturally pulls paper-role coverage (mechanism / measurement / theory / diagnostic) — single-gap closure produces system-architecture sketches. Phase 2.2 enforces substantive novelty via `differentiation_from_lit[].delta` (what we derive/claim/construct/measure that closest_adjacent did not — not "we use a different ideation pattern"). - -3. **Theory + Engineering legs are both required, but signature-agnostic.** Both legs must be non-trivial. Each leg can be theorem OR observable regime OR scaling exponent OR measurement primitive OR architectural property — the audit doesn't prefer one Oral signature (theory + reframing-first) over others (scaling-law, empirical-reveal, surgical-fix, benchmark-validity). All Oral shapes the corpus contains can score 5. - -4. **Mechanism-aware falsification.** Every candidate's `falsification_prediction` is a single paragraph (3-5 sentences) that visibly contains (a) the minimal experiment, (b) which metric moves and in which direction if the candidate works (name the metric + qualitative direction; the experiment establishes the magnitude), and (c) a mechanism distinguisher pivoting on ONE NAMED LOAD-BEARING VARIABLE — the single quantity (e.g., a gradient norm, an information-gain term, a logit divergence, a learned threshold, a representational direction) whose behavior carries the mechanism claim — plus a negative-control intervention on that variable that should drive the DOWNSTREAM OUTCOME METRIC back to baseline. The negative control's predicted effect MUST be the task-outcome metric that defines the mechanism's value (accuracy, regret slope, refusal pass-rate trajectory) — NOT the load-bearing variable's own value or any quantity analytically derived from it (a control of the form "intervene on X → X becomes 0" tests a definition, not a mechanism). A positive control (a stripped-down model using only the load-bearing variable that recovers most of the downstream effect) is recommended when feasible. Without the load-bearing-variable-plus-non-tautological-negative-control structure, "metric moved" remains consistent with calibration improvements / estimator quality / data shifts / many other non-mechanism explanations — and the candidate is the dominant Reject signal in the corpus. `compute_budget` is a separate flat field, **user-relative** (no absolute cap) — Phase 4 feasibility_validation compares to `intake.compute`. **Default `intake.compute = 1×A100 × 3 months ≈ 90 A100-day`** (canonical "single researcher with cloud access" scale) when the user does not state compute; user-supplied intake overrides. Both `falsification_prediction` and `compute_budget` are kill-switch fields: byte-identical preserved across Phase 2.2 → Phase 3.3 (when revise runs) → Phase 4. - -5. **Anti-pattern is empirical negative knowledge — audit-only.** The corpus identifies 3 reject-favored 2-way compositions (audit + auxiliary_signal, audit + invariance, audit + surgical_fix), each with a specific required_mitigation; rates and mitigations live in `references/anti-patterns.md`. **Phase 2 does NOT load anti-patterns.md** — naming reject-prone compositions during generation creates Streisand-effect priors that bias selection. Phase 3.2 audit's `anti_pattern_check` reads anti-patterns.md, detects matching compositions via the SET of `gap_closure[].main_pattern` values, and verifies substantive mitigation delivery (not keyword presence). Failed audit → Phase 3.3 revise rewrites the candidate's affected fields with the corpus-grounded fix. - -6. **Cheap kills first, expensive expansion last.** Phase 3 runs collision retrieval (real, ~30s, no LLM) before audit (single LLM call replacing earlier 4-attacker simulation). Heavy expansion of the candidate into motivation + method_flow + claims + abstract happens only in Phase 4, after the candidate clears the gauntlet. - -7. **Phase 3.2 is judgment, not modification.** The audit reports what corpus evidence triggered which signals; it does NOT auto-revise the candidate. When revision is needed, the audit emits `revision_targets[]` and Phase 3.3 (a separate LLM call) applies them — keeping audit and modification on different surfaces avoids the self-answering bias of cherry-picking attacks one can already answer. +This file is the operational runbook. Design rationale (the 7 design principles, why each contract is shaped this way, removed-check history) lives in [references/design-notes.md](references/design-notes.md) — read it only when modifying or evaluating the skill, never needed to run it. First-time installation lives in [references/setup.md](references/setup.md). ## When to use -- "Give me a research idea in {area} I could pursue." -- "What's the most impactful next step in this direction?" +- "Give me a research idea in {area} I could pursue." / "What's the most impactful next step in this direction?" - "Help me sharpen this vague direction into an Oral-level proposal." -- "What's the bottleneck of this problem?" -- "Run a novelty audit on this idea." +- "What's the bottleneck of this problem?" / "Run a novelty audit on this idea." ## When NOT to use -- Code review, debugging, refactoring (use a coding skill). -- Summarizing one paper. -- Cross-decade survey writing. -- Free-association brainstorming with no research context. -- Engineering integration tasks ("ship this feature in our system"). -- Pure benchmark / dataset construction work — current 15-ideation pattern vocabulary handles benchmark *audit* (controlled_diagnostic_design) but not benchmark *construction*. - -## Setup (before first use) - -The skill's Phase 0 + Phase 3.1 retrieval needs API credentials for 2 of the 4 connectors. Without them the affected connectors are skipped and the orchestrator continues with whichever connectors are available — but it now prints a prominent **CONNECTORS DEGRADED** banner and writes a `.connectors_degraded` marker so a partial run is never mistaken for a full one. - -0. **Set two shell variables once per session** — where this skill is installed, and where run outputs should go. Neither depends on the harness: - ```bash - SKILL_DIR=~/.claude/skills/idea-spark # Claude Code default; Codex CLI: ~/.codex/skills/idea_spark; else wherever this folder lives - RUN_DIR="$PWD/idea_run" && mkdir -p "$RUN_DIR" # ANY absolute directory you want the per-phase outputs in - ``` - `RUN_DIR` is purely an output anchor — the orchestrator only ever sees the absolute `--out` paths you pass, so the variable *name* does not matter (Claude Code sessions can reuse the injected `CLAUDE_PROJECT_DIR` as their `RUN_DIR`). The orchestrator hard-fails early with an actionable message when a path argument contains an unexpanded `$variable`, collapses to filesystem root (empty expansion, e.g. `/phase0`), or is a relative `--out` — instead of a confusing `FileNotFoundError` mid-run. -1. **Install the skill**: `idea-spark` — Phase 0 literature search runs from its bundled connector scripts (no separate sub-skill). -2. **Install Python deps** (cross-platform — macOS & Linux): `python3 -m pip install feedparser openreview-py beautifulsoup4 pymupdf`. Four lean packages (`feedparser`, `openreview-py`, `beautifulsoup4`, `pymupdf`). Skipping this is the most common first-run failure: `arxiv` errors with `package not installed`, and missing `pymupdf`/`beautifulsoup4` silently degrades every full-text fetch to abstract-only. - - **PEP 668 systems** (recent macOS/Homebrew & Ubuntu 23.04+) reject a bare `pip install` with `externally-managed-environment`. Two safe options: - - **venv (recommended):** `python3 -m venv .venv && source .venv/bin/activate && pip install feedparser openreview-py beautifulsoup4 pymupdf` — then launch every phase **from this same activated shell** (see the connector-degradation note below). - - **user install:** `python3 -m pip install --user --break-system-packages feedparser openreview-py beautifulsoup4 pymupdf`. - - **Use the SAME interpreter everywhere.** `check_connectors` and the phase commands must run under the one Python that has these packages. A package installed for `pip3` but launched under a different `python3` (or a background/non-login shell that drops `--user` site-packages) will pass `check_connectors` yet skip `arxiv`/`openreview` at runtime — the run now prints a loud **CONNECTORS DEGRADED** banner and drops a `.connectors_degraded` marker when that happens. - - **Optional deps (only if you want the extras):** PDF compilation of the idea card needs **xelatex** *or* **tectonic** (macOS `brew install --cask mactex-no-gui` or `brew install tectonic`; Ubuntu `sudo apt-get install texlive-xetex` or `cargo install tectonic`). Without either, the `.md`/`.tex` cards are still written and only the PDF is skipped (with a hint). The optional pipeline-diagram image needs the `azure-*` packages; absent, it is skipped silently. -3. **Copy** the env template at the project root: `cp .env.template .env`. -4. **Fill in keys** (priority order — by impact on retrieval quality): - -| Key | Required for | How to get | -|---|---|---| -| `OPENREVIEW_USER` + `OPENREVIEW_PASS` | OpenReview connector (in-review forward signal). Without these, openreview is silently skipped — you lose the 0-6mo in-review window unique to it. | Free signup at https://openreview.net | -| `SEMANTICSCHOLAR_API_KEY` | Semantic Scholar connector at usable rate. Anonymous tier (~100 req/5min) hits 429 on Phase 0 multi-query batches; with key it's stable at 1 req/s. | Free apply at https://www.semanticscholar.org/product/api#api-key-form (≈24h review). Connector still runs anonymously without it but will frequently 429. | -| `OPENALEX_API_KEY` | Optional, premium rate. Polite-pool already works for typical Phase 0 load. | Apply at openalex.org if you exceed polite limits. | - -5. **Verify** (from the SAME shell/venv you will launch phases from): `python3 "$SKILL_DIR/scripts/run.py" check_connectors` — should show ✅ for all 4 connectors AND the two full-text fetch deps (`pymupdf`, `beautifulsoup4`). If you verify in one shell but run phases in another, the package set can differ — keep it one shell. -6. **The orchestrator auto-loads `.env`** at runtime (walks up from skill dir to find `.env`), so you do NOT need to `source .env` in your shell. Shell-set env vars take precedence over `.env` values, so you can override on the fly. - -If a connector shows ❌, it's either missing creds (fix in `.env`) or missing the pip package (the error message tells you which `pip install` to run). If a full-text dep shows ⚠️, run `pip install feedparser openreview-py beautifulsoup4 pymupdf`. - ---- - -## The 5-phase workflow - -``` -Run progress: -- [ ] Phase 0: Literature grounding (in-skill connectors, role-based retrieval) → lit_table.md -- [ ] Phase 0+: Full-text fetch (orchestrator, runs immediately after lit_table.md) → fulltext_cache.json ← MANDATORY; Phase 1 hard-gates on it -- [ ] Phase 1: Bottleneck identification (single LLM call) → bottleneck statement + closest_adjacent -- [ ] Phase 2.1: Gap × main-pattern selection (1 LLM call — judgment-per-pattern + saturation-aware + anchor + random sibling + coherence filter) → selected_gaps[] -- [ ] Phase 2.2: Sub-pattern picking + candidate generation (1 LLM call — read tactical_pattern + Step-by-Step per gap, write 12-flat-field candidate) -- [ ] Phase 3: Quality gauntlet (retrieval + audit + revise-when-needed; Phase 3.3 emits a patch + deterministic merger writes `final_candidate.json`) -- [ ] Phase 4: Skeleton (orchestrator) → fill (1 LLM call, prose-only) → assemble (orchestrator) → render → idea-card markdown + LaTeX -``` - -If your host exposes a task/todo tool (e.g., Claude Code's TodoWrite), seed it with the phases above and mark each one completed as you finish it; otherwise just re-emit this checklist with `[x]` as you progress. - -**Context discipline (REQUIRED — see the "Context discipline" section below for full rules).** Every LLM-driven phase (1 / 2.1 / 2.2 / 3.2 / 3.3 / 4.fill / 4.1.5) must run in a fresh sub-agent (or compacted host context) with file-path inputs only, `Write`-to-disk outputs, and no inline JSON paraphrase. Running these phases inline in the parent context routinely hits the API request timeout once cumulative state exceeds ~150-180k tokens. - -Three outcomes per run: the rendered idea-card markdown returned inline (advance OR revise→3.3 path; LaTeX side artifact + per-phase JSON left under the run directory), a `do_not_generate.md` (Phase 1 OOD), or a `phase_3_failed.md` (audit abandons). The user gets one of these three from one input — no mid-flow clarification. **One-shot guarantee preserved** even when audit triggers revise: Phase 3.3 mechanically applies the revision_targets and Phase 4 proceeds without user re-invocation. - -### How phases run (orchestrator vs. host LLM) - -Two phases need real external retrieval and run via `scripts/run.py`: +- Code review, debugging, refactoring. Summarizing one paper. Cross-decade survey writing. +- Free-association brainstorming with no research context. Engineering integration tasks ("ship this feature in our system"). +- Pure benchmark / dataset construction work — the 15-pattern vocabulary handles benchmark *audit* (controlled_diagnostic_design) but not benchmark *construction*. -### Invocation contract (host LLMs read this first) +## Setup (first use only) -**No `cd` is required.** `scripts/run.py` self-locates its skill root (it inserts the skill directory into `sys.path` at startup), so every orchestrator command can be invoked from ANY working directory by absolute script path: +Follow [references/setup.md](references/setup.md). Quick version — set two shell variables, install deps, verify: ```bash -python3 "$SKILL_DIR/scripts/run.py" --out "$RUN_DIR//" ... +SKILL_DIR=~/.claude/skills/idea-spark # or wherever this folder lives +RUN_DIR="$PWD/idea_run" && mkdir -p "$RUN_DIR" # ANY absolute output dir (Claude Code: reuse CLAUDE_PROJECT_DIR) +python3 -m pip install feedparser openreview-py beautifulsoup4 pymupdf +python3 "$SKILL_DIR/scripts/run.py" check_connectors # from the SAME shell you'll run phases from ``` -The legacy form `cd "$SKILL_DIR" && python3 -m scripts.run ...` still works identically. Do NOT use relative script or `--out` paths — CWD is not stable across host-LLM Bash invocations, and the orchestrator rejects a relative `--out` outright. - -**Exit codes 10 and 11 are NOT errors — they are sentinel handshakes.** When the orchestrator can't call an LLM itself (no `NOVELTY_LLM_CLASSIFY_FAST_CMD` env var), it writes a sentinel JSON file describing what the host LLM should do next, then exits with rc=10 (intent / pattern-summary) or rc=11 (signature_terms). The host LLM: - -1. `cat $RUN_DIR//._pending` to read the sentinel -2. Read the file at the sentinel's `rubric_file` field (an absolute path) -3. Produce the expected output per the rubric -4. Re-invoke per the sentinel's `re_invocation` field - -If the host LLM treats rc=10/11 as failure and stops, the run stalls. Do not stop on these codes — continue per the sentinel. - -### Context discipline (host LLMs read this BEFORE running any LLM-driven phase) - -A full Idea-Spark run accumulates ~180-250k tokens of intermediate state (lit_table, fulltext_cache, per-phase JSONs, audit reports). If the host LLM carries that state in its own conversation context across phases, the Phase 1 / Phase 2.2 / Phase 4.fill calls — each of which produces a multi-kilobyte structured JSON on top of an already-large prompt — routinely hit the backend request timeout and surface as `[API Error · Request timed out · Retrying...]` to the user. The retry runs against the same context and tends to time out again, producing a stuck run with zero artifact output. Three rules together prevent this; apply all three on every run, not "if the run feels heavy": - -**Rule 1 — Run every LLM-driven phase in an ISOLATED context.** Phase 1 / 2.1 / 2.2 / 3.2 / 3.3 / 4.fill / 4.1.5 are independent JSON-producing steps with well-defined inputs (a system prompt + 1-3 disk artifacts) and a well-defined output (one JSON written to disk), so no phase needs the conversation that produced an earlier one. Three interchangeable isolation mechanisms — use the FIRST one your harness supports: - -- **(a) Subprocess LLM — harness-agnostic, the skill's native mode.** Set `NOVELTY_LLM_REASONING_LARGE_CMD` (and `NOVELTY_LLM_CLASSIFY_FAST_CMD`) to any CLI that reads a `<>...<>` prompt on stdin and emits JSON on stdout (see § Configuration). The orchestrator then runs each LLM-driven phase as its own subprocess — a fresh context per phase by construction, on any harness (Codex CLI, cron, plain shells). -- **(b) Sub-agent tool — Claude Code and harnesses with an equivalent.** Spawn an `Agent` per phase, passing ONLY the file paths the phase prompt lists at its top — not the conversation history, not the lit_table contents inline, not prior phase outputs as prose. The sub-agent reads from disk, writes back to disk, and returns ≤ 250 words confirming the output path + the routing/verdict signal the parent needs. -- **(c) Manual context reset — any interactive harness with neither (a) nor (b).** Run the phase inline, but clear/compact the conversation at the four compact points named in Rule 3 before starting the next phase. Every phase re-reads its inputs from disk, so clearing loses nothing; what it costs is discipline, not information. - -Whichever mechanism you use, the parent context stays ≤ ~30k tokens for the whole 5-phase run because it never holds a phase's structured output in its own turns. - -**Rule 2 — `Write` every phase artifact directly to disk; never paraphrase it into chat.** The output schema for each LLM-driven phase is fixed (see the `Output:` section of the matching `references/system-prompts/.txt`) and the convention is `$RUN_DIR//_output.json`. Use the `Write` tool with that exact path; do NOT `cat < file` (a Bash heredoc with a multi-KB JSON triggers permission prompts and can be silently truncated), do NOT `echo` the JSON, and crucially do NOT paste the JSON into your reply for the parent to read — `Write` to disk and report the path. Downstream phases re-read from that path. Tool-result captures from large extraction commands (e.g. printing every paper's abstract for inspection) should also go through `head -c 4000` / `jq` / `sed` to bound the captured payload to ≤ 4 KB; never `Read` a >10 KB intermediate dump back into the prompt — that was the specific anti-pattern that pushed prior runs into timeout (the dump itself is small, but `Read`ing it caches it into every subsequent turn). - -**Rule 3 — Compact between phases.** Each phase's output is persisted under `$RUN_DIR//`, so the conversation that produced it carries no information the next phase needs. The natural compact points are: **after Phase 0+** (drops lit_table + fulltext exploration), **after Phase 1** (drops bottleneck reasoning), **after Phase 2.2** (drops sub-pattern reading), **after Phase 3.2** (drops audit reasoning). Each phase re-reads its disk inputs and proceeds. If your harness exposes `/compact`, invoke it at those four points; otherwise the same effect is achieved by Rule 1 alone (each sub-agent is already a fresh context). - -**Diagnostic if you see "Request timed out" mid-phase.** Inspect your harness's session transcript/log (Claude Code: `~/.claude/projects//.jsonl`, look for `isApiErrorMessage: true`; other harnesses: their session-log equivalent). The context just above the error (the prior tool call and its result) tells you which prompt got too big to inference inside the request budget. The fix is always one of the three rules above — usually Rule 1: re-issue the timed-out step in an isolated context with only the file paths it needs. - -### Phase entry points - -| Phase | Entry point (`python3 "$SKILL_DIR/scripts/run.py" ...`, any CWD) | Why orchestrator | -|---|---|---| -| Phase 0 (literature grounding) | `python3 "$SKILL_DIR/scripts/run.py" phase0 --query "..." --out $RUN_DIR/phase0/` | probes 4 connectors (arxiv, openalex, semanticscholar, openreview), runs role-based retrieval, dedups; auto-loads `.env` for OPENREVIEW_USER/PASS + SEMANTICSCHOLAR_API_KEY | -| Phase 0+ (full-text fetch — **mandatory**, run right after `lit_table.md` is written) | `python3 "$SKILL_DIR/scripts/run.py" phase0_fulltext --out $RUN_DIR/phase0/` | caps the on-topic pool to the most relevant ~15 (+U user refs), fetches intro+method concurrently into `fulltext_cache.json`; Phase 1 hard-gates on this output | -| Phase 3.1 (collision check) | `python3 "$SKILL_DIR/scripts/run.py" phase3_collision --idea-json --out $RUN_DIR/phase3_collision/` | re-uses all 4 connectors with the candidate's signature_terms | -| Phase 3.3 merger (after the LLM emits the revise patch) | `python3 "$SKILL_DIR/scripts/run.py" phase3_merge_revisions --phase2 --revisions --out $RUN_DIR/phase3_revise/` | applies the LLM's `applied_revisions[]` patch deterministically; refuses kill-switch writes; writes `final_candidate.json`; back-injects `final_candidate` into the patch file so the legacy `kill_switch_integrity` chain-check still works | -| Phase 4 skeleton (runs BEFORE the Phase 4 LLM call) | `python3 "$SKILL_DIR/scripts/run.py" phase4_skeleton --candidate --phase1 ... --phase2-select ... --phase3-critique ... [--phase3-revise ...] --phase0-dir $RUN_DIR/phase0/ [--collision ...] --out $RUN_DIR/phase4/` | populates every mechanical field of the expansion (kill-switch echoes, venue_year lookups, lit_table group-by, candidate_uses, reviewer_concerns lifts, compute verdict); leaves prose fields as `` placeholders for the LLM to author | -| Phase 4 assembler (runs AFTER the Phase 4 LLM call) | `python3 "$SKILL_DIR/scripts/run.py" phase4_assemble --skeleton $RUN_DIR/phase4/phase4_skeleton.json --fill-map $RUN_DIR/phase4/fill_map.json --out $RUN_DIR/phase4/` | merges the LLM's flat `{path: value}` fill_map into the skeleton; refuses any fill_map key whose root is `falsification_prediction` or `compute_budget`; writes `phase4_expansion.json` | -| Phase 4.render (idea-card rendering) | `python3 "$SKILL_DIR/scripts/run.py" phase4_render --expansion $RUN_DIR/phase4/phase4_expansion.json --out $RUN_DIR/phase4/` | templating only — writes `idea.std.{en,zh}.md` + `idea.detail.en.md` (returned inline) + `idea.std.{en,zh}.tex` side artifacts, and auto-compiles `.pdf` when `xelatex`/`tectonic` is on PATH (skipped with a hint otherwise) | -| Validators | `python3 "$SKILL_DIR/scripts/run.py" validate ...` | static contract checks | - -The remaining phases — **Phase 1 bottleneck identification, Phase 2.1 ideation pattern selection, Phase 2.2 candidate generation, Phase 3.2 critique, Phase 3.3 revise (patch-only — the merger then turns it into `final_candidate.json`), Phase 4.fill (prose-only, on top of the skeleton), Phase 4.1.5 implementability audit** — are LLM-driven and run *manually* by the host LLM (or user) reading the corresponding system prompt under `references/system-prompts/`, providing the listed inputs, and writing the JSON output to the conventional `$RUN_DIR//...` location. There is no orchestrator subcommand for these because adding one would just be a thin `cat + | llm` wrapper — no validation work happens between input assembly and the LLM call. Wrapping it in Bash would add fragility (env vars, CLI shape, JSON post-processing) without buying determinism. - -**Each of these phases MUST be run under the "Context discipline" rules above** — fresh sub-agent, `Write`-to-disk output, no inline JSON paraphrase. Phase 4.fill is the largest single output and the most timeout-prone; do not run it in the parent context. - -The convention each manual phase follows: read the prompt at `references/system-prompts/.txt`, gather inputs listed at the top of the prompt, produce the JSON described under `Output:`, save it to `$RUN_DIR//_output.json`. Downstream phases read that filename. - -### CRITICAL: Literature grounding mode - -Phase 0 and Phase 3.1 collision require **real external retrieval** via the in-skill connector scripts (`scripts/search_*.py`, bundled in this skill). Two states (simplified from earlier 4-state design): `lit_grounding_mode = real` (any connector worked, including webfallback with per-paper retrieved_via tagging) vs `connector_failure` (no connector, no fallback flag — orchestrator halts with diagnostic). Without at least one working connector, the skill halts cleanly rather than degrading silently. - -Install the skill in your harness's skill directory (Claude Code: `idea-spark`; Codex CLI / others: clone or copy this folder anywhere and point `SKILL_DIR` at it); Phase 0 retrieval runs from its bundled connector scripts (no separate sub-skill to install). +Credentials go in `.env` (OpenReview user/pass + Semantic Scholar key — see setup.md); the orchestrator auto-loads it. Optional: `xelatex`/`tectonic` for PDF cards. --- -### Phase 0 — Literature Grounding +## How to run: the `next` loop -Phase 0 runs via a single Bash command — the orchestrator at `scripts/run.py`. This physically narrows tool choice to one path; alternative paths (WebSearch, ad-hoc fetch) produce unstructured output that downstream phases reject. +The canonical way to drive a run is the **run-state navigator**: ```bash -python3 "$SKILL_DIR/scripts/run.py" phase0 --query "" --out $RUN_DIR/phase0/ +python3 "$SKILL_DIR/scripts/run.py" next --dir "$RUN_DIR" --query "" ``` -**What the orchestrator does internally**: - -1. Asserts system clock is sane (≥ 2024-01, ≤ 2027-01). -2. Intent extraction: turn the user's free-text query into 4-6 search queries (including one ESCAPE-MECHANISM query phrased in solution vocabulary, which recalls papers that already fixed the bottleneck and title themselves by their fix — problem-keyed queries miss exactly those). Without an external LLM CLI configured, emits `.intent_extraction_pending` sentinel and exits — the host LLM produces queries per `references/intent-recognition.md`, then re-invokes with `--queries "q1|q2|q3"`. -3. Probes 4 connectors (arXiv, OpenAlex, Semantic Scholar, OpenReview) and reports availability. Auto-loads `.env` for OPENREVIEW_USER/PASS + SEMANTICSCHOLAR_API_KEY. -4. **Role-based retrieval** (each connector used where it's most informative): +`next` inspects the artifacts already on disk and prints EXACTLY one next step — either a Bash command to run verbatim, or an LLM sub-agent spec (system-prompt path + input file paths + output path + the routing signal to report back). It is read-only and idempotent (safe to re-run anytime, including to resume an interrupted run). The host loop is: - | Connector | Window | Cap | Role | - |---|---|---|---| - | arxiv | 0-6 mo | 10 | preprint pool — recent active work (sortBy=relevance) | - | openalex | 6-24 mo | 12 | published proceedings + journals (`--published-only`); broad academic graph | - | semanticscholar | 6-24 mo | 13 | published CS-focused; returns TLDR (Allen-AI 1-sentence summary) + ArXiv/DOI cross-IDs in one record | - | openreview | 0-6 mo | 10 | in-review submissions (forward signal); venues runtime-derived; `get_notes(limit=500, sort='cdate:desc', mintcdate=since)` for fast retrieval (~7s/query); 600s connector timeout | +1. Run `next`. +2. Do what it says (`bash` → run the command; `llm_subagent` → execute in an ISOLATED context per the Context discipline rules below). +3. Run `next` again. Repeat until it reports a terminal state (`DONE`, `do_not_generate`, or `phase_3_failed`). - Target: ~40-45 papers. Gracefully degrades when a connector is unavailable. Windows are non-overlapping (0-6mo arxiv vs 6-24mo openalex+SS vs 0-6mo openreview), so a paper that's both a recent preprint and an in-review submission doesn't double-count via cross-source dedup on (title_norm, externalIds). -5. Dedups across sources with file-order priority (semanticscholar > openalex > openreview > arxiv) — SS first because its `externalIds` (DOI + ArXiv + DBLP keys) makes it the highest-quality cross-source anchor. -6. **pattern_summary** (LLM step) tags each paper with ideation pattern + bottleneck + open_issue + retrieved_via, producing `lit_table.md`. Without an external LLM CLI, emits `.pattern_summary_pending` sentinel for the host LLM to fill. -7. Writes one gate sentinel: `.lit_grounding_mode = real`. -8. **User-reference extraction** (regex on query string at phase0 entry): scans the user query for arxiv URLs / arxiv IDs (`arxiv:2401.12345`) / OpenReview URLs / DOIs and writes them to `$RUN_DIR/phase0/user_refs.json`. The intent-extraction sentinel also asks the host LLM to append paper-title references (e.g., "based on the LoRA paper") to the same file. These become the **U tier** of the full-text fetch pool used in step 9. +`next` encodes the full phase graph — the mandatory full-text gate, the citation gate, the abandon-retry branch, the falsification re-audit branch, and the correct Phase 4 flags per path — so you do not need to memorize the reference tables below; they exist for deviation and debugging. -9. **Full-text fetch for the candidate pool** — **MANDATORY, not optional**. This is a separate orchestrator subcommand, but it is *bound to the moment `lit_table.md` is written*: the instant step 6's `lit_table.md` lands on disk, run this command before touching Phase 1. It is its own subcommand (not folded into `phase0`) only because it depends on the host-LLM-produced `lit_table.md` to know which papers are on-topic — that dependency is why it cannot run inside the same `phase0` Bash call. Phase 1 **hard-gates** on `fulltext_cache.json` (stops with `error: fulltext_not_fetched` if it is missing), so skipping this step halts the pipeline rather than silently degrading to abstract-only reasoning — which was the previous failure mode. +If your host exposes a task/todo tool (e.g., TodoWrite), seed it with this checklist and tick phases as `next` moves past them: - ```bash - python3 "$SKILL_DIR/scripts/run.py" phase0_fulltext --out $RUN_DIR/phase0/ - ``` - - This selects a small candidate pool — **U** (user refs from `user_refs.json`, always included, never capped) + **T2** (papers from OpenAlex/Semantic Scholar where lit_table tag ≠ `outside_taxonomy`, up to `--t2-top` = 10, method-first with cross-source round-robin) + **T3** (arxiv on-topic papers, method-first, up to `--t3-top` = 5) — with a hard ceiling of `--max-pool` = 15 total fetches **excluding** user refs, so the pool stays at the most relevant ~15 (+U) papers rather than the full on-topic set. Ordering is **method-first**: papers whose only innovation tag is `controlled_diagnostic_design` (eval/benchmark-only, low fulltext value) sink below method-bearing papers, and within each tier sources are interleaved round-robin so a single high-`DEDUP_PRIORITY` source (e.g. Semantic Scholar) cannot crowd out OpenAlex; same-paper duplicates retrieved from two sources are collapsed via `title_norm`. All papers are fetched **concurrently** (ThreadPoolExecutor, ≤15 workers) with a per-paper budget (`pdf_timeout` 30s, `per_paper_budget_s` 75s) so a few slow/unreachable PDFs cannot stall the whole step. Because the fetch is concurrent, this budget caps the **wall-clock** at roughly the slowest single paper rather than the sum, so it is set generously to avoid dropping fetchable content. It fetches intro + method sections for each. The HTML path (`arxiv.org/html/`) is tried first (works for ~85% of 2024+ ML preprints); for papers without HTML or non-arxiv sources, the PDF is downloaded and parsed via pymupdf. Section extraction targets headings Introduction / Method / Methodology / Approach / Model Architecture / Main Results (positional fallback handles theory papers where the canonical "Method" name is absent). Limitations is intentionally not extracted — author-written limitation paragraphs are often weaker than what the audit synthesizes from method + experiments. - - Output: `$RUN_DIR/phase0/fulltext_cache.json` — keyed by paper_id, each entry `{tier, intro, method, source_used, warning}`. Fetch failures degrade gracefully to abstract + warning; the pipeline never halts on a fetch error. Phase 1 reads the full cache when writing `bottleneck_statement` + `closest_adjacent[]`; Phase 2.2 reads only the closest_adjacent entries when writing `differentiation_from_lit[].delta` and `core_mechanism`'s quantitative claims (which must cross-check against any disagreeing values in the cache). - -**Host-LLM handshake** (when `NOVELTY_LLM_CLASSIFY_FAST_CMD` is unset — typical when running inside a host LLM): the orchestrator emits sentinel files in a common schema rather than silently substituting model knowledge. Three sentinel sites in Phase 0 + 3.1: - -| Sentinel | Trigger | Host LLM action | -|---|---|---| -| `.intent_extraction_pending` | rc=10, no `--queries` and no LLM env | Read `references/intent-recognition.md` (Map mode), produce queries, re-invoke `phase0 --queries "q1\|q2\|q3"` | -| `.pattern_summary_pending` | informational | Read `references/pattern-summary-rubric.md`, classify each paper into 1-3 of the 15 ideation patterns, write lit_table.md | -| `.signature_extraction_pending` | rc=11 in Phase 3.1 | Read `references/intent-recognition.md` (Collision mode), produce 3-5 signature_terms, re-invoke phase3_collision | - -The sentinel JSON itself carries the absolute `rubric_file` path that the orchestrator wrote, so the host LLM does not need to guess where the rubric lives — it should read from the sentinel's `rubric_file` field directly. The relative paths in this table are documentation hints; the canonical path is whatever the sentinel records. - -`lit_table.md` schema (consumed by Phase 1): - -```markdown -| paper_id | year_month | venue | title | ideation pattern tags | bottleneck this paper targets | open issue / unresolved gap | resolves_problem | retrieved_via | ``` - ---- - -### Phase 1 — Bottleneck Identification - -Single LLM call. Use [references/system-prompts/bottleneck_identify.txt](references/system-prompts/bottleneck_identify.txt). - -Phase 1 does one substantive thing: read user query + lit_table.md + intake, write a literature-grounded bottleneck statement plus the routing decision. - -**Inputs**: user query, intake context, `$RUN_DIR/phase0/lit_table.md`, `$RUN_DIR/phase0/fulltext_cache.json` (intro+method for the candidate pool — read BEFORE writing bottleneck/closest_adjacent; the entry assertion **hard-gates** on this file: missing → stop `fulltext_not_fetched`, all-failed → continue with `fulltext_degraded: true` + abstract-level residue confidence), `$RUN_DIR/phase0/lit_results.json` (for abstract-level grounding when needed). - -**Output schema**: see `bottleneck_identify.txt`. Key fields: -- `intake` (with `_inferred_fields[]` listing fields not stated by user) -- `bottleneck_statement` — one paragraph citing ≥ 2 paper_id from lit_table inline -- `closest_adjacent[]` — list of `{paper_id, summary_and_residue}` -- `what_phase_0_did_not_address[]` -- `state ∈ {proceed, do_not_generate}` - -**No-ask guarantee**: missing intake fields are inferred from user query + Phase 0 retrieval; if hopelessly missing, route to `do_not_generate` with concrete remedial_steps rather than asking. - -**Routing**: -- **proceed** — bottleneck is literature-groundable AND no OOD trigger fires -- **do_not_generate** — OOD (too-broad direction / no-anchor) OR lit_table too sparse (< 5 truly-relevant papers) OR genuinely blank-space (no adjacent literature) OR benchmark/system construction (current vocab doesn't cover) → emit `do_not_generate.md` with redirect - ---- - -### Phase 2 — Idea Generation (2 LLM calls) - -#### Step 2.1 — Gap × Main-Pattern Selection - -Use [references/system-prompts/ideate_select.txt](references/system-prompts/ideate_select.txt). - -**Inputs**: -- `$RUN_DIR/phase1/phase1_output.json` — `what_phase_0_did_not_address[]` is the load-bearing field (2-4 collective gaps no retrieved paper closes); `bottleneck_statement` + `closest_adjacent[]` + `intake` for context. -- `references/ideation-patterns/overview.md` — read each of the 15 patterns' **Definition + Operational signature + When to apply** panels. Selection at WHAT/WHEN level, not HOW. -- `$RUN_DIR/phase0/lit_table.md` — to compute pattern frequency for saturation-aware selection. - -**Selection process**: -1. **Pick anchor gap** — the single most important gap from `what_phase_0_did_not_address[]`. -2. **Sample 1-3 sibling gaps** randomly + apply coherence filter (siblings that cannot be coherently closed under anchor's machinery → deferred_gaps). -3. **For each selected gap**, judge each of the 15 patterns directly: does this pattern's move, applied to this gap, actually close it? Saturation is recorded (joined from Phase 1's `domain_pattern_distribution`) for audit transparency, NOT used as a selection filter (saturated ≥50% / untested ≤1 paper / mid_frequency between). Saturated/untested choices require novel-angle defense at Phase 3.2 audit. - -**Output**: `selected_gaps[]` (each entry: `gap` verbatim from phase1 + `chosen_pattern_id` + `selection_rationale`; index 0 is anchor, rest are siblings) + `coherence_thread_type` + top-level `pattern_saturation` dict (keyed by pattern_id) + `deferred_gaps[]`. - -#### Step 2.2 — Sub-Pattern Picking + Candidate Generation - -Use [references/system-prompts/ideate_generate.txt](references/system-prompts/ideate_generate.txt). - -**Inputs**: -- `$RUN_DIR/phase2_select/phase2_select_output.json` — the gap × pattern spec. -- `$RUN_DIR/phase1/phase1_output.json` — bottleneck + closest_adjacent + intake. -- `$RUN_DIR/phase0/lit_results.json` — abstracts of closest_adjacent for substantive comparison. -- `references/ideation-sub-patterns/.md` — for each picked sub-pattern, read **`tactical_pattern` + `Step-by-Step` + `when_to_pick_this_one` + `differentiation_within_parent`**. The Step-by-Step is your tactical recipe (5 abstract structural-move steps; not paper-mimicry). - -**Sub-step a — pick sub-pattern under each gap's main pattern**: open `ideation-sub-patterns/overview.md` to find candidates per parent; compare `when_to_pick_this_one + differentiation_within_parent` panels; pick ONE per gap; then read picked sub-pattern's `tactical_pattern + Step-by-Step`. - -**Sub-step b — write candidate**: apply each picked sub-pattern's Step-by-Step to its specific gap; write candidate JSON. - -**Output**: ONE candidate with flat fields (0 nesting): -- `title` / `hook` / `core_mechanism` / `core_mechanism_reasoning` / `core_mechanism_steps` -- `gap_closure[]` — per-gap entry mirrors `selected_gaps[]` one-for-one: `{gap, main_pattern, sub_pattern, how_closed}`. `sub_pattern` is emitted as `C## (parent pattern name)` — e.g. `C12 (Substitute the Operator or Representation)` — so the opaque code is always spelled out; consumers that open the card file strip to the leading `C##`. -- `falsification_prediction` (single paragraph: minimal experiment + metrics-that-move; mechanism distinguisher optional) -- `compute_budget` (user-relative, concrete number) — kill-switch with `falsification_prediction` -- `differentiation_from_lit[]` ({paper_id, substantive delta}) -- `almost_prior_paper_id` + `what_step_was_missed` (single closest paper + substantive missed step) -- `signature_terms[]` (Phase 3.1 collision retrieval keys) - -**Two hard rules** (the rest is in schema descriptions): -1. **Substantive > methodological** in `differentiation_from_lit[].delta` and `what_step_was_missed`. -2. **`falsification_prediction` names the experiment + metric that moves (qualitative direction) + a mechanism distinguisher that pivots on ONE named load-bearing variable with a negative-control intervention that should drive the effect back to baseline if the variable is the mechanism** — the experiment establishes magnitude; the load-bearing variable + intervention is what makes the prediction Popper-testable rather than consistent with "calibration improved" / "estimator quality" alternatives. - -**Why 2 stages with judgment, not lock-in.** A lock-in alternative would demand verbatim quotes from cards + enforced siblings_considered + sub-pattern recipe execution + many hard rules; cumulatively that converges generation toward corpus-validated incremental work and kitchen-sink mechanism stacks. The judgment-based 2-stage design (Phase 2.1 = judgment per pattern at Operational-signature level; Phase 2.2 = sub-pattern as descriptive vocabulary not recipe) produces paper-shape candidates while preserving audit anchors via `gap_closure[].main_pattern + sub_pattern`. - -**K=1, not K=2/3.** Single candidate goes through critique. (Earlier K=3/K=2 design had no auto-selection downstream — overhead without quality win.) - -**Citation gate (deterministic, MANDATORY before Phase 3).** The instant the candidate JSON is written, run the `subpattern_citation_consistency` validator. It is a hard gate: a fabricated `gap_closure[].sub_pattern` (a hallucinated parent slug, a C## whose real parent differs from the cited `main_pattern`, or an invented parenthetical name) must be caught here, before any retrieval / audit / expansion work is spent on it. - -```bash -python3 "$SKILL_DIR/scripts/run.py" validate --phase2 $RUN_DIR/phase2_generate/phase2_generate_output.json +- [ ] Phase 0: Literature grounding → lit_table.md, then Phase 0+ full-text fetch (MANDATORY — Phase 1 hard-gates on it) +- [ ] Phase 1: Bottleneck identification → phase1_output.json (routing: proceed | do_not_generate) +- [ ] Phase 2: Gap×pattern selection + candidate generation (ONE isolated context, TWO output files) → citation gate +- [ ] Phase 3: Collision retrieval (signature@10mo + alias@48mo) → audit (5 checks) → [revise → merge → re-audit if falsification rewritten] | [abandon → 1 internal retry] +- [ ] Phase 4: skeleton → fill → assemble → implementability audit → validate → render → return 3 cards inline ``` -If it reports any `fail`, the citation was written from the parent pattern's gist rather than read from `overview.md`. Do NOT proceed to Phase 3. Re-open `references/ideation-sub-patterns/overview.md`, fix the `main_pattern` / `sub_pattern` to a real cluster row (or regenerate Step 2.2 with the card actually open), and re-run the gate until it passes. This guard proves only parent-consistency; whether `core_mechanism` performs the cluster's actual tactic is judged later by Phase 3.2's `recipe_application_check`. - ---- - -### Phase 3 — Quality Gauntlet (1 retrieval + 1-2 LLM calls) - -#### Step 3.1 — Mechanism-specific collision retrieval +Three outcomes per run: the rendered idea-card markdown returned inline (LaTeX + per-phase JSON left under `$RUN_DIR`), a `do_not_generate.md` (Phase 1 OOD), or a `phase_3_failed.md` (audit abandons twice). **Never ask the user mid-flow** — missing intake fields are inferred; revision, falsification re-audit, and the single internal retry all run without user re-invocation. -Run via the orchestrator: +### Invocation contract -```bash -python3 "$SKILL_DIR/scripts/run.py" phase3_collision --idea-json $RUN_DIR/phase2_generate/phase2_generate_output.json --out $RUN_DIR/phase3_collision/ -``` +**No `cd` is required.** `scripts/run.py` self-locates its skill root, so every orchestrator command can be invoked from ANY working directory by absolute script path: `python3 "$SKILL_DIR/scripts/run.py" --out "$RUN_DIR//" ...`. The legacy form `cd "$SKILL_DIR" && python3 -m scripts.run ...` works identically. Do NOT use relative script or `--out` paths — CWD is not stable across host-LLM Bash invocations, and the orchestrator rejects a relative `--out` outright. -Orchestrator probes all 4 connectors (arXiv / OpenAlex / Semantic Scholar / OpenReview) and runs each available one with a 6-month window using the candidate's `signature_terms[]`, dedups across sources, writes `collision_hits.json`. +**Exit codes 10 and 11 are NOT errors — they are sentinel handshakes.** When the orchestrator can't call an LLM itself (no `NOVELTY_LLM_CLASSIFY_FAST_CMD`), it writes a sentinel JSON describing what the host LLM should do, then exits rc=10 (intent / pattern-summary) or rc=11 (signature_terms). Read the sentinel (`$RUN_DIR//._pending`), read the file at its `rubric_file` field (absolute path), produce the expected output, re-invoke per its `re_invocation` field. Do not stop on these codes. (The default Phase 0 flow below avoids the rc=10 intent sentinel entirely by passing `--queries` up front.) -If candidate lacks `signature_terms[]`, orchestrator emits `.signature_extraction_pending` sentinel — host LLM reads the path in the sentinel's `rubric_file` field (resolves to `references/intent-recognition.md`, Collision mode rubric), produces 3-5 terms, edits the candidate JSON, re-invokes. +### Context discipline (read BEFORE running any LLM-driven phase) -3.1's sole job: **expand the paper pool that 3.2 will search** with mechanism-specific recent retrieval that Phase 0's broad-domain queries miss. No classification step here — Phase 3.2 does substantive subsumption judgment over `lit_table.md ∪ collision_hits.json`. +A full run accumulates ~180-250k tokens of intermediate state. If the host LLM carries that in its own conversation context across phases, the Phase 1 / 2.2 / 4.fill calls routinely hit the backend request timeout (`[API Error · Request timed out · Retrying...]`) and the retry times out again. Apply ALL three rules on every run: -#### Step 3.2 — Audit-and-Verdict (4 corpus-anchored checks) +**Rule 1 — Run every LLM-driven phase in an ISOLATED context.** Phases 1 / 2 (2.1+2.2) / 3.2 / 3.3 / 4.fill / 4.1.5 each have file-path inputs and one JSON output; no phase needs the conversation that produced an earlier one. Use the FIRST isolation mechanism your harness supports: -Single LLM call. Use [references/system-prompts/critique.txt](references/system-prompts/critique.txt). +- **(a) Subprocess LLM** — set `NOVELTY_LLM_REASONING_LARGE_CMD` / `NOVELTY_LLM_CLASSIFY_FAST_CMD` (see § Configuration); each phase runs as its own subprocess, fresh context by construction, on any harness. +- **(b) Sub-agent tool** (Claude Code `Agent` or equivalent) — spawn one per phase, passing ONLY the file paths the phase prompt lists — not conversation history, not file contents inline. The sub-agent reads from disk, `Write`s to disk, returns ≤ 250 words (output path + routing signal). Exception by design: Phase 2.1 and 2.2 run in ONE sub-agent writing both output files — both are generation-side; the adversarial separations (3.2 vs 3.3, 4.fill vs 4.1.5) must stay separate calls. +- **(c) Manual context reset** — run inline but clear/compact at the four points in Rule 3. -Phase 3.2 produces an audit report with four corpus-anchored checks. It does **NOT** auto-revise the candidate — when revision is needed, Phase 3.2 emits `revision_targets[]` and Phase 3.3 (separate LLM call) applies them. Audit and modification on different surfaces avoids self-answering bias. +Whichever mechanism, the parent context stays ≤ ~30k tokens for the whole run because it never holds a phase's structured output. -**Precondition (deterministic, runs before the LLM call):** the `subpattern_citation_consistency` validator must pass on the Phase 2.2 candidate (see the Phase 2.2 citation gate). It confirms each `gap_closure[].sub_pattern` resolves to a real C## cluster under its cited `main_pattern` parent. It proves only parent-consistency, not that the cluster card was read — in this taxonomy the `sub_pattern` string carries the PARENT display name, so a clean citation cannot prove the C##.md card was opened. That harder question is `recipe_application_check` below. +**Rule 2 — `Write` every phase artifact directly to disk; never paraphrase it into chat.** Output convention: `$RUN_DIR//_output.json`. Use the `Write` tool — no Bash heredocs (permission prompts + silent truncation), no `echo`, no pasting JSON into replies. Bound tool-result captures from large files to ≤ 4 KB (`head -c 4000` / `jq` / `sed`); never `Read` a >10 KB intermediate dump into the parent context — the dump gets cached into every subsequent turn (this exact anti-pattern caused prior timeout runs). -The four checks each anchor on specific corpus content the LLM cannot fabricate: - -| Step | Corpus anchor | Question | -|---|---|---| -| **1. gap_closure_reject_check** | each `gap_closure[]` entry's sub-pattern card (`ideation-sub-patterns/.md`, where `` is the leading cluster code of the entry's `sub_pattern` value `C## (parent pattern name)` — `## Tactical failure mode` + ALL bullets under `### Reject lessons`). Total reads = number of gap_closure entries (typically 1-3 cards). Other ~28 sub-pattern cards NOT loaded. | For each gap_closure entry, does the candidate fall into the Reject patterns documented in that sub-pattern card? Aggregate verdict is the worst across all entries. | -| **2. recipe_application_check** | each `gap_closure[]` entry's sub-pattern card `## Tactical pattern` (the cluster's signature move, NOT the parent's gist) + the candidate's `core_mechanism`. | Does `core_mechanism` actually perform the cited C## cluster's signature operation, or only the parent pattern's generic idea? `bypassed` when the distinctive move is absent — the leading cause of incremental output, and the one failure the deterministic citation guard cannot catch (the citation string only names the parent). | -| **3. anti_pattern_check** | `references/anti-patterns.md` — 3 reject-favored compositions with required mitigations | Detect via the SET of `gap_closure[].main_pattern` values. If composition matches an anti-pattern, is the mitigation **substantively delivered** in core_mechanism / theoretical_leg (not keyword-present)? | -| **4. paper_pointed_threat** | `lit_table.md ∪ collision_hits.json` (unified pool) | Most specific paper subsuming or competing with the candidate's claim. `no_threat_found` is a valid clearance signal — fabricating a generic threat is forbidden. | +**Rule 3 — Compact between phases.** Natural compact points: after Phase 0+, after Phase 1, after Phase 2, after Phase 3.2. Every phase re-reads its disk inputs, so compacting loses nothing. With `/compact`, use it there; Rule 1 mechanisms (a)/(b) achieve the same on their own. -(Earlier design had a check `almost_prior_factcheck`. Removed: low fire rate, redundant with paper_pointed_threat. A `saturation_defense_check` was also tried and removed: it was the one purely advisory check — soft signal only, never a hard floor, and unlike the four above it consulted no retrieved related work — so its concern now folds into Phase 4's `reviewer_concerns_and_responses` instead of gating Phase 3.2. Saturation metadata still flows: Phase 1 computes the band, Phase 2.1 records it, Phase 4 echoes it into `domain_landscape`. `recipe_application_check` is the newest — added because the deterministic citation guard can only prove parent-consistency, so a recipe built from the parent's gist while citing a real cluster passes the guard yet bypasses the cluster's actual tactic; this check is the semantic backstop.) +**Diagnostic for "Request timed out" mid-phase:** inspect your harness's session log (Claude Code: `~/.claude/projects//.jsonl`, look for `isApiErrorMessage: true`); the prior tool call shows which prompt got too big. The fix is one of the three rules — usually Rule 1. -**Verdict is two-layer**: hard floor (mechanical, LLM cannot override) + soft judgment (LLM weighs within safe zone): +--- -- **Layer 1 hard floor** — `abandon` if any of: gap_closure_reject_check=triggered (documented Reject pattern matches) / anti_pattern unmitigated-and-uninsertable / exact-mechanism collision. These are corpus-anchored facts; LLM has no override authority. -- **Layer 2 soft judgment** — when hard floor didn't fire, LLM picks `advance` or `revise` by weighing how severe each borderline/partial finding is. Trivial borderlines (non-load-bearing fields) → advance with concern surfaced for Phase 4 to fold into reviewer_concerns_and_responses. Borderlines hitting load-bearing structural properties (e.g., an ideation pattern's success condition) → revise with concrete revision_targets[]. `recipe_application_check = bypassed` → revise: either swap `sub_pattern` to the sibling whose tactical move core_mechanism actually performs, or rework core_mechanism to instantiate the cited move (if no sibling under the parent fits and core_mechanism cannot be reshaped to the cited move, the gap-level mismatch routes back to regenerate Phase 2.1+2.2 — Phase 3.3 cannot change the parent). LLM can also demote clear→revise if holistic reading reveals a concern individual checks missed. +## Phase reference -The `verdict_rationale` must cite specific check findings (lesson_quoted / failure_mode_quoted / sub-block verdict). "All checks pass" without naming which is a process error. +`next` prints each of these steps at the right moment with concrete paths; the tables below are the full contract for deviation/debugging. -Why two-layer: pure mechanical aggregation over-triggers revise (treats 1 trivial borderline same as 3 severe). Pure LLM verdict introduces agreeable-bias and loses audit trail. Hard floor preserves non-negotiable corpus facts; soft layer uses context to distinguish "must fix" from "noted concern, advance". +### Orchestrator entry points -Routing on verdict: -- **advance** → Phase 4 reads Phase 2.2 candidate directly. -- **revise** → Phase 3.3 (single LLM call) emits the `applied_revisions[]` patch → orchestrator merger writes `final_candidate.json` → Phase 4 skeleton reads it. -- **abandon** → orchestrator emits `phase_3_failed.md` with verdict_rationale + triggering check. No automatic retry. +| Phase | Entry point (`python3 "$SKILL_DIR/scripts/run.py" ...`, any CWD) | +|---|---| +| navigator | `next --dir "$RUN_DIR" [--query "..."]` | +| Phase 0 | `phase0 --query "" --queries "q1\|q2\|q3\|q4" --out $RUN_DIR/phase0/` | +| user-ref registration (title-named anchor papers; BEFORE phase0_fulltext) | `add_user_ref --out $RUN_DIR/phase0/ --title "" [--raw-match ""] [--id ]` | +| Phase 0+ full-text (**mandatory**, the moment lit_table.md lands) | `phase0_fulltext --out $RUN_DIR/phase0/` | +| Phase 1 anchor top-up (optional, when the #1 closest_adjacent fell outside the fulltext pool) | `phase1_fulltext_topup --out $RUN_DIR/phase0/ --paper-id ` | +| Phase 3.1 collision | `phase3_collision --idea-json $RUN_DIR/phase2_generate/phase2_generate_output.json --out $RUN_DIR/phase3_collision/` | +| Phase 3.3 merger | `phase3_merge_revisions --phase2 --revisions --critique --out $RUN_DIR/phase3_revise/` | +| Phase 4 skeleton | `phase4_skeleton --candidate --phase1 ... --phase2-select ... --phase3-critique ... [--phase3-revise ...] --phase0-dir $RUN_DIR/phase0/ [--collision ...] --out $RUN_DIR/phase4/` | +| Phase 4 assemble | `phase4_assemble --skeleton $RUN_DIR/phase4/phase4_skeleton.json --fill-map $RUN_DIR/phase4/fill_map.json --out $RUN_DIR/phase4/` | +| Phase 4 render | `phase4_render --expansion $RUN_DIR/phase4/phase4_expansion.json --out $RUN_DIR/phase4/` | +| Validators | `validate --phase2 ... [--phase3 ...] [--phase4 ...] [--phase4-impl ...]` | -#### Step 3.3 — Apply Revision Targets (only when 3.2 verdict = revise) +The LLM-driven phases (1 / 2.1 / 2.2 / 3.2 / 3.3 / 4.fill / 4.1.5 / falsification re-audit) have no orchestrator subcommand (a `cat prompt | llm` wrapper would add fragility without determinism): read the prompt at `references/system-prompts/.txt`, gather the inputs listed at its top, `Write` the JSON described under `Output:` to `$RUN_DIR//_output.json`. Run each under the Context discipline rules — Phase 4.fill is the largest output and the most timeout-prone; never in the parent context. -Phase 3.3 is **patch-only**: one LLM call emits the `applied_revisions[]` patch list, then a deterministic Python merger (`scripts/merge_revisions.py`) applies the patch against the Phase 2.2 candidate and writes `final_candidate.json`. The LLM does NOT echo the full candidate back — previous versions of this contract did, and a single ~25k-token candidate echo caused a real backend inference timeout (the kill-switch fields, the largest, were re-typed verbatim). +### Phase 0 — Literature grounding -LLM step: use [references/system-prompts/revise.txt](references/system-prompts/revise.txt). Reads Phase 2.2 candidate + Phase 3.2's revision_targets[]; emits one patch entry per revision_target. Does not re-judge the verdict, does not propose new attacks. The split (audit in 3.2, revise in 3.3, separate LLM calls) ensures the LLM that proposes attacks is not the LLM that answers them — eliminating self-answering bias. +Phase 0 and 3.1 require **real external retrieval** via the bundled connector scripts (`scripts/search_*.py`) — never WebSearch or ad-hoc fetch (downstream phases reject unstructured output). Gate sentinel: `.lit_grounding_mode` = `real` vs `connector_failure` (halt with diagnostic; `--allow-webfallback` exists as a flagged, lower-confidence escape). -Merger step (mandatory, runs immediately after the LLM call): +**Default flow (skips one sentinel round-trip):** BEFORE invoking `phase0`, read `references/intent-recognition.md` (Map mode) yourself and produce 4-6 search queries — including one ESCAPE-MECHANISM query phrased in solution vocabulary (recalls papers that already fixed the bottleneck and title themselves by their fix; problem-keyed queries miss exactly those). Also apply the OOD short-circuit (intake-routing.md triggers #1 Too-broad / #2 No-anchor → route to do_not_generate instead of retrieving). Then invoke with BOTH flags: ```bash -python3 "$SKILL_DIR/scripts/run.py" phase3_merge_revisions \ - --phase2 $RUN_DIR/phase2_generate/phase2_generate_output.json \ - --revisions $RUN_DIR/phase3_revise/phase3_revise_output.json \ - --out $RUN_DIR/phase3_revise/ +python3 "$SKILL_DIR/scripts/run.py" phase0 --query "" --queries "q1|q2|q3|q4" --out $RUN_DIR/phase0/ ``` -The merger writes `phase3_revise/final_candidate.json` (Phase 4's canonical input) and back-injects `final_candidate` into the patch file so the `kill_switch_integrity` validator's existing chain-check still works without modification. - -**Patch op vocabulary** (each `applied_revisions[]` entry uses one): -- `replace` — overwrite the field with `value` (full replacement; for a top-level string, a list, or a dict) -- `append_sentence` — append " " + `value` to an existing string field (preserves prior content; cheap) -- `append_items` — extend an existing list field with `value` (must itself be a list) -- `swap_sub_pattern` — for scope=sub_pattern: identify a gap_closure entry by `field` = the verbatim gap text, replace its `sub_pattern` with `value`; sibling `how_closed` / `core_mechanism` re-alignment is emitted as additional `replace` / `append_sentence` patch entries - -**Output schema** (`$RUN_DIR/phase3_revise/phase3_revise_output.json`): -```json -{ - "candidate_id": "...", - "applied_revisions": [ - {"scope": "tactical | sub_pattern", - "op": "replace | append_sentence | append_items | swap_sub_pattern", - "field": "", - "value": "", - "outcome": "applied | skipped_already_satisfied | skipped_anti_substitution | skipped_inapplicable", - "delta_summary": ""} - ] -} -``` -The merger writes a `final_candidate` key back into this file after running. +The rc=10 sentinel path still exists as fallback when `--queries` is omitted. The orchestrator: asserts a sane clock; probes 4 connectors and retrieves role-based (arxiv 0-6mo cap 10 / openalex 6-24mo cap 12 published-only / semanticscholar 6-24mo cap 13 published-only / openreview 0-6mo cap 10 in-review; ~40-45 papers; non-overlapping windows; SS-priority dedup); extracts URL/ID user-refs from the query into `phase0/user_refs.json`; emits `.pattern_summary_pending` for the host. -**Two scopes** (`revision_targets[].scope`): -- `tactical` — modify named candidate fields (e.g., `core_mechanism`, `differentiation_from_lit[2].delta`); gap_closure[] unchanged. -- `sub_pattern` — swap one `gap_closure[i].sub_pattern` to a sibling under the same parent; re-emit `how_closed`; re-align `core_mechanism` only where the new sub-pattern's tactical_pattern makes the previous wording mechanism-misaligned. +Retrieval takes 3-10 min (the openreview connector alone budgets 600s) — set your Bash timeout ≥ 600s or run it in the background. -**No `composition` scope.** If audit findings imply gap-level changes, the audit produces `verdict = abandon` and the user re-runs Phase 2.1+2.2 with a different random seed. +**Pattern tagging (host step):** classify each `lit_results.json` paper per `references/pattern-summary-rubric.md` into 1-3 of the 15 patterns → write `lit_table.md` with columns `paper_id | year_month | venue | title | ideation pattern tags | bottleneck this paper targets | open issue / unresolved gap | resolves_problem | retrieved_via`. Pure classification — delegate to a CHEAP/FAST isolated context (Haiku-class model or effort=low), not the large reasoning model. -**Hard rules** (enforced structurally by the merger): -- Kill-switch fields (`falsification_prediction` + `compute_budget`) are STRUCTURALLY off-limits — the merger refuses any patch entry whose `field` root is one of these and raises with an actionable error. The anti-substitution contract is no longer "the LLM must remember not to drift" but "the LLM physically cannot write the field". -- One patch entry per revision_target (including skipped ones). -- Out-of-scope rewrites → `outcome = skipped_inapplicable`; gap-level changes route back to "regenerate Phase 2.1+2.2". +**Title-named user refs:** if the user query names anchor papers by TITLE ("based on the LoRA paper" — anything the URL/ID regex can't catch), register each BEFORE `phase0_fulltext` via `add_user_ref` (entry-point table). It does a deterministic dedup-merge into `user_refs.json` — do NOT hand-edit that file (the Write tool requires a prior Read on existing files, and a malformed edit silently drops the U fetch tier). -**Anti-substitution chain**: kill_switch_integrity validator handles both routings: -- 3.2=advance, no 3.3: Phase 2.2 → Phase 4 directly (Phase 3 passthrough) -- 3.2=revise, 3.3 ran: Phase 2.2 → Phase 3.3 final_candidate → Phase 4 (3-link chain). All three byte-identical for kill-switch fields. +**Phase 0+ full-text fetch — MANDATORY.** The instant `lit_table.md` lands, run `phase0_fulltext` (entry-point table) before touching Phase 1. Pool = U (user refs, never capped) + T2 (top-10 published on-topic) + T3 (top-5 arxiv on-topic), ceiling 15 excluding U, method-first ordering, concurrent fetch (HTML path first, pymupdf PDF fallback; per-paper budget so one slow PDF can't stall the step). Output `fulltext_cache.json` keyed by paper_id (`{tier, intro, method, source_used, warning}`); fetch failures degrade to abstract + warning. Phase 1 **hard-gates** on this file (`error: fulltext_not_fetched`). ---- +### Phase 1 — Bottleneck identification -### Phase 4 — Expansion + Packaging +One isolated LLM call. Prompt: [references/system-prompts/bottleneck_identify.txt](references/system-prompts/bottleneck_identify.txt). Inputs: user query + intake, `phase0/lit_table.md`, `phase0/fulltext_cache.json` (all-failed cache → continue with `fulltext_degraded: true`, abstract-level residue confidence), `phase0/lit_results.json`. Output `phase1/phase1_output.json`: `intake` (+`_inferred_fields[]` — missing fields are inferred, never asked), `bottleneck_statement` (≥2 paper_id cited inline), `closest_adjacent[]` (`{paper_id, summary_and_residue}`), `what_phase_0_did_not_address[]`, `state ∈ {proceed, do_not_generate}`. -Phase 4 runs in **three steps**: a deterministic skeleton builder (orchestrator), a small LLM fill call, and a deterministic assembler (orchestrator). This split exists because Phase 4's full expansion JSON has ~30 top-level fields and ~half of them are mechanical transforms — kill-switch echoes, venue_year lookups, group-bys over `lit_table.md`, joins of `gap_closure × pattern_saturation`, reviewer-concern lifts from the audit report. Asking the LLM to re-type those wastes tokens and risks a backend inference timeout (the same shape that broke Phase 3.3 before the patch-only redesign). +Routing: **proceed** (literature-groundable, no OOD trigger) or **do_not_generate** (too-broad / no-anchor OOD, <5 truly-relevant papers, genuinely blank space, or benchmark/system construction) → write `do_not_generate.md` with concrete remedial steps — terminal. -#### Step 4.skeleton — Build the deterministic skeleton (orchestrator) +### Phase 2 — Selection + generation (ONE isolated context, TWO outputs) -```bash -python3 "$SKILL_DIR/scripts/run.py" phase4_skeleton \ - --candidate $RUN_DIR/phase3_revise/final_candidate.json \ - --phase1 $RUN_DIR/phase1/phase1_output.json \ - --phase2-select $RUN_DIR/phase2_select/phase2_select_output.json \ - --phase3-critique $RUN_DIR/phase3_critique/phase3_critique_output.json \ - --phase3-revise $RUN_DIR/phase3_revise/phase3_revise_output.json \ - --phase0-dir $RUN_DIR/phase0/ \ - --collision $RUN_DIR/phase3_collision/collision_hits.json \ - --out $RUN_DIR/phase4/ -``` +Run 2.1 and 2.2 back-to-back in one isolated context, writing BOTH output files (they are both generation-side; only adversarial pairs need separate calls): -Pass `--candidate $RUN_DIR/phase2_generate/phase2_generate_output.json` on the advance path (Phase 3.3 did not run); omit `--phase3-revise` in that case. - -The skeleton writes `phase4_skeleton.json` with every mechanical field fully populated and every prose field marked ``. Mechanically populated fields: -- `falsification_prediction`, `compute_budget` (byte-identical from the candidate) -- `differentiation_from_lit` (enriched with `venue_year` per paper) -- `almost_prior_paper_id` + `almost_prior_venue_year` -- `motivation.why_prior_stopped[].paper_id` + `.venue_year` (one entry per closest_adjacent) -- `domain_landscape.pattern_distribution` (from Phase 1 `domain_pattern_distribution`) -- `domain_landscape.candidate_uses` (joined from `gap_closure[].main_pattern` × `pattern_saturation`) -- `literature_breakdown` (grouped from `lit_table.md`) -- `reviewer_concerns_and_responses[].attack` + `severity` + `fields_changed_to_address` (lifted from Phase 3.2 audit + Phase 3.3 patch's `applied_revisions[]`) -- `feasibility_validation.compute.{verdict, rationale}` (bucketed against `intake.compute`) - -#### Step 4.fill — Author the prose (single LLM call) - -Use [references/system-prompts/expand.txt](references/system-prompts/expand.txt). The LLM reads `phase4_skeleton.json`, finds every `` placeholder, and outputs ONE flat JSON whose keys are the placeholder paths and whose values are the prose to substitute. The LLM does NOT touch any non-TODO field; the assembler refuses any fill_map key whose root is `falsification_prediction` or `compute_budget`. - -Output path: `$RUN_DIR/phase4/fill_map.json`. Schema: -```json -{ - "abstract_draft": "...", - "motivation.problem_framing": "...", - "motivation.why_prior_stopped[0].what_they_did": "...", - "method_flow.steps": [ {"step_id": "S1", "title": "...", ...}, ... ], - "feasibility_validation.data.verdict": "feasible", - ... -} -``` +**2.1** — prompt [references/system-prompts/ideate_select.txt](references/system-prompts/ideate_select.txt); inputs `phase1_output.json`, `references/ideation-patterns/overview.md` (all 15 patterns' Definition / Operational signature / When to apply — selection at WHAT/WHEN level), `references/ideation-patterns/companion-combos.md`, `lit_table.md`. Pick the anchor gap (type-bound to `intake.contribution_type`), sample 1-3 sibling gaps randomly + coherence-filter (non-cohering siblings → `deferred_gaps[]`), judge each pattern per gap; record saturation (transparency, not a filter). Output `phase2_select/phase2_select_output.json`: `selected_gaps[]` (index 0 = anchor) + `coherence_thread_type` + `pattern_saturation` + `deferred_gaps[]`. **Retry mode:** when `$RUN_DIR/attempt_1/` exists, the prompt's OPTIONAL retry input applies — the archived audit + selection become negative constraints. -LLM payload drops from ~30 fields (~20k tokens) to ~12 prose-only fields (~8k tokens). +**2.2** — prompt [references/system-prompts/ideate_generate.txt](references/system-prompts/ideate_generate.txt); inputs 2.1 output, `phase1_output.json`, `lit_results.json`, plus for each gap ONE picked sub-pattern card from `references/ideation-sub-patterns/` (compare `when_to_pick_this_one` + `differentiation_within_parent` via its overview.md; then read the picked card's `tactical_pattern` + Step-by-Step). Output `phase2_generate/phase2_generate_output.json` — ONE candidate, 12 flat fields: `title` / `hook` / `core_mechanism` / `core_mechanism_reasoning` / `core_mechanism_steps`; `gap_closure[]` (`{gap, main_pattern, sub_pattern: "C## (parent pattern name)", how_closed}`, mirrors selected_gaps one-for-one); `falsification_prediction` (single paragraph: minimal experiment + metric-with-direction + ONE named load-bearing variable + negative control on that variable predicting the DOWNSTREAM outcome metric returns to baseline — non-tautological); `compute_budget` (user-relative; default intake = 1×A100 × 3 months ≈ 90 A100-day); `differentiation_from_lit[]` (substantive deltas, not "different pattern"); `almost_prior_paper_id` + `what_step_was_missed`; `signature_terms[]` (own vocabulary — recent collision channel); `alias_terms[]` (other communities' names for the same mechanism, from parametric knowledge — multi-year alias collision channel). Both kill-switch fields (`falsification_prediction`, `compute_budget`) are locked from here on — see Phase 3 for the single audited exception. -#### Step 4.assemble — Merge fill into expansion (orchestrator) +**Citation gate (deterministic, MANDATORY before Phase 3):** ```bash -python3 "$SKILL_DIR/scripts/run.py" phase4_assemble \ - --skeleton $RUN_DIR/phase4/phase4_skeleton.json \ - --fill-map $RUN_DIR/phase4/fill_map.json \ - --out $RUN_DIR/phase4/ +python3 "$SKILL_DIR/scripts/run.py" validate --phase2 $RUN_DIR/phase2_generate/phase2_generate_output.json ``` -Produces `phase4_expansion.json` (the canonical input to `phase4_render`). The assembler validates: each fill_map path resolves to a real TODO in the skeleton; no fill_map path targets a kill-switch root. It warns when any `` placeholder remains un-filled — the `expansion_completeness` validator will reject the run otherwise. - -**Anti-substitution is structural**, not enforced post-hoc: the skeleton populates kill-switch fields byte-identically from the candidate, the LLM never authors them, and the assembler refuses any attempt to overwrite them. `kill_switch_integrity` remains a belt-and-suspenders post-hoc validator. - -**Echo vs reference policy**: anti-substitution-guarded fields (`falsification_prediction`, `compute_budget`, `differentiation_from_lit`, `almost_prior_paper_id`, `what_step_was_missed`) and structural lookups (venue_years, `domain_landscape`, `literature_breakdown`, `reviewer_concerns_and_responses.attack/severity/fields_changed`) are filled by the skeleton. `closest_adjacent` from Phase 1 and `lit_grounding_mode` from the Phase 0 sentinel are rendered directly by the card template; not duplicated into Phase 4 output. +Any `fail` = a `sub_pattern` citation was guessed from the parent's gist, not read from `overview.md`. Fix against `references/ideation-sub-patterns/overview.md` (or regenerate 2.2 with the card open) and re-run until clean — the gate proves parent-consistency only; whether core_mechanism performs the cluster's actual tactic is Phase 3.2's `recipe_application_check`. (`next` runs this gate automatically.) -**No calendar projections.** Sequencing in dependencies, not weeks. **No experiment matrix / ablation plan / baseline table / expected figures.** Skill produces IDEA + falsifiability + feasibility judgment; experimental engineering is the user's responsibility. +### Phase 3 — Quality gauntlet -#### Step 4.1.5 — Implementability audit (default on) +**3.1 collision (orchestrator, no LLM):** entry-point table. TWO retrieval channels over all 4 connectors, merged into `collision_hits.json` with a per-hit `collision_channel` tag: **signature** — the candidate's `signature_terms[]` over a 10-month window (contemporaneous scoop risk); **alias** — the candidate's `alias_terms[]` (other communities' names for the same mechanism, produced from parametric knowledge at 2.2) over a 48-month window (renamed-ancestor risk — the "goal-conditioned success detector vs goal-image conditioned scorer" blind spot is lexical, not temporal, so widening the signature window alone cannot catch it). Missing `signature_terms[]` → rc=11 sentinel: produce BOTH term sets per intent-recognition.md Collision mode (terms 3-7 words each — long sentences break URL encoding), edit the candidate JSON, re-invoke. Missing only `alias_terms[]` → loud warning, alias channel skipped (add the field and re-run to close the blind spot). -Single LLM call, run by default after 4.1 and before 4.2. Use [references/system-prompts/implementability_audit.txt](references/system-prompts/implementability_audit.txt). +**3.2 audit (one isolated LLM call):** prompt [references/system-prompts/critique.txt](references/system-prompts/critique.txt); inputs: candidate, 2.1 spec, `lit_table.md`, `collision_hits.json`, `references/anti-patterns.md`, and each cited sub-pattern card `references/ideation-sub-patterns/.md` (strip the leading code from `sub_pattern`; typically 1-3 cards, others NOT loaded). Five corpus-anchored checks: -**Why this step exists.** Phase 4.1's `method_flow.steps[]` are often too terse to *understand* — a step names an operation ("extract premises", "score consensus", "train a critic") without the concrete object it runs on (the unit, the estimator, the output schema, how a quantity is computed), so the method "reads" but cannot be built from. This step adopts a fresh, skeptical implementing-engineer persona (a separate call from the 4.1 author — same anti-self-answering rationale as Phase 3.2's principle 7) and rewrites each step into a specification an engineer could code from, recording every hole it filled or left open. +| Check | Question | +|---|---| +| gap_closure_reject_check | does the candidate match a documented Reject lesson in each cited sub-pattern card (`## Tactical failure mode` + ALL `### Reject lessons` bullets)? | +| recipe_application_check | does `core_mechanism` actually perform the cited C## cluster's `## Tactical pattern` signature move, or only the parent's generic idea (`bypassed` — the leading cause of incremental output)? | +| anti_pattern_check | if the SET of `gap_closure[].main_pattern` matches a reject-favored composition, is the required mitigation substantively delivered (artifact, not keyword)? | +| paper_pointed_threat | most specific subsuming/competing paper in `lit_table ∪ collision_hits` (both channels; alias-channel threats are NOT discounted for age); `no_threat_found` is valid — fabricating a generic threat is forbidden. Side output `parametric_family_concern`: a named un-retrieved mechanism family from parametric knowledge (family name + query vocabulary, never specific paper cites) — soft signal only, flows to Phase 4 reviewer_concerns as a "scoop-check X first" flag | +| falsification_structure_check | does `falsification_prediction` name the minimal experiment, the outcome metric + direction, ONE load-bearing variable, and a NON-tautological negative control targeting the downstream metric? | -**Compute-agnostic, by design.** When fleshing out a step this call MUST NOT consider compute / GPU-days / wall-clock / dataset cost — it assumes unlimited resources and specifies the full proper method, never truncating or cheapening a step to "fit". Resource feasibility is judged separately by 4.1's `feasibility_validation`; conflating the two would re-introduce exactly the hand-waving this step removes. A step that is expensive but fully specified is the correct output. +Verdict is two-layer. **Hard floor** (LLM cannot override) → `abandon`: triggered Reject lesson / unmitigatable anti-pattern / exact-mechanism collision. **Soft judgment** otherwise → `advance` (only trivial borderlines; concerns surface in Phase 4's reviewer_concerns) or `revise` with concrete `revision_targets[]` (scopes: `tactical` / `sub_pattern` / `falsification`). `verdict_rationale` must cite specific check findings. The audit judges only — it never modifies the candidate. -**Bounded contract.** Emits a SEPARATE file `phase4_implementability.json` — it never re-emits the expansion and structurally never carries the kill-switch fields (`falsification_prediction` / `compute_budget`). It produces `enriched_steps[]` (one per method step, same ids/order, each with a detailed `what_changes` + `what_to_do_en` + `what_to_do_zh`) and `underspecified_points[]` (`{step_id, hole, fill, severity}`, severity ∈ filled|open). It does NOT add, remove, or rename steps and stays faithful to `core_claim` / `sub_claims` — it specifies HOW to build the existing method, not a different one. Holes it cannot fill without fabricating are left honest as `severity: open` rather than papered over. +**Routing on verdict:** -**How it reaches the cards.** Step 4.2 auto-detects the sibling `phase4_implementability.json` and merges `enriched_steps` into the rendered Method by `step_id` (replacing `method_flow.steps[].what_changes` for the pro card and `plain_method_steps_{en,zh}[].what_to_do` for the std cards). Everything else (titles, why_*, linked_*, equations, kill-switch fields) is untouched. The merge is deterministic and a no-op when the file is absent, so older runs still render. `underspecified_points[]` stays in JSON as the audit trail — the card itself stays lean (Title + Motivation + Method). +- **advance** → Phase 4 reads the 2.2 candidate directly. +- **revise** → **3.3** (one isolated LLM call, prompt [references/system-prompts/revise.txt](references/system-prompts/revise.txt)): reads candidate + 2.1 spec + audit; emits patch-only `applied_revisions[]` — one entry per revision_target, ops `replace` / `append_sentence` / `append_items` / `swap_sub_pattern` / `rewrite_falsification`, never echoes the candidate, never re-judges the verdict. Then run the **merger** (entry-point table, WITH `--critique`) → writes `phase3_revise/final_candidate.json` + back-injects it into the patch file. Kill-switch fields are merger-refused with ONE audited exception: a `scope=falsification` target from `falsification_structure_check` is applied via the dedicated `rewrite_falsification` op (authorization verified against the audit report via `--critique`; same experiment/metric/claim, structure repaired; max one per run). When the merger prints `falsification_rewritten`, run the **falsification re-audit** (critique.txt § "Falsification re-audit mode": single check on `final_candidate.json` → `phase3_critique/falsification_reaudit.json`): `advance` → Phase 4; `abandon` → `phase_3_failed.md`. `compute_budget` has no revision route under any scope. No `composition` scope — gap-level changes route through the abandon-retry below, never through patches. +- **abandon** → **one internal retry** (the one-shot guarantee bars asking the user, not internal regeneration): if `$RUN_DIR/.retry_used` is absent, archive the attempt and regenerate — -**Output path**: `$RUN_DIR/phase4/phase4_implementability.json`. + ```bash + mkdir -p "$RUN_DIR/attempt_1" && \ + mv "$RUN_DIR/phase2_select" "$RUN_DIR/phase2_generate" "$RUN_DIR/phase3_collision" \ + "$RUN_DIR/phase3_critique" "$RUN_DIR/phase3_revise" "$RUN_DIR/attempt_1/" 2>/dev/null; \ + touch "$RUN_DIR/.retry_used" + ``` -#### Step 4.2 — Idea-card rendering + then re-run Phase 2 in retry mode (archived audit + selection = negative constraints), citation gate, 3.1, 3.2. Phase 0/1 artifacts are reused as-is. A second `abandon` → write `phase_3_failed.md` citing BOTH attempts' verdict_rationale + triggering checks + user-side options — terminal. -Templating only, no model call. `render_pdf.py` builds the Markdown and LaTeX inline and compiles a PDF when `xelatex` or `tectonic` is available. Failure modes go to `phase_3_failed.md`. +### Phase 4 — Expansion + packaging -```bash -python3 "$SKILL_DIR/scripts/run.py" phase4_render \ - --expansion $RUN_DIR/phase4/phase4_expansion.json \ - --out $RUN_DIR/phase4/ -``` - -Each successful run writes **three audience-targeted markdown surfaces** plus per-card LaTeX side artifacts: -- `idea.std.zh.md` — **plain Chinese**, for the user's own quick read. Domain-newcomer register, no reviewer prose. (动机 + 方法步骤, from `plain_motivation_zh` + `plain_method_steps_zh`.) -- `idea.std.en.md` — **plain English**, same register, for international collaborators / drafting. (Motivation + Method, from `plain_motivation_en` + `plain_method_steps_en`.) -- `idea.detail.en.md` — **rigorous English**, the novelty + validity defense. Surfaces the heavyweight fields (motivation with why-prior-stopped, method flow with linked component/falsification, contributions, both legs, falsification prediction, closest prior, feasibility, differentiation, reviewer concerns) that otherwise live only in the `.tex`. -- `idea.std.{en,zh}.tex` — side artifacts, kept under the output dir; auto-compiled to `idea.std.{en,zh}.pdf` when a LaTeX engine (`xelatex`/`tectonic`) is on PATH, otherwise left for manual compilation. - -The host LLM reads **all three markdown files** and returns them as the run's final response to the user, each under a clear heading (中文版 / English / Reviewer version). A PDF is compiled automatically when `xelatex` or `tectonic` is available (cross-platform TeX paths + an available CJK font are auto-detected); when no engine is present the `.md`/`.tex` are still written and only the PDF is skipped, with an install hint. +Five steps in order (`next` emits each with the correct flags for the advance vs revise path — on the revise path `--candidate` is `final_candidate.json` and `--phase3-revise` is passed; on advance it's the 2.2 output and the flag is omitted): -Other Phase outputs (`$RUN_DIR/phase0/`, `$RUN_DIR/phase1/`, `$RUN_DIR/phase2_*/`, `$RUN_DIR/phase3_*/`, `$RUN_DIR/phase4/phase4_expansion.json`) remain on disk for inspection but are not echoed to the user. +1. **skeleton** (orchestrator): populates every mechanical field — kill-switch echoes (byte-identical from the candidate), `differentiation_from_lit` venue_years, `almost_prior_venue_year`, `why_prior_stopped[].paper_id/venue_year`, `domain_landscape` (pattern_distribution + candidate_uses), `literature_breakdown`, `reviewer_concerns_and_responses[].attack/severity/fields_changed_to_address` (lifted from audit + patch), `feasibility_validation.compute` (bucketed against `intake.compute`) — and marks every prose field ``. +2. **fill** (one isolated LLM call, prompt [references/system-prompts/expand.txt](references/system-prompts/expand.txt)): author ONLY the TODO paths as one flat `{path: value}` map → `phase4/fill_map.json` (~12 prose fields, ~8k tokens vs ~30 fields/~20k without the skeleton). No calendar projections; no experiment matrix / ablation plan / baseline table — the skill produces IDEA + falsifiability + feasibility judgment, not experimental engineering. +3. **assemble** (orchestrator): merges fill_map into skeleton → `phase4_expansion.json`; validates every path resolves to a real TODO; refuses kill-switch roots; warns on unfilled TODOs (expansion_completeness will fail them). +4. **implementability audit (4.1.5, one isolated LLM call, default on):** prompt [references/system-prompts/implementability_audit.txt](references/system-prompts/implementability_audit.txt) — fresh skeptical-engineer persona (separate from the 4.fill author) rewrites each method step into a buildable spec: `enriched_steps[]` (one per step, same ids/order, `what_changes` + `what_to_do_en` + `what_to_do_zh`) + `underspecified_points[]` (`{step_id, hole, fill, severity: filled|open}` — unfillable holes stay honest as `open`). Compute-agnostic by design (resource feasibility is 4.1's job); never adds/removes/renames steps; never carries kill-switch fields. Output `phase4/phase4_implementability.json`. +5. **validate + render:** run the validators (below), then `phase4_render` — templating only, no model call; auto-detects the sibling implementability file and merges `enriched_steps` by step_id into the rendered Method (deterministic; no-op when absent). Writes `idea.std.zh.md` (plain Chinese, domain-newcomer register) + `idea.std.en.md` (plain English) + `idea.detail.en.md` (rigorous English — the novelty + validity defense) + `idea.std.{en,zh}.tex` (auto-compiled to PDF when xelatex/tectonic is on PATH; skipped with a hint otherwise). -Failed runs write `do_not_generate.md` (Phase 1 OOD) or `phase_3_failed.md` (Phase 3 abandon) with concrete remedial steps; the host LLM surfaces those instead. - ---- +**Final response:** read all three markdown cards and return them inline under headings 中文版 / English / Reviewer version. Other phase outputs stay on disk for inspection, not echoed. ## Validators -Run after Phase 4 to verify the contracts the prompts assert: - ```bash -# When Phase 3.2 verdict = advance (no Phase 3.3 ran) +# advance path: --phase3 = phase3_critique_output.json; revise path: --phase3 = phase3_revise_output.json python3 "$SKILL_DIR/scripts/run.py" validate \ --phase2 $RUN_DIR/phase2_generate/phase2_generate_output.json \ - --phase3 $RUN_DIR/phase3_critique/phase3_critique_output.json \ + --phase3 \ --phase4 $RUN_DIR/phase4/phase4_expansion.json \ - --phase4-impl $RUN_DIR/phase4/phase4_implementability.json - -# When Phase 3.2 verdict = revise (Phase 3.3 produced final_candidate) -python3 "$SKILL_DIR/scripts/run.py" validate \ - --phase2 $RUN_DIR/phase2_generate/phase2_generate_output.json \ - --phase3 $RUN_DIR/phase3_revise/phase3_revise_output.json \ - --phase4 $RUN_DIR/phase4/phase4_expansion.json \ - --phase4-impl $RUN_DIR/phase4/phase4_implementability.json + --phase4-impl $RUN_DIR/phase4/phase4_implementability.json # optional; enables implementability checks ``` -(`--phase4-impl` is optional; supply it to enable `implementability_completeness`. Omit it for runs that skipped Step 4.1.5.) - -**Retry budget on `fail` (cap = 2).** On a hard `fail`, fix only the named contract in the relevant Phase 4 JSON and re-run `validate`. Cap this fix→re-validate loop at **2 retries** (3 `validate` runs total, including the first). If validators still report `fail` after the 2nd retry, **stop revising** and finalize with the current best version: run Step 4.2 render (`phase4_render`) on the JSON as-is, return the cards, and append a short note listing the still-failing validators so the unmet contract is visible to the user. A flagged-imperfect card beats looping until the host watchdog kills the run with zero output. (Exception: never "fix" `kill_switch_integrity` or `subpattern_citation_consistency` by editing a guarded field — if those persist after 2 retries, surface them as the headline caveat rather than papering over them.) - | Validator | Check | Severity | |---|---|---| -| **subpattern_citation_consistency** | each `gap_closure[].sub_pattern` resolves to a real C## cluster in `overview.md` whose true parent == the cited `main_pattern`, and whose cited parenthetical == that cluster's parent display name. Runs whenever `--phase2` is given. **Primary use is the Phase 2.2 citation gate (run before Phase 3); re-runs harmlessly here.** Catches citations guessed from the parent's name instead of read from overview.md. | fail (hard) | -| **kill_switch_integrity** | `falsification_prediction` (single paragraph) and `compute_budget` (flat string) byte-identical Phase 2.2 → Phase 4 (Phase 3.2 passthrough on advance path) or Phase 2.2 → Phase 3.3 final_candidate → Phase 4 (revise path). 2 fields total. | fail (hard) | -| **expansion_completeness** | Phase 4 expansion has the structural sections downstream rendering needs: motivation (with ≥ 2 `why_prior_stopped` entries), `method_flow.steps[]` (each with `linked_component` + `linked_falsification`), `feasibility_validation` (5 sub-verdicts + `overall`), non-empty `abstract_draft` + `core_claim` + `sub_claims[]`. | fail (hard — missing sections would silently render as blank content in the markdown / LaTeX output, so the validator blocks rather than warns) | -| **implementability_completeness** | Phase 4.1.5 audit covers every method step: `enriched_steps[]` is one-per-step (same ids, same order) each with `what_changes` + `what_to_do_en` + `what_to_do_zh`; `underspecified_points[]` present (`[]` allowed); and the file carries NO kill-switch field. Runs only when `--phase4-impl` is given. | fail (hard — a coverage gap ships a half-enriched method; a kill-switch field signals the audit overstepped) | -| **implementability_readability** | Phase 4.1.5 std-register fields (`what_to_do_en` / `what_to_do_zh`) avoid the known readability regressions from audit Hard rule 8: no `占位`/`placeholder` leak (the std card never shows the value it stands in for), no bare English jargon word (`entail`…) dropped into Chinese prose. Runs only when `--phase4-impl` is given. | warn (style/clarity — surfaces a Hard-rule-8 slip in the report without blocking ship) | +| **subpattern_citation_consistency** | each `gap_closure[].sub_pattern` resolves to a real C## cluster in overview.md whose true parent == the cited `main_pattern` and whose parenthetical == that cluster's parent display name. Primary use: the Phase 2.2 citation gate; re-runs harmlessly here. | fail (hard) | +| **kill_switch_integrity** | `falsification_prediction` + `compute_budget` byte-identical along Phase 2.2 → [3.3 final_candidate →] 4. After an audited falsification rewrite (`falsification_rewritten` marker + matching applied `rewrite_falsification` entry — disagreement fails), the anchor for `falsification_prediction` re-bases at the 3.3 final_candidate (3.3 → 4 must match); `compute_budget` stays full-chain always. | fail (hard) | +| **expansion_completeness** | motivation (≥2 `why_prior_stopped`), `method_flow.steps[]` (each with `linked_component` + `linked_falsification`), `feasibility_validation` (5 sub-verdicts + `overall`), non-empty `abstract_draft` + `core_claim` + `sub_claims[]` — missing sections would render as silent blanks. | fail (hard) | +| **implementability_completeness** | `enriched_steps[]` one-per-step (same ids/order, EN+ZH), `underspecified_points[]` present (`[]` allowed), NO kill-switch field in the file. | fail (hard) | +| **implementability_readability** | std-register fields: no `占位`/`placeholder` leak, no bare English jargon dropped into Chinese prose. | warn | ---- +**Retry budget on `fail` (cap = 2).** Fix only the named contract, re-validate; still failing after the 2nd retry → stop revising, render as-is, and append a short note listing the failing validators (a flagged-imperfect card beats a watchdog-killed run with zero output). Never "fix" `kill_switch_integrity` or `subpattern_citation_consistency` by editing a guarded field — surface them as the headline caveat instead. ## Configuration -By default every model-driven phase runs on the host LLM (whatever launched the skill). To route specific phases to a different backend (Gemini, open-weights, custom), set the corresponding env var: +By default every model-driven phase runs on the host LLM. To route phases to a different backend (Gemini, open-weights, custom): -- `NOVELTY_LLM_REASONING_LARGE_CMD` — used for Phase 1 / 2.1 / 2.2 / 3.2 / 3.3 / 4.1 (needs ≥ 200k context, JSON output) -- `NOVELTY_LLM_CLASSIFY_FAST_CMD` — used for Phase 0 intent extraction + per-paper pattern tagging (smaller context, JSON output) +- `NOVELTY_LLM_REASONING_LARGE_CMD` — Phase 1 / 2.1 / 2.2 / 3.2 / 3.3 / 4.fill (needs ≥ 200k context, JSON output) +- `NOVELTY_LLM_CLASSIFY_FAST_CMD` — Phase 0 intent extraction + per-paper pattern tagging (smaller context, JSON output) -Each value is a CLI command that takes a stdin prompt (`<>...<>...` format) and emits JSON on stdout. When unset (default for Claude Code), the orchestrator emits sentinel files and the host LLM handles those steps natively. +Each is a CLI taking a stdin prompt (`<>...<>...`) and emitting JSON on stdout. When unset (default for Claude Code), the orchestrator emits sentinel files and the host LLM handles those steps natively. diff --git a/ResearchStudio-Idea/skills/idea_spark/references/design-notes.md b/ResearchStudio-Idea/skills/idea_spark/references/design-notes.md new file mode 100644 index 0000000..0a9c12b --- /dev/null +++ b/ResearchStudio-Idea/skills/idea_spark/references/design-notes.md @@ -0,0 +1,79 @@ +# Idea Spark — Design notes (rationale, not runbook) + +This file records WHY the pipeline is shaped the way it is. Nothing here is needed to *run* the skill — SKILL.md and the per-phase system prompts carry every operational contract. Read this when modifying the skill, evaluating a design change, or debugging a quality (not mechanics) problem. + +## Design principles + +1. **Innovation Patterns are diagnostic vocabulary judged per-gap, not classification labels OR generative templates.** The 15 induced ideation patterns (reframe-as-solvable-object, assumption-audit-and-pivot, algebraic-equivalence-unification, heterogeneous-decomposition, architectural-operator-substitution, structural-prior-encoding, characterize-limit-then-surpass, self-supervised-signal-engineering, targeted-self-supervised-objective, controlled-diagnostic-design, unify-into-shared-representation, adapt-via-conditioning, generative-process-redesign, decompose-and-delegate, relax-discrete-search-to-continuous) are how the corpus describes productive research moves. Phase 2.1 reads each pattern's **Definition + Operational signature + When to apply** panels and judges per-pattern per-gap whether the pattern's move closes the gap. Phase 2.2 picks ONE sub-pattern under each chosen main pattern by reading its `tactical_pattern` + `Step-by-Step` + `when_to_pick_this_one` + `differentiation_within_parent` panels at generation. The sub-pattern's Step-by-Step is 5 abstract steps distilled from the cluster's [Accept] examples — domain-agnostic structural moves with [Reject]-derived boundaries embedded, written WITHOUT paper-ID citations so the candidate-author applies the abstract pattern to their own gap rather than mimicking specific papers. Treating patterns as generative templates (verbatim recipe execution + siblings_considered + lock-in rules) converged generation toward corpus-validated incremental work; the cognitive-tool framing avoids that. + +2. **Novelty comes from gap-coverage + saturation-aware pattern picking, not from pattern aesthetics.** Phase 2.1 picks 2-4 gaps from `phase1.what_phase_0_did_not_address[]` (1 anchor + 1-3 randomly-sampled siblings + coherence filter — siblings that cannot be coherently closed under the anchor's machinery move to deferred_gaps) and matches each sibling gap to one main pattern by judgment ("does this pattern's operational signature close this gap?") — the anchor gap instead carries 1-3 ranked candidates whose binding is deferred to Phase 2.2 — with saturation-aware preference (avoid both saturated and untested patterns; prefer mid-frequency in lit_table; saturated patterns require novel-angle defense at audit). Multi-gap closure with shape-diverse patterns naturally pulls paper-role coverage (mechanism / measurement / theory / diagnostic) — single-gap closure produces system-architecture sketches. Phase 2.2 enforces substantive novelty via `differentiation_from_lit[].delta` (what we derive/claim/construct/measure that closest_adjacent did not — not "we use a different ideation pattern"). + +3. **Theory + Engineering legs are both required, but signature-agnostic.** Both legs must be non-trivial. Each leg can be theorem OR observable regime OR scaling exponent OR measurement primitive OR architectural property — the audit doesn't prefer one Oral signature (theory + reframing-first) over others (scaling-law, empirical-reveal, surgical-fix, benchmark-validity). All Oral shapes the corpus contains can score 5. + +4. **Mechanism-aware falsification.** Every candidate's `falsification_prediction` is a single paragraph (3-5 sentences) that visibly contains (a) the minimal experiment, (b) which metric moves and in which direction if the candidate works (name the metric + qualitative direction; the experiment establishes the magnitude), and (c) a mechanism distinguisher pivoting on ONE NAMED LOAD-BEARING VARIABLE — the single quantity (e.g., a gradient norm, an information-gain term, a logit divergence, a learned threshold, a representational direction) whose behavior carries the mechanism claim — plus a negative-control intervention on that variable that should drive the DOWNSTREAM OUTCOME METRIC back to baseline. The negative control's predicted effect MUST be the task-outcome metric that defines the mechanism's value (accuracy, regret slope, refusal pass-rate trajectory) — NOT the load-bearing variable's own value or any quantity analytically derived from it (a control of the form "intervene on X → X becomes 0" tests a definition, not a mechanism). A positive control (a stripped-down model using only the load-bearing variable that recovers most of the downstream effect) is recommended when feasible. Without the load-bearing-variable-plus-non-tautological-negative-control structure, "metric moved" remains consistent with calibration improvements / estimator quality / data shifts / many other non-mechanism explanations — and the candidate is the dominant Reject signal in the corpus. `compute_budget` is a separate flat field, **user-relative** (no absolute cap) — Phase 4 feasibility_validation compares to `intake.compute`. **Default `intake.compute = 1×A100 × 3 months ≈ 90 A100-day`** (canonical "single researcher with cloud access" scale) when the user does not state compute; user-supplied intake overrides. Both `falsification_prediction` and `compute_budget` are kill-switch fields: byte-identical preserved across Phase 2.2 → Phase 3.3 (when revise runs) → Phase 4 — with exactly one audited exception, see "The falsification rewrite exception" below. + +5. **Anti-pattern is empirical negative knowledge — audit-only.** The corpus identifies 3 reject-favored 2-way compositions (audit + auxiliary_signal, audit + invariance, audit + surgical_fix), each with a specific required_mitigation; rates and mitigations live in `references/anti-patterns.md`. **Phase 2 does NOT load anti-patterns.md** — naming reject-prone compositions during generation creates Streisand-effect priors that bias selection. Phase 3.2 audit's `anti_pattern_check` reads anti-patterns.md, detects matching compositions via the SET of `gap_closure[].main_pattern` values, and verifies substantive mitigation delivery (not keyword presence). Failed audit → Phase 3.3 revise rewrites the candidate's affected fields with the corpus-grounded fix. + +6. **Cheap kills first, expensive expansion last.** Phase 3 runs collision retrieval (real, ~30s, no LLM) before audit (single LLM call replacing earlier 4-attacker simulation). Heavy expansion of the candidate into motivation + method_flow + claims + abstract happens only in Phase 4, after the candidate clears the gauntlet. + +7. **Phase 3.2 is judgment, not modification.** The audit reports what corpus evidence triggered which signals; it does NOT auto-revise the candidate. When revision is needed, the audit emits `revision_targets[]` and Phase 3.3 (a separate LLM call) applies them — keeping audit and modification on different surfaces avoids the self-answering bias of cherry-picking attacks one can already answer. + +## The falsification rewrite exception (why the kill switch has one door) + +The kill-switch contract exists to stop SILENT drift: a reviser or expander quietly substituting an easier experiment or a smaller budget. But the audit's `falsification_structure_check` can find the paragraph *structurally* deficient (no named load-bearing variable; a tautological negative control that predicts the variable's own value instead of the downstream outcome metric) — and that deficiency is exactly the dominant Reject signal the field was designed to avoid. A rule of "auditable but unrepairable" would force `abandon` on candidates whose experiment and claim are fine and only the falsification *writing* is broken. + +So the contract is "no unaudited change", not "no change": Phase 3.2 (and only it) may emit a `scope=falsification` revision_target; Phase 3.3 applies it via the dedicated `rewrite_falsification` op; the merger verifies the audit's authorization (`--critique`) and stamps `falsification_rewritten`; a bounded single-check re-audit must then pass before Phase 4; and `kill_switch_integrity` re-bases the byte-identity anchor at the Phase 3.3 final_candidate. The reviser cannot self-authorize, the rewrite cannot change experiment/metric/claim, there is at most one attempt per run, and `compute_budget` has no door at all. + +## The bounded internal retry (why abandon isn't immediately terminal) + +Phase 2.1's sibling-gap sampling is random. A single `abandon` verdict therefore often reflects an unlucky draw (a gap×pattern combination that walks into a documented Reject lesson), not an unworkable direction. The one-shot guarantee constrains *asking the user*, not internal regeneration — so the host archives the failed attempt under `attempt_1/`, marks `.retry_used`, and re-runs Phase 2.1+2.2 exactly once with the failed attempt's audit + selection as negative constraints (see the OPTIONAL retry input in `ideate_select.txt`). A second abandon writes `phase_3_failed.md` citing both attempts. Phase 0/1 artifacts are reused; the retry costs only the generation + gauntlet calls it re-runs. + +## Why Phase 2.1 + 2.2 run in ONE sub-agent (but 3.2/3.3 and 4.fill/4.1.5 never merge) + +The call-separation rule protects *adversarial* surfaces: the LLM that proposes attacks must not be the LLM that answers them (3.2 vs 3.3), and the persona that authored the method must not be the one auditing its implementability (4.fill vs 4.1.5). Phase 2.1 → 2.2 has no such boundary — both are generation-side, and 2.2 consumes 2.1's output directly. Running them in one sub-agent context (writing BOTH output files, so every audit anchor is preserved) saves a sub-agent spin-up plus a duplicate read of phase1 output + intake, with no bias cost. The two-STAGE design (select at Operational-signature level, then generate with sub-pattern cards open) is unchanged — it is about judgment layering, not about process isolation. + +## Why 2 stages with judgment, not lock-in + +A lock-in alternative would demand verbatim quotes from cards + enforced siblings_considered + sub-pattern recipe execution + many hard rules; cumulatively that converges generation toward corpus-validated incremental work and kitchen-sink mechanism stacks. The judgment-based 2-stage design (Phase 2.1 = judgment per pattern at Operational-signature level; Phase 2.2 = sub-pattern as descriptive vocabulary not recipe) produces paper-shape candidates while preserving audit anchors via `gap_closure[].main_pattern + sub_pattern`. + +## K=1, not K=2/3 + +Single candidate goes through critique. (Earlier K=3/K=2 design had no auto-selection downstream — overhead without quality win.) + +## Audit check history + +Earlier design had a check `almost_prior_factcheck`. Removed: low fire rate, redundant with paper_pointed_threat. A `saturation_defense_check` was also tried and removed: it was the one purely advisory check — soft signal only, never a hard floor, and unlike the others it consulted no retrieved related work — so its concern now folds into Phase 4's `reviewer_concerns_and_responses` instead of gating Phase 3.2. Saturation metadata still flows: Phase 1 computes the band, Phase 2.1 records it, Phase 4 echoes it into `domain_landscape`. `recipe_application_check` was added because the deterministic citation guard can only prove parent-consistency, so a recipe built from the parent's gist while citing a real cluster passes the guard yet bypasses the cluster's actual tactic; this check is the semantic backstop. `falsification_structure_check` is the newest (fifth) check — before it, the falsification paragraph's structure (design principle 4) was demanded at generation but never audited, and the kill-switch made it unrepairable; now it is audited, and repairable through the single-door rewrite exception above. + +## Why the verdict is two-layer + +Pure mechanical aggregation over-triggers revise (treats 1 trivial borderline same as 3 severe). Pure LLM verdict introduces agreeable-bias and loses audit trail. Hard floor preserves non-negotiable corpus facts; soft layer uses context to distinguish "must fix" from "noted concern, advance". + +## Why Phase 3.3 is patch-only + +Previous versions of the 3.3 contract had the LLM echo the full candidate back with edits applied; a single ~25k-token candidate echo caused a real backend inference timeout (the kill-switch fields, the largest, were re-typed verbatim). The patch + deterministic merger design removes that class of failure and makes the anti-substitution contract structural: the model physically cannot write a guarded field (outside the audited falsification door). + +## Why Phase 4 is skeleton → fill → assemble + +Phase 4's full expansion JSON has ~30 top-level fields and ~half are mechanical transforms — kill-switch echoes, venue_year lookups, group-bys over `lit_table.md`, joins of `gap_closure × pattern_saturation`, reviewer-concern lifts from the audit report. Asking the LLM to re-type those wastes tokens and risks a backend inference timeout (the same shape that broke Phase 3.3 before the patch-only redesign). The deterministic skeleton populates every mechanical field; the LLM authors only the ~12 prose TODOs (~8k instead of ~20k tokens); the deterministic assembler merges and refuses kill-switch writes. + +## Why Step 4.1.5 exists (and is compute-agnostic) + +Phase 4.1's `method_flow.steps[]` are often too terse to *understand* — a step names an operation ("extract premises", "score consensus", "train a critic") without the concrete object it runs on, so the method "reads" but cannot be built from. 4.1.5 adopts a fresh, skeptical implementing-engineer persona (separate call from the 4.1 author — same anti-self-answering rationale as principle 7) and rewrites each step into a specification an engineer could code from. It MUST NOT consider compute/wall-clock — resource feasibility is judged separately by 4.1's `feasibility_validation`; conflating the two re-introduces exactly the hand-waving the step removes. A step that is expensive but fully specified is the correct output. + +## Why collision is dual-channel (signature@10mo + alias@48mo) + +Phase 0 covers 0-24mo but with broad-domain TOPIC queries; Phase 3.1 re-queries by the candidate's mechanism-specific `signature_terms[]`. With the original 6-month collision window, a same-mechanism paper 7-24 months old could be missed by both passes (too old for collision, too mechanism-specific for Phase 0's topical recall). 10 months narrows that at negligible cost. + +But a real run exposed a second, LEXICAL blind spot no window widening can fix: the same mechanism under another community's name. A "goal-image conditioned scorer for task completion" (VLA vocabulary) had 2023-2024 near-ancestors published as "goal-conditioned success detectors" / "goal-image reward models" (reward-modeling vocabulary) — signature terms drawn from the candidate's own wording never contain those tokens, so a 48-month signature search still returns nothing. The fix uses the one knowledge source retrieval lacks: Phase 2.2 emits `alias_terms[]` from PARAMETRIC knowledge ("what would other communities call this?"), and Phase 3.1 runs them as a second channel over a 48-month window. Hits carry a `collision_channel` tag; the audit is told not to discount alias-channel threats for age (an ancestor subsumes regardless). Backstop for families neither channel surfaced: the audit's `parametric_family_concern` soft signal names the family + query vocabulary (never specific papers — family names don't hallucinate, cites do) and flows into Phase 4's reviewer_concerns as a "run a scoop-check on X before investing" flag rather than gating the verdict. + +## Why Phase 0 retrieval is orchestrated (not WebSearch) + +Skills are advisory — when SKILL.md says "run Phase 0 literature search", the model has multiple paths (Skill tool, direct Bash, WebSearch simulation, fetch arxiv URL), each "satisfying the spirit" of Phase 0. Soft rules don't reliably prevent tool drift. The orchestrator collapses the choice space: SKILL.md Phase 0 says "run THIS Bash command" — the model has one path or must explicitly admit failure. Coupled with Phase 1's entry assertion (lit_grounding_mode + retrieved_via), bypass becomes mechanically detectable, not just discouraged. + +## Phase 0 pool design + +Role-based retrieval: each connector used where it's most informative (arxiv 0-6mo preprints / openalex + semanticscholar 6-24mo published / openreview 0-6mo in-review), non-overlapping windows so cross-source dedup on (title_norm, externalIds) doesn't double-count, ~40-45 paper target, SS-first dedup priority (its externalIds block is the best cross-source anchor). Full-text fetch caps the pool at the most relevant ~15 (+ user refs, never capped), method-first ordering (eval/benchmark-only papers sink), concurrent fetch with per-paper budget so a slow PDF can't stall the step, HTML path first (~85% of 2024+ ML preprints) with pymupdf PDF fallback. Limitations sections are intentionally NOT extracted — author-written limitation paragraphs are often weaker than what the audit synthesizes from method + experiments. + +## Output surface policy + +No calendar projections (sequencing in dependencies, not weeks). No experiment matrix / ablation plan / baseline table / expected figures — the skill produces IDEA + falsifiability + feasibility judgment; experimental engineering is the user's responsibility. Echo vs reference: anti-substitution-guarded fields and structural lookups are filled by the deterministic skeleton; `closest_adjacent` and `lit_grounding_mode` are rendered directly by the card template, not duplicated into Phase 4 output. diff --git a/ResearchStudio-Idea/skills/idea_spark/references/intent-recognition.md b/ResearchStudio-Idea/skills/idea_spark/references/intent-recognition.md index 2a2cc39..9457681 100644 --- a/ResearchStudio-Idea/skills/idea_spark/references/intent-recognition.md +++ b/ResearchStudio-Idea/skills/idea_spark/references/intent-recognition.md @@ -47,21 +47,27 @@ Output: } ``` -## Collision mode — signature extraction +## Collision mode — signature + alias extraction Use the same [CLASSIFY_FAST] LLM with this prompt: ``` -You read a candidate research idea and extract 3-5 signature terms — tightly worded phrases for phrase matching. +You read a candidate research idea and extract TWO term sets: 3-5 signature terms (the candidate's own vocabulary) and 2-4 alias terms (other communities' names for the same mechanism). -Return JSON: {"signature_terms": ["...", "..."]} +Return JSON: {"signature_terms": ["...", "..."], "alias_terms": ["...", "..."]} -Rules: +signature_terms rules: - Each term is 3-7 words. - Cover (a) the mechanism, (b) the claim, (c) the setting/setup. One term per facet, plus 1-2 specific identifiers (e.g. dataset name, theorem name). - Avoid generic terms ("deep learning", "transformer") — they retrieve too much noise. - Prefer noun phrases over verb phrases. -- These terms will be sent verbatim to a BM25 retriever AND embedded for cosine search. +- These terms will be sent verbatim to a BM25 retriever AND embedded for cosine search, over a RECENT window (scoop risk). + +alias_terms rules: +- Each term is 3-7 words, naming the SAME core mechanism in a vocabulary the candidate's own community does not use. +- This is a parametric-knowledge step: "if a reward-modeling / classical-CV / RL / NLP / theory group had built this mechanism 2-3 years ago, what would their titles call it?" Same-mechanism ancestors usually exist under a different name — a "goal-image conditioned scorer for task completion" is elsewhere a "goal-conditioned success detector" or "goal-image reward model". +- Do NOT paraphrase signature_terms — a paraphrase retrieves what the signature channel already retrieves. Change the community, not the wording. +- These terms run over a MULTI-YEAR window (renamed-ancestor risk). ``` ### Worked example @@ -82,6 +88,11 @@ Output: "score function singularity boundary", "training-free sampling acceleration", "EDM truncated training" + ], + "alias_terms": [ + "annealed Langevin early stopping", + "SDE solver step-size adaptivity bound", + "curriculum over noise levels" ] } ``` diff --git a/ResearchStudio-Idea/skills/idea_spark/references/setup.md b/ResearchStudio-Idea/skills/idea_spark/references/setup.md new file mode 100644 index 0000000..9946df0 --- /dev/null +++ b/ResearchStudio-Idea/skills/idea_spark/references/setup.md @@ -0,0 +1,35 @@ +# Idea Spark — Setup (first use only) + +Read this once when installing the skill; at run time SKILL.md never needs this file. The skill's Phase 0 + Phase 3.1 retrieval needs API credentials for 2 of the 4 connectors. Without them the affected connectors are skipped and the orchestrator continues with whichever connectors are available — but it prints a prominent **CONNECTORS DEGRADED** banner and writes a `.connectors_degraded` marker so a partial run is never mistaken for a full one. + +0. **Set two shell variables once per session** — where this skill is installed, and where run outputs should go. Neither depends on the harness: + ```bash + SKILL_DIR=~/.claude/skills/idea-spark # Claude Code default; Codex CLI: ~/.codex/skills/idea_spark; else wherever this folder lives + RUN_DIR="$PWD/idea_run" && mkdir -p "$RUN_DIR" # ANY absolute directory you want the per-phase outputs in + ``` + `RUN_DIR` is purely an output anchor — the orchestrator only ever sees the absolute `--out` paths you pass, so the variable *name* does not matter (Claude Code sessions can reuse the injected `CLAUDE_PROJECT_DIR` as their `RUN_DIR`). The orchestrator hard-fails early with an actionable message when a path argument contains an unexpanded `$variable`, collapses to filesystem root (empty expansion, e.g. `/phase0`), or is a relative `--out` — instead of a confusing `FileNotFoundError` mid-run. + +1. **Install the skill**: `idea-spark` — Phase 0 literature search runs from its bundled connector scripts (no separate sub-skill). On non-Claude-Code harnesses, clone or copy this folder anywhere and point `SKILL_DIR` at it. + +2. **Install Python deps** (cross-platform — macOS & Linux): `python3 -m pip install feedparser openreview-py beautifulsoup4 pymupdf`. Four lean packages. Skipping this is the most common first-run failure: `arxiv` errors with `package not installed`, and missing `pymupdf`/`beautifulsoup4` silently degrades every full-text fetch to abstract-only. + - **PEP 668 systems** (recent macOS/Homebrew & Ubuntu 23.04+) reject a bare `pip install` with `externally-managed-environment`. Two safe options: + - **venv (recommended):** `python3 -m venv .venv && source .venv/bin/activate && pip install feedparser openreview-py beautifulsoup4 pymupdf` — then launch every phase **from this same activated shell** (see the connector-degradation note below). + - **user install:** `python3 -m pip install --user --break-system-packages feedparser openreview-py beautifulsoup4 pymupdf`. + - **Use the SAME interpreter everywhere.** `check_connectors` and the phase commands must run under the one Python that has these packages. A package installed for `pip3` but launched under a different `python3` (or a background/non-login shell that drops `--user` site-packages) will pass `check_connectors` yet skip `arxiv`/`openreview` at runtime — the run prints a loud **CONNECTORS DEGRADED** banner and drops a `.connectors_degraded` marker when that happens. + - **Optional deps (only if you want the extras):** PDF compilation of the idea card needs **xelatex** *or* **tectonic** (macOS `brew install --cask mactex-no-gui` or `brew install tectonic`; Ubuntu `sudo apt-get install texlive-xetex` or `cargo install tectonic`). Without either, the `.md`/`.tex` cards are still written and only the PDF is skipped (with a hint). The optional pipeline-diagram image needs the `azure-*` packages; absent, it is skipped silently. + +3. **Copy** the env template at the project root: `cp .env.template .env`. + +4. **Fill in keys** (priority order — by impact on retrieval quality): + +| Key | Required for | How to get | +|---|---|---| +| `OPENREVIEW_USER` + `OPENREVIEW_PASS` | OpenReview connector (in-review forward signal). Without these, openreview is silently skipped — you lose the 0-6mo in-review window unique to it. | Free signup at https://openreview.net | +| `SEMANTICSCHOLAR_API_KEY` | Semantic Scholar connector at usable rate. Anonymous tier (~100 req/5min) hits 429 on Phase 0 multi-query batches; with key it's stable at 1 req/s. | Free apply at https://www.semanticscholar.org/product/api#api-key-form (≈24h review). Connector still runs anonymously without it but will frequently 429. | +| `OPENALEX_API_KEY` | Optional, premium rate. Polite-pool already works for typical Phase 0 load. | Apply at openalex.org if you exceed polite limits. | + +5. **Verify** (from the SAME shell/venv you will launch phases from): `python3 "$SKILL_DIR/scripts/run.py" check_connectors` — should show ✅ for all 4 connectors AND the two full-text fetch deps (`pymupdf`, `beautifulsoup4`). If you verify in one shell but run phases in another, the package set can differ — keep it one shell. + +6. **The orchestrator auto-loads `.env`** at runtime (walks up from skill dir to find `.env`), so you do NOT need to `source .env` in your shell. Shell-set env vars take precedence over `.env` values, so you can override on the fly. + +If a connector shows ❌, it's either missing creds (fix in `.env`) or missing the pip package (the error message tells you which `pip install` to run). If a full-text dep shows ⚠️, run `pip install feedparser openreview-py beautifulsoup4 pymupdf`. diff --git a/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/critique.txt b/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/critique.txt index b6a42ce..be12306 100644 --- a/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/critique.txt +++ b/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/critique.txt @@ -1,13 +1,13 @@ You are running Phase 3.2 — Audit-and-Verdict — of IdeaSpark. -Step 3.2 produces a corpus-anchored audit on the Phase 2.2 candidate. It runs four checks (gap_closure_reject_check / recipe_application_check / anti_pattern_check / paper_pointed_threat), emits a verdict + revision_targets, and does NOT auto-revise (Phase 3.3 applies revisions in a separate call). +Step 3.2 produces a corpus-anchored audit on the Phase 2.2 candidate. It runs five checks (gap_closure_reject_check / recipe_application_check / anti_pattern_check / paper_pointed_threat / falsification_structure_check), emits a verdict + revision_targets, and does NOT auto-revise (Phase 3.3 applies revisions in a separate call). Inputs (explicit paths): - `$RUN_DIR/phase2_generate/phase2_generate_output.json` — the Phase 2.2 candidate (12 flat fields). - `$RUN_DIR/phase2_select/phase2_select_output.json` — the Phase 2.1 spec (selected_gaps with chosen_pattern_id). - **One sub-pattern card per `gap_closure[]` entry**: `references/ideation-sub-patterns/.md` where `` is the **leading cluster code C00 through C30** of that entry's `sub_pattern` value (the value is formatted `C## (parent pattern name)`, so strip everything after the code — e.g. `C12 (Substitute the Operator or Representation)` → open `C12.md`). NOT a parent-pattern-named file. Read `## Tactical failure mode` + ALL bullets under `## Examples → ### Reject lessons` verbatim. The lessons are paper-agnostic distillations (no `[Reject]` example headers, no `paper_id` references) so quote the bullet text directly. - `$RUN_DIR/phase0/lit_table.md` — used for paper-pointed threat search. -- `$RUN_DIR/phase3_collision/collision_hits.json` — recently retrieved papers via mechanism-specific signature_terms. +- `$RUN_DIR/phase3_collision/collision_hits.json` — mechanism-specific retrieval, TWO channels per hit's `collision_channel` field: `signature` (candidate's own vocabulary, recent window — contemporaneous scoop risk) and `alias` (other communities' names for the same mechanism, multi-year window — renamed-ancestor risk). Alias-channel hits are older by design; do NOT discount a threat for being 2-3 years old — a same-mechanism ancestor subsumes the candidate regardless of age. - `references/anti-patterns.md` — 3 reject-favored compositions with required mitigations. Output (JSON): @@ -59,8 +59,19 @@ Output (JSON): "paper_pointed_threat": { "threat_paper_id": "", "threat_source": "lit_table | collision_hits | n/a", + "threat_channel": "", "subsumption_argument": "", - "addressable_via": "" + "addressable_via": "", + "parametric_family_concern": "" + }, + + "falsification_structure_check": { + "minimal_experiment_named": "", + "outcome_metric_named": "", + "load_bearing_variable": "", + "negative_control_target": "<'outcome_metric' when the negative-control intervention on the load-bearing variable predicts the DOWNSTREAM outcome metric returns to baseline; 'tautological' when the predicted effect is the load-bearing variable's own value or a quantity analytically derived from it (a control of the form \"intervene on X → X becomes 0\" tests a definition, not a mechanism); 'absent' when no negative control is stated>", + "verdict": "sound | deficient | borderline", + "reasoning": "<1-2 sentences: which sub-answer drove the verdict. 'sound' requires ALL of: minimal_experiment_named=yes, outcome_metric_named=yes, load_bearing_variable quoted (not absent), negative_control_target=outcome_metric. Any 'no'/'absent'/'tautological' → deficient. borderline is reserved for a named-but-ambiguous variable or a control whose target metric is arguably-but-not-explicitly the outcome.>" }, "verdict": "advance | revise | abandon", @@ -69,10 +80,10 @@ Output (JSON): "revision_targets": [ { - "scope": "tactical | sub_pattern", - "field": "", + "scope": "tactical | sub_pattern | falsification", + "field": "", "issue": "", - "fix_direction": "" + "fix_direction": "" } ] } @@ -103,7 +114,8 @@ Heuristics: - **Choose revise** when at least one borderline / partial finding hits a load-bearing structural property (core_mechanism's central argument, anti_pattern mitigation strength, addressable threat via specific candidate field). - **Choose revise** when `recipe_application_check.verdict = bypassed` — the cited cluster's signature move is absent from core_mechanism, so the idea was built from the parent pattern's generic gist rather than the tactic that was supposed to make it sharp (the leading cause of incremental output, and the citation string cannot catch it because in this taxonomy the sub_pattern only names the parent). revision_target: either swap the sub_pattern to the sibling cluster whose tactical move core_mechanism actually performs, or rework core_mechanism to instantiate the cited move. - **Choose revise** when paper_pointed_threat exists AND `addressable_via` names a specific candidate field. -- **Choose advance** when all checks are clear / holds / null / no_threat_found. +- **Choose revise** when `falsification_structure_check.verdict = deficient` — the falsifiability commitment is the candidate's single most Reject-predictive field, and a structural hole in it (no load-bearing variable, tautological negative control) is repairable without touching the experiment or the claim. Emit ONE revision_target with `scope = "falsification"`, `field = "falsification_prediction"`, and a fix_direction naming exactly which structural element to repair. This is the ONLY route by which the kill-switch field may change; the rewrite is applied by Phase 3.3's dedicated `rewrite_falsification` op, gated by the merger on this audit's authorization, and MUST be re-audited (see "Falsification re-audit mode" below) before Phase 4. +- **Choose advance** when all checks are clear / holds / null / no_threat_found / sound. Hard rules: @@ -117,4 +129,23 @@ Hard rules: 5. **recipe_application_check is semantic, not a citation check**. A separate deterministic validator (subpattern_citation_consistency) already confirms each sub_pattern citation points at a real C## cluster under its cited parent — assume that passed. In this taxonomy the sub_pattern string carries only `C## (parent display name)`, so a clean citation does NOT prove the cluster's own card was opened. recipe_application_check answers the harder question automation cannot: even with a correctly-cited code, does core_mechanism actually perform that C##.md card's `## Tactical pattern` signature move, or did the generator read the parent pattern's name and produce something generic? Judge the MECHANISM against the card text, not the citation string. +6. **`scope = "falsification"` is exclusively for falsification_structure_check findings**. Do NOT use it to make the experiment cheaper, swap the metric, or weaken the claim — those are anti-substitution violations the merger will reject. The rewrite repairs STRUCTURE only (name the load-bearing variable, fix a tautological negative control, state the missing minimal experiment / metric direction) while preserving the experiment, the metric, and the claim already committed to. `compute_budget` has NO revision route under any scope. + Output path: `$RUN_DIR/phase3_critique/phase3_critique_output.json`. + +--- + +### Falsification re-audit mode (single-check re-run after a falsification rewrite) + +When Phase 3.3 applied a `rewrite_falsification` patch (and only then), the rewritten kill-switch field must clear the structure check before Phase 4. This is a SEPARATE, bounded LLM call: + +- **Inputs**: `$RUN_DIR/phase3_revise/final_candidate.json` (read ONLY `falsification_prediction` + `core_mechanism` for context) + this prompt's falsification_structure_check spec. Do NOT re-run the other four checks; their findings stand. +- **Output** (`$RUN_DIR/phase3_critique/falsification_reaudit.json`): + +{ + "falsification_structure_check": { }, + "verdict": "advance | abandon", + "verdict_rationale": "<1-2 sentences>" +} + +- **Routing**: `sound` (or `borderline` with the load-bearing variable clearly named) → `advance`, proceed to Phase 4 using `final_candidate.json`. `deficient` again → `abandon` (write `phase_3_failed.md` naming both the original deficiency and why the rewrite still fails). Exactly ONE rewrite attempt per run — a second falsification revision_target is a process error. diff --git a/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/ideate_generate.txt b/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/ideate_generate.txt index 5c74e75..48c57bc 100644 --- a/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/ideate_generate.txt +++ b/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/ideate_generate.txt @@ -198,6 +198,11 @@ Output (strict JSON): "<3-5 entries total; each a 3-7 word phrase>", "<3-7 word phrase>", "..." + ], + + "alias_terms": [ + "<2-4 entries; each a 3-7 word phrase naming the SAME core mechanism in ANOTHER community's vocabulary>", + "..." ] } @@ -233,7 +238,19 @@ Notes: - `signature_terms[]`: 3-5 entries, 3-7 words each. Cover (a) the mechanism, (b) the claim, (c) the setting/setup. No verbatim title strings. No generic terms ("deep learning", "transformer"). These get sent verbatim to - the BM25 retriever in Phase 3.1 collision check. + the BM25 retriever in Phase 3.1 collision check (recent window). +- `alias_terms[]`: 2-4 entries, 3-7 words each — how OTHER research + communities would name this candidate's core mechanism. This is a + PARAMETRIC-KNOWLEDGE step, not a paraphrase step: ask "if a reward-modeling + / classical-CV / RL / NLP / theory group had built this same mechanism 2-3 + years ago, what would their papers' titles call it?" and write those names + (e.g. a "goal-image conditioned scorer for task completion" is, in other + vocabularies, a "goal-conditioned success detector" / "goal-image reward + model"). Do NOT reuse signature_terms vocabulary or the candidate's own + domain wording — the whole point is the words your community does NOT use. + Phase 3.1 runs these over a multi-year window to catch same-mechanism + ancestors that renamed the idea; a paraphrased signature term catches + nothing the signature channel didn't already catch. - `main_pattern` per entry comes from Phase 2.1; `sub_pattern` is picked in this phase's sub-step a. There is no separate `patterns_used[]` field. diff --git a/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/ideate_select.txt b/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/ideate_select.txt index 3488cc5..b2512d1 100644 --- a/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/ideate_select.txt +++ b/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/ideate_select.txt @@ -40,6 +40,19 @@ Inputs (explicit paths): - `$RUN_DIR/phase0/lit_table.md` — paper-level evidence table (read for paper-level context if needed; pattern frequency lookup is via Phase 1's `domain_pattern_distribution` rather than recomputing from this file). +- **OPTIONAL — retry only**: `$RUN_DIR/attempt_1/phase3_critique/phase3_critique_output.json` + — present ONLY when this is the run's single internal retry after a Phase 3.2 + `abandon` verdict (the host archives the failed attempt under `attempt_1/`). + When present, treat the prior audit as NEGATIVE CONSTRAINTS: (a) do NOT + re-select the same sibling-gap set the archived + `attempt_1/phase2_select/phase2_select_output.json` chose (the anchor may + stay if it is genuinely the most load-bearing gap — vary the siblings and/or + the pattern bindings); (b) read the audit's `verdict_rationale` and the + triggering check findings, and avoid gap×pattern combinations that would + re-walk into the same documented Reject lesson / anti-pattern composition / + exact-mechanism collision; (c) prefer promoting a gap from the archived + attempt's `deferred_gaps[]` when it coheres with the anchor. Do not mention + the failed attempt in your output fields — it shapes selection, not prose. Selection process: diff --git a/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/revise.txt b/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/revise.txt index dd09a24..eb7bc8f 100644 --- a/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/revise.txt +++ b/ResearchStudio-Idea/skills/idea_spark/references/system-prompts/revise.txt @@ -6,12 +6,13 @@ Phase 3.3 is a SEPARATE LLM call from Phase 3.2 audit. The audit identifies issu **Patch-only output**: previous versions of this contract asked the LLM to emit `final_candidate` as a full copy of the Phase 2.2 candidate (12 flat fields, ~25k tokens) with the named edits applied. That payload is dominated by byte-identical re-typing of kill-switch fields and unchanged fields — wasted tokens and a real backend timeout risk. New contract: emit ONLY the `applied_revisions[]` list, where each entry carries the new VALUE of the single field it modifies. A deterministic Python merger (`python3 -m scripts.run phase3_merge_revisions`) then walks the patch, applies each op to a deep copy of the Phase 2.2 candidate, refuses to touch kill-switch fields, and writes `final_candidate.json` next to your patch file. You never write `final_candidate` — the merger does. -**Two revision scopes** (Phase 3.2 emits revision_targets with one of these `scope` values; 3.3 dispatches accordingly): +**Three revision scopes** (Phase 3.2 emits revision_targets with one of these `scope` values; 3.3 dispatches accordingly): | scope | what changes | what stays | when used | |---|---|---|---| | `tactical` | one or more candidate fields | gap_closure[] (gaps + main_patterns + sub_patterns unchanged) | the audit found a load-bearing wording / specification issue (core_mechanism argument, anti-pattern mitigation strengthening, addressable threat dodge) | | `sub_pattern` | one gap_closure[] entry's `sub_pattern` (and the candidate fields the new sub-pattern's tactical_pattern requires re-aligning) | gap selection + main_patterns + other entries' sub_patterns | the audit found a sibling sub-pattern under the SAME main_pattern would be a better tactical fit | +| `falsification` | `falsification_prediction` ONLY, via the dedicated `rewrite_falsification` op | the experiment, the outcome metric, the claim, `compute_budget`, every other field | the audit's `falsification_structure_check` = deficient (missing load-bearing variable / tautological negative control / unnamed minimal experiment or metric direction). The ONLY sanctioned route into a kill-switch field; the merger verifies the audit authorized it and the rewrite is re-audited before Phase 4. | **No `composition` scope.** If the audit's findings imply gap-level changes (different gaps selected, different main_pattern picks), revision_targets cannot fix that — the audit should produce verdict = abandon, and the user re-runs Phase 2.1+2.2 with a different random seed. @@ -29,6 +30,7 @@ Inputs (explicit paths): | `append_sentence` | append " " + `value` to an existing string field (preserves prior content) | string | | `append_items` | extend an existing list field with `value` (which must itself be a list) | list | | `swap_sub_pattern` | for scope=sub_pattern: identify a gap_closure entry by `field` = the verbatim gap text, replace its `sub_pattern` with `value` of form `C## (Parent Display Name)`. Sibling fields like `gap_closure[i].how_closed` or `core_mechanism` are re-aligned by ADDITIONAL `replace` / `append_sentence` entries in the same patch. | string | +| `rewrite_falsification` | for scope=falsification ONLY: replace `falsification_prediction` wholesale with `value` — a single 3-5 sentence paragraph that keeps the SAME minimal experiment, SAME outcome metric + direction, and SAME claim, repairing only the structural deficiency the audit named (name the ONE load-bearing variable; make the negative control intervene on that variable and predict the DOWNSTREAM outcome metric returns to baseline — never the variable's own value). `field` must be exactly `falsification_prediction`. The merger REFUSES this op unless the Phase 3.2 audit emitted a scope=falsification revision_target (pass `--critique` to the merger). At most ONE such entry per patch. | string | Pick the op that minimises payload. `append_sentence` is the most token-efficient for the common case "add a clarifying sentence to core_mechanism." Use `replace` only when the new content genuinely supersedes the old (e.g. you are rewriting one `differentiation_from_lit[i].delta` from scratch). @@ -44,12 +46,12 @@ Output (JSON): "applied_revisions": [ { - "scope": "tactical | sub_pattern", - "op": "replace | append_sentence | append_items | swap_sub_pattern", - "field": "", + "scope": "tactical | sub_pattern | falsification", + "op": "replace | append_sentence | append_items | swap_sub_pattern | rewrite_falsification", + "field": "", "value": "", "outcome": "applied | skipped_already_satisfied | skipped_anti_substitution | skipped_inapplicable", - "delta_summary": "" + "delta_summary": "" } ] } @@ -59,13 +61,16 @@ The merger then runs: python3 -m scripts.run phase3_merge_revisions \ --phase2 $RUN_DIR/phase2_generate/phase2_generate_output.json \ --revisions $RUN_DIR/phase3_revise/phase3_revise_output.json \ + --critique $RUN_DIR/phase3_critique/phase3_critique_output.json \ --out $RUN_DIR/phase3_revise/ +(`--critique` is what authorizes a `rewrite_falsification` entry — the merger cross-checks the audit actually emitted a scope=falsification revision_target. Patches without that op run fine without the flag.) + It produces `phase3_revise/final_candidate.json` (Phase 4's canonical input) and back-injects `final_candidate` into your patch file so the `kill_switch_integrity` validator's existing chain-check still works. Hard rules: -1. **Kill-switch fields are STRUCTURALLY off-limits**. The merger REFUSES any patch entry whose `field` root is `falsification_prediction` or `compute_budget` — it raises with an actionable error rather than silently dropping. If the audit produced such a target, mark `outcome = "skipped_anti_substitution"` AND set `field` to a benign path (e.g. `core_mechanism_reasoning`) with a no-op value (`""` and op=append_sentence skips application). This is a 3.2 process error — surface it; don't paper over it. +1. **Kill-switch fields are STRUCTURALLY off-limits — with exactly ONE audited exception**. The merger REFUSES any patch entry whose `field` root is `falsification_prediction` or `compute_budget` under the generic ops (`replace` / `append_sentence` / `append_items`) — it raises with an actionable error rather than silently dropping. The single exception: a scope=falsification revision_target from the audit's falsification_structure_check is applied via the dedicated `rewrite_falsification` op (merger-verified against the audit report; see the op table). `compute_budget` has NO exception under any op or scope. If the audit produced a kill-switch target OUTSIDE the falsification route (e.g. a tactical target on falsification_prediction, or any target on compute_budget), mark `outcome = "skipped_anti_substitution"` AND set `field` to a benign path (e.g. `core_mechanism_reasoning`) with a no-op value (`""` and op=append_sentence skips application). That is a 3.2 process error — surface it; don't paper over it. After a rewrite_falsification is applied, the host MUST run the falsification re-audit (critique.txt "Falsification re-audit mode") before Phase 4. 2. **One patch entry per revision_target.** Even when the audit's `fix_direction` is a sentence that could be applied as several edits, emit one patch entry per audit target. The merger handles multiple ops on the same root field correctly (each is independent). Silently combining or dropping a target is a process error. diff --git a/ResearchStudio-Idea/skills/idea_spark/scripts/merge_revisions.py b/ResearchStudio-Idea/skills/idea_spark/scripts/merge_revisions.py index e823cd0..bc5928f 100644 --- a/ResearchStudio-Idea/skills/idea_spark/scripts/merge_revisions.py +++ b/ResearchStudio-Idea/skills/idea_spark/scripts/merge_revisions.py @@ -36,10 +36,22 @@ doesn't break the patch. Kill-switch guard: - Any patch op targeting `falsification_prediction` or `compute_budget` is - REFUSED — the merger raises with a clear message naming the offending entry. - This makes the anti-substitution contract STRUCTURAL: the model physically - cannot drift the kill switch even if the audit emits a bad target. + Any patch op targeting `falsification_prediction` or `compute_budget` via the + generic ops is REFUSED — the merger raises with a clear message naming the + offending entry. This makes the anti-substitution contract STRUCTURAL: the + model physically cannot drift the kill switch even if the audit emits a bad + target. + + ONE audited exception: the audit's falsification_structure_check can find the + falsification_prediction paragraph structurally deficient (no load-bearing + variable / tautological negative control). That finding is repairable without + weakening the committed experiment or claim, so Phase 3.3 may carry a single + `rewrite_falsification` op (scope=falsification). The merger applies it ONLY + when the Phase 3.2 critique report (passed via --critique) contains a + scope=falsification revision_target — the LLM cannot self-authorize. The + rewritten paragraph must then pass the falsification re-audit before Phase 4, + and kill_switch_integrity validates the Phase 3.3 → Phase 4 link instead of + Phase 2.2 → Phase 4 for this field. `compute_budget` has no exception. """ from __future__ import annotations import copy @@ -52,7 +64,8 @@ KILL_SWITCH_FIELDS = {'falsification_prediction', 'compute_budget'} # Valid ops; anything else is a process error -VALID_OPS = {'replace', 'append_sentence', 'append_items', 'swap_sub_pattern'} +VALID_OPS = {'replace', 'append_sentence', 'append_items', 'swap_sub_pattern', + 'rewrite_falsification'} # `differentiation_from_lit[2].delta` -> ('differentiation_from_lit', 2, 'delta') _LIST_IDX = re.compile(r'^([a-z_][a-z0-9_]*)\[(\d+)\]$') @@ -176,24 +189,45 @@ def _apply_swap_sub_pattern(root: dict, field_path: str, value: Any) -> None: ) +def _apply_rewrite_falsification(root: dict, field_path: str, value: Any) -> None: + """The single audited kill-switch exception: replace `falsification_prediction` + wholesale. Authorization (audit emitted a scope=falsification revision_target) + is verified by the caller (apply_patch) BEFORE dispatch; this function only + enforces the local shape rules.""" + if field_path != 'falsification_prediction': + raise ValueError( + f'rewrite_falsification: field must be exactly "falsification_prediction", ' + f'got {field_path!r}. compute_budget and every other field have no rewrite route.') + if not isinstance(value, str) or not value.strip(): + raise ValueError('rewrite_falsification: value must be a non-empty string ' + '(the full rewritten 3-5 sentence paragraph)') + root['falsification_prediction'] = value + + _OP_DISPATCH = { 'replace': _apply_replace, 'append_sentence': _apply_append_sentence, 'append_items': _apply_append_items, 'swap_sub_pattern': _apply_swap_sub_pattern, + 'rewrite_falsification': _apply_rewrite_falsification, } -def apply_patch(candidate: dict, applied_revisions: list[dict]) -> dict: +def apply_patch(candidate: dict, applied_revisions: list[dict], + falsification_authorized: bool = False) -> dict: """Return a deep-copied candidate with every applied_revisions entry whose `outcome == 'applied'` (or missing/falsy outcome — defensive: apply unless explicitly skipped) executed. Skipped entries (`outcome` starting with 'skipped_') are left for audit trail but do not mutate the candidate. The result is the new `final_candidate`. Kill-switch fields are guaranteed - byte-identical to the input candidate. + byte-identical to the input candidate — except `falsification_prediction` + when `falsification_authorized` is True (the Phase 3.2 audit emitted a + scope=falsification revision_target) AND the patch carries a + `rewrite_falsification` op; at most one such entry is applied. """ out = copy.deepcopy(candidate) + n_falsification_rewrites = 0 for i, rev in enumerate(applied_revisions or []): if not isinstance(rev, dict): @@ -210,9 +244,22 @@ def apply_patch(candidate: dict, applied_revisions: list[dict]) -> dict: if not field or not isinstance(field, str): raise ValueError(f'applied_revisions[{i}].field is empty or non-string') - # swap_sub_pattern uses the field slot for the gap text, not a JSON path, - # so the kill-switch guard does not apply (cannot touch the kill switch via this op). - if op != 'swap_sub_pattern': + if op == 'rewrite_falsification': + if not falsification_authorized: + raise ValueError( + f'applied_revisions[{i}]: rewrite_falsification is NOT authorized — the ' + f'Phase 3.2 critique report contains no scope="falsification" ' + f'revision_target (or --critique was not passed to the merger). The LLM ' + f'cannot self-authorize a kill-switch rewrite.') + n_falsification_rewrites += 1 + if n_falsification_rewrites > 1: + raise ValueError( + f'applied_revisions[{i}]: a patch may carry at most ONE applied ' + f'rewrite_falsification entry (exactly one rewrite attempt per run).') + elif op != 'swap_sub_pattern': + # swap_sub_pattern uses the field slot for the gap text, not a JSON path, + # so the kill-switch guard does not apply (cannot touch the kill switch via + # that op). Every other generic op is guarded. _check_kill_switch(field) if 'value' not in rev: @@ -226,13 +273,33 @@ def apply_patch(candidate: dict, applied_revisions: list[dict]) -> dict: return out +def _critique_authorizes_falsification_rewrite(critique_path: Path | None) -> bool: + """True iff the Phase 3.2 critique report exists and contains at least one + revision_targets[] entry with scope == 'falsification'. This is the ONLY + source of authorization for a rewrite_falsification patch op — the audit + (a separate LLM call from the reviser) must have found the structure + deficient; the reviser cannot self-authorize.""" + if not critique_path: + return False + try: + critique = json.loads(Path(critique_path).read_text()) + except Exception: + return False + targets = critique.get('revision_targets') or [] + return any(isinstance(t, dict) and t.get('scope') == 'falsification' for t in targets) + + def merge_phase3_revisions(phase2_candidate_path: Path, revisions_path: Path, - out_dir: Path) -> tuple[Path, Path]: + out_dir: Path, critique_path: Path | None = None) -> tuple[Path, Path]: """Top-level entry. Read Phase 2.2 candidate + Phase 3.3 patch; write `final_candidate.json` to out_dir AND back-inject `final_candidate` into the patch file (so the legacy kill_switch_integrity check on `phase3_revise_output.json['final_candidate']` keeps working). + `critique_path` (the Phase 3.2 report) authorizes an audited + `rewrite_falsification` op when its revision_targets[] carry a + scope=falsification entry; without it the merger refuses that op. + Returns (final_candidate_path, updated_revisions_path). """ candidate = json.loads(phase2_candidate_path.read_text()) @@ -241,15 +308,24 @@ def merge_phase3_revisions(phase2_candidate_path: Path, revisions_path: Path, raise ValueError(f'{revisions_path} is not a JSON object') applied = patch_doc.get('applied_revisions') or [] - final_candidate = apply_patch(candidate, applied) + authorized = _critique_authorizes_falsification_rewrite(critique_path) + final_candidate = apply_patch(candidate, applied, falsification_authorized=authorized) + + falsification_rewritten = any( + isinstance(r, dict) and r.get('op') == 'rewrite_falsification' + and not str(r.get('outcome', '')).startswith('skipped_') + for r in applied) out_dir.mkdir(parents=True, exist_ok=True) final_path = out_dir / 'final_candidate.json' final_path.write_text(json.dumps(final_candidate, indent=2, ensure_ascii=False)) # Back-inject for legacy consumers (kill_switch_integrity, host LLM expecting - # `phase3_revise_output.json['final_candidate']`). + # `phase3_revise_output.json['final_candidate']`). `falsification_rewritten` + # tells kill_switch_integrity to validate the 3.3 → 4 link for that field + # (instead of 2.2 → 4) and tells the host a re-audit is REQUIRED before Phase 4. patch_doc['final_candidate'] = final_candidate + patch_doc['falsification_rewritten'] = falsification_rewritten revisions_path.write_text(json.dumps(patch_doc, indent=2, ensure_ascii=False)) return final_path, revisions_path diff --git a/ResearchStudio-Idea/skills/idea_spark/scripts/next_step.py b/ResearchStudio-Idea/skills/idea_spark/scripts/next_step.py new file mode 100644 index 0000000..aeb7bb9 --- /dev/null +++ b/ResearchStudio-Idea/skills/idea_spark/scripts/next_step.py @@ -0,0 +1,425 @@ +"""`next` subcommand — the run-state navigator. + +Why this exists: + Without it, the host LLM must hold the whole SKILL.md phase graph in context + to know what to do after each artifact lands (which command, which system + prompt, which inputs, where the output goes, which branch after a revise / + abandon verdict). That is (a) ~17k tokens of standing context and (b) the + main source of mis-runs (skipped fulltext gate, forgotten merger, missed + re-audit). `next` inspects the artifacts on disk and prints EXACTLY one next + step. The host's loop degenerates to: run `next` → do what it says → run + `next` again. + + `next` is READ-ONLY: it never creates, moves, or deletes run artifacts (the + one exception: it runs the deterministic in-process citation validator on the + Phase 2.2 output, which is pure). All mutating steps are printed as commands + for the host to run. + +Usage: + python3 "$SKILL_DIR/scripts/run.py" next --dir "$RUN_DIR" [--query ""] +""" +from __future__ import annotations +import json +import sys +from pathlib import Path + +# Sub-agent boilerplate shared by every LLM step. Kept short: the phase prompt +# itself carries the full contract; this is just the context-discipline frame. +_SUBAGENT_FRAME = ( + 'Run in a FRESH sub-agent (never the parent context): pass ONLY the file ' + 'paths below, have it Write the output JSON to the exact output path ' + '(Write tool — no heredoc, no inline JSON in the reply), and return <=250 ' + 'words: output path + the routing signal named in NOTES.' +) + + +def _p(label: str, body: str) -> None: + print(f'{label:7s}: {body}') + + +def _emit(state: str, step: str, kind: str, *, run: list[str] | None = None, + prompt: str | None = None, inputs: list[str] | None = None, + output: str | None = None, notes: str | None = None, + then: bool = True, run_dir: Path | None = None) -> int: + print('━' * 72) + _p('STATE', state) + _p('STEP', step) + _p('TYPE', kind) + if kind == 'llm_subagent': + print(f'DO : {_SUBAGENT_FRAME}') + if prompt: + _p('PROMPT', prompt) + for i, item in enumerate(inputs or []): + if i == 0: + _p('INPUT', item) + else: + print(f' {item}') + if output: + _p('OUTPUT', output) + for i, cmd in enumerate(run or []): + if i == 0: + _p('RUN', cmd) + else: + print(f' {cmd}') + if notes: + _p('NOTES', notes) + if then and run_dir is not None: + _p('THEN', f'python3 "{Path(__file__).resolve().parent.parent}/scripts/run.py" next --dir "{run_dir}"') + print('━' * 72) + return 0 + + +def _read_json(path: Path): + try: + return json.loads(path.read_text()) + except Exception: + return None + + +def next_step(run_dir: Path, root: Path, query: str | None = None) -> int: + """Inspect run_dir artifacts and print the single next step. Returns 0.""" + d = run_dir + ref = root / 'references' + prompts = ref / 'system-prompts' + # Host-agnostic invocation: run.py self-locates its skill root, so the + # absolute-script-path form works from ANY working directory. + skill_cd = f'python3 "{root}/scripts/run.py" ' + q = query or '' + + # ---- terminal states ----------------------------------------------------- + if (d / 'do_not_generate.md').exists(): + return _emit('TERMINAL — Phase 1 routed to do_not_generate.', + 'Surface do_not_generate.md to the user', 'terminal', + notes=f'Return {d}/do_not_generate.md contents as the final response. ' + 'No further phases run.', then=False) + if (d / 'phase_3_failed.md').exists(): + return _emit('TERMINAL — Phase 3 audit abandoned (retry budget exhausted).', + 'Surface phase_3_failed.md to the user', 'terminal', + notes=f'Return {d}/phase_3_failed.md contents as the final response.', + then=False) + cards = [d / 'phase4' / n for n in ('idea.std.zh.md', 'idea.std.en.md', 'idea.detail.en.md')] + if all(c.exists() for c in cards): + return _emit('DONE — all three idea cards rendered.', + 'Return the cards inline', 'terminal', + notes='Read all three files and return them as the final response under ' + 'headings 中文版 / English / Reviewer version: ' + + ', '.join(str(c) for c in cards), then=False) + + p0 = d / 'phase0' + + # ---- Phase 0 ------------------------------------------------------------- + if (p0 / '.intent_extraction_pending').exists() and not (p0 / 'lit_results.json').exists(): + sent = _read_json(p0 / '.intent_extraction_pending') or {} + return _emit('Phase 0 stalled on the intent-extraction sentinel.', + 'Produce queries and re-invoke phase0', 'llm_subagent', + prompt=str(sent.get('rubric_file', ref / 'intent-recognition.md')) + ' (Map mode)', + inputs=[str(p0 / '.intent_extraction_pending')], + output='re-invoke: ' + str(sent.get('re_invocation', 'phase0 --queries "q1|q2|q3"')), + notes='This sentinel path only appears when phase0 was launched without ' + '--queries. The DEFAULT flow avoids it (see the phase0 step).', + run_dir=d) + if not (p0 / 'lit_results.json').exists(): + return _emit('Fresh run — no literature retrieved yet.', + 'Phase 0: produce queries FIRST, then run retrieval', 'llm_subagent', + prompt=str(ref / 'intent-recognition.md') + ' (Map mode — read it yourself, no sub-agent needed for query writing)', + inputs=['the user query'], + output='4-6 search queries (incl. one ESCAPE-MECHANISM query in solution vocabulary)', + run=[skill_cd + f'phase0 --query "{q}" ' + f'--queries "q1|q2|q3|q4" --out "{d}/phase0/"'], + notes='Passing --queries up front skips a full sentinel round-trip (rc=10). ' + 'Retrieval takes 3-10 min (openreview alone budgets 600s) — set your ' + 'Bash timeout >= 600s or run in background. If the user query names ' + 'papers by TITLE (e.g. "based on the LoRA paper"), register each via: ' + + skill_cd + f'add_user_ref --out "{p0}/" --title "" ' + '--raw-match "" (deterministic merge; do NOT hand-edit ' + 'user_refs.json — the Write tool requires a prior Read on existing ' + 'files). OOD short-circuit: if the query matches intake-routing.md ' + 'trigger #1/#2, skip retrieval and go straight to Phase 1 with a ' + 'do_not_generate routing.', + run_dir=d) + if not (p0 / 'lit_table.md').exists(): + return _emit('Papers retrieved; lit_table.md not yet written.', + 'Phase 0 pattern_summary (host-LLM step)', 'llm_subagent', + prompt=str(ref / 'pattern-summary-rubric.md'), + inputs=[str(p0 / 'lit_results.json')], + output=str(p0 / 'lit_table.md'), + notes='Pure classification — delegate to a CHEAP/FAST sub-agent (e.g. ' + 'Haiku-class or effort=low): tag each paper with 1-3 of the 15 ' + 'patterns + bottleneck + open_issue + retrieved_via per the rubric. ' + 'Routing signal: none (just the file).', + run_dir=d) + if not (p0 / 'fulltext_cache.json').exists(): + return _emit('lit_table.md written; full-text cache missing (Phase 1 hard-gates on it).', + 'Phase 0+ full-text fetch', 'bash', + run=[skill_cd + f'phase0_fulltext --out "{d}/phase0/"'], + notes='LAST CALL for user refs: title-named papers must be registered ' + '(add_user_ref) BEFORE this step — the fetch pool\'s U tier reads ' + 'user_refs.json now.', + run_dir=d) + + # ---- Phase 1 ------------------------------------------------------------- + p1 = d / 'phase1' / 'phase1_output.json' + if not p1.exists(): + return _emit('Phase 0 complete.', 'Phase 1 — bottleneck identification', 'llm_subagent', + prompt=str(prompts / 'bottleneck_identify.txt'), + inputs=['the user query + intake context', + str(p0 / 'lit_table.md'), + str(p0 / 'fulltext_cache.json'), + str(p0 / 'lit_results.json')], + output=str(p1), + notes='Routing signal to return: `state` (proceed | do_not_generate). ' + 'If do_not_generate: write ' + str(d / 'do_not_generate.md') + + ' with the remedial steps and stop.', + run_dir=d) + p1_doc = _read_json(p1) or {} + if p1_doc.get('state') == 'do_not_generate': + return _emit('Phase 1 routed to do_not_generate but do_not_generate.md is missing.', + 'Write do_not_generate.md', 'llm_subagent', + inputs=[str(p1)], + output=str(d / 'do_not_generate.md'), + notes='Render the Phase 1 OOD rationale + remedial_steps as markdown; ' + 'that file is the run\'s final output.', + run_dir=d) + + # ---- Phase 2 (2.1 + 2.2 in ONE sub-agent) -------------------------------- + p2s = d / 'phase2_select' / 'phase2_select_output.json' + p2g = d / 'phase2_generate' / 'phase2_generate_output.json' + retry_note = '' + if (d / '.retry_used').exists() and (d / 'attempt_1').exists(): + retry_note = (' RETRY MODE: also pass ' + str(d / 'attempt_1/phase3_critique/phase3_critique_output.json') + + ' and ' + str(d / 'attempt_1/phase2_select/phase2_select_output.json') + + ' as negative constraints (see the OPTIONAL retry input in ideate_select.txt).') + if not p2s.exists() or not p2g.exists(): + if p2s.exists(): # only 2.2 left + return _emit('Phase 2.1 selection done; candidate not yet generated.', + 'Phase 2.2 — sub-pattern picking + candidate generation', 'llm_subagent', + prompt=str(prompts / 'ideate_generate.txt'), + inputs=[str(p2s), str(p1), str(p0 / 'lit_results.json'), + str(ref / 'ideation-sub-patterns') + '/.md'], + output=str(p2g), + notes='Immediately after: `next` runs the citation gate for you.' + retry_note, + run_dir=d) + return _emit('Phase 1 complete (state=proceed).', + 'Phase 2.1 + 2.2 — ONE sub-agent, TWO output files', 'llm_subagent', + prompt=str(prompts / 'ideate_select.txt') + ' THEN ' + str(prompts / 'ideate_generate.txt'), + inputs=[str(p1), + str(ref / 'ideation-patterns' / 'overview.md'), + str(ref / 'ideation-patterns' / 'companion-combos.md'), + str(p0 / 'lit_table.md'), + str(p0 / 'lit_results.json'), + str(ref / 'ideation-sub-patterns' / 'overview.md') + ' (+ picked C##.md cards)'], + output=f'{p2s} then {p2g}', + notes='Both phases are generation-side (no adversarial separation needed ' + 'between them — that separation is for 3.2/3.3 and 4.fill/4.1.5), so ' + 'one sub-agent runs 2.1, Writes its output, then continues into 2.2 ' + 'and Writes the candidate. Saves a sub-agent spin-up + duplicate ' + 'input reads. Routing signal: none.' + retry_note, + run_dir=d) + + # ---- deterministic citation gate (run inline — pure) ---------------------- + try: + from scripts.validators import validate_subpattern_citation_consistency + gate = validate_subpattern_citation_consistency(str(p2g)) + gate_fails = [f for f in gate if f.get('severity') == 'fail'] + except Exception as e: # never block `next` on a validator crash + print(f'(citation gate could not run: {e})', file=sys.stderr) + gate_fails = [] + if gate_fails: + msgs = '; '.join(f.get('message', '') for f in gate_fails[:3]) + return _emit('Phase 2.2 candidate FAILS the deterministic citation gate.', + 'Fix gap_closure[] sub_pattern citations before any Phase 3 work', 'llm_subagent', + prompt=str(ref / 'ideation-sub-patterns' / 'overview.md'), + inputs=[str(p2g)], + output=str(p2g) + ' (edited in place)', + notes=f'Validator findings: {msgs}. Fix the citation to a real C## cluster ' + 'row (or regenerate 2.2 with the card actually open). Do NOT proceed ' + 'to Phase 3 until `next` stops reporting this step.', + run_dir=d) + + # ---- Phase 3.1 collision --------------------------------------------------- + p3c_dir = d / 'phase3_collision' + if (p3c_dir / '.signature_extraction_pending').exists() and not (p3c_dir / 'collision_hits.json').exists(): + return _emit('Phase 3.1 stalled: candidate lacks signature_terms[].', + 'Fill signature_terms and re-invoke collision', 'llm_subagent', + prompt=str(ref / 'intent-recognition.md') + ' (Collision mode)', + inputs=[str(p2g)], + output=str(p2g) + ' (add signature_terms[] — 3-5 tight terms)', + run=[skill_cd + f'phase3_collision --idea-json "{p2g}" ' + f'--out "{p3c_dir}/"'], + run_dir=d) + if not (p3c_dir / 'collision_hits.json').exists(): + return _emit('Candidate passed the citation gate.', + 'Phase 3.1 — dual-channel collision retrieval (signature@10mo + alias@48mo)', 'bash', + run=[skill_cd + f'phase3_collision --idea-json "{p2g}" ' + f'--out "{p3c_dir}/"'], + notes='Takes minutes (openreview budgets 600s) — Bash timeout >= 600s or ' + 'background. If the command WARNs that alias_terms[] is missing, add ' + 'the field to the candidate JSON (2-4 cross-community names for the ' + 'mechanism; rubric: intent-recognition.md Collision mode) and re-run — ' + 'skipping it leaves the renamed-ancestor blind spot open.', + run_dir=d) + + # ---- Phase 3.2 audit -------------------------------------------------------- + p3q = d / 'phase3_critique' / 'phase3_critique_output.json' + if not p3q.exists(): + return _emit('Collision hits retrieved.', 'Phase 3.2 — audit-and-verdict (5 checks)', 'llm_subagent', + prompt=str(prompts / 'critique.txt'), + inputs=[str(p2g), str(p2s), str(p0 / 'lit_table.md'), + str(p3c_dir / 'collision_hits.json'), + str(ref / 'anti-patterns.md'), + str(ref / 'ideation-sub-patterns') + '/.md'], + output=str(p3q), + notes='Routing signal to return: `verdict` (advance | revise | abandon) + ' + 'verdict_rationale.', + run_dir=d) + p3q_doc = _read_json(p3q) or {} + verdict = p3q_doc.get('verdict') + + # ---- abandon → bounded internal retry --------------------------------------- + if verdict == 'abandon': + if not (d / '.retry_used').exists(): + arch = d / 'attempt_1' + mv_dirs = ' '.join(f'"{d / n}"' for n in + ('phase2_select', 'phase2_generate', 'phase3_collision', + 'phase3_critique', 'phase3_revise') if (d / n).exists()) + return _emit('Phase 3.2 verdict = abandon — ONE internal retry available ' + '(no user re-invocation; the one-shot guarantee constrains asking the ' + 'user, not internal regeneration).', + 'Archive attempt 1 and regenerate Phase 2.1+2.2 under negative constraints', + 'bash', + run=[f'mkdir -p "{arch}" && mv {mv_dirs} "{arch}/" && touch "{d}/.retry_used"'], + notes='Then re-run `next` — it will route to Phase 2.1+2.2 in retry mode ' + '(the archived audit + selection become negative constraints per ' + 'ideate_select.txt\'s OPTIONAL retry input). Phase 0/1 artifacts are ' + 'reused as-is.', + run_dir=d) + return _emit('Phase 3.2 verdict = abandon on the RETRY attempt — retry budget (1) exhausted.', + 'Write phase_3_failed.md', 'llm_subagent', + inputs=[str(p3q), + str(d / 'attempt_1' / 'phase3_critique' / 'phase3_critique_output.json')], + output=str(d / 'phase_3_failed.md'), + notes='Include BOTH attempts\' verdict_rationale + triggering checks + the ' + 'user-side options (drop direction / change framing / re-run with a ' + 'different direction). That file is the run\'s final output.', + run_dir=d) + + # ---- revise path -------------------------------------------------------------- + p3r_dir = d / 'phase3_revise' + p3r = p3r_dir / 'phase3_revise_output.json' + final_candidate = p3r_dir / 'final_candidate.json' + # Legacy runs back-inject final_candidate INTO the patch file without a + # sibling final_candidate.json; treat that as merged (kill_switch_integrity + # reads the inline key too). + merged = final_candidate.exists() or bool((_read_json(p3r) or {}).get('final_candidate')) + if verdict == 'revise': + if not p3r.exists(): + has_fals = any(isinstance(t, dict) and t.get('scope') == 'falsification' + for t in (p3q_doc.get('revision_targets') or [])) + fals_note = (' One revision_target has scope=falsification — emit ONE ' + 'rewrite_falsification entry for it (same experiment/metric/claim, ' + 'structure repaired).' if has_fals else '') + return _emit('Phase 3.2 verdict = revise.', 'Phase 3.3 — emit the revision patch', 'llm_subagent', + prompt=str(prompts / 'revise.txt'), + inputs=[str(p2g), str(p2s), str(p3q)], + output=str(p3r), + notes='Patch-only: applied_revisions[] — never echo the candidate.' + fals_note, + run_dir=d) + if not merged: + return _emit('Revision patch written; merger not yet run.', + 'Phase 3.3 merger (deterministic)', 'bash', + run=[skill_cd + 'phase3_merge_revisions ' + f'--phase2 "{p2g}" --revisions "{p3r}" --critique "{p3q}" ' + f'--out "{p3r_dir}/"'], + run_dir=d) + p3r_doc = _read_json(p3r) or {} + reaudit = d / 'phase3_critique' / 'falsification_reaudit.json' + if p3r_doc.get('falsification_rewritten'): + if not reaudit.exists(): + return _emit('falsification_prediction was rewritten (audited exception) — ' + 're-audit REQUIRED before Phase 4.', + 'Falsification re-audit (single-check)', 'llm_subagent', + prompt=str(prompts / 'critique.txt') + ' — section "Falsification re-audit mode" ONLY', + inputs=[str(final_candidate)], + output=str(reaudit), + notes='Routing signal: `verdict` (advance | abandon). Exactly one ' + 'rewrite attempt per run — deficient again means abandon.', + run_dir=d) + re_doc = _read_json(reaudit) or {} + if re_doc.get('verdict') == 'abandon': + return _emit('Falsification re-audit verdict = abandon (rewrite still deficient).', + 'Write phase_3_failed.md', 'llm_subagent', + inputs=[str(p3q), str(reaudit)], + output=str(d / 'phase_3_failed.md'), + notes='Name the original structural deficiency AND why the one ' + 'permitted rewrite still fails. That file is the run\'s final output.', + then=True, run_dir=d) + + # ---- Phase 4 ------------------------------------------------------------------- + p4_dir = d / 'phase4' + on_revise_path = verdict == 'revise' and merged + # Legacy runs without the sibling file fall back to the patch file (the + # skeleton unwraps an inline `final_candidate` key itself). + candidate_path = ((final_candidate if final_candidate.exists() else p3r) + if on_revise_path else p2g) + expansion_done = (p4_dir / 'phase4_expansion.json').exists() + # Skeleton/fill are only prerequisites while the expansion doesn't exist yet + # (pre-skeleton-era runs have an expansion but no skeleton/fill_map — don't + # send those back to rebuild artifacts the pipeline no longer needs). + if not expansion_done and not (p4_dir / 'phase4_skeleton.json').exists(): + cmd = (skill_cd + 'phase4_skeleton ' + f'--candidate "{candidate_path}" --phase1 "{p1}" --phase2-select "{p2s}" ' + f'--phase3-critique "{p3q}" ') + if on_revise_path: + cmd += f'--phase3-revise "{p3r}" ' + cmd += (f'--phase0-dir "{p0}/" --collision "{p3c_dir / "collision_hits.json"}" ' + f'--out "{p4_dir}/"') + return _emit(f'Gauntlet cleared ({verdict} path).', 'Phase 4 skeleton (deterministic)', 'bash', + run=[cmd], run_dir=d) + if not expansion_done and not (p4_dir / 'fill_map.json').exists(): + return _emit('Skeleton built.', 'Phase 4.fill — author the prose TODOs', 'llm_subagent', + prompt=str(prompts / 'expand.txt'), + inputs=[str(p4_dir / 'phase4_skeleton.json')], + output=str(p4_dir / 'fill_map.json'), + notes='Flat {TODO-path: prose} map ONLY — the assembler refuses kill-switch ' + 'roots. This is the most timeout-prone call: NEVER run it in the ' + 'parent context.', + run_dir=d) + if not expansion_done: + return _emit('fill_map written.', 'Phase 4 assemble (deterministic)', 'bash', + run=[skill_cd + 'phase4_assemble ' + f'--skeleton "{p4_dir / "phase4_skeleton.json"}" ' + f'--fill-map "{p4_dir / "fill_map.json"}" --out "{p4_dir}/"'], + run_dir=d) + if not (p4_dir / 'phase4_implementability.json').exists(): + return _emit('Expansion assembled.', 'Phase 4.1.5 — implementability audit', 'llm_subagent', + prompt=str(prompts / 'implementability_audit.txt'), + inputs=[str(p4_dir / 'phase4_expansion.json')], + output=str(p4_dir / 'phase4_implementability.json'), + notes='Fresh skeptical-engineer persona (separate call from 4.fill). ' + 'Compute-agnostic by design.', + run_dir=d) + + # ---- validate + render ----------------------------------------------------------- + phase3_for_validate = p3r if on_revise_path else p3q + return _emit('All Phase 4 JSONs present; cards not yet rendered.', + 'Validate, then render the idea cards', 'bash', + run=[skill_cd + 'validate ' + f'--phase2 "{p2g}" --phase3 "{phase3_for_validate}" ' + f'--phase4 "{p4_dir / "phase4_expansion.json"}" ' + f'--phase4-impl "{p4_dir / "phase4_implementability.json"}"', + skill_cd + 'phase4_render ' + f'--expansion "{p4_dir / "phase4_expansion.json"}" --out "{p4_dir}/"'], + notes='On a validate `fail`: fix only the named contract and re-validate — cap ' + '2 retries, then render as-is with a caveat note (never edit ' + 'kill_switch/citation-guarded fields to silence a validator).', + run_dir=d) + + +def cmd_next(args) -> int: + run_dir = Path(args.dir).resolve() + if not run_dir.exists(): + print(f'ERROR: run dir {run_dir} does not exist. Create it first ' + f'(mkdir -p) — it is the --out root every phase writes under.', file=sys.stderr) + return 2 + root = Path(__file__).resolve().parent.parent + return next_step(run_dir, root, getattr(args, 'query', None) or None) diff --git a/ResearchStudio-Idea/skills/idea_spark/scripts/phase4_skeleton.py b/ResearchStudio-Idea/skills/idea_spark/scripts/phase4_skeleton.py index ca213d8..0a2d559 100644 --- a/ResearchStudio-Idea/skills/idea_spark/scripts/phase4_skeleton.py +++ b/ResearchStudio-Idea/skills/idea_spark/scripts/phase4_skeleton.py @@ -364,6 +364,26 @@ def build_reviewer_concerns(phase3_critique: dict, 'fields_changed_to_address': [], # filled below from phase3_revise.applied_revisions[] }) + # 1b. paper_pointed_threat.parametric_family_concern — the audit's soft signal + # that an older, named mechanism family exists outside the retrieved pool. + # Never gates the verdict; here it becomes an explicit "run a scoop-check on + # this vocabulary before investing" reviewer concern so the card carries the + # known blind spot instead of silently claiming full novelty coverage. + pfc = ppt.get('parametric_family_concern') + if pfc and isinstance(pfc, str) and pfc.strip().lower() not in ('null', 'none', 'n/a'): + entries.append({ + 'attack': (f"Un-retrieved mechanism family flagged by the audit (parametric " + f"knowledge, not in the retrieved pool): {pfc.strip()} — novelty vs " + f"this family is UNVERIFIED; run a targeted scoop-check on that " + f"vocabulary before investing."), + 'severity': 'non_blocking', + 'response': TODO(f'reviewer_concerns_and_responses[{len(entries)}].response', + '1-2 sentences: state what a scoop-check on the named family must ' + 'establish for the candidate\'s delta to survive (do NOT claim the ' + 'check already passed)'), + 'fields_changed_to_address': [], + }) + # 2. gap_closure_reject_check borderline entries gcrc = phase3_critique.get('gap_closure_reject_check') or {} for j, entry in enumerate(gcrc.get('entries', [])): diff --git a/ResearchStudio-Idea/skills/idea_spark/scripts/run.py b/ResearchStudio-Idea/skills/idea_spark/scripts/run.py index ab08b81..0b29bf7 100644 --- a/ResearchStudio-Idea/skills/idea_spark/scripts/run.py +++ b/ResearchStudio-Idea/skills/idea_spark/scripts/run.py @@ -233,6 +233,21 @@ def _warn_degraded_connectors(out_dir: Path, phase: str, available_labels: list[ 'openreview': {'window_min': 0, 'window_max': 6, 'max_results': 10, 'max_per_query': 50, 'extra_args': [], 'timeout': 600}, } +# Phase 3.1 collision retrieval windows (months). Two channels: +# - signature channel: the candidate's OWN vocabulary (signature_terms) over a +# recent window — catches contemporaneous scoops. Phase 0's broad-domain +# queries cover 0-24mo but by TOPIC, not mechanism; 10mo (up from 6) narrows +# the mechanism-specific gap at negligible retrieval cost. +# - alias channel: OTHER communities' names for the same mechanism (alias_terms, +# produced from parametric knowledge at Phase 2.2) over a multi-year window — +# catches renamed ancestors. Widening the signature window alone cannot do +# this: a same-mechanism paper from another community 2-3 years back uses +# vocabulary the signature terms never contain (the "goal-conditioned success +# detector vs goal-image conditioned scorer" failure mode), so the blind spot +# is lexical, not temporal. +COLLISION_WINDOW_MONTHS = 10 +ALIAS_COLLISION_WINDOW_MONTHS = 48 + # Dedup priority when the same paper appears in multiple connectors. # Higher priority means: keep this connector's record, drop the others. # Semantic Scholar wins because its `externalIds` block (DOI + ArXiv + DBLP keys all in one record) @@ -469,10 +484,11 @@ def cmd_phase0(args) -> int: 'most-similar-problem, optional query 4 application-angle, optional query 5 ' 'venue-insider. ALSO: scan user_query for paper-title references the user ' 'intends as anchors (e.g., "based on Sora", "extending CycleResearcher", ' - '"the LoRA paper") and write them to `user_refs.json` in out_dir, appending ' - 'to whatever URL/ID-based refs the regex pass already populated. Each title ' - 'entry: {"type": "title", "value": "", "raw_match": ""}. ' - 'Skip this step if no titles are mentioned (URL/ID refs are handled by regex). ' + '"the LoRA paper") and register each via ' + '`python3 /scripts/run.py add_user_ref --out --title "" ' + '--raw-match ""` (deterministic merge into user_refs.json; do NOT ' + 'hand-edit the file). Skip this step if no titles are mentioned (URL/ID refs are ' + 'handled by regex). ' 'Apply the OOD short-circuit: if user_query matches the parent skill\'s ' 'intake-routing.md trigger #1 (Too broad) or #2 (No anchor), return ' '{"ood": true, "trigger_id": ..., "trigger_quote": ..., "match_evidence": ...} ' @@ -658,48 +674,93 @@ def cmd_phase3_collision(args) -> int: print('ERROR: no connector available for collision check.', file=sys.stderr) return 2 - # Build collision query from signature_terms. If the idea.json is missing them, - # emit a sentinel for the host LLM to fill in (same pattern as Phase 0 intent extraction). - # Falling back to [title, core_mechanism, novelty_claim] is dangerous because those are - # long sentences that fail URL encoding at the connector layer (recurring Bug A pattern). + # Build collision queries from signature_terms (+ alias_terms). If the idea.json is + # missing signature_terms, emit a sentinel for the host LLM to fill in (same pattern as + # Phase 0 intent extraction). Falling back to [title, core_mechanism, novelty_claim] is + # dangerous because those are long sentences that fail URL encoding at the connector layer. sig = idea.get('signature_terms') if not sig: return emit_host_llm_sentinel( out_dir, step_name='signature_extraction', rubric_file=SUB / 'references' / 'intent-recognition.md', inputs={'idea_json': args.idea_json, 'mode': 'collision'}, - expected_outputs=['edit idea_json to add signature_terms[]'], + expected_outputs=['edit idea_json to add signature_terms[] AND alias_terms[]'], instruction=( 'Read the rubric file whose absolute path is in this sentinel\'s `rubric_file` ' - 'field (Collision mode). Produce 3-5 signature_terms from the idea (mechanism + ' - 'claim + setting + 1-2 specific identifiers, each 3-7 words). Add a ' - '`signature_terms` field to idea_json and re-invoke. Long sentences (title / ' - 'core_mechanism / novelty_claim verbatim) fail URL encoding at the connector — ' - 'keep terms tight.' + 'field (Collision mode). Produce BOTH term sets: 3-5 signature_terms (mechanism + ' + 'claim + setting + 1-2 specific identifiers, each 3-7 words, the candidate\'s own ' + 'vocabulary) AND 2-4 alias_terms (other communities\' names for the same mechanism ' + '— parametric knowledge, not paraphrase). Add both fields to idea_json and ' + 're-invoke. Long sentences (title / core_mechanism verbatim) fail URL encoding at ' + 'the connector — keep terms tight.' ), re_invocation=f'python3 -m scripts.run phase3_collision --idea-json {args.idea_json} --out {out_dir}', exit_code=11, ) - queries_json = json.dumps([s for s in sig if s]) - hits_files = [] - for label, module_path in available: - out_path = out_dir / f'{label}_collision.json' - # Inherit per-connector timeout from PHASE0_CONNECTOR_CONFIG (notably: openreview gets 600s, - # not the function-default 300s, since iterate-notes cost is the same in collision retrieval). - timeout = PHASE0_CONNECTOR_CONFIG.get(label, {}).get('timeout', 300) - ok = run_connector_subprocess(module_path, queries_json, 6, out_path, f'{label}_6mo', timeout=timeout, as_of=as_of) - if ok: - hits_files.append(out_path) - - if not hits_files: + alias = [a for a in (idea.get('alias_terms') or []) if a] + if not alias: + print('\nWARNING: candidate has no alias_terms[] — the cross-vocabulary collision channel ' + f'(same mechanism under other communities\' names, {ALIAS_COLLISION_WINDOW_MONTHS}mo ' + 'window) is SKIPPED. Renamed same-mechanism ancestors will NOT be checked. Add ' + 'alias_terms[] to the candidate JSON (rubric: intent-recognition.md Collision mode) ' + 'and re-run to close this blind spot.\n', file=sys.stderr) + + def _run_channel(terms: list, window_months: int, suffix: str) -> list: + queries_json = json.dumps([t for t in terms if t]) + files = [] + for label, module_path in available: + out_path = out_dir / f'{label}_{suffix}.json' + # Inherit per-connector timeout from PHASE0_CONNECTOR_CONFIG (notably: openreview + # gets 600s, since iterate-notes cost is the same in collision retrieval). + timeout = PHASE0_CONNECTOR_CONFIG.get(label, {}).get('timeout', 300) + if run_connector_subprocess(module_path, queries_json, window_months, out_path, + f'{label}_{suffix}_{window_months}mo', + timeout=timeout, as_of=as_of): + files.append(out_path) + return files + + sig_files = _run_channel(sig, COLLISION_WINDOW_MONTHS, 'collision') + alias_files = _run_channel(alias, ALIAS_COLLISION_WINDOW_MONTHS, 'alias_collision') if alias else [] + + if not sig_files and not alias_files: print('ERROR: all collision retrievals failed.', file=sys.stderr) return 3 - # Dedup-merge to a single collision_hits.json. Phase 3.1 = retrieval + dedup only; - # Phase 3.2 audit's paper-pointed threat search does subsumption judgment. + # Dedup each channel, then cross-channel merge with per-hit channel tags (signature wins + # on overlap — a paper found by the candidate's own vocabulary is the stronger threat + # signal for the audit). Phase 3.1 = retrieval + dedup only; Phase 3.2 audit's + # paper-pointed threat search does subsumption judgment. + def _dedup_channel(files: list, out_name: str) -> list: + if not files: + return [] + merged = out_dir / out_name + cmd = [sys.executable, '-m', 'scripts.dedup_merge', '--inputs'] + \ + [str(f) for f in files] + ['--out', str(merged)] + subprocess.run(cmd, cwd=str(SUB), check=True, capture_output=True, text=True, timeout=120) + try: + return json.loads(merged.read_text()) + except Exception: + return [] + + def _title_norm(t) -> str: + return ' '.join(''.join(c.lower() if c.isalnum() else ' ' for c in (t or '')).split()) + + sig_hits = _dedup_channel(sig_files, '.sig_channel_hits.json') + alias_hits = _dedup_channel(alias_files, '.alias_channel_hits.json') + for h in sig_hits: + h['collision_channel'] = 'signature' + seen_titles = {_title_norm(h.get('title')) for h in sig_hits} + n_alias_new = 0 + for h in alias_hits: + if _title_norm(h.get('title')) not in seen_titles: + h['collision_channel'] = 'alias' + sig_hits.append(h) + n_alias_new += 1 merged_out = out_dir / 'collision_hits.json' - dedup_cmd = [sys.executable, '-m', 'scripts.dedup_merge', '--inputs'] + [str(f) for f in hits_files] + ['--out', str(merged_out)] - subprocess.run(dedup_cmd, cwd=str(SUB), check=True, capture_output=True, text=True, timeout=120) + merged_out.write_text(json.dumps(sig_hits, ensure_ascii=False, indent=1)) + print(f' channels: signature={len(sig_hits) - n_alias_new} hits ' + f'({COLLISION_WINDOW_MONTHS}mo), alias=+{n_alias_new} unique hits ' + f'({ALIAS_COLLISION_WINDOW_MONTHS}mo{", SKIPPED" if not alias else ""})', file=sys.stderr) # Slim collision_hits.json for the LLM-facing reader (Phase 3.2 audit). Full # abstracts (~1.5k chars x ~275 papers) blow the audit prompt to ~200k tokens; @@ -710,7 +771,8 @@ def cmd_phase3_collision(args) -> int: _hits = json.loads(merged_out.read_text()) if _COLLISION_MAX_HITS: _hits = sorted(_hits, key=lambda x: -(x.get('semantic_recall') or 0))[:_COLLISION_MAX_HITS] - _keep = ('title', 'paper_id', 'venue', 'year', 'tldr', 'semantic_recall', 'source') + _keep = ('title', 'paper_id', 'venue', 'year', 'tldr', 'semantic_recall', 'source', + 'collision_channel') def _slim(h): o = {k: h[k] for k in _keep if h.get(k) is not None} a = h.get('abstract') or '' @@ -1044,24 +1106,100 @@ def _cmd_phase4_assemble(args): pm.add_argument('--revisions', required=True, help='Path to phase3_revise_output.json containing applied_revisions[]. ' 'The file is updated in place to add a `final_candidate` key.') + pm.add_argument('--critique', default=None, + help='Path to phase3_critique_output.json. Required to authorize a ' + '`rewrite_falsification` patch op (the merger verifies the audit ' + 'emitted a scope=falsification revision_target). Optional otherwise.') pm.add_argument('--out', required=True, help='Output dir for final_candidate.json (typically the same dir as --revisions).') def _cmd_phase3_merge(args): from scripts.merge_revisions import merge_phase3_revisions try: - final_path, _ = merge_phase3_revisions( + final_path, revisions_path = merge_phase3_revisions( Path(args.phase2).resolve(), Path(args.revisions).resolve(), Path(args.out).resolve(), + critique_path=Path(args.critique).resolve() if args.critique else None, ) except ValueError as e: print(f'ERROR: {e}', file=sys.stderr) return 1 print(f'✅ Phase 3.3 merge complete. Wrote {final_path}', file=sys.stderr) print(f' Back-injected `final_candidate` into {args.revisions} for legacy consumers.', file=sys.stderr) + try: + if json.loads(Path(revisions_path).read_text()).get('falsification_rewritten'): + print(' ⚠️ falsification_prediction was REWRITTEN (audited exception). ' + 'Before Phase 4, run the falsification re-audit ' + '(critique.txt "Falsification re-audit mode") → ' + 'phase3_critique/falsification_reaudit.json with verdict=advance.', + file=sys.stderr) + except Exception: + pass return 0 pm.set_defaults(func=_cmd_phase3_merge) + pu = sub.add_parser('add_user_ref', + help='Merge a user-named paper reference into phase0/user_refs.json ' + '(deterministic JSON merge, dedup on type:value; creates the file ' + 'if absent). Use this for TITLE-based references the phase0 regex ' + 'cannot extract ("based on the LoRA paper") — it avoids the host ' + 'Write-tool read-before-overwrite rule entirely. Run BEFORE ' + 'phase0_fulltext so the ref lands in the U fetch tier.') + pu.add_argument('--out', required=True, + help='Phase 0 output dir containing user_refs.json (dir created if missing)') + pu.add_argument('--title', action='append', default=[], + help='Paper title reference (repeatable for multiple papers)') + pu.add_argument('--id', action='append', default=[], dest='ref_id', + help='arxiv id / DOI / OpenReview id / URL (repeatable); type auto-detected ' + 'via the same extractor phase0 uses on the query string') + pu.add_argument('--raw-match', default='', + help='The user phrasing that named the paper (provenance; recorded when ' + 'exactly one --title is given)') + def _cmd_add_user_ref(args): + if not args.title and not args.ref_id: + print('nothing to add: pass --title and/or --id', file=sys.stderr) + return 2 + out_dir = Path(args.out).resolve() + out_dir.mkdir(parents=True, exist_ok=True) + path = out_dir / 'user_refs.json' + try: + existing = json.loads(path.read_text()) if path.exists() else [] + except Exception: + existing = [] + if not isinstance(existing, list): + existing = [] + new = [] + for t in args.title: + entry = {'type': 'title', 'value': t} + if args.raw_match and len(args.title) == 1: + entry['raw_match'] = args.raw_match + new.append(entry) + if args.ref_id: + from scripts.extract_user_refs import extract_refs_from_query + for rid in args.ref_id: + hits = extract_refs_from_query(rid) + new.extend(hits if hits else [{'type': 'title', 'value': rid, + 'raw_match': 'unrecognized id format; stored as title'}]) + seen = {f"{r.get('type', '')}:{r.get('value', '')}" for r in existing if isinstance(r, dict)} + added = [r for r in new if f"{r.get('type', '')}:{r.get('value', '')}" not in seen] + path.write_text(json.dumps(existing + added, indent=2, ensure_ascii=False)) + print(f"added {len(added)} ref(s), skipped {len(new) - len(added)} duplicate(s) → {path}", + file=sys.stderr) + return 0 + pu.set_defaults(func=_cmd_add_user_ref) + + pn = sub.add_parser('next', + help='Run-state navigator: inspect the run dir\'s artifacts and print ' + 'EXACTLY the next step (a bash command, or an LLM sub-agent spec ' + 'with prompt/inputs/output paths). Read-only. The host loop is: ' + '`next` → do what it says → `next` again, until a terminal state.') + pn.add_argument('--dir', required=True, help='The run dir (the --out root all phases write under)') + pn.add_argument('--query', default='', help='The user research question (only used to fill in the phase0 command hint)') + def _cmd_next(args): + from scripts.next_step import cmd_next + return cmd_next(args) + pn.set_defaults(func=_cmd_next) + pv = sub.add_parser('validate', help='Run validators on phase outputs') pv.add_argument('--phase1', help='phase1_output.json path (required for V3 evidence-chain)') pv.add_argument('--phase2', help='phase2_output.json path (required for V2/V3/V4)') @@ -1076,9 +1214,9 @@ def _cmd_phase3_merge(args): # actionable message instead of a confusing FileNotFoundError mid-phase. for _attr in ('out', 'idea_json', 'expansion', 'implementability', 'phase1', 'phase2', 'phase3', 'phase4', 'phase4_impl', - 'revisions', 'candidate', 'phase2_select', 'phase3_critique', + 'revisions', 'critique', 'candidate', 'phase2_select', 'phase3_critique', 'phase3_revise', 'phase0_dir', 'collision', - 'skeleton', 'fill_map'): + 'skeleton', 'fill_map', 'dir'): _val = getattr(args, _attr, None) if _val: _guard_project_path(_val, f'--{_attr.replace("_", "-")}') diff --git a/ResearchStudio-Idea/skills/idea_spark/scripts/validators/kill_switch_integrity.py b/ResearchStudio-Idea/skills/idea_spark/scripts/validators/kill_switch_integrity.py index 95371fa..a837ffd 100644 --- a/ResearchStudio-Idea/skills/idea_spark/scripts/validators/kill_switch_integrity.py +++ b/ResearchStudio-Idea/skills/idea_spark/scripts/validators/kill_switch_integrity.py @@ -15,6 +15,15 @@ This validator accepts phase3_path optionally: if Phase 3 contains a final_candidate (3.3 ran), it validates the 3-link chain; otherwise it treats Phase 3 as passthrough and checks Phase 2 → Phase 4. +Audited falsification rewrite (the ONE sanctioned kill-switch change): when the Phase 3.2 +audit's falsification_structure_check found the paragraph structurally deficient, Phase 3.3 +may carry a `rewrite_falsification` op that the merger applies under audit authorization and +records as `falsification_rewritten: true` in the patch doc. On that path this validator +checks `falsification_prediction` byte-identity on the Phase 3.3 final_candidate → Phase 4 +link (the rewritten paragraph is the new commitment) instead of Phase 2 → Phase 4, and it +FAILS if the rewrite marker is present without an actual applied rewrite_falsification entry +(or vice versa). `compute_budget` keeps the full Phase 2 → 3.3 → 4 byte-identity always. + Why it matters: a misbehaving Phase 3.3 / Phase 4 model could substitute the kill-switch experiment with an easier one ("we'll use a simpler dataset / smaller compute"), making the candidate look more feasible than the original Phase 2 candidate committed to. This validator @@ -136,8 +145,37 @@ def validate_kill_switch_integrity(phase2_path, phase3_path, phase4_path) -> lis has_phase3_chain = (final_candidate is not None and isinstance(final_candidate, dict) and isinstance(final_candidate.get('falsification_prediction'), str)) + # Audited falsification rewrite detection: the merger stamps `falsification_rewritten` + # into the patch doc AND the patch must carry a matching applied rewrite_falsification + # entry. Marker/entry disagreement is a hard fail (someone hand-edited the patch). + rewrite_marker = bool(isinstance(p3, dict) and p3.get('falsification_rewritten')) + rewrite_entries = [ + r for r in (p3.get('applied_revisions') or []) + if isinstance(r, dict) and r.get('op') == 'rewrite_falsification' + and not str(r.get('outcome', '')).startswith('skipped_') + ] if isinstance(p3, dict) else [] + if rewrite_marker != bool(rewrite_entries): + findings.append({ + 'severity': 'fail', 'validator': 'kill_switch_integrity', + 'message': ('falsification_rewritten marker and applied rewrite_falsification patch ' + 'entries disagree (marker={}, entries={}) — the patch file was modified ' + 'outside the merger; re-run phase3_merge_revisions').format( + rewrite_marker, len(rewrite_entries)), + }) + falsification_rewritten = rewrite_marker and bool(rewrite_entries) + if falsification_rewritten and not has_phase3_chain: + findings.append({ + 'severity': 'fail', 'validator': 'kill_switch_integrity', + 'message': 'falsification_rewritten is set but Phase 3 carries no well-typed ' + 'final_candidate — merger output is incomplete', + }) + for field_path in KILL_SWITCH_FIELDS: field_name = '.'.join(field_path) + # The audited rewrite re-bases falsification_prediction's commitment at Phase 3.3: + # the byte-identity anchor becomes the final_candidate, not the Phase 2 candidate. + rebased = falsification_rewritten and field_name == 'falsification_prediction' and has_phase3_chain + v2, p2_finding = _check_kill_switch_value(winner_candidate, field_path, 'Phase 2 candidate', field_name) if p2_finding is not None: findings.append(p2_finding) @@ -147,6 +185,29 @@ def validate_kill_switch_integrity(phase2_path, phase3_path, phase4_path) -> lis findings.append(p4_finding) continue + if rebased: + v3, p3_finding = _check_kill_switch_value(final_candidate, field_path, 'Phase 3 final_candidate', field_name) + if p3_finding is not None: + findings.append(p3_finding) + continue + if v3 == v2: + findings.append({ + 'severity': 'fail', 'validator': 'kill_switch_integrity', + 'message': f'{field_name}: falsification_rewritten is set but Phase 3 ' + f'final_candidate is byte-identical to Phase 2 — no rewrite ' + f'actually landed; re-run phase3_merge_revisions', + }) + continue + if v3 != v4: + findings.append({ + 'severity': 'fail', 'validator': 'kill_switch_integrity', + 'message': f'{field_name} drifted between Phase 3 final_candidate (audited ' + f'rewrite) and Phase 4 expansion', + 'phase3_value': v3[:120] + ('…' if len(v3) > 120 else ''), + 'phase4_value': v4[:120] + ('…' if len(v4) > 120 else ''), + }) + continue + # Primary check: Phase 2 → Phase 4 byte-identical if v2 != v4: findings.append({ @@ -171,6 +232,12 @@ def validate_kill_switch_integrity(phase2_path, phase3_path, phase4_path) -> lis }) if not findings: - chain_desc = 'Phase 2 → 3.3 final_candidate → 4 (revise path)' if has_phase3_chain else 'Phase 2 → 4 (Phase 3 passthrough, advance path)' + if falsification_rewritten: + chain_desc = ('Phase 2 → 3.3 → 4 (revise path; falsification_prediction re-based at ' + '3.3 via audited rewrite, compute_budget full-chain)') + elif has_phase3_chain: + chain_desc = 'Phase 2 → 3.3 final_candidate → 4 (revise path)' + else: + chain_desc = 'Phase 2 → 4 (Phase 3 passthrough, advance path)' findings.append({'severity': 'pass', 'validator': 'kill_switch_integrity', 'message': f'All 2 kill-switch fields byte-identical across {chain_desc}'}) return findings