From 31211a0e7c512d63972b4f0438877d3777ae0e85 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 6 Jul 2026 02:34:28 +0100 Subject: [PATCH 001/292] docs(readme): visual overhaul (hero, capability table, query demo, honest benchmarks table) Rework the top of the README into a self-contained pitch: - New logo + Trendshift badge; collapse the 31-language row into a toggle so the tagline and hero land in the first screen. - Hero screenshot of the FastAPI graph, tagline with selective bold. - 'What it does' capability table and a 'See it in action' section with verbatim explain/path output (real relations + confidence tags). - Benchmarks as a scannable table, honest: shows where competitors lead, only bolds rows graphify wins or ties, names the tie. - Collapse platform picker + optional extras into
; typography and identifier-formatting polish; fix orphaned paragraphs; reword the confidence-tag gloss; drop the redundant callflow CTA from the hero. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 90 ++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 70 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index c9e4ad701..ea3c862e3 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,17 @@

- 🇺🇸 English | 🇨🇳 简体中文 | 🇯🇵 日本語 | 🇰🇷 한국어 | 🇩🇪 Deutsch | 🇫🇷 Français | 🇪🇸 Español | 🇮🇳 हिन्दी | 🇧🇷 Português | 🇷🇺 Русский | 🇸🇦 العربية | 🇮🇷 فارسی | 🇮🇹 Italiano | 🇵🇱 Polski | 🇳🇱 Nederlands | 🇹🇷 Türkçe | 🇺🇦 Українська | 🇻🇳 Tiếng Việt | 🇮🇩 Bahasa Indonesia | 🇸🇪 Svenska | 🇬🇷 Ελληνικά | 🇷🇴 Română | 🇨🇿 Čeština | 🇫🇮 Suomi | 🇩🇰 Dansk | 🇳🇴 Norsk | 🇭🇺 Magyar | 🇹🇭 ภาษาไทย | 🇺🇿 Oʻzbekcha | 🇹🇼 繁體中文 | 🇵🇭 Filipino + Graphify-Labs%2Fgraphify | Trendshift

+
+
Read this in other languages + +🇺🇸 English | 🇨🇳 简体中文 | 🇯🇵 日本語 | 🇰🇷 한국어 | 🇩🇪 Deutsch | 🇫🇷 Français | 🇪🇸 Español | 🇮🇳 हिन्दी | 🇧🇷 Português | 🇷🇺 Русский | 🇸🇦 العربية | 🇮🇷 فارسی | 🇮🇹 Italiano | 🇵🇱 Polski | 🇳🇱 Nederlands | 🇹🇷 Türkçe | 🇺🇦 Українська | 🇻🇳 Tiếng Việt | 🇮🇩 Bahasa Indonesia | 🇸🇪 Svenska | 🇬🇷 Ελληνικά | 🇷🇴 Română | 🇨🇿 Čeština | 🇫🇮 Suomi | 🇩🇰 Dansk | 🇳🇴 Norsk | 🇭🇺 Magyar | 🇹🇭 ภาษาไทย | 🇺🇿 Oʻzbekcha | 🇹🇼 繁體中文 | 🇵🇭 Filipino + +
+
+

YC S26 Discord @@ -18,7 +26,7 @@ X

-Type `/graphify` in your AI coding assistant and it maps your entire project — code, docs, PDFs, images, videos — into a knowledge graph you can query instead of grepping through files. +Type `/graphify` in your AI coding assistant and it maps your entire project (code, docs, PDFs, images, videos) into a **knowledge graph** you can **query instead of grepping** through files.

graphify's interactive graph.html showing the FastAPI codebase as a force-directed knowledge graph with a legend of detected communities @@ -27,13 +35,13 @@ Type `/graphify` in your AI coding assistant and it maps your entire project — The FastAPI codebase mapped by graphify. Every node is a concept, colors are detected communities, and the whole thing is clickable in graph.html.

-Works in Claude Code, Codex, OpenCode, Kilo Code, Cursor, Gemini CLI, GitHub Copilot CLI, VS Code Copilot Chat, Aider, Amp, OpenClaw, Factory Droid, Trae, Hermes, Kimi Code, Kiro, Pi, Devin CLI, and Google Antigravity. +**Works in:** Claude Code, Codex, OpenCode, Kilo Code, Cursor, Gemini CLI, GitHub Copilot CLI, VS Code Copilot Chat, Aider, Amp, OpenClaw, Factory Droid, Trae, Hermes, Kimi Code, Kiro, Pi, Devin CLI, and Google Antigravity. ``` /graphify . ``` -That's it. You get three files: +That's it. You get **three files**: ``` graphify-out/ @@ -42,12 +50,49 @@ graphify-out/ └── graph.json the full graph — query it anytime without re-reading your files ``` -For a readable architecture page with Mermaid call-flow diagrams, run: +--- -```bash -graphify export callflow-html +## What it does + +A knowledge graph is not a vector index: it is **deterministic**, **every edge is explained**, and it costs **zero tokens to build**. + +| Capability | What you get | +|---|---| +| **God nodes** | The most-connected concepts, so you see what everything flows through | +| **Communities** | The graph split into subsystems (Leiden), with LLM-free labels | +| **Cross-file links** | `calls` / `imports` / `inherits` / `mixes_in` resolved across ~40 languages via tree-sitter AST | +| **Query, path, explain** | Ask a question, trace the path between two things, or explain one concept, all against `graph.json` | +| **Rationale + doc refs** | `# NOTE:` / `# WHY:` comments and ADR/RFC citations become first-class nodes linked to the code | +| **Beyond code** | Docs, PDFs, images, and video/audio all map into the same graph | +| **Local-first** | AST plus a local embedder build the graph with zero LLM credits; your code never leaves your machine | + +--- + +## See it in action + +Once the graph is built you query it instead of reading files. Real output, graphify run on the FastAPI codebase shown above: + +```text +$ graphify explain "APIRouter" +Node: APIRouter + Source: routing.py L2210 + Community: 2 + Degree: 47 + +Connections (47): + --> RequestValidationError [uses] [INFERRED] + --> Dependant [uses] [INFERRED] + --> .get() [method] [EXTRACTED] + <-- __init__.py [imports] [EXTRACTED] + ... + +$ graphify path "FastAPI" "ModelField" +Shortest path (3 hops): + FastAPI --uses--> DefaultPlaceholder <--references-- get_request_handler() --references--> ModelField ``` +Every edge carries a **confidence tag** (`EXTRACTED` = explicit in the source, `INFERRED` = derived by resolution), so you can tell what was read directly from what was inferred. `graphify query ""` returns a scoped subgraph for a plain-language question, and `graphify path A B` traces how any two things connect. + --- ## Benchmarks @@ -55,8 +100,8 @@ graphify export callflow-html | Benchmark | Metric | graphify | Field | |---|---|---|---| | LOCOMO (n=300) | recall@10 | **0.497** | mem0 0.048, supermemory 0.149 | -| LOCOMO (n=300) | QA accuracy | **45.3%** | supermemory 49.7%, mem0 27.3% | -| LongMemEval-S (n=50) | QA accuracy | **76%** | tied best | +| LOCOMO (n=300) | QA accuracy | 45.3% | supermemory 49.7%, mem0 27.3% | +| LongMemEval-S (n=50) | QA accuracy | **76%** | tied with dense RAG | | Graph build | LLM credits | **0** | per-token for most systems | Every system ran on the same harness with the same model and budgets, scored by a judge blind-validated against a second judge (90.6% agreement, Cohen's kappa 0.81). Full per-system tables, the code-intelligence result, and reproduction commands: **[BENCHMARKS.md](./BENCHMARKS.md)**. @@ -167,7 +212,7 @@ for example `graphify claude install --project` or `graphify codex install --pro | Devin CLI | `graphify devin install` | | Google Antigravity | `graphify antigravity install` | -Codex users also need `multi_agent = true` under `[features]` in `~/.codex/config.toml` for parallel extraction. CodeBuddy uses the same Agent tool and PreToolUse hook mechanism as Claude Code. Factory Droid uses the `Task` tool for parallel subagent dispatch. OpenClaw and Aider use sequential extraction (parallel agent support is still early on those platforms). Trae uses the Agent tool for parallel subagent dispatch and does **not** support PreToolUse hooks — AGENTS.md is the always-on mechanism. +Codex users also need `multi_agent = true` under `[features]` in `~/.codex/config.toml` for parallel extraction. CodeBuddy uses the same Agent tool and PreToolUse hook mechanism as Claude Code. Factory Droid uses the `Task` tool for parallel subagent dispatch. OpenClaw and Aider use sequential extraction (parallel agent support is still early on those platforms). Trae uses the Agent tool for parallel subagent dispatch and does **not** support `PreToolUse` hooks, so AGENTS.md is the always-on mechanism. `--platform agents` (alias `--platform skills`) targets the generic cross-framework [Agent-Skills](https://github.com/anthropics/skills) locations: the spec's user-global `~/.agents/skills/` (read by `npx skills` and spec-compliant frameworks) for a global install, and `./.agents/skills/` for a project (`--project`) install. The bare `graphify install` stays single-platform (Claude Code) by design — use the named `agents` platform when you want the skill discoverable by any framework that reads `.agents/skills`. @@ -236,19 +281,24 @@ Run this once in your project after building a graph: | Devin CLI | `graphify devin install` | | Google Antigravity | `graphify antigravity install` | -This writes a small config file that tells your assistant to consult the knowledge graph for codebase questions — preferring scoped queries like `graphify query ""` over reading the full report or grepping raw files. On platforms that support payload-bearing hooks (Claude Code, Gemini CLI), a hook fires automatically before search-style tool calls (and, on Claude Code, before reading source files one by one via the Read/Glob tools) and nudges your assistant toward the graph path. On the others (Codex, OpenCode, Cursor, etc.), the persistent instruction files (`AGENTS.md`, `.cursor/rules/`, etc.) provide the same query-first guidance. `GRAPH_REPORT.md` is still available for broad architecture review. +This writes a small config file that tells your assistant to consult the knowledge graph for codebase questions, preferring scoped queries like `graphify query ""` over reading the full report or grepping raw files. -**CodeBuddy** does the same two things as Claude Code: writes a `CODEBUDDY.md` section telling CodeBuddy to read `graphify-out/GRAPH_REPORT.md` before answering architecture questions, and installs **PreToolUse hooks** (`.codebuddy/settings.json`) that fire before Bash search commands and file reads, nudging toward `graphify query` instead. +- **Hook platforms** (Claude Code, Gemini CLI): a hook fires automatically before search-style tool calls (and, on Claude Code, before reading source files one by one via the Read/Glob tools) and nudges your assistant toward the graph path. +- **Instruction-file platforms** (Codex, OpenCode, Cursor, etc.): persistent instruction files (`AGENTS.md`, `.cursor/rules/`, etc.) provide the same query-first guidance. -**Codex** writes to `AGENTS.md` and also installs a **PreToolUse hook** in `.codex/hooks.json` that fires before every Bash tool call — same always-on mechanism as Claude Code. +`GRAPH_REPORT.md` is still available for broad architecture review. -To remove graphify from all platforms at once: `graphify uninstall` (add `--purge` to also delete `graphify-out/`). Or use the per-platform command (e.g. `graphify claude uninstall`). +**CodeBuddy** does the same two things as Claude Code: writes a `CODEBUDDY.md` section telling CodeBuddy to read `graphify-out/GRAPH_REPORT.md` before answering architecture questions, and installs `PreToolUse` hooks (`.codebuddy/settings.json`) that fire before Bash search commands and file reads, nudging toward `graphify query` instead. ---- +**Codex** writes to `AGENTS.md` and also installs a `PreToolUse` hook in `.codex/hooks.json` that fires before every Bash tool call, same always-on mechanism as Claude Code. + +**Kilo Code** installs the Graphify skill to `~/.config/kilo/skills/graphify/SKILL.md` and a native `/graphify` command to `~/.config/kilo/command/graphify.md`. `graphify kilo install` also writes `AGENTS.md` plus a native `tool.execute.before` plugin (`.kilo/plugins/graphify.js` + `.kilo/kilo.json` or `.kilo/kilo.jsonc` registration) so Kilo gets the same always-on graph reminder behavior through native `.kilo` config. -**Kilo Code** installs the Graphify skill to `~/.config/kilo/skills/graphify/SKILL.md` and a native `/graphify` command to `~/.config/kilo/command/graphify.md`. `graphify kilo install` also writes `AGENTS.md` plus a native **`tool.execute.before` plugin** (`.kilo/plugins/graphify.js` + `.kilo/kilo.json` or `.kilo/kilo.jsonc` registration) so Kilo gets the same always-on graph reminder behavior through native `.kilo` config. +**Cursor** writes `.cursor/rules/graphify.mdc` with `alwaysApply: true`, so Cursor includes it in every conversation automatically, no hook needed. -**Cursor** writes `.cursor/rules/graphify.mdc` with `alwaysApply: true` — Cursor includes it in every conversation automatically, no hook needed. +To remove graphify from all platforms at once: `graphify uninstall` (add `--purge` to also delete `graphify-out/`). Or use the per-platform command (e.g. `graphify claude uninstall`). + +--- ## What's in the report @@ -277,7 +327,7 @@ To remove graphify from all platforms at once: `graphify uninstall` (add `--purg | Video / Audio | `.mp4 .mov .mp3 .wav` and more (requires `uv tool install graphifyy[video]`) | | YouTube / URLs | any video URL (requires `uv tool install graphifyy[video]`) | -Code is extracted locally with no API calls (AST via tree-sitter). Everything else goes through your AI assistant's model API. +Code is extracted **locally with no API calls** (AST via tree-sitter). Everything else goes through your AI assistant's model API. Google Drive for desktop `.gdoc`, `.gsheet`, and `.gslides` files are shortcut pointers, not document content. To include native Google Docs, Sheets, and Slides @@ -464,7 +514,7 @@ These are only needed for **headless / CI extraction** (`graphify extract`). Whe - **Video / audio** — transcribed locally with faster-whisper. Nothing leaves your machine. - **Docs, PDFs, images** — sent to your AI assistant for semantic extraction (via the `/graphify` skill, using whatever model your IDE session runs). Headless `graphify extract` requires `GEMINI_API_KEY` / `GOOGLE_API_KEY` (Gemini), `MOONSHOT_API_KEY` (Kimi), `ANTHROPIC_API_KEY` (Claude), `OPENAI_API_KEY` (OpenAI), `DEEPSEEK_API_KEY` (DeepSeek), a running Ollama instance (`OLLAMA_BASE_URL`), AWS credentials via the standard provider chain (Bedrock - no API key needed, uses IAM), or the `claude` CLI binary (Claude Code - no API key needed, uses your Claude subscription). The `--dedup-llm` flag uses the same key. - **Data residency** — `graphify extract` auto-detects which provider to use based on which API key is set (priority: Gemini → Kimi → Claude → OpenAI → DeepSeek → Azure → Bedrock → Ollama). For code with data-residency requirements, use `--backend ollama` (fully local) or pass an explicit `--backend` flag. Kimi (`MOONSHOT_API_KEY`) routes to Moonshot AI servers in China. -- No telemetry, no usage tracking, no analytics. +- **No telemetry**, no usage tracking, no analytics. - **Query logging** — every `graphify query`, `graphify path`, `graphify explain`, and MCP `query_graph` call is logged to `~/.cache/graphify-queries.log` in JSON Lines format (timestamp, question, corpus, nodes returned, duration). Full subgraph responses are **not** stored by default. Set `GRAPHIFY_QUERY_LOG_DISABLE=1` to opt out, or `GRAPHIFY_QUERY_LOG=/dev/null` to silence without disabling the code path. --- @@ -706,7 +756,7 @@ graphify label ./my-project --backend=openai --model gpt-4o # force a specific --- -## Built on graphify — Penpax +## Built on graphify: Penpax [**Penpax**](https://graphifylabs.ai) is the always-on layer built on top of graphify — it applies the same graph approach to your entire working life: meetings, browser history, emails, files, and code, updating continuously in the background. From 97a1371478e108fdfefe96cb581b0f3fd0d0393e Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 6 Jul 2026 12:36:12 +0100 Subject: [PATCH 002/292] Fix: exact-match section heading in _replace_or_append_section to prevent CLAUDE.md data loss (#1688) The updater located its managed block by substring (`marker in content` and `next(... if marker in line)`), so a heading that merely appeared as a substring of another line, or a duplicate heading, matched the wrong offset and the rewrite could truncate or drop unrelated content in CLAUDE.md / AGENTS.md. It now matches the section heading exactly (`line.strip() == marker`), appends when the section is absent, and prefers the last exact match when several exist. Thanks @bdfinst for the report. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 4 ++ graphify/__main__.py | 35 ++++++++------ tests/test_replace_or_append_section.py | 62 +++++++++++++++++++++++++ 3 files changed, 86 insertions(+), 15 deletions(-) create mode 100644 tests/test_replace_or_append_section.py diff --git a/CHANGELOG.md b/CHANGELOG.md index fe0bc82e7..9f4e67bed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ Full release notes with details on each version: [GitHub Releases](https://github.com/safishamsi/graphify/releases) +## Unreleased + +- Fix: `--update`-style section writes to `CLAUDE.md`/`AGENTS.md` no longer corrupt or drop content (#1688, thanks @bdfinst). `_replace_or_append_section` located its managed block by substring (`marker in content`) and `next(... if marker in line)`, so a heading that appeared as a substring of another line (or duplicate headings) matched the wrong offset and the rewrite could truncate the file. It now matches the section heading exactly (`line.strip() == marker`), appends when absent, and prefers the last exact match when several exist, so unrelated content is preserved. + ## 0.9.7 (2026-07-06) - Fix: Java standard-library types are no longer emitted as `references` noise (#1603, thanks @NydiaChung). A `_JAVA_BUILTIN_TYPES` skip list now suppresses ubiquitous `java.lang`/`java.util`/`java.io`/`java.time`/`java.math`/`java.nio.file` type names (`String`, `List`, `Map`, `Optional`, `Integer`, `Exception`, ...) at the type-ref walker; they never resolve to a project node, so edges to them were pure noise (mirrors `_GO_PREDECLARED_TYPES`/`_PYTHON_ANNOTATION_NOISE`). Nested user-type generic arguments still resolve: `List` drops the `List` edge but keeps `Item`. diff --git a/graphify/__main__.py b/graphify/__main__.py index e620d97a5..9367e5754 100644 --- a/graphify/__main__.py +++ b/graphify/__main__.py @@ -658,26 +658,31 @@ def _canonical_platform(platform_name: str) -> str: def _replace_or_append_section(content: str, marker: str, new_section: str) -> str: """Idempotently update or append a graphify-owned section in shared files. - If ``marker`` is not in ``content``, append ``new_section`` to the end - (with a blank-line separator if there's existing content). - - If ``marker`` IS in ``content``, replace the existing section in place. - The section runs from the first line containing ``marker`` to the line - before the next H2 heading (``## `` at line start), or to EOF if no later - H2 exists. This lets older installs receive the updated copy without - users having to uninstall and reinstall — important for the issue #580 - fix where existing report-first text would otherwise silently linger. + If no line is exactly ``marker`` (the heading, at column 0), append + ``new_section`` to the end (with a blank-line separator if there's existing + content). + + If a real ``marker`` heading exists, replace the existing section in place. + The section runs from that heading to the line before the next H2 heading + (``## `` at line start), or to EOF if no later H2 exists. This lets older + installs receive the updated copy without users having to uninstall and + reinstall (issue #580). + + The heading is matched only when a line *is* exactly ``marker`` (after + stripping surrounding whitespace), never as a substring. Matching ``## + graphify`` inside a bullet or an inline reference used to anchor the replace + on that mention and delete every line from there to the next heading, + silently destroying hand-curated content (#1688). When several exact + headings exist, the last one is used, since graphify's section is appended. """ - if marker not in content: + lines = content.split("\n") + starts = [i for i, line in enumerate(lines) if line.strip() == marker] + if not starts: if content.strip(): return content.rstrip() + "\n\n" + new_section.lstrip() return new_section.lstrip() - lines = content.split("\n") - start = next((i for i, line in enumerate(lines) if marker in line), None) - if start is None: - return content.rstrip() + "\n\n" + new_section.lstrip() - + start = starts[-1] end = len(lines) for j in range(start + 1, len(lines)): if lines[j].startswith("## "): diff --git a/tests/test_replace_or_append_section.py b/tests/test_replace_or_append_section.py new file mode 100644 index 000000000..f098a0070 --- /dev/null +++ b/tests/test_replace_or_append_section.py @@ -0,0 +1,62 @@ +"""#1688 - graphify's shared-file section update must not destroy user content. + +_replace_or_append_section used to locate its marker (`## graphify`) as a +substring, so a bullet or inline reference to the section became the replace +anchor and every line from there to the next heading was deleted. The marker is +now matched only as an exact heading line. +""" +from __future__ import annotations + +from graphify.__main__ import _replace_or_append_section + +MARKER = "## graphify" +NEW = "## graphify\n\nThis project has a knowledge graph at graphify-out/.\n" + + +def test_inline_reference_to_marker_is_not_treated_as_the_section(): + before = ( + "# My Project\n\n" + "## Setup\n" + "- See the `## graphify` section for graph usage.\n\n" + "## Release Process\n" + "Critical steps that must not be lost.\n" + ) + after = _replace_or_append_section(before, MARKER, NEW) + assert "See the `## graphify` section" in after # bullet preserved + assert "Critical steps that must not be lost" in after # later section preserved + assert "knowledge graph at graphify-out/" in after # section still added + + +def test_real_section_is_replaced_in_place(): + before = ( + "# P\n\n## Setup\n- do things\n\n" + "## graphify\n\nOLD text.\n\n" + "## Release\nkeep me\n" + ) + after = _replace_or_append_section(before, MARKER, NEW) + assert "OLD text." not in after + assert "knowledge graph at graphify-out/" in after + assert "do things" in after and "keep me" in after + + +def test_reinstall_is_idempotent(): + once = _replace_or_append_section("# P\n\n## Setup\n- x\n", MARKER, NEW) + twice = _replace_or_append_section(once, MARKER, NEW) + assert once.split("\n").count(MARKER) == 1 + assert twice.split("\n").count(MARKER) == 1 + + +def test_append_when_no_real_heading(): + before = "# P\n\n## Setup\n- x\n" + after = _replace_or_append_section(before, MARKER, NEW) + assert "- x" in after + assert after.split("\n").count(MARKER) == 1 + + +def test_prefers_last_heading_when_duplicated(): + before = "## graphify\nstale early copy\n\n## Other\nmid\n\n## graphify\nreal trailing copy\n" + after = _replace_or_append_section(before, MARKER, NEW) + # the trailing real section is replaced; the earlier stray heading + the + # user's "mid" content are left intact + assert "mid" in after + assert "knowledge graph at graphify-out/" in after From 0ff584f070ba36c14866b39b5c7aff54d0e59b70 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 6 Jul 2026 12:42:39 +0100 Subject: [PATCH 003/292] Fix: tolerate tiktoken special-token text in token estimation (#1685) `_TOKENIZER.encode(content)` raises ValueError by default when the text contains a special token such as `<|endoftext|>`, so a doc or corpus that merely mentions these strings crashed the entire semantic pass. Both `encode` sites in `_estimate_file_tokens` now pass `disallowed_special=()` so such text is tokenized as ordinary bytes. Thanks @Kyzcreig for the report. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 1 + graphify/llm.py | 4 ++-- tests/test_chunking.py | 31 ++++++++++++++++++++++++++++++- 3 files changed, 33 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f4e67bed..2fe4e3330 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ Full release notes with details on each version: [GitHub Releases](https://githu ## Unreleased - Fix: `--update`-style section writes to `CLAUDE.md`/`AGENTS.md` no longer corrupt or drop content (#1688, thanks @bdfinst). `_replace_or_append_section` located its managed block by substring (`marker in content`) and `next(... if marker in line)`, so a heading that appeared as a substring of another line (or duplicate headings) matched the wrong offset and the rewrite could truncate the file. It now matches the section heading exactly (`line.strip() == marker`), appends when absent, and prefers the last exact match when several exist, so unrelated content is preserved. +- Fix: token estimation no longer crashes on files containing tiktoken special-token text like `<|endoftext|>` (#1685, thanks @Kyzcreig). `_TOKENIZER.encode(content)` raises `ValueError` by default when the text contains a special token, which aborted packing on docs/corpora that merely mention these strings. Both `encode` sites now pass `disallowed_special=()` so such text is tokenized as ordinary bytes. ## 0.9.7 (2026-07-06) diff --git a/graphify/llm.py b/graphify/llm.py index 16644e6dc..62dd4b6f3 100644 --- a/graphify/llm.py +++ b/graphify/llm.py @@ -1507,7 +1507,7 @@ def _estimate_file_tokens(unit: "Path | FileSlice") -> int: content = read_slice_text(unit)[:_FILE_CHAR_CAP] except OSError: return 0 - return len(_TOKENIZER.encode(content)) + (_PER_FILE_OVERHEAD_CHARS // _CHARS_PER_TOKEN) + return len(_TOKENIZER.encode(content, disallowed_special=())) + (_PER_FILE_OVERHEAD_CHARS // _CHARS_PER_TOKEN) path = unit # Raster images are not read as text; a vision model bills them at a roughly @@ -1526,7 +1526,7 @@ def _estimate_file_tokens(unit: "Path | FileSlice") -> int: content = path.read_text(encoding="utf-8", errors="replace")[:_FILE_CHAR_CAP] except OSError: return 0 - return len(_TOKENIZER.encode(content)) + (_PER_FILE_OVERHEAD_CHARS // _CHARS_PER_TOKEN) + return len(_TOKENIZER.encode(content, disallowed_special=())) + (_PER_FILE_OVERHEAD_CHARS // _CHARS_PER_TOKEN) def _pack_chunks_by_tokens( diff --git a/tests/test_chunking.py b/tests/test_chunking.py index 21b28eec4..2706dfd79 100644 --- a/tests/test_chunking.py +++ b/tests/test_chunking.py @@ -118,7 +118,9 @@ def test_estimate_file_tokens_uses_tiktoken_when_available(tmp_path): # Force the tokenizer to be a mock that records calls and returns a known # token list, so we can assert the tiktoken path is taken. - fake_encoder = type("E", (), {"encode": staticmethod(lambda s: [0] * 999)})() + # Match tiktoken's real signature: encode(text, *, disallowed_special=...) + # so the #1685 hardening call (disallowed_special=()) reaches the mock. + fake_encoder = type("E", (), {"encode": staticmethod(lambda s, **kw: [0] * 999)})() with patch.object(llm, "_TOKENIZER", fake_encoder): n = llm._estimate_file_tokens(f) assert n == 999 + (llm._PER_FILE_OVERHEAD_CHARS // llm._CHARS_PER_TOKEN) @@ -497,3 +499,30 @@ def stub(chunk, **kwargs): assert len(chunk_done_args) == 1 assert chunk_done_args[0] == (0, 1, 4) assert len(result["nodes"]) == 4 + + +# ---- #1685: special-token strings in docs must not crash token estimation ---- + +def test_estimate_file_tokens_handles_tiktoken_special_token(tmp_path): + """A doc containing a literal tiktoken special token (e.g. <|endoftext|>) + must not crash token estimation. tiktoken's default encode() raises on such + strings appearing as ordinary text; we pass disallowed_special=() since this + is only an estimate (#1685).""" + import graphify.llm as llm + if llm._TOKENIZER is None: + import pytest + pytest.skip("tiktoken not installed; estimation uses the char heuristic") + f = tmp_path / "tokenizer-notes.md" + f.write_text("The GPT end-of-text token is <|endoftext|> in the vocab.\n") + n = llm._estimate_file_tokens(f) # must not raise + assert isinstance(n, int) and n > 0 + + +def test_pack_chunks_with_special_token_doc_does_not_crash(tmp_path): + """End to end: packing a corpus that includes a special-token doc must not + raise (the crash in #1685 happened during token-budget packing).""" + from graphify.llm import _pack_chunks_by_tokens + doc = tmp_path / "doc.md"; doc.write_text("see <|endoftext|> and <|im_start|> tokens\n") + code = tmp_path / "code.py"; code.write_text("def f():\n return 1\n") + chunks = _pack_chunks_by_tokens([doc, code], token_budget=60_000) + assert chunks # produced at least one chunk, no exception From b78248f22a6247fe9b0f23900d5bd1b84355818f Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 6 Jul 2026 12:44:47 +0100 Subject: [PATCH 004/292] Fix: cap Ollama client-side retries so a hung request cannot multiply the stall (#1686) A stalled local model wedged for `timeout * (max_retries + 1)`, which with the default 6 retries turned one long stall into a ~21-minute block with no progress. `_call_openai_compat` now defaults the Ollama backend to zero client-side retries (a local model that stalls will not un-stall on retry); set `GRAPHIFY_MAX_RETRIES` to opt back in. Other backends are unchanged. This bounds the wait; the underlying stall is driven by the model server and is non-deterministic, so it is not eliminated. Thanks @Kyzcreig for the report. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 1 + graphify/llm.py | 12 ++++++- tests/test_ollama_retry_cap.py | 60 ++++++++++++++++++++++++++++++++++ 3 files changed, 72 insertions(+), 1 deletion(-) create mode 100644 tests/test_ollama_retry_cap.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 2fe4e3330..7d96c6823 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ Full release notes with details on each version: [GitHub Releases](https://githu - Fix: `--update`-style section writes to `CLAUDE.md`/`AGENTS.md` no longer corrupt or drop content (#1688, thanks @bdfinst). `_replace_or_append_section` located its managed block by substring (`marker in content`) and `next(... if marker in line)`, so a heading that appeared as a substring of another line (or duplicate headings) matched the wrong offset and the rewrite could truncate the file. It now matches the section heading exactly (`line.strip() == marker`), appends when absent, and prefers the last exact match when several exist, so unrelated content is preserved. - Fix: token estimation no longer crashes on files containing tiktoken special-token text like `<|endoftext|>` (#1685, thanks @Kyzcreig). `_TOKENIZER.encode(content)` raises `ValueError` by default when the text contains a special token, which aborted packing on docs/corpora that merely mention these strings. Both `encode` sites now pass `disallowed_special=()` so such text is tokenized as ordinary bytes. +- Fix: the Ollama backend no longer multiplies a hang by the retry count (#1686, thanks @Kyzcreig). A stalled local model would wedge for `timeout * (max_retries + 1)`, which with the default 6 retries turned one long stall into a very long one. Ollama now defaults to zero client-side retries (a local model that stalls will not un-stall on retry); set `GRAPHIFY_MAX_RETRIES` to opt back in. Other backends are unchanged. Note: the underlying stall is non-deterministic and driven by the model server, so this bounds the wait rather than eliminating the hang. ## 0.9.7 (2026-07-06) diff --git a/graphify/llm.py b/graphify/llm.py index 62dd4b6f3..7933403e6 100644 --- a/graphify/llm.py +++ b/graphify/llm.py @@ -962,8 +962,18 @@ def _call_openai_compat( # default. Honour GRAPHIFY_API_TIMEOUT (seconds) for explicit override; # default to 600s, which is long enough for a 31B model on a 16k chunk # but still bounds runaway connections (issue #792 addendum). + # The SDK's transient-error retries (default 6) exist for cloud rate limits + # (429). A local Ollama server does not rate-limit, and if it wedges it will + # not recover by retrying, so 6 retries turn a 180s --api-timeout into a + # ~21min block (7 attempts x 180s) with no progress (#1686). Default ollama + # to 0 SDK retries so --api-timeout is the hard wall-clock bound and a hung + # request fails fast into the chunk-level retry/skip. An explicit + # GRAPHIFY_MAX_RETRIES still wins for users who want it. + _retries = _resolve_max_retries() + if backend == "ollama" and not os.environ.get("GRAPHIFY_MAX_RETRIES", "").strip(): + _retries = 0 client = OpenAI(api_key=api_key, base_url=base_url, timeout=_resolve_api_timeout(), - max_retries=_resolve_max_retries()) + max_retries=_retries) kwargs: dict = { "model": model, "messages": [ diff --git a/tests/test_ollama_retry_cap.py b/tests/test_ollama_retry_cap.py new file mode 100644 index 000000000..5ad43cc73 --- /dev/null +++ b/tests/test_ollama_retry_cap.py @@ -0,0 +1,60 @@ +"""#1686 - a wedged local Ollama request must not multiply --api-timeout by the +SDK's 6 transient-error retries into a ~20min block. Ollama defaults to 0 SDK +retries so the timeout is the effective wall-clock bound; an explicit +GRAPHIFY_MAX_RETRIES still wins. +""" +from __future__ import annotations + +from unittest.mock import MagicMock + +import graphify.llm as llm + + +def _capture_client_kwargs(monkeypatch): + captured: dict = {} + + def _factory(**kwargs): + captured.update(kwargs) + client = MagicMock() + resp = MagicMock() + resp.choices[0].message.content = '{"nodes": [], "edges": [], "hyperedges": []}' + resp.choices[0].finish_reason = "stop" + resp.usage.prompt_tokens = 1 + resp.usage.completion_tokens = 1 + client.chat.completions.create.return_value = resp + return client + + monkeypatch.setattr("openai.OpenAI", _factory) + return captured + + +def test_ollama_defaults_to_zero_sdk_retries(monkeypatch): + monkeypatch.delenv("GRAPHIFY_MAX_RETRIES", raising=False) + captured = _capture_client_kwargs(monkeypatch) + llm._call_openai_compat("http://localhost:11434/v1", "ollama", "m", + "def f(): pass", backend="ollama") + assert captured.get("max_retries") == 0 + + +def test_ollama_honors_explicit_max_retries(monkeypatch): + monkeypatch.setenv("GRAPHIFY_MAX_RETRIES", "3") + captured = _capture_client_kwargs(monkeypatch) + llm._call_openai_compat("http://localhost:11434/v1", "ollama", "m", + "def f(): pass", backend="ollama") + assert captured.get("max_retries") == 3 + + +def test_cloud_backend_keeps_default_retries(monkeypatch): + monkeypatch.delenv("GRAPHIFY_MAX_RETRIES", raising=False) + captured = _capture_client_kwargs(monkeypatch) + llm._call_openai_compat("https://api.moonshot.cn/v1", "sk-x", "m", + "def f(): pass", backend="kimi") + assert captured.get("max_retries") == 6 # default retained for rate-limited clouds + + +def test_api_timeout_is_passed_to_client(monkeypatch): + monkeypatch.setenv("GRAPHIFY_API_TIMEOUT", "180") + captured = _capture_client_kwargs(monkeypatch) + llm._call_openai_compat("http://localhost:11434/v1", "ollama", "m", + "def f(): pass", backend="ollama") + assert captured.get("timeout") == 180.0 From 21b851b3d25849ff4f5bb5e65daeb766d8f64877 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 6 Jul 2026 12:46:12 +0100 Subject: [PATCH 005/292] Fix: salvage truncated labeling replies and account for labeling token cost (#1690, #1694) Two related fixes in the community-labeling path: #1690 (thanks @vdgbcrypto): a truncated or slightly malformed reply no longer discards the whole batch with "Expecting value: line 1 column 6". `_parse_label_response` now salvages the complete `"id": "name"` pairs from a reply that failed a strict `json.loads` (e.g. one truncated mid-object), raising only when no pairs can be recovered. The per-batch token budget was also raised (256 + 48*n, was 64 + 24*n) so models that prepend a short preamble have headroom to finish the JSON. The exact provider truncation could not be reproduced without a live key; the parser and budget address the mechanism. #1694 (thanks @sub4biz): cluster-only mode reported a hardcoded `0 input * 0 output` token cost because the labeling LLM calls were never accounted for. `_call_llm` now accumulates per-response usage into an optional accumulator threaded through the labeling path and surfaced in GRAPH_REPORT.md. Backends that do not return usage (the Claude Code CLI) contribute nothing, which is honest rather than estimated. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 2 + graphify/__main__.py | 7 +++- graphify/llm.py | 86 ++++++++++++++++++++++++++++++++++++++---- tests/test_labeling.py | 68 ++++++++++++++++++++++++++++++++- 4 files changed, 153 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d96c6823..691b1c7ef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,8 @@ Full release notes with details on each version: [GitHub Releases](https://githu - Fix: `--update`-style section writes to `CLAUDE.md`/`AGENTS.md` no longer corrupt or drop content (#1688, thanks @bdfinst). `_replace_or_append_section` located its managed block by substring (`marker in content`) and `next(... if marker in line)`, so a heading that appeared as a substring of another line (or duplicate headings) matched the wrong offset and the rewrite could truncate the file. It now matches the section heading exactly (`line.strip() == marker`), appends when absent, and prefers the last exact match when several exist, so unrelated content is preserved. - Fix: token estimation no longer crashes on files containing tiktoken special-token text like `<|endoftext|>` (#1685, thanks @Kyzcreig). `_TOKENIZER.encode(content)` raises `ValueError` by default when the text contains a special token, which aborted packing on docs/corpora that merely mention these strings. Both `encode` sites now pass `disallowed_special=()` so such text is tokenized as ordinary bytes. - Fix: the Ollama backend no longer multiplies a hang by the retry count (#1686, thanks @Kyzcreig). A stalled local model would wedge for `timeout * (max_retries + 1)`, which with the default 6 retries turned one long stall into a very long one. Ollama now defaults to zero client-side retries (a local model that stalls will not un-stall on retry); set `GRAPHIFY_MAX_RETRIES` to opt back in. Other backends are unchanged. Note: the underlying stall is non-deterministic and driven by the model server, so this bounds the wait rather than eliminating the hang. +- Fix: a truncated or slightly malformed community-labeling reply no longer discards the whole batch (#1690, thanks @vdgbcrypto). `_parse_label_response` now salvages the complete `"id": "name"` pairs from a reply that failed a strict `json.loads` (e.g. a reply truncated mid-object), raising only when no pairs can be recovered. The per-batch token budget was also raised (`256 + 48*n`, was `64 + 24*n`) to give models that prepend a short preamble enough headroom to finish the JSON. The exact provider truncation in the report could not be reproduced without a live key; the parser and budget fixes address the mechanism. +- Fix: cluster-only mode now reports the real token cost of community labeling instead of a hardcoded zero (#1694, thanks @sub4biz). The labeling LLM calls were never accounted for, so `GRAPH_REPORT.md`'s "Token cost" line always read `0 input · 0 output` in cluster-only runs. `_call_llm` now accumulates per-response usage into an optional accumulator that is threaded through the labeling path and surfaced in the report. Backends that do not return usage (the Claude Code CLI) still contribute nothing, which is honest rather than estimated. ## 0.9.7 (2026-07-06) diff --git a/graphify/__main__.py b/graphify/__main__.py index 9367e5754..3a8113587 100644 --- a/graphify/__main__.py +++ b/graphify/__main__.py @@ -3560,6 +3560,10 @@ def main() -> None: } except Exception: existing_labels = {} + # Accumulate token usage from the labeling LLM calls so cluster-only mode + # reports real cost instead of a hardcoded zero (#1694). Stays {0, 0} on + # the reuse / no-label paths, which make no LLM calls. + label_token_usage = {"input": 0, "output": 0} if labels_path.exists() and not force_relabel: # Reuse saved labels, but don't blindly trust them: the graph may have # been re-scoped/re-clustered since labeling, in which case a cid now @@ -3643,6 +3647,7 @@ def main() -> None: generated_labels, _ = generate_community_labels( G, label_communities_input, backend=label_backend, model=label_model, gods=gods, max_concurrency=label_max_concurrency, batch_size=label_batch_size, + usage_out=label_token_usage, ) # Only let the LLM OVERRIDE where it produced a real name — its no-backend # fallback returns "Community {cid}" placeholders, which must not clobber @@ -3653,7 +3658,7 @@ def main() -> None: }) stages.mark("label") questions = suggest_questions(G, communities, labels) - tokens = {"input": 0, "output": 0} + tokens = label_token_usage from graphify.export import _git_head as _gh _commit = _gh() from graphify.report import load_learning_for_report as _llfr diff --git a/graphify/llm.py b/graphify/llm.py index 7933403e6..a0c073afa 100644 --- a/graphify/llm.py +++ b/graphify/llm.py @@ -1949,9 +1949,15 @@ def _call_llm( backend: str, max_tokens: int = 200, model: str | None = None, + usage_out: dict | None = None, ) -> str: """Send a plain-text prompt to `backend` and return the model's text reply. + When ``usage_out`` is provided it is accumulated in place with ``input`` and + ``output`` token counts from the response, so callers (community labeling) + can total the cost of otherwise-uninstrumented LLM calls (#1694). Existing + callers that omit it are unaffected. + Used by lightweight callers (e.g. `graphify.dedup` LLM tiebreaker) that don't need the full extraction prompt or JSON-shaped output. Mirrors the backend dispatch logic of `extract_files_direct` but skips the @@ -1975,6 +1981,11 @@ def _call_llm( ) mdl = model or _default_model_for_backend(backend) + def _rec(inp, out) -> None: + if usage_out is not None: + usage_out["input"] = usage_out.get("input", 0) + int(inp or 0) + usage_out["output"] = usage_out.get("output", 0) + int(out or 0) + if backend == "claude": try: import anthropic @@ -1986,6 +1997,9 @@ def _call_llm( max_tokens=max_tokens, messages=[{"role": "user", "content": prompt}], ) + u = getattr(resp, "usage", None) + if u is not None: + _rec(getattr(u, "input_tokens", 0), getattr(u, "output_tokens", 0)) return resp.content[0].text if resp.content else "" if backend == "claude-cli": @@ -2019,6 +2033,14 @@ def _call_llm( if proc.returncode != 0: raise RuntimeError(f"claude -p exited {proc.returncode}: {proc.stderr.strip()[:500]}") envelope = _claude_cli_envelope(proc.stdout) + cli_usage = envelope.get("usage") or {} + if cli_usage: + _rec( + (cli_usage.get("input_tokens", 0) or 0) + + (cli_usage.get("cache_read_input_tokens", 0) or 0) + + (cli_usage.get("cache_creation_input_tokens", 0) or 0), + cli_usage.get("output_tokens", 0), + ) return envelope.get("result", "") @@ -2036,6 +2058,9 @@ def _call_llm( messages=[{"role": "user", "content": [{"text": prompt}]}], inferenceConfig=_bedrock_inference_config(max_tokens, mdl), ) + bu = resp.get("usage") or {} + if bu: + _rec(bu.get("inputTokens", 0), bu.get("outputTokens", 0)) return resp.get("output", {}).get("message", {}).get("content", [{}])[0].get("text", "") if backend == "azure": @@ -2056,6 +2081,9 @@ def _call_llm( resp = azure_client.chat.completions.create(**azure_kwargs) if not resp.choices or resp.choices[0].message is None: raise ValueError("Azure OpenAI returned empty or filtered response") + au = getattr(resp, "usage", None) + if au is not None: + _rec(getattr(au, "prompt_tokens", 0), getattr(au, "completion_tokens", 0)) return resp.choices[0].message.content or "" # OpenAI-compatible (kimi, openai, gemini, ollama) @@ -2088,6 +2116,9 @@ def _call_llm( resp = client.chat.completions.create(**kwargs) if not resp.choices or resp.choices[0].message is None: raise ValueError("LLM returned empty or filtered response") + ou = getattr(resp, "usage", None) + if ou is not None: + _rec(getattr(ou, "prompt_tokens", 0), getattr(ou, "completion_tokens", 0)) return resp.choices[0].message.content or "" @@ -2257,9 +2288,25 @@ def _parse_label_response(text: str, labeled_cids: list[int]) -> dict[int, str]: start, end = cleaned.find("{"), cleaned.rfind("}") if start != -1 and end > start: cleaned = cleaned[start:end + 1] - data = json.loads(cleaned) - if not isinstance(data, dict): - raise ValueError("label response is not a JSON object") + data: dict | None = None + try: + parsed = json.loads(cleaned) + if isinstance(parsed, dict): + data = parsed + except (json.JSONDecodeError, ValueError): + data = None + if data is None: + # Salvage: pull the complete "": "" pairs directly. A model + # can truncate its reply mid-object (a stingy token budget or a preamble + # eating the completion), which used to hard-fail the whole batch with + # e.g. `Expecting value: line 1 column 6` on a `{"0":` fragment (#1690). + # Recovering the pairs that DID arrive labels those communities instead + # of dropping the entire batch to placeholders. + pairs = re.findall(r'"?(-?\d+)"?\s*:\s*"([^"\\]*(?:\\.[^"\\]*)*)"', cleaned) + if pairs: + data = {k: v for k, v in pairs} + else: + raise ValueError(f"label response is not parseable JSON: {text[:120]!r}") out: dict[int, str] = {} for cid in labeled_cids: name = data.get(str(cid)) @@ -2278,6 +2325,7 @@ def _label_batch_with_retry( model: str | None, depth: int = 0, max_depth: int = 3, + usage_out: dict | None = None, ) -> dict[int, str]: """Label a batch of communities, splitting in half and retrying on parse failure. @@ -2301,10 +2349,18 @@ def _label_batch_with_retry( "Respond ONLY with a JSON object mapping the community id (as a string) to " "its name - no prose, no markdown fences.\n\n" + "\n".join(batch_lines) ) - max_tokens = _resolve_max_tokens(min(64 + 24 * len(batch_cids), 8192)) + # Budget generously: a 2-5 word name is ~10 tokens, but models (notably + # gemini) often prepend a short preamble or reasoning that eats the + # completion and truncates the JSON mid-object, which used to fail the whole + # batch (#1690). The old 64 + 24*n floor left no headroom. + max_tokens = _resolve_max_tokens(min(256 + 48 * len(batch_cids), 8192)) call_kwargs: dict = {"backend": backend, "max_tokens": max_tokens} if model is not None: call_kwargs["model"] = model + # Only forward usage_out when the caller wants accounting, so existing + # callers (and their test doubles) see the unchanged _call_llm signature. + if usage_out is not None: + call_kwargs["usage_out"] = usage_out try: text = _call_llm(prompt, **call_kwargs) @@ -2325,10 +2381,12 @@ def _label_batch_with_retry( left = _label_batch_with_retry( batch_cids[:mid], batch_lines[:mid], backend=backend, model=model, depth=depth + 1, max_depth=max_depth, + usage_out=usage_out, ) right = _label_batch_with_retry( batch_cids[mid:], batch_lines[mid:], backend=backend, model=model, depth=depth + 1, max_depth=max_depth, + usage_out=usage_out, ) return left | right @@ -2344,6 +2402,7 @@ def label_communities( top_k: int = _LABEL_TOP_K, batch_size: int = _LABEL_BATCH_SIZE, max_concurrency: int = 4, + usage_out: dict | None = None, ) -> dict[int, str]: """Return a complete ``{cid: name}`` map using ``backend`` for naming. @@ -2386,19 +2445,30 @@ def label_communities( def _run_batch(batch_idx: int): start = batch_idx * batch_size end = min(start + batch_size, len(labeled_cids)) + # Accumulate token usage into a per-batch dict so concurrent workers + # never race on the shared accumulator; it is merged on the main thread + # in _merge (#1694). + batch_usage: dict = {} if usage_out is not None else None + batch_kwargs = {"usage_out": batch_usage} if usage_out is not None else {} try: parsed = _label_batch_with_retry( labeled_cids[start:end], lines[start:end], backend=backend, model=model, + **batch_kwargs, ) - return batch_idx, parsed, None + return batch_idx, parsed, None, batch_usage except Exception as exc: # noqa: BLE001 - reported per-batch; surfaced below - return batch_idx, None, exc + return batch_idx, None, exc, batch_usage written = 0 errors: dict[int, Exception] = {} - def _merge(batch_idx: int, parsed, exc) -> None: + def _merge(batch_idx: int, parsed, exc, batch_usage=None) -> None: nonlocal written + # Count tokens even for a failed batch: the LLM call was billed whether + # or not the reply parsed. + if usage_out is not None and batch_usage: + usage_out["input"] = usage_out.get("input", 0) + batch_usage.get("input", 0) + usage_out["output"] = usage_out.get("output", 0) + batch_usage.get("output", 0) if exc is not None: errors[batch_idx] = exc start = batch_idx * batch_size @@ -2440,6 +2510,7 @@ def generate_community_labels( quiet: bool = False, max_concurrency: int = 4, batch_size: int = _LABEL_BATCH_SIZE, + usage_out: dict | None = None, ) -> tuple[dict[int, str], str]: """CLI entry point: resolve a backend, name communities, and degrade to ``Community N`` placeholders on any failure (no backend, API error, malformed @@ -2462,6 +2533,7 @@ def generate_community_labels( labels = label_communities( G, communities, backend=backend, model=model, gods=gods, max_concurrency=max_concurrency, batch_size=batch_size, + usage_out=usage_out, ) return labels, "llm" except Exception as exc: diff --git a/tests/test_labeling.py b/tests/test_labeling.py index d3a68067c..7acbd30e6 100644 --- a/tests/test_labeling.py +++ b/tests/test_labeling.py @@ -82,7 +82,7 @@ def test_label_cli_passes_model_override(tmp_path, monkeypatch): captured = {} def fake_generate(G, communities, *, backend=None, model=None, gods=None, - quiet=False, max_concurrency=4, batch_size=100): + quiet=False, max_concurrency=4, batch_size=100, usage_out=None): captured["backend"] = backend captured["model"] = model captured["max_concurrency"] = max_concurrency @@ -143,7 +143,7 @@ def test_label_cli_missing_only_preserves_existing_labels(tmp_path, monkeypatch) captured = {} def fake_generate(G, communities, *, backend=None, model=None, gods=None, - quiet=False, max_concurrency=4, batch_size=100): + quiet=False, max_concurrency=4, batch_size=100, usage_out=None): captured["communities"] = dict(communities) return {1: "Payment Flow"}, "llm" @@ -410,3 +410,67 @@ def test_label_communities_forces_serial_for_ollama(monkeypatch): monkeypatch.delenv("GRAPHIFY_OLLAMA_PARALLEL", raising=False) label_communities(G, communities, backend="ollama", batch_size=1, max_concurrency=8) assert state["peak"] == 1, "ollama must be forced serial" + + +def test_label_communities_salvages_truncated_reply(monkeypatch): + # #1690: a reply truncated mid-object (a stingy token budget or model + # preamble) used to hard-fail the whole batch with `Expecting value: line 1 + # column 6`. The complete pairs that arrived are now salvaged. + G, communities = _graph() + monkeypatch.setattr( + "graphify.llm._call_llm", + lambda p, *, backend, max_tokens=200: '{"0": "Order Management", "1":', + ) + labels = label_communities(G, communities, backend="gemini") + assert labels[0] == "Order Management" # salvaged + assert labels[1] == "Community 1" # truncated cid falls back to placeholder + + +def test_label_communities_accumulates_token_usage(monkeypatch): + # #1694: cluster-only mode reported zero labeling cost because token usage + # from the naming LLM calls was never accumulated. label_communities now + # fills a caller-supplied usage_out accumulator, summed across all batches. + G, communities = _many_communities(6) + + def fake_call(prompt, *, backend, max_tokens=200, usage_out=None): + if usage_out is not None: + usage_out["input"] = usage_out.get("input", 0) + 100 + usage_out["output"] = usage_out.get("output", 0) + 10 + # one name per community id present in this batch + cids = [int(line.split()[1].rstrip(":")) for line in prompt.splitlines() + if line.startswith("Community ")] + return json.dumps({str(c): f"Name {c}" for c in cids}) + + monkeypatch.setattr("graphify.llm._call_llm", fake_call) + usage = {"input": 0, "output": 0} + # batch_size=2 -> 3 batches, run serially so the count is deterministic + labels = label_communities( + G, communities, backend="gemini", batch_size=2, max_concurrency=1, + usage_out=usage, + ) + assert len(labels) == 6 + assert usage == {"input": 300, "output": 30} # 3 batches * (100, 10) + + +def test_label_communities_counts_tokens_for_failed_batch(monkeypatch): + # A batch whose reply can't be parsed was still billed by the provider, so + # its tokens must be counted even though it contributes no label (#1694). + G, communities = _graph() + + def fake_call(prompt, *, backend, max_tokens=200, usage_out=None): + if usage_out is not None: + usage_out["input"] = usage_out.get("input", 0) + 50 + usage_out["output"] = usage_out.get("output", 0) + 5 + return "not json at all" + + monkeypatch.setattr("graphify.llm._call_llm", fake_call) + usage = {"input": 0, "output": 0} + # single community -> no split retry; the only batch fails to parse, so + # label_communities re-raises (every batch failed) after counting tokens. + G2 = nx.Graph() + G2.add_node("a", label="alpha") + with pytest.raises((ValueError, json.JSONDecodeError)): + label_communities( + G2, {0: ["a"]}, backend="gemini", usage_out=usage, + ) + assert usage == {"input": 50, "output": 5} From e801196e4ec65b90efa69d35e85ac2487947017c Mon Sep 17 00:00:00 2001 From: adirbuskila Date: Mon, 6 Jul 2026 12:58:08 +0100 Subject: [PATCH 006/292] docs: add Hebrew (he-IL) README translation (#1639) Adds the Hebrew translation under docs/translations/README.he-IL.md and links it from the language row (30 -> 31 languages). Merged from PR #1639 by @AdirBuskila; the language-row link was adapted to the current README structure by hand since the English README changed after the PR was opened. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 2 +- docs/translations/README.he-IL.md | 849 ++++++++++++++++++++++++++++++ 2 files changed, 850 insertions(+), 1 deletion(-) create mode 100644 docs/translations/README.he-IL.md diff --git a/README.md b/README.md index ea3c862e3..9f1e11f06 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@
Read this in other languages -🇺🇸 English | 🇨🇳 简体中文 | 🇯🇵 日本語 | 🇰🇷 한국어 | 🇩🇪 Deutsch | 🇫🇷 Français | 🇪🇸 Español | 🇮🇳 हिन्दी | 🇧🇷 Português | 🇷🇺 Русский | 🇸🇦 العربية | 🇮🇷 فارسی | 🇮🇹 Italiano | 🇵🇱 Polski | 🇳🇱 Nederlands | 🇹🇷 Türkçe | 🇺🇦 Українська | 🇻🇳 Tiếng Việt | 🇮🇩 Bahasa Indonesia | 🇸🇪 Svenska | 🇬🇷 Ελληνικά | 🇷🇴 Română | 🇨🇿 Čeština | 🇫🇮 Suomi | 🇩🇰 Dansk | 🇳🇴 Norsk | 🇭🇺 Magyar | 🇹🇭 ภาษาไทย | 🇺🇿 Oʻzbekcha | 🇹🇼 繁體中文 | 🇵🇭 Filipino +🇺🇸 English | 🇨🇳 简体中文 | 🇯🇵 日本語 | 🇰🇷 한국어 | 🇩🇪 Deutsch | 🇫🇷 Français | 🇪🇸 Español | 🇮🇳 हिन्दी | 🇧🇷 Português | 🇷🇺 Русский | 🇸🇦 العربية | 🇮🇷 فارسی | 🇮🇹 Italiano | 🇵🇱 Polski | 🇳🇱 Nederlands | 🇹🇷 Türkçe | 🇺🇦 Українська | 🇻🇳 Tiếng Việt | 🇮🇩 Bahasa Indonesia | 🇸🇪 Svenska | 🇬🇷 Ελληνικά | 🇷🇴 Română | 🇨🇿 Čeština | 🇫🇮 Suomi | 🇩🇰 Dansk | 🇳🇴 Norsk | 🇭🇺 Magyar | 🇹🇭 ภาษาไทย | 🇺🇿 Oʻzbekcha | 🇹🇼 繁體中文 | 🇵🇭 Filipino | 🇮🇱 עברית
diff --git a/docs/translations/README.he-IL.md b/docs/translations/README.he-IL.md new file mode 100644 index 000000000..c1c193e84 --- /dev/null +++ b/docs/translations/README.he-IL.md @@ -0,0 +1,849 @@ +

+ Graphify +

+ +

+ 🇺🇸 English | 🇨🇳 简体中文 | 🇯🇵 日本語 | 🇰🇷 한국어 | 🇩🇪 Deutsch | 🇫🇷 Français | 🇪🇸 Español | 🇮🇳 हिन्दी | 🇧🇷 Português | 🇷🇺 Русский | 🇸🇦 العربية | 🇮🇷 فارسی | 🇮🇹 Italiano | 🇵🇱 Polski | 🇳🇱 Nederlands | 🇹🇷 Türkçe | 🇺🇦 Українська | 🇻🇳 Tiếng Việt | 🇮🇩 Bahasa Indonesia | 🇸🇪 Svenska | 🇬🇷 Ελληνικά | 🇷🇴 Română | 🇨🇿 Čeština | 🇫🇮 Suomi | 🇩🇰 Dansk | 🇳🇴 Norsk | 🇭🇺 Magyar | 🇹🇭 ภาษาไทย | 🇺🇿 Oʻzbekcha | 🇹🇼 繁體中文 | 🇵🇭 Filipino | 🇮🇱 עברית +

+ +

+ YC S26 + Discord + The Memory Layer + CI + PyPI + Downloads + Sponsor + LinkedIn + X +

+ +

+ + Star History Chart + +

+ +
+ +הקלידו `‎/graphify` בעוזר ה-AI לכתיבת קוד שלכם, והוא ימפה את הפרויקט כולו — קוד, מסמכים, קובצי PDF, תמונות, סרטונים — לגרף ידע שאפשר לשאול עליו שאלות, במקום לחפש בקבצים עם grep. + +עובד ב-Claude Code, Codex, OpenCode, Kilo Code, Cursor, Gemini CLI, GitHub Copilot CLI, VS Code Copilot Chat, Aider, Amp, OpenClaw, Factory Droid, Trae, Hermes, Kimi Code, Kiro, Pi, Devin CLI ו-Google Antigravity. + +
+ +``` +/graphify . +``` + +
+ +זה הכול. מקבלים שלושה קבצים: + +
+ +``` +graphify-out/ +├── graph.html נפתח בכל דפדפן — לחיצה על צמתים, סינון, חיפוש +├── GRAPH_REPORT.md עיקרי הדברים: מושגי מפתח, קשרים מפתיעים, שאלות מוצעות +└── graph.json הגרף המלא — אפשר לשאול עליו בכל רגע בלי לקרוא שוב את הקבצים +``` + +
+ +לדף ארכיטקטורה קריא עם דיאגרמות זרימת-קריאות ב-Mermaid, הריצו: + +
+ +```bash +graphify export callflow-html +``` + +--- + +
+ +## דרישות מקדימות + +| דרישה | מינימום | בדיקה | התקנה | +|---|---|---|---| +| Python | 3.10+ | `python --version` | [python.org](https://www.python.org/downloads/) | +| uv *(מומלץ)* | כל גרסה | `uv --version` | `curl -LsSf https://astral.sh/uv/install.sh \| sh` | +| pipx *(חלופה)* | כל גרסה | `pipx --version` | `pip install pipx` | + +**התקנה מהירה ב-macOS (עם Homebrew):** + +
+ +```bash +brew install python@3.12 uv +``` + +
+ +**התקנה מהירה ב-Windows:** + +
+ +```powershell +winget install astral-sh.uv +``` + +
+ +**Ubuntu/Debian:** + +
+ +```bash +sudo apt install python3.12 python3-pip pipx +# או התקנת uv: +curl -LsSf https://astral.sh/uv/install.sh | sh +``` + +--- + +
+ +## התקנה + +> **החבילה הרשמית:** חבילת ה-PyPI היא `graphifyy` (עם y כפולה). חבילות `graphify*` אחרות ב-PyPI אינן קשורות לפרויקט. פקודת ה-CLI היא עדיין `graphify`. + +**שלב 1 — התקנת החבילה:** + +
+ +```bash +# מומלץ (סביבה מבודדת; אם הפקודה 'graphify' לא נמצאת אחר כך, הריצו: uv tool update-shell): +uv tool install graphifyy + +# חלופות: +pipx install graphifyy +pip install graphifyy # עשוי לדרוש הגדרת PATH — ראו הערה בהמשך +``` + +
+ +**שלב 2 — רישום המיומנות (skill) אצל עוזר ה-AI שלכם:** + +
+ +```bash +graphify install +``` + +
+ +זהו. פתחו את עוזר ה-AI והקלידו `‎/graphify .` + +כדי להתקין את המיומנות בתוך המאגר הנוכחי במקום בפרופיל המשתמש, הוסיפו `--project`: + +
+ +```bash +graphify install --project +graphify install --project --platform codex +``` + +
+ +התקנה ברמת הפרויקט כותבת לתוך התיקייה הנוכחית, למשל `.claude/skills/graphify/SKILL.md` או `.agents/skills/graphify/SKILL.md` (בתוספת תיקיית `references/` שהמיומנות טוענת לפי הצורך), ומדפיסה רמז `git add` לקבצים שאפשר לבצע להם commit. פקודות פר-פלטפורמה שתומכות בהתקנה ברמת הפרויקט מקבלות את אותו דגל, למשל `graphify claude install --project` או `graphify codex install --project`. + +> **הערת PowerShell:** השתמשו ב-`graphify .` ולא ב-`‎/graphify .` — הלוכסן המוביל הוא מפריד נתיבים ב-PowerShell. + +> **‏`graphify: command not found`?** ‏`uv tool install` / `pipx install` מציבים את פקודת `graphify` בתיקיית הכלים שלהם (`~/.local/bin`). אם המעטפת (shell) לא מוצאת אותה מיד אחרי ההתקנה — נפוץ בהתקנת macOS + zsh טרייה — התיקייה הזו עדיין לא ב-`PATH`: הריצו `uv tool update-shell` (או `pipx ensurepath`) ופתחו טרמינל חדש. עם `pip` רגיל, הוסיפו את `~/.local/bin` (בלינוקס) או `~/Library/Python/3.x/bin` (במק) ל-PATH, או הריצו `python -m graphify`. + +> **מריצים עם `uvx` / `uv tool run` בלי להתקין?** ציינו את שם החבילה, לא את שם הפקודה: `uvx --from graphifyy graphify install`. ‏`uvx graphify …` רגיל נכשל (`No solution found … no versions of graphify`) כי `uv tool run` קורא את המילה הראשונה כשם *חבילה*, והחבילה היא `graphifyy` — פקודת `graphify` נמצאת בתוכה. + +> **הימנעו מ-`pip install` במק/Windows** אם אפשר. המיומנות מאתרת את Python בזמן ריצה מתוך `graphify-out/.graphify_python`; אם הוא מצביע על סביבה שונה מזו שבה `pip` התקין את החבילה, תקבלו `ModuleNotFoundError: No module named 'graphify'`. ‏`uv tool install` ו-`pipx install` מבודדים את החבילה בסביבה משלהם ונמנעים מהבעיה לחלוטין. + +> **הוקים של Git עם uv tool / pipx:** ‏`graphify hook install` מטמיע את נתיב המפרש הנוכחי ישירות בסקריפטי ההוק בזמן ההתקנה, כך שהוק ה-post-commit יופעל כראוי גם בלקוחות Git גרפיים וב-CI שבהם `~/.local/bin` אינו ב-PATH. אם התקנתם מחדש או שדרגתם את graphify, הריצו שוב `graphify hook install` כדי לרענן את הנתיב המוטמע. + +### בחרו את הפלטפורמה שלכם + +| פלטפורמה | פקודת התקנה | +|----------|----------------| +| Claude Code (לינוקס/מק) | `graphify install` | +| Claude Code (Windows) | `graphify install` (זיהוי אוטומטי) או `graphify install --platform windows` | +| CodeBuddy | `graphify install --platform codebuddy` | +| Codex | `graphify install --platform codex` | +| OpenCode | `graphify install --platform opencode` | +| Kilo Code | `graphify install --platform kilo` | +| GitHub Copilot CLI | `graphify install --platform copilot` | +| VS Code Copilot Chat | `graphify vscode install` | +| Aider | `graphify install --platform aider` | +| OpenClaw | `graphify install --platform claw` | +| Factory Droid | `graphify install --platform droid` | +| Trae | `graphify install --platform trae` | +| Trae CN | `graphify install --platform trae-cn` | +| Gemini CLI | `graphify install --platform gemini` | +| Hermes | `graphify install --platform hermes` | +| Kimi Code | `graphify install --platform kimi` | +| Amp | `graphify amp install` | +| Agent Skills (חוצה-פלטפורמות) | `graphify install --platform agents` (כינוי: `--platform skills`) | +| Kiro IDE/CLI | `graphify kiro install` | +| Pi coding agent | `graphify install --platform pi` | +| Cursor | `graphify cursor install` | +| Devin CLI | `graphify devin install` | +| Google Antigravity | `graphify antigravity install` | + +משתמשי Codex צריכים גם `multi_agent = true` תחת `[features]` בקובץ `~/.codex/config.toml` לצורך חילוץ מקבילי. CodeBuddy משתמש באותו מנגנון Agent tool והוק PreToolUse כמו Claude Code. ‏Factory Droid משתמש בכלי `Task` לשיגור תת-סוכנים במקביל. OpenClaw ו-Aider משתמשים בחילוץ טורי (תמיכה בסוכנים מקביליים עדיין מוקדמת בפלטפורמות אלו). Trae משתמש ב-Agent tool לשיגור תת-סוכנים במקביל ו**אינו** תומך בהוקים מסוג PreToolUse — ‏AGENTS.md הוא המנגנון הקבוע שם. + +`--platform agents` (כינוי: `--platform skills`) מכוון למיקומים הגנריים חוצי-הפלטפורמות של [Agent-Skills](https://github.com/anthropics/skills): ‏`~/.agents/skills/` הגלובלי של המשתמש (נקרא על ידי `npx skills` ומסגרות תואמות-מפרט) בהתקנה גלובלית, ו-`./.agents/skills/` בהתקנת פרויקט (`--project`). ‏`graphify install` החשוף נשאר חד-פלטפורמי (Claude Code) בכוונה — השתמשו בפלטפורמת `agents` כשתרצו שהמיומנות תהיה זמינה לכל מסגרת שקוראת `.agents/skills`. + +> Codex משתמש ב-`‎$graphify` במקום `‎/graphify`. + +### תוספים אופציונליים + +התקינו רק מה שצריך: + +| תוסף | מה הוא מוסיף | התקנה | +|---|---|---| +| `pdf` | חילוץ PDF | `uv tool install "graphifyy[pdf]"` | +| `office` | תמיכה ב-`.docx` ו-`.xlsx` | `uv tool install "graphifyy[office]"` | +| `google` | רינדור Google Sheets | `uv tool install "graphifyy[google]"` | +| `video` | תמלול וידאו/אודיו (faster-whisper + yt-dlp) | `uv tool install "graphifyy[video]"` | +| `mcp` | שרת MCP stdio | `uv tool install "graphifyy[mcp]"` | +| `neo4j` | דחיפה ל-Neo4j | `uv tool install "graphifyy[neo4j]"` | +| `falkordb` | דחיפה ל-FalkorDB | `uv tool install "graphifyy[falkordb]"` | +| `svg` | ייצוא גרף ל-SVG | `uv tool install "graphifyy[svg]"` | +| `leiden` | זיהוי קהילות Leiden ‏(Python < 3.13 בלבד) | `uv tool install "graphifyy[leiden]"` | +| `ollama` | הרצה מקומית עם Ollama | `uv tool install "graphifyy[ollama]"` | +| `openai` | OpenAI / ממשקי API תואמי-OpenAI | `uv tool install "graphifyy[openai]"` | +| `gemini` | Google Gemini API | `uv tool install "graphifyy[gemini]"` | +| `anthropic` | Anthropic Claude API ‏(`--backend claude`, משתמש ב-`ANTHROPIC_API_KEY`) | `uv tool install "graphifyy[anthropic]"` | +| `bedrock` | AWS Bedrock (משתמש ב-IAM, ללא מפתח API) | `uv tool install "graphifyy[bedrock]"` | +| `azure` | Azure OpenAI Service ‏(`--backend azure`, משתמש ב-`AZURE_OPENAI_API_KEY` + ‏`AZURE_OPENAI_ENDPOINT`) | `uv tool install "graphifyy[openai]"` | +| `sql` | חילוץ סכמות SQL | `uv tool install "graphifyy[sql]"` | +| `postgres` | אינטרוספקציה של PostgreSQL חי (`--postgres DSN`) | `uv tool install "graphifyy[postgres]"` | +| `dm` | חילוץ AST של BYOND DreamMaker ‏`.dm`/`.dme` (עשוי לדרוש קומפיילר C + ‏`python3-dev` אם אין wheel מתאים לפלטפורמה) | `uv tool install "graphifyy[dm]"` | +| `terraform` | חילוץ AST של Terraform / HCL ‏`.tf`/`.tfvars`/`.hcl` | `uv tool install "graphifyy[terraform]"` | +| `chinese` | פילוח שאילתות בסינית (jieba) | `uv tool install "graphifyy[chinese]"` | +| `all` | כל מה שלמעלה | `uv tool install "graphifyy[all]"` | + +--- + +## גרמו לעוזר שלכם להשתמש בגרף תמיד + +הריצו פעם אחת בפרויקט אחרי בניית גרף: + +| פלטפורמה | פקודה | +|----------|---------| +| Claude Code | `graphify claude install` | +| CodeBuddy | `graphify codebuddy install` | +| Codex | `graphify codex install` | +| OpenCode | `graphify opencode install` | +| Kilo Code | `graphify kilo install` | +| GitHub Copilot CLI | `graphify copilot install` | +| VS Code Copilot Chat | `graphify vscode install` | +| Aider | `graphify aider install` | +| OpenClaw | `graphify claw install` | +| Factory Droid | `graphify droid install` | +| Trae | `graphify trae install` | +| Trae CN | `graphify trae-cn install` | +| Cursor | `graphify cursor install` | +| Gemini CLI | `graphify gemini install` | +| Hermes | `graphify hermes install` | +| Kimi Code | `graphify install --platform kimi` | +| Amp | `graphify amp install` | +| Agent Skills (חוצה-פלטפורמות) | `graphify agents install` (כינוי: `graphify skills install`) | +| Kiro IDE/CLI | `graphify kiro install` | +| Pi coding agent | `graphify pi install` | +| Devin CLI | `graphify devin install` | +| Google Antigravity | `graphify antigravity install` | + +הפקודה כותבת קובץ תצורה קטן שמנחה את העוזר שלכם להתייעץ עם גרף הידע בשאלות על בסיס הקוד — ולהעדיף שאילתות ממוקדות כמו `graphify query "<שאלה>"` על פני קריאת הדוח המלא או grep על קבצים גולמיים. בפלטפורמות שתומכות בהוקים נושאי-מטען (Claude Code, ‏Gemini CLI), הוק מופעל אוטומטית לפני קריאות כלי בסגנון חיפוש (וב-Claude Code גם לפני קריאת קובצי מקור אחד-אחד דרך הכלים Read/Glob) ומכוון את העוזר לנתיב הגרף. באחרות (Codex, ‏OpenCode, ‏Cursor וכו'), קובצי ההנחיות הקבועים (`AGENTS.md`, ‏`.cursor/rules/` וכו') מספקים את אותה הנחיית "קודם הגרף". ‏`GRAPH_REPORT.md` עדיין זמין לסקירת ארכיטקטורה רחבה. + +**CodeBuddy** עושה את אותם שני דברים כמו Claude Code: כותב קטע `CODEBUDDY.md` שמנחה את CodeBuddy לקרוא את `graphify-out/GRAPH_REPORT.md` לפני מענה על שאלות ארכיטקטורה, ומתקין **הוקים מסוג PreToolUse** ‏(`.codebuddy/settings.json`) שמופעלים לפני פקודות חיפוש ב-Bash וקריאת קבצים, ומכוונים ל-`graphify query` במקום. + +**Codex** כותב ל-`AGENTS.md` וגם מתקין **הוק PreToolUse** ב-`.codex/hooks.json` שמופעל לפני כל קריאת כלי Bash — אותו מנגנון קבוע כמו ב-Claude Code. + +להסרת graphify מכל הפלטפורמות בבת אחת: `graphify uninstall` (הוסיפו `--purge` כדי למחוק גם את `graphify-out/`). או השתמשו בפקודה הפר-פלטפורמית (למשל `graphify claude uninstall`). + +--- + +**Kilo Code** מתקין את מיומנות Graphify ל-`~/.config/kilo/skills/graphify/SKILL.md` ופקודת `‎/graphify` נטיבית ל-`~/.config/kilo/command/graphify.md`. ‏`graphify kilo install` כותב גם `AGENTS.md` וגם **תוסף `tool.execute.before` נטיבי** (`.kilo/plugins/graphify.js` + רישום ב-`.kilo/kilo.json` או `.kilo/kilo.jsonc`) כך ש-Kilo מקבל את אותה התנהגות תזכורת-גרף קבועה דרך תצורת `.kilo` נטיבית. + +**Cursor** כותב `.cursor/rules/graphify.mdc` עם `alwaysApply: true` — ‏Cursor מכליל אותו בכל שיחה אוטומטית, ללא צורך בהוק. + +## מה יש בדוח + +- **צומתי מפתח (God nodes)** — המושגים המקושרים ביותר בפרויקט. הכול עובר דרכם. +- **קשרים מפתיעים** — קישורים בין דברים שחיים בקבצים או מודולים שונים. מדורגים לפי מידת ההפתעה. +- **ה"למה"** — הערות בקוד (`# NOTE:`‏, `# WHY:`‏, `# HACK:`), ‏docstrings ורציונל עיצובי מהמסמכים מחולצים כצמתים נפרדים המקושרים לקוד שהם מסבירים. +- **שאלות מוצעות** — 4–5 שאלות שהגרף נמצא בעמדה ייחודית לענות עליהן. +- **תגי ביטחון** — כל קשר מוסק מסומן `EXTRACTED`‏, `INFERRED` או `AMBIGUOUS`. תמיד יודעים מה נמצא ומה נוחש. + +--- + +## אילו קבצים הוא מטפל + +| סוג | סיומות | +|------|-----------| +| קוד (36 דקדוקי tree-sitter) | `.py .ts .js .jsx .tsx .mjs .go .rs .java .c .cpp .h .hpp .cu .cuh .metal .rb .cs .kt .scala .php .swift .lua .luau .zig .ps1 .psm1 .ex .exs .m .mm .jl .vue .svelte .astro .groovy .gradle .dart .v .sv .svh .sql .f .f90 .f95 .f03 .f08 .pas .pp .dpr .dpk .lpr .inc .dfm .lfm .lpk .sh .bash .json .dm .dme .dmi .dmm .dmf .sln .slnx .csproj .fsproj .vbproj .xaml .razor .cshtml` ‏(`.dm`/`.dme` דורש `uv tool install graphifyy[dm]`; ‏CUDA ‏`.cu`/`.cuh` ו-Metal ‏`.metal` משתמשים בדקדוק C++) | +| Salesforce Apex | `.cls .trigger` (מבוסס regex; מחלקות, ממשקים, enums, מתודות, טריגרים, קשתות SOQL/DML) | +| Terraform / HCL | `.tf .tfvars .hcl` (דורש `uv tool install graphifyy[terraform]`) | +| תצורות MCP | `.mcp.json` ‏`mcp.json` ‏`mcp_servers.json` ‏`claude_desktop_config.json` — מחלץ צומתי שרתים, הפניות לחבילות, דרישות משתני סביבה | +| מניפסטים של חבילות | `apm.yml` ‏`pyproject.toml` ‏`go.mod` ‏`pom.xml` — צומת חבילה קנוני אחד לכל חבילה (לפי שם) בתוספת קשתות `depends_on`, כך שחבילה שמופנית מכמה מניפסטים היא מוקד (hub) יחיד | +| מסמכים | `.md .mdx .qmd .html .txt .rst .yaml .yml` (קישורי markdown ‏`[text](./other.md)` ו-`[[wikilinks]]` הופכים לקשתות `references` בין מסמכים) | +| Office | `.docx .xlsx` (דורש `uv tool install graphifyy[office]`) | +| Google Workspace | `.gdoc .gsheet .gslides` ‏(opt-in; דורש אימות `gws` ו-`--google-workspace`; ‏Sheets דורש `uv tool install graphifyy[google]`) | +| PDF | `.pdf` | +| תמונות | `.png .jpg .webp .gif` | +| וידאו / אודיו | `.mp4 .mov .mp3 .wav` ועוד (דורש `uv tool install graphifyy[video]`) | +| YouTube / כתובות URL | כל כתובת וידאו (דורש `uv tool install graphifyy[video]`) | + +קוד מחולץ מקומית ללא קריאות API ‏(AST באמצעות tree-sitter). כל השאר עובר דרך ה-API של מודל עוזר ה-AI שלכם. + +קובצי `.gdoc`‏, `.gsheet` ו-`.gslides` של Google Drive לשולחן העבודה הם קיצורי דרך, לא תוכן מסמכים. כדי לכלול מסמכי Google Docs, ‏Sheets ו-Slides נטיביים בחילוץ headless, התקינו ואמתו את [`gws` CLI](https://github.com/googleworkspace/cli), ואז הריצו: + +
+ +```bash +uv tool install "graphifyy[google]" # נדרש לרינדור טבלאות Google Sheets +gws auth login -s drive +graphify extract ./docs --google-workspace +``` + +
+ +אפשר גם להגדיר `GRAPHIFY_GOOGLE_WORKSPACE=1`. ‏graphify מייצא קיצורי דרך אל `graphify-out/converted/` כקובצי Markdown נלווים, ואז מחלץ אותם. + +--- + +## פקודות נפוצות + +
+ +```bash +/graphify . # בניית גרף לתיקייה הנוכחית +/graphify ./docs --update # חילוץ מחדש של קבצים שהשתנו בלבד +/graphify . --cluster-only # הרצת אשכול מחדש בלי לחלץ מחדש +/graphify . --cluster-only --resolution 1.5 # קהילות מפורטות יותר +/graphify . --cluster-only --exclude-hubs 99 # הסתרת מוקדי-עזר מדירוג צומתי המפתח +/graphify . --no-viz # דילוג על ה-HTML, רק הדוח וה-JSON +/graphify . --wiki # בניית ויקי markdown מהגרף +graphify export callflow-html # ‏HTML ארכיטקטורה/זרימת-קריאות ב-Mermaid (מתחדש אוטומטית בכל commit אם ההוק מותקן) + +/graphify query "מה מחבר את האימות למסד הנתונים?" +/graphify path "UserService" "DatabasePool" +/graphify explain "RateLimiter" + +/graphify add https://arxiv.org/abs/1706.03762 # הבאת מאמר והוספתו +/graphify add # תמלול והוספת סרטון + +graphify hook install # בנייה מחדש אוטומטית בכל commit +graphify merge-graphs a.json b.json # מיזוג שני גרפים + +graphify prs # לוח PR: מצב CI, סטטוס ביקורת, מיפוי worktree +graphify prs 42 # צלילה עמוקה ל-PR ‏#42 עם השפעת גרף +graphify prs --triage # ‏AI מדרג את תור הביקורות שלכם (משתמש ב-backend המוגדר) +graphify prs --conflicts # ‏PRs שחולקים קהילות גרף — סיכון בסדר המיזוג +``` + +
+ +ראו את [רשימת הפקודות המלאה](#רשימת-הפקודות-המלאה) בהמשך. + +--- + +## התעלמות מקבצים + +צרו קובץ `.graphifyignore` בשורש הפרויקט — אותו תחביר כמו `.gitignore`, כולל שלילה עם `!`. + +**‏`.gitignore` נאכף אוטומטית.** ‏graphify קורא את ה-`.gitignore` בכל תיקייה. אם קיים גם `.graphifyignore`, השניים **ממוזגים** — תבניות `.graphifyignore` מוערכות אחרונות, כך שהן גוברות במקרה של התנגשות (כולל שלילות `!`). הוספת `.graphifyignore` רק מחריגה עוד; היא לעולם לא תחזיר קובץ שה-`.gitignore` כבר החריג. תחולת תת-תיקיות עובדת כמו ב-git — קובץ ignore משפיע רק על תת-העץ שלו. + +
+ +``` +# .graphifyignore +node_modules/ +dist/ +*.generated.py + +# לאנדקס רק את src/, להתעלם מכל השאר +* +!src/ +!src/** +``` + +
+ +--- + +## עבודת צוות + +‏`graphify-out/` מיועד להיכנס ל-git כדי שכל חברי הצוות יתחילו עם מפה מוכנה. + +**תוספות מומלצות ל-`.gitignore`:** + +
+ +``` +graphify-out/cost.json # מקומי בלבד +# graphify-out/cache/ # אופציונלי: commit למהירות, דילוג לשמירה על ריפו קטן +``` + +
+ +> ‏`manifest.json` הוא כעת נייד — המפתחות נשמרים כנתיבים יחסיים ומעוגנים מחדש בטעינה, כך שבטוח לבצע לו commit והדבר חוסך בנייה מלאה מחדש ב-checkout הראשון. + +**תהליך העבודה:** +1. אחד מחברי הצוות מריץ `‎/graphify .` ומבצע commit ל-`graphify-out/`. +2. כולם מושכים — העוזר שלהם קורא את הגרף מיד. +3. הריצו `graphify hook install` לבנייה אוטומטית מחדש אחרי כל commit ‏(AST בלבד, ללא עלות API). זה גם מגדיר merge driver של git כך ש-`graph.json` לעולם לא יישאר עם סימוני קונפליקט — שני מפתחים שמבצעים commit במקביל מקבלים מיזוג-איחוד אוטומטי של הגרפים. +4. כשמסמכים או מאמרים משתנים, הריצו `‎/graphify --update` לרענון הצמתים הללו. + +--- + +## שימוש ישיר בגרף + +
+ +```bash +# שאילתת גרף מהטרמינל +graphify query "הצג את זרימת האימות" +graphify query "מה מחבר בין DigestAuth ל-Response?" --graph graphify-out/graph.json + +# חשיפת הגרף כשרת MCP (לגישת כלים חוזרת) +python -m graphify.serve graphify-out/graph.json +python -m graphify.serve --graph graphify-out/graph.json # גם הדגל --graph מתקבל + +# רישום ב-Kimi Code: +kimi mcp add --transport stdio graphify -- python -m graphify.serve graphify-out/graph.json + +# או הגשה על HTTP כך שכל הצוות מצביע על URL אחד (בלי graphify מקומי): +python -m graphify.serve graphify-out/graph.json --transport http --port 8080 +python -m graphify.serve graphify-out/graph.json --transport http --host 0.0.0.0 --api-key "$SECRET" +``` + +
+ +שרת ה-MCP נותן לעוזר שלכם גישה מובנית: `query_graph`‏, `get_node`‏, `get_neighbors`‏, `shortest_path`‏, `list_prs`‏, `get_pr_impact`‏, `triage_prs`. + +### שרת HTTP משותף + +`--transport stdio` (ברירת המחדל) מריץ שרת מקומי אחד לכל מפתח. `--transport http` מגיש את אותם כלים על גבי MCP Streamable HTTP, כך שתהליך משותף אחד יכול לשרת את הגרף לכל הצוות — הלקוחות מכוונים את תצורת ה-MCP של ה-IDE אל `http://:8080/mcp` במקום להריץ graphify מקומית. + +| דגל | ברירת מחדל | תפקיד | +|---|---|---| +| `--transport {stdio,http}` | `stdio` | סוג התעבורה | +| `--host` | `127.0.0.1` | כתובת ההאזנה ל-HTTP (השתמשו ב-`0.0.0.0` לחשיפה מעבר ל-localhost) | +| `--port` | `8080` | פורט ההאזנה | +| `--api-key` | משתנה סביבה `GRAPHIFY_API_KEY` | דרישת `Authorization: Bearer ` (או `X-API-Key`) | +| `--path` | `/mcp` | נתיב העיגון ב-HTTP | +| `--json-response` | כבוי | החזרת JSON רגיל במקום זרמי SSE | +| `--stateless` | כבוי | ללא מצב פר-סשן (לפריסות מאוזנות-עומס / CI) | +| `--session-timeout` | `3600` | ניקוי סשנים לא פעילים אחרי N שניות (`0` מבטל) | + +ברירת המחדל `127.0.0.1` היא loopback בלבד. הגדירו `--host 0.0.0.0` **וגם** `--api-key` יחד כשחושפים על מארח משותף. אפשר להריץ בקונטיינר: + +
+ +```bash +docker build -t graphify . +docker run -p 8080:8080 -v "$(pwd)/graphify-out:/data" graphify \ + /data/graph.json --transport http --host 0.0.0.0 --api-key "$SECRET" +``` + +
+ +> **הערת WSL / לינוקס:** אובונטו מגיעה עם `python3`, לא `python`. השתמשו ב-venv כדי להימנע מהתנגשויות: + +
+ +```bash +python3 -m venv .venv && .venv/bin/pip install "graphifyy[mcp]" +``` + +
+ +--- + +## משתני סביבה + +אלו נדרשים רק ל**חילוץ headless / CI** ‏(`graphify extract`). בהרצה דרך מיומנות `‎/graphify` בתוך ה-IDE, ה-API של המודל מסופק על ידי סשן ה-IDE — לא נדרשים מפתחות נוספים. + +| משתנה | משמש עבור | מתי נדרש | +|---|---|---| +| `ANTHROPIC_API_KEY` | ‏backend של Claude ‏(Anthropic) | `--backend claude` | +| `ANTHROPIC_BASE_URL` | כתובת endpoint תואם-Anthropic ‏(פרוקסי LiteLLM, שערים, ...) | `--backend claude` (ברירת מחדל: `https://api.anthropic.com`) | +| `ANTHROPIC_MODEL` | שם המודל ל-backend של Claude — ל-endpoints מותאמים, השתמשו בשם/כינוי שהשרת שלכם חושף | `--backend claude` (ברירת מחדל: `claude-sonnet-4-6`) | +| `GEMINI_API_KEY` או `GOOGLE_API_KEY` | ‏backend של Google Gemini | `--backend gemini` | +| `OPENAI_API_KEY` | ‏OpenAI או ממשקים תואמי-OpenAI | `--backend openai` (שרתים מקומיים מקבלים כל ערך לא ריק) | +| `OPENAI_BASE_URL` | כתובת שרת תואם-OpenAI ‏(llama.cpp, ‏vLLM, ‏LM Studio, ...) | `--backend openai` (ברירת מחדל: `https://api.openai.com/v1`) | +| `OPENAI_MODEL` | שם המודל ל-backend של OpenAI — לשרתים בהרצה עצמית, השתמשו בשם/כינוי שהשרת חושף (בדקו את endpoint ה-`/v1/models` שלו) | `--backend openai` (ברירת מחדל: `gpt-4.1-mini`) | +| `DEEPSEEK_API_KEY` | ‏backend של DeepSeek | `--backend deepseek` | +| `MOONSHOT_API_KEY` | ‏backend של Kimi Code | `--backend kimi` | +| `OLLAMA_BASE_URL` | כתובת הרצה מקומית של Ollama | `--backend ollama` (ברירת מחדל: `http://localhost:11434`) | +| `OLLAMA_MODEL` | שם מודל Ollama | `--backend ollama` (ברירת מחדל: זיהוי אוטומטי) | +| `GRAPHIFY_OLLAMA_NUM_CTX` | דריסת גודל חלון ה-KV-cache של Ollama | אופציונלי — מותאם אוטומטית כברירת מחדל | +| `GRAPHIFY_OLLAMA_KEEP_ALIVE` | דקות להשארת מודל Ollama טעון | אופציונלי — ‏`0` לפריקה אחרי כל chunk | +| `AZURE_OPENAI_API_KEY` | ‏backend של Azure OpenAI Service | `--backend azure` | +| `AZURE_OPENAI_ENDPOINT` | כתובת ה-endpoint של משאב Azure | `--backend azure` (נדרש יחד עם מפתח ה-API) | +| `AZURE_OPENAI_API_VERSION` | דריסת גרסת ה-API של Azure | אופציונלי — ברירת מחדל `2024-12-01-preview` | +| `AZURE_OPENAI_DEPLOYMENT` או `GRAPHIFY_AZURE_MODEL` | שם הפריסה ב-Azure | אופציונלי — ברירת מחדל `gpt-4o` | +| `AWS_*` / `~/.aws/credentials` | ‏AWS Bedrock — שרשרת אישורים סטנדרטית | `--backend bedrock` (ללא מפתח API, משתמש ב-IAM) | +| `GRAPHIFY_MAX_WORKERS` | מספר threads למקביליות AST | אופציונלי — גם דגל `--max-workers` | +| `GRAPHIFY_MAX_OUTPUT_TOKENS` | העלאת תקרת הפלט לקורפוסים צפופים | אופציונלי — למשל `32768` לקבצים גדולים | +| `GRAPHIFY_API_TIMEOUT` | ‏timeout לקריאה בשניות עבור HTTP, ‏claude-cli ו-Anthropic SDK ‏(ברירת מחדל: 600) | אופציונלי — גם דגל `--api-timeout` | +| `GRAPHIFY_MAX_RETRIES` | מספר ניסיונות חוזרים לבקשה שנחסמה בקצב (429) לפני ויתור (ברירת מחדל: 6; מכבד `Retry-After`) | אופציונלי — העלו למגבלות ארגוניות נוקשות; ‏`0` מבטל | +| `GRAPHIFY_FORCE` | כפיית בנייה מחדש של הגרף גם עם פחות צמתים | אופציונלי — גם דגל `--force` | +| `GRAPHIFY_GOOGLE_WORKSPACE` | הפעלה אוטומטית של ייצוא Google Workspace | אופציונלי — הגדירו `1` | +| `GRAPHIFY_TRIAGE_BACKEND` | ‏backend עבור `graphify prs --triage` | אופציונלי — מזוהה אוטומטית מהמפתחות הזמינים | +| `GRAPHIFY_TRIAGE_MODEL` | דריסת מודל לטריאז' | אופציונלי — למשל `claude-opus-4-7` | +| `GRAPHIFY_QUERY_LOG` | דריסת נתיב יומן השאילתות (ברירת מחדל: `~/.cache/graphify-queries.log`) | אופציונלי — ערך ריק או `/dev/null` להשתקה | +| `GRAPHIFY_QUERY_LOG_DISABLE` | הגדירו `1` לביטול מוחלט של יומן השאילתות | אופציונלי | +| `GRAPHIFY_QUERY_LOG_RESPONSES` | הגדירו `1` לרישום גם של תשובות תת-גרף מלאות (כבוי כברירת מחדל) | אופציונלי | +| `GRAPHIFY_MAX_GRAPH_BYTES` | דריסת תקרת הגודל של graph.json ‏(512 MiB) — למשל `700MB`, ‏`2GB` או בייטים | אופציונלי — שימושי לקורפוסים גדולים מאוד | +| `GRAPHIFY_LLM_TEMPERATURE` | דריסת טמפרטורת ה-LLM לחילוץ סמנטי — למשל `0.7`, או `none` להשמטה | אופציונלי — מושמט אוטומטית למודלי היסק o1/o3/o4/gpt-5 | + +--- + +## פרטיות + +- **קובצי קוד** — מעובדים מקומית עם tree-sitter. שום דבר לא עוזב את המחשב. קורפוס של קוד בלבד אינו דורש מפתח API — ‏`graphify extract` רץ לגמרי offline. +- **וידאו / אודיו** — מתומלל מקומית עם faster-whisper. שום דבר לא עוזב את המחשב. +- **מסמכים, PDF, תמונות** — נשלחים לעוזר ה-AI שלכם לחילוץ סמנטי (דרך מיומנות `‎/graphify`, עם המודל שסשן ה-IDE שלכם מריץ). ‏`graphify extract` בחילוץ headless דורש `GEMINI_API_KEY` / ‏`GOOGLE_API_KEY` ‏(Gemini), ‏`MOONSHOT_API_KEY` ‏(Kimi), ‏`ANTHROPIC_API_KEY` ‏(Claude), ‏`OPENAI_API_KEY` ‏(OpenAI), ‏`DEEPSEEK_API_KEY` ‏(DeepSeek), מופע Ollama רץ (`OLLAMA_BASE_URL`), אישורי AWS דרך שרשרת הספקים הסטנדרטית (Bedrock — ללא מפתח API, משתמש ב-IAM), או קובץ ההרצה `claude` ‏(Claude Code — ללא מפתח API, משתמש במנוי Claude שלכם). הדגל `--dedup-llm` משתמש באותו מפתח. +- **מיקום נתונים (Data residency)** — ‏`graphify extract` מזהה אוטומטית באיזה ספק להשתמש לפי המפתח שמוגדר (עדיפות: Gemini ‏→ Kimi ‏→ Claude ‏→ OpenAI ‏→ DeepSeek ‏→ Azure ‏→ Bedrock ‏→ Ollama). לקוד עם דרישות מיקום נתונים, השתמשו ב-`--backend ollama` (מקומי לחלוטין) או העבירו דגל `--backend` מפורש. ‏Kimi ‏(`MOONSHOT_API_KEY`) מנתב לשרתי Moonshot AI בסין. +- ללא טלמטריה, ללא מעקב שימוש, ללא אנליטיקה. +- **יומן שאילתות** — כל קריאת `graphify query`‏, `graphify path`‏, `graphify explain` ו-`query_graph` של MCP נרשמת ל-`~/.cache/graphify-queries.log` בפורמט JSON Lines ‏(חותמת זמן, שאלה, קורפוס, צמתים שהוחזרו, משך). תשובות תת-גרף מלאות **אינן** נשמרות כברירת מחדל. הגדירו `GRAPHIFY_QUERY_LOG_DISABLE=1` לביטול, או `GRAPHIFY_QUERY_LOG=/dev/null` להשתקה בלי לבטל את המנגנון. + +--- + +## פתרון בעיות + +**‏`graphify: command not found` אחרי ההתקנה** +ה-CLI מותקן אבל תיקיית ה-bin שלו אינה ב-`PATH` של המעטפת. בחרו את התיקון לפי אופן ההתקנה: +- **uv** ‏(`uv tool install graphifyy`): הפקודה מגיעה לתיקיית הכלים של uv ‏(`~/.local/bin`), שהתקנת macOS/zsh טרייה לרוב לא כוללת ב-`PATH`. הריצו `uv tool update-shell` ופתחו טרמינל חדש. (מצאו את התיקייה עם `uv tool dir --bin`.) +- **pipx** ‏(`pipx install graphifyy`): הריצו `pipx ensurepath` ופתחו טרמינל חדש. +- **pip** ‏(`pip install graphifyy`): ‏pip מתקין סקריפטים לתיקיית bin של המשתמש שאולי אינה ב-`PATH` — הוסיפו את `~/Library/Python/3.x/bin` ‏(macOS) או `~/.local/bin` ‏(לינוקס) ל-`PATH` ב-`~/.zshrc`/`~/.bashrc`, או פשוט הריצו `python -m graphify`. + +**‏`uvx graphify …` או `uv tool run graphify …` לא מצליחים לפתור את `graphify`** +חבילת ה-PyPI היא `graphifyy`; ‏`graphify` הוא רק הפקודה שהיא מספקת. ‏`uv tool run` מתייחס למילה הראשונה כשם *חבילה*, מחפש חבילה בשם `graphify` ומדווח `No solution found … no versions of graphify`. ציינו את החבילה במפורש: `uvx --from graphifyy graphify install` (זהה ל-`uv tool run --from graphifyy graphify install`). או התקינו פעם אחת עם `uv tool install graphifyy` וקראו ל-`graphify` ישירות. + +**‏`python -m graphify` עובד אבל פקודת `graphify` לא** +ה-`PATH` של המעטפת לא כולל את תיקיית ה-bin שאליה הותקנה הפקודה. העדיפו `uv tool install` / ‏`pipx install` על פני `pip` רגיל, ואז הריצו `uv tool update-shell` / ‏`pipx ensurepath` ופתחו טרמינל חדש (ראו הערות ההתקנה לעיל). + +**‏`‎/graphify .` גורם ל-"path not recognized" ב-PowerShell** +‏PowerShell מתייחס ל-`/` מוביל כמפריד נתיבים. השתמשו ב-`graphify .` (בלי לוכסן) ב-Windows. + +**לגרף יש פחות צמתים אחרי `--update` או בנייה מחדש** +אם refactor מחק קבצים, הצמתים הישנים נשארים. העבירו `--force` (או הגדירו `GRAPHIFY_FORCE=1`) לדריסה גם כשהבנייה החדשה קטנה יותר. + +**לגרף יש צמתים כפולים לאותה ישות (ghost duplicates)** +כפילויות רפאים (אותו סימבול מופיע פעמיים — פעם מחילוץ AST עם מיקום מקור, ופעם מחילוץ סמנטי בלעדיו) ממוזגות כעת אוטומטית בזמן הבנייה. אם אתם רואים זאת בגרף שנבנה לפני v0.8.33, הריצו חילוץ מלא מחדש לניקוי: + +
+ +```bash +graphify extract . --force +``` + +
+ +**‏Ollama נגמר לו ה-VRAM / חריגה מחלון ההקשר** +חלון ה-KV-cache מותאם אוטומטית אך עשוי להיות גדול מדי ל-GPU שלכם. הקטינו אותו: + +
+ +```bash +GRAPHIFY_OLLAMA_NUM_CTX=8192 graphify extract ./docs --backend ollama --token-budget 4000 +``` + +
+ +**אזהרות `LLM returned invalid JSON` / ‏`Unterminated string`** +תשובת ה-JSON של המודל פגעה במגבלת אסימוני הפלט ונחתכה באמצע מחרוזת. ‏graphify מתאושש אוטומטית (הוא מפצל את ה-chunk ומחלץ מחדש את החצאים, ומסמך יחיד גדול מדי נחתך תחילה בגבולות כותרות/פסקאות כך שהקובץ כולו עדיין מכוסה), כך שהאזהרות רועשות אך אין אובדן נתונים. להפחתת התופעה, העלו את תקרת הפלט או הקטינו את פלט ה-chunk: + +
+ +```bash +GRAPHIFY_MAX_OUTPUT_TOKENS=16384 graphify extract . --mode deep # הרמת התקרה +graphify extract . --mode deep --token-budget 4000 # קלטים קטנים יותר -> פלט קטן יותר +``` + +
+ +עם שער ענן כמו OpenRouter, העדיפו `--backend openai` (הגדירו `OPENAI_BASE_URL`) על פני שכבת ה-Ollama — זה נתיב תואם-OpenAI נקי יותר. אם למודל יש תקרת פלט משלו, הורדת `--token-budget` היא המנוף האמין. + +**‏HTML של הגרף גדול מדי לפתיחה בדפדפן (מעל 5000 צמתים)** +דלגו על יצירת ה-HTML והשתמשו ב-JSON ישירות: + +
+ +```bash +graphify cluster-only ./my-project --no-viz +graphify query "..." +``` + +
+ +**ל-`graph.json` יש סימוני קונפליקט אחרי ששני מפתחים ביצעו commit במקביל** +הריצו `graphify hook install` — הוא מגדיר merge driver של git שממזג-מאחד את `graph.json` אוטומטית כך שקונפליקטים לא קורים בכלל. + +**החילוץ מחזיר צמתים/קשתות ריקים למסמכים או PDF** +מסמכים, PDF ותמונות דורשים קריאת LLM — קורפוסים של קוד בלבד אינם דורשים מפתח. ודאו שמפתח ה-API מוגדר וה-backend נכון: + +
+ +```bash +ANTHROPIC_API_KEY=sk-... graphify extract ./docs --backend claude +``` + +
+ +**אזהרת חוסר התאמה בגרסת המיומנות ב-IDE** +גרסת graphify המותקנת שונה מקובץ המיומנות. עדכנו: + +
+ +```bash +uv tool upgrade graphifyy +graphify install # דורס את קובץ המיומנות +``` + +
+ +--- + +## רשימת הפקודות המלאה + +
+ +``` +/graphify # הרצה על התיקייה הנוכחית +/graphify ./raw # הרצה על תיקייה מסוימת +/graphify ./raw --mode deep # חילוץ קשרים אגרסיבי יותר +/graphify ./raw --update # חילוץ מחדש של קבצים שהשתנו בלבד +/graphify ./raw --directed # שמירת כיוון הקשתות +/graphify ./raw --cluster-only # הרצת אשכול מחדש על גרף קיים +/graphify ./raw --no-viz # דילוג על ויזואליזציית HTML +/graphify ./raw --obsidian # יצירת כספת Obsidian +/graphify ./raw --obsidian --obsidian-dir ~/vault # כתיבה לכספת קיימת (לעולם לא דורס פתקים או תצורת .obsidian שלכם) +/graphify ./raw --wiki # בניית ויקי markdown שסוכנים יכולים לסרוק +/graphify ./raw --svg # ייצוא graph.svg +/graphify ./raw --graphml # ייצוא ל-Gephi / yEd +/graphify ./raw --neo4j # יצירת cypher.txt ל-Neo4j +/graphify ./raw --neo4j-push bolt://localhost:7687 +/graphify ./raw --falkordb # יצירת cypher.txt ל-FalkorDB +/graphify ./raw --falkordb-push falkordb://localhost:6379 +/graphify ./raw --watch # סנכרון אוטומטי כשקבצים משתנים +/graphify ./raw --mcp # הפעלת שרת MCP stdio + +/graphify add https://arxiv.org/abs/1706.03762 +/graphify add +/graphify add https://... --author "Name" --contributor "Name" + +/graphify query "מה מחבר את ה-attention לאופטימייזר?" +/graphify query "..." --dfs --budget 1500 +/graphify path "DigestAuth" "Response" +/graphify explain "SwinTransformer" + +graphify save-result --question "Q" --answer "A" --nodes Foo Bar --outcome useful # תיעוד תוצאת שאלה-תשובה (זיכרון עבודה; outcome ∈ useful|dead_end|corrected) +graphify reflect # איחוד תוצאות graphify-out/memory/ אל reflections/LESSONS.md +graphify reflect --if-stale # לא עושה דבר אם LESSONS.md כבר חדש מכל הקלטים (זול להרצה בכל סשן) +graphify reflect --out docs/LESSONS.md # כתיבת מסמך הלקחים למקום אחר +graphify reflect --graph graphify-out/graph.json # קיבוץ לקחים לפי קהילה + כתיבת שכבת זיכרון העבודה (.graphify_learning.json) + # השכבה מתייגת צמתים preferred/tentative/contested (משוקלל-עדכניות, עם מקור); + # graphify explain / query מציגים אז רמז "Lesson:", מסומן "code changed — re-verify" כשהמקור התקדם + +graphify uninstall # הסרה מכל הפלטפורמות במכה אחת +graphify uninstall --purge # מחיקה גם של graphify-out/ +graphify uninstall --project --platform codex # הסרת קובצי התקנה ברמת פרויקט בלבד + +graphify hook install # הוקים של post-commit + post-checkout +graphify hook uninstall +graphify hook status + +# הנחיות עוזר קבועות - פר פלטפורמה +graphify claude install # CLAUDE.md + הוק PreToolUse ‏(Claude Code) +graphify claude uninstall +graphify codebuddy install # CODEBUDDY.md + הוק PreToolUse ‏(CodeBuddy) +graphify codebuddy uninstall +graphify codex install # AGENTS.md + הוק PreToolUse ב-.codex/hooks.json ‏(Codex) +graphify opencode install # AGENTS.md + תוסף tool.execute.before ‏(OpenCode) +graphify kilo install # מיומנות Kilo נטיבית + פקודת /graphify + ‏AGENTS.md + תוסף .kilo +graphify kilo uninstall +graphify cursor install # .cursor/rules/graphify.mdc ‏(Cursor) +graphify cursor uninstall +graphify gemini install # GEMINI.md + הוק BeforeTool ‏(Gemini CLI) +graphify gemini uninstall +graphify copilot install # קובץ מיומנות (GitHub Copilot CLI) +graphify copilot uninstall +graphify aider install # AGENTS.md ‏(Aider) +graphify aider uninstall +graphify claw install # AGENTS.md ‏(OpenClaw) +graphify claw uninstall +graphify droid install # AGENTS.md ‏(Factory Droid) +graphify droid uninstall +graphify trae install # AGENTS.md ‏(Trae) +graphify trae uninstall +graphify trae-cn install # AGENTS.md ‏(Trae CN) +graphify trae-cn uninstall +graphify hermes install # AGENTS.md + ~/.hermes/skills/ ‏(Hermes) +graphify hermes uninstall +graphify amp install # קובץ מיומנות (Amp) +graphify amp uninstall +graphify agents install # ~/.agents/skills/ + AGENTS.md ‏(חוצה-פלטפורמות; כינוי: graphify skills) +graphify agents uninstall +graphify kiro install # .kiro/skills/ + .kiro/steering/graphify.md ‏(Kiro IDE/CLI) +graphify kiro uninstall +graphify pi install # קובץ מיומנות (Pi coding agent) +graphify pi uninstall +graphify devin install # קובץ מיומנות + .windsurf/rules/graphify.md ‏(Devin CLI) +graphify devin uninstall +graphify antigravity install # .agents/rules + .agents/workflows ‏(Google Antigravity) +graphify antigravity uninstall + +graphify extract ./docs # חילוץ LLM ‏headless ל-CI ‏(ללא IDE) +graphify extract ./docs --backend gemini # ‏backend מפורש: gemini, kimi, claude, openai, deepseek, ollama, bedrock או claude-cli +graphify extract ./docs --backend gemini --model gemini-3.1-pro-preview +graphify extract ./docs --backend ollama # ‏Ollama מקומי (הגדירו OLLAMA_BASE_URL / OLLAMA_MODEL) - ללא מפתח API ל-loopback +OPENAI_BASE_URL=http://localhost:8080/v1 OPENAI_MODEL=my-model graphify extract ./docs --backend openai # כל שרת תואם-OpenAI ‏(llama.cpp, vLLM, LM Studio) +ANTHROPIC_BASE_URL=http://localhost:4000 ANTHROPIC_MODEL=my-model graphify extract ./docs --backend claude # כל endpoint תואם-Anthropic ‏(פרוקסי LiteLLM, שערים) +GRAPHIFY_OLLAMA_NUM_CTX=32768 graphify extract ./docs --backend ollama # דריסת חלון ה-KV-cache (מותאם אוטומטית כברירת מחדל) +GRAPHIFY_OLLAMA_KEEP_ALIVE=0 graphify extract ./docs --backend ollama # פריקת המודל אחרי כל chunk (חוסך VRAM ב-GPU קטן) +graphify extract ./docs --backend bedrock # AWS Bedrock דרך IAM - ללא מפתח API, שרשרת אישורי AWS +graphify extract ./docs --backend claude-cli # ניתוב דרך Claude Code CLI - ללא מפתח API, דרך מנוי Claude שלכם +graphify extract ./docs --backend azure # Azure OpenAI (הגדירו AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT) +graphify extract ./docs --max-workers 16 # מקביליות AST ‏(גם GRAPHIFY_MAX_WORKERS) +graphify extract --postgres "postgresql://user:pass@host/db" # אינטרוספקציה ישירה של סכמת PostgreSQL חיה +graphify extract ./my-workspace --cargo # אינטרוספקציה ישירה של תלויות Cargo workspace ברוסט +graphify extract ./docs --token-budget 30000 # ‏chunks סמנטיים קטנים יותר למודלים מקומיים/קטנים +graphify extract ./docs --max-concurrency 2 # פחות קריאות LLM מקביליות (שימושי להרצה מקומית) +graphify extract ./docs --api-timeout 900 # ‏timeout ארוך יותר למודלים מקומיים איטיים (ברירת מחדל 600 שניות) +graphify extract ./docs --google-workspace # ייצוא .gdoc/.gsheet/.gslides דרך gws לפני החילוץ +graphify extract ./docs --mode deep # חילוץ סמנטי עשיר יותר עם system prompt מורחב +graphify extract ./docs --no-cluster # חילוץ גולמי בלבד, דילוג על אשכול +graphify extract ./docs --timing # הדפסת זמני ריצה פר-שלב ל-stderr ‏(עובד גם ב-cluster-only) +graphify extract ./docs --force # דריסת graph.json גם אם לגרף החדש פחות צמתים (אחרי refactors או לניקוי כפילויות רפאים) +graphify extract ./docs --dedup-llm # ‏LLM כמכריע לזוגות ישויות עמומים (משתמש באותו מפתח API) +graphify extract ./docs --global --as myrepo # חילוץ ורישום בגרף הגלובלי חוצה-הפרויקטים +GRAPHIFY_MAX_OUTPUT_TOKENS=32768 graphify extract ./docs --backend claude # הרמת תקרת הפלט לקורפוסים צפופים + +graphify export callflow-html # graphify-out/-callflow.html +graphify export callflow-html --max-sections 8 # הגבלת מספר קטעי הארכיטקטורה שנוצרים +graphify export callflow-html --output docs/arch.html +graphify export callflow-html ./some-repo/graphify-out + +graphify global add graphify-out/graph.json --as myrepo # רישום גרף פרויקט אל ~/.graphify/global-graph.json +graphify global remove myrepo # הסרת פרויקט מהגרף הגלובלי +graphify global list # הצגת כל המאגרים הרשומים + ספירות צמתים/קשתות +graphify global path # הדפסת הנתיב לקובץ הגרף הגלובלי + +graphify prs # לוח PR: ‏CI, ביקורת, worktree, השפעת גרף +graphify prs 42 # צלילה עמוקה ל-PR ‏#42 +graphify prs --triage # דירוג טריאז' AI ‏(מזהה backend אוטומטית מהסביבה) +graphify prs --worktrees # מיפוי worktree ← branch ← PR +graphify prs --conflicts # ‏PRs שחולקים קהילות גרף (סיכון בסדר המיזוג) +graphify prs --base main # סינון ל-PRs שמכוונים ל-branch בסיס מסוים +graphify prs --repo owner/repo # הרצה מול מאגר GitHub אחר +GRAPHIFY_TRIAGE_BACKEND=kimi graphify prs --triage # ‏backend מסוים לטריאז' + +graphify clone https://github.com/karpathy/nanoGPT +graphify merge-graphs a.json b.json --out merged.json +graphify --version # הדפסת הגרסה המותקנת +graphify watch ./src +graphify check-update ./src +graphify update ./src +graphify update ./src --no-cluster # דילוג על אשכול מחדש, כתיבת גרף AST גולמי בלבד +graphify update ./src --force # דריסה גם אם לגרף החדש פחות צמתים +graphify cluster-only ./my-project +graphify cluster-only ./my-project --graph path/to/graph.json # מיקום גרף מותאם +graphify cluster-only ./my-project --max-concurrency 16 --batch-size 200 # תיוג קהילות מקבילי (גרפים גדולים) +graphify cluster-only ./my-project --resolution 1.5 # יותר קהילות, קטנות יותר +graphify cluster-only ./my-project --exclude-hubs 99 # החרגת צומתי p99 מהחלוקה +graphify cluster-only ./my-project --no-label # השארת מצייני "Community N" +graphify cluster-only ./my-project --backend=gemini # ‏backend לשמות הקהילות +graphify cluster-only ./my-project --backend=gemini --model gemini-2.5-pro # מודל מסוים +graphify label ./my-project # מתן שמות (מחדש) לקהילות עם ה-backend המוגדר +graphify label ./my-project --backend=openai --model gpt-4o # כפיית backend ומודל מסוימים +``` + +
+ +> **שמות קהילות:** בתוך סוכן (Claude Code, ‏Gemini CLI) הסוכן נותן שמות לקהילות בעצמו. בהרצת ה-CLI החשוף, `cluster-only` נותן שמות אוטומטית עם ה-backend המוגדר (מובנה או ספק תואם-OpenAI מותאם) — העבירו `--no-label` להשארת `Community N`, או הריצו `graphify label` ליצירת שמות מחדש לפי דרישה. + +--- + +## ללמוד עוד + +- [איך זה עובד](../how-it-works.md) — צינור החילוץ, זיהוי קהילות, ניקוד ביטחון, מדדים +- [ARCHITECTURE.md](../../ARCHITECTURE.md) — פירוק מודולים, איך מוסיפים שפה +- [אינטגרציות אופציונליות](../docker-mcp-sqlite.md) — ‏Docker MCP Toolkit + ‏SQLite + +--- + +## נבנה על graphify — ‏Penpax + +[**Penpax**](https://graphifylabs.ai) היא השכבה התמידית שנבנתה מעל graphify — היא מיישמת את אותה גישת גרף על כל חיי העבודה שלכם: פגישות, היסטוריית דפדפן, מיילים, קבצים וקוד, ומתעדכנת ברציפות ברקע. + +נבנתה לאנשים שהעבודה שלהם מפוזרת על פני מאות שיחות ומסמכים שהם לעולם לא יוכלו לשחזר במלואם. ללא ענן, לגמרי על המכשיר. + +**גרסת ניסיון חינמית תושק בקרוב.** [הצטרפו לרשימת ההמתנה ←](https://graphifylabs.ai) + +--- + +
+תרומה לפרויקט + +### הקמת סביבת פיתוח + +הפרויקט משתמש ב-[uv](https://docs.astral.sh/uv/) לתהליך הפיתוח. התקינו אותו פעם אחת, ואז: + +
+ +```bash +git clone https://github.com/safishamsi/graphify.git +cd graphify +git checkout v8 # ‏branch הפיתוח הפעיל + +# יצירת venv לפרויקט והתקנת graphify + כל התוספים + קבוצת dev +# ‏(pytest). ‏uv מתקין את קבוצת התלויות dev כברירת מחדל; העבירו --no-dev לדילוג. +uv sync --all-extras +``` + +
+ +אימות ההתקנה במצב עריכה: + +
+ +```bash +uv run graphify --version +uv run python -c "import graphify; print(graphify.__file__)" +``` + +
+ +### הרצת בדיקות + +
+ +```bash +uv run pytest tests/ -q # הרצת החבילה המלאה +uv run pytest tests/test_extract.py -q # מודול אחד +uv run pytest tests/ -q -k "python" # סינון לפי שם +``` + +
+ +> הערת macOS: חבילת הבדיקות כוללת גם `sample.f90` וגם `sample.F90`. אלו מתנגשים במערכות קבצים HFS+ / APFS שאינן רגישות לרישיות. הריצו בלינוקס או בקונטיינר Docker אם צריך לבדוק את שתי גרסאות ה-Fortran יחד. + +### תהליך עבודה ב-Git + +- הפיתוח הפעיל מתרחש ב-branch ‏`v8`. +- סגנון commit: ‏`fix: ` / ‏`feat: ` / ‏`docs: ` +- לפני פתיחת PR, הריצו `uv run pytest tests/ -q` וודאו שהוא עובר. +- הוסיפו קובץ fixture ל-`tests/fixtures/` ובדיקות ל-`tests/test_languages.py` לכל מחלץ שפה חדש. + +### מה כדאי לתרום + +**דוגמאות עבודה (worked examples)** הן התרומה השימושית ביותר. הריצו `‎/graphify` על קורפוס אמיתי, שמרו את הפלט ב-`worked/{slug}/`, כתבו `review.md` כן שמכסה מה הגרף קלע ומה פספס, ופתחו PR. + +**באגים בחילוץ** — פתחו issue עם קובץ הקלט, רשומת ה-cache ‏(`graphify-out/cache/`) ומה חסר או שגוי. + +ראו [ARCHITECTURE.md](../../ARCHITECTURE.md) לאחריות המודולים ואיך מוסיפים שפה. + +
From bbc3be2238d05b56813be67f382ceeacb5b40720 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 6 Jul 2026 13:22:31 +0100 Subject: [PATCH 007/292] docs(readme): add animated "path lights up" demo to See it in action An animated SVG that mirrors the real `graphify path` output: a terminal types the query on the left while the answer draws itself hop by hop across the knowledge graph on the right, ending on "3 hops. Zero files opened." Pure SMIL inside a self-contained dark card, so it renders inline on both GitHub themes with no JS and no external fonts. Palette is brand-only: muted emerald (sampled from the logo) as the single accent, no neon. Generated by scripts/gen_demo_path.py (re-run to regenerate; STATIC=1 bakes a still frame for QA). Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 4 + docs/demo-path.svg | 1 + scripts/gen_demo_path.py | 236 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 241 insertions(+) create mode 100644 docs/demo-path.svg create mode 100644 scripts/gen_demo_path.py diff --git a/README.md b/README.md index 9f1e11f06..1952acbc2 100644 --- a/README.md +++ b/README.md @@ -70,6 +70,10 @@ A knowledge graph is not a vector index: it is **deterministic**, **every edge i ## See it in action +

+ graphify path query: a terminal asks for the shortest path between FastAPI and ModelField, and the answer lights up hop by hop across the knowledge graph +

+ Once the graph is built you query it instead of reading files. Real output, graphify run on the FastAPI codebase shown above: ```text diff --git a/docs/demo-path.svg b/docs/demo-path.svg new file mode 100644 index 000000000..4bf10a6e8 --- /dev/null +++ b/docs/demo-path.svg @@ -0,0 +1 @@ +graphify$graphifypath"FastAPI""ModelField"Shortest path (3 hops):FastAPI --uses--> DefaultPlaceholder <--references-- get_request_handler() --references--> ModelField3 hops. Zero files opened.usesreferencesreferencesFastAPIDefaultPlaceholderget_request_handler()ModelField \ No newline at end of file diff --git a/scripts/gen_demo_path.py b/scripts/gen_demo_path.py new file mode 100644 index 000000000..acb0dae78 --- /dev/null +++ b/scripts/gen_demo_path.py @@ -0,0 +1,236 @@ +#!/usr/bin/env python3 +"""Generate docs/demo-path.svg: the animated 'path lights up' README hero-companion. + +Concept 1 ("The Path Lights Up"): a terminal types `graphify path ...` on the +left; on the right the same answer draws itself as a graph, a pulse igniting +each hop while the rest of the constellation stays dim. + +Constraints honored: +- Pure SMIL/CSS inside an -referenced SVG (survives GitHub's sanitizer, + no JS, no external fonts). +- Brand palette only: muted emerald green (#4db18f family) as the single + accent on a dark green-black card. No shiny/neon colors. +- One 10s master period; every is dur=10s repeatCount=indefinite and + fades back to its start state before the loop point, so the loop is seamless. + +Run: python3 scripts/gen_demo_path.py -> writes docs/demo-path.svg +""" +import math +import os +import random + +STATIC = bool(os.environ.get("STATIC")) # bake the lit held-frame for visual QA + +T = 10.0 # master loop seconds +HOLD_END = 0.96 # everything fades out over the last 0.4s +FADE = 0.12 # generic fade-in duration (fraction handled per item) + +# ---- brand palette (sampled from docs/logo.png) ---------------------------- +BG = "#0c1712" # dark green-black card +BG2 = "#0f1e18" # subtle gradient partner +BORDER = "#20342b" +DIVIDER = "#1c2c25" +DIM_NODE = "#2c3a34" # unlit nodes +DIM_EDGE = "#22302a" # unlit edges +GREEN = "#4db18f" # brand green (lit) +GREEN_HI = "#62c4a2" # brand green light (accents/caption) +GREEN_DK = "#41a884" # brand green deep +TXT = "#c6d6ce" # primary terminal text +TXT_DIM = "#6f8a7f" # secondary terminal text +PROMPT = "#4db18f" +DOTS = ["#ff5f56", "#ffbd2e", "#27c93f"] # macOS traffic lights: close / minimize / zoom + +W, H = 900, 360 +CHARW = 6.62 # mono advance at 11px +FS = 11 +LX = 20 # left text margin +PANEL = 336 # terminal panel width + +def kt(*pairs): + """pairs of (keyTime, value) -> (values_str, keyTimes_str).""" + ks = ";".join(f"{k:.4f}".rstrip("0").rstrip(".") if k not in (0, 1) else str(int(k)) for k, _ in pairs) + vs = ";".join(str(v) for _, v in pairs) + return vs, ks + +def op0(): + """initial opacity for a revealable element (1 when baking a static frame).""" + return "1" if STATIC else "0" + +def reveal(t, hold_val="1", start_val="0"): + """opacity reveal at time t (s), hold, fade out before loop.""" + if STATIC: + return "" + a = t / T + b = min((t + FADE) / T, HOLD_END - 0.001) + vs, ks = kt((0, start_val), (a, start_val), (b, hold_val), (HOLD_END, hold_val), (1, start_val)) + return f'' + +out = [] +def e(s): out.append(s) + +# ---------------------------------------------------------------- svg header +e(f'') +e('') +e(f'' + f'') +e(f'' + f'' + f'') +e('') + +# card +e(f'') +e(f'') + +# window dots (muted) +for i in range(3): + e(f'') +e(f'graphify') + +# ---------------------------------------------------------------- terminal +line_y = 74 +prompt_x = LX +# static prompt "$" +e(f'$') +cmd = 'graphify path "FastAPI" "ModelField"' +cmd_x0 = prompt_x + CHARW * 2 +type_start = 0.35 +type_iv = 0.058 +for i, ch in enumerate(cmd): + if ch == " ": + continue + cx = cmd_x0 + i * CHARW + t = type_start + i * type_iv + disp = ch.replace("&", "&").replace("<", "<").replace(">", ">") + e(f'{disp}{reveal(t)}') +type_end = type_start + len(cmd) * type_iv + +# blinking caret that steps along while typing, then blinks at the end +caret_xs = [cmd_x0 + i * CHARW for i in range(len(cmd) + 1)] +cvals = ";".join(f"{x:.1f}" for x in caret_xs) + f";{caret_xs[-1]:.1f}" +ckeys = ";".join(f"{(type_start + i*type_iv)/T:.4f}" for i in range(len(cmd) + 1)) + ";1" +# prepend a 0 keyframe +cvals = f"{caret_xs[0]:.1f};" + cvals +ckeys = "0;" + ckeys +if not STATIC: + e(f'' + f'' + f'') + +# output lines: (text, color, appear_time) +oy = line_y + 30 +out_lines = [ + ("Shortest path (3 hops):", TXT_DIM, 2.95), + ("FastAPI", GREEN_HI, 3.25), + (" --uses--> DefaultPlaceholder", TXT, 3.95), + (" <--references-- get_request_handler()", TXT, 4.65), + (" --references--> ModelField", TXT, 5.35), +] +for i, (txt, col, t) in enumerate(out_lines): + y = oy + i * 22 + disp = txt.replace("&", "&").replace("<", "<").replace(">", ">") + e(f'{disp}{reveal(t)}') + +# caption in the terminal footer +cap_y = H - 30 +e(f'' + f'3 hops. Zero files opened.{reveal(5.95)}') + +# ---------------------------------------------------------------- graph +# path nodes (visiting order, reads left-to-right) +GX0, GY0, GXW, GYH = PANEL, 0, W - PANEL, H +path_nodes = [ + ("FastAPI", (PANEL + 70, 205), "below"), + ("DefaultPlaceholder", (PANEL + 195, 118), "above"), + ("get_request_handler()", (PANEL + 335, 210), "below"), + ("ModelField", (PANEL + 470, 120), "above"), +] +path_light_t = [3.25, 3.95, 4.65, 5.35] +rel_labels = ["uses", "references", "references"] + +# distractor constellation +random.seed(42) +distractors = [] +attempts = 0 +avoid = [p[1] for p in path_nodes] +while len(distractors) < 26 and attempts < 4000: + attempts += 1 + x = random.uniform(PANEL + 30, W - 28) + y = random.uniform(30, H - 26) + if all((x-ax)**2 + (y-ay)**2 > 44**2 for ax, ay in avoid) and \ + all((x-dx)**2 + (y-dy)**2 > 34**2 for dx, dy in distractors): + distractors.append((x, y)) + +# dim edges: connect each distractor to a couple of near neighbors / path nodes +allpts = distractors + [p[1] for p in path_nodes] +dim_edges = set() +for i, (x, y) in enumerate(distractors): + dists = sorted(range(len(allpts)), key=lambda j: (allpts[j][0]-x)**2 + (allpts[j][1]-y)**2) + for j in dists[1:3]: + a, b = min(i, j + 0 if j < len(distractors) else j), max(i, j) + dim_edges.add((i, j)) + +e('') +for i, j in dim_edges: + x1, y1 = allpts[i]; x2, y2 = allpts[j] + e(f'') +for (x, y) in distractors: + e(f'') +e('') + +# path base (dim) edges + green draw-on overlays +def line_len(p, q): + return math.hypot(q[0]-p[0], q[1]-p[1]) + +for i in range(3): + p = path_nodes[i][1]; q = path_nodes[i+1][1] + e(f'') + +for i in range(3): + p = path_nodes[i][1]; q = path_nodes[i+1][1] + L = line_len(p, q) + start = path_light_t[i] + 0.05 + draw = 0.55 + a = start / T; b = (start + draw) / T + vs, ks = kt((0, f"{L:.1f}"), (a, f"{L:.1f}"), (b, "0"), (HOLD_END, "0"), (1, f"{L:.1f}")) + off0 = "0" if STATIC else f"{L:.1f}" + anim = "" if STATIC else f'' + e(f'{anim}') + # relationship label at edge midpoint + mx, my = (p[0]+q[0])/2, (p[1]+q[1])/2 - 6 + e(f'{rel_labels[i]}{reveal(start + 0.25)}') + +# path nodes: glow + circle pop + label +for (name, (x, y), pos), t in zip(path_nodes, path_light_t): + a = t / T + # glow halo + ga, gb = a, (t + 0.15) / T + gvs, gks = kt((0, "0"), (ga, "0"), (gb, "1"), (HOLD_END, "1"), (1, "0")) + ganim = "" if STATIC else f'' + e(f'{ganim}') + # node: fill dim->green, radius pop + fvs, fks = kt((0, DIM_NODE), (a, DIM_NODE), ((t+0.06)/T, GREEN_HI), ((t+0.2)/T, GREEN), (HOLD_END, GREEN), (1, DIM_NODE)) + rvs, rks = kt((0, "4"), (a, "4"), ((t+0.05)/T, "8"), ((t+0.18)/T, "5.6"), (HOLD_END, "5.6"), (1, "4")) + nfill = GREEN if STATIC else DIM_NODE + nr = "5.6" if STATIC else "4" + nanim = "" if STATIC else ( + f'' + f'') + e(f'{nanim}') + # label + ly = y - 14 if pos == "above" else y + 22 + e(f'{name}{reveal(t + 0.1)}') + +e('') + +with open("docs/demo-path.svg", "w", encoding="utf-8") as f: + f.write("".join(out)) +print("wrote docs/demo-path.svg", sum(len(s) for s in out), "bytes") From 912d832a6036c37d9b22687648fa7ff291912552 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 6 Jul 2026 16:00:31 +0100 Subject: [PATCH 008/292] Fix: stop silently dropping source files during discovery via two over-broad filters (#1666) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @krishnateja7 root-caused this precisely: the files were never reaching extraction, so the 0.9.7 no-cache-on-empty mitigation could not surface them. Two discovery-layer filters were the cause: (a) A bare `snapshots/` directory was pruned as a Jest/Vitest artifact, which killed legitimate code namespaces like a Rails `app/services/snapshots/`. It is now pruned only when it actually contains `.snap` files or sits directly under a JS test root (`__tests__`/`__test__`). `__snapshots__` stays unconditionally pruned. (b) `_is_sensitive` dropped files on a bare name-keyword hit (device_token.rb, passwords_controller.rb) even when `classify_file` had already resolved them to source code. A genuine programming-language source file is now exempt from the weak keyword heuristic, while real secret stores in data/config formats (credentials.json, secrets.yaml, .env, .pem, ...) are still caught — those route through the CODE path for manifest parsing but are deliberately not exempted. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 1 + graphify/detect.py | 47 +++++++++++++++++++++++++++++++++++++++++--- tests/test_detect.py | 45 ++++++++++++++++++++++++++++++++++++++---- 3 files changed, 86 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 691b1c7ef..34200b835 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ Full release notes with details on each version: [GitHub Releases](https://githu - Fix: the Ollama backend no longer multiplies a hang by the retry count (#1686, thanks @Kyzcreig). A stalled local model would wedge for `timeout * (max_retries + 1)`, which with the default 6 retries turned one long stall into a very long one. Ollama now defaults to zero client-side retries (a local model that stalls will not un-stall on retry); set `GRAPHIFY_MAX_RETRIES` to opt back in. Other backends are unchanged. Note: the underlying stall is non-deterministic and driven by the model server, so this bounds the wait rather than eliminating the hang. - Fix: a truncated or slightly malformed community-labeling reply no longer discards the whole batch (#1690, thanks @vdgbcrypto). `_parse_label_response` now salvages the complete `"id": "name"` pairs from a reply that failed a strict `json.loads` (e.g. a reply truncated mid-object), raising only when no pairs can be recovered. The per-batch token budget was also raised (`256 + 48*n`, was `64 + 24*n`) to give models that prepend a short preamble enough headroom to finish the JSON. The exact provider truncation in the report could not be reproduced without a live key; the parser and budget fixes address the mechanism. - Fix: cluster-only mode now reports the real token cost of community labeling instead of a hardcoded zero (#1694, thanks @sub4biz). The labeling LLM calls were never accounted for, so `GRAPH_REPORT.md`'s "Token cost" line always read `0 input · 0 output` in cluster-only runs. `_call_llm` now accumulates per-response usage into an optional accumulator that is threaded through the labeling path and surfaced in the report. Backends that do not return usage (the Claude Code CLI) still contribute nothing, which is honest rather than estimated. +- Fix: source files are no longer silently dropped during discovery by two over-broad filters (#1666, thanks @krishnateja7 for the precise root-cause). (a) A bare `snapshots/` directory was pruned as a Jest/Vitest artifact, which killed legitimate code namespaces like a Rails `app/services/snapshots/`; it is now pruned only when it actually contains `.snap` files or sits directly under a JS test root (`__snapshots__` stays unconditionally pruned). (b) `_is_sensitive` dropped files on a bare name-keyword hit (`device_token.rb`, `passwords_controller.rb`) even when `classify_file` had already resolved them to source code; a genuine programming-language source file is now exempt from the weak keyword heuristic, while real secret stores in data/config formats (`credentials.json`, `secrets.yaml`, `.env`, `.pem`, ...) are still caught. This is the discovery-layer fix; the 0.9.7 no-cache-on-empty change could not surface these because the files never reached extraction. ## 0.9.7 (2026-07-06) diff --git a/graphify/detect.py b/graphify/detect.py index 080dab813..c9d7792c7 100644 --- a/graphify/detect.py +++ b/graphify/detect.py @@ -125,6 +125,15 @@ def _zip_within_caps(path: Path) -> bool: re.compile(r'(? bool: name = path.name if any(p.search(name) for p in _SENSITIVE_PATTERNS): return True - # Stage 3: generic keywords, only when load-bearing in the name - return _generic_keyword_hit(name) + # Stage 3: generic keywords, only when load-bearing in the name. Do NOT let a + # bare name keyword silently drop a genuine programming-language source file: + # a .rb/.py named device_token or passwords_controller is a module, not a secret + # store (#1666). Data/config formats (.json, .yaml, .toml, ...) are deliberately + # NOT exempt even though .json routes through the CODE path for manifest parsing, + # because credentials.json / oauth_token.json / secrets.yaml are exactly the + # secret stores this stage must catch. The specific Stage 2 patterns (.env, .pem, + # id_rsa, ...) still apply to everything regardless of extension. + if _generic_keyword_hit(name): + ext = path.suffix.lower() + is_source_code = classify_file(path) == FileType.CODE and ext not in _SECRET_PRONE_DATA_EXTS + return not is_source_code + return False def _looks_like_paper(path: Path) -> bool: @@ -677,7 +697,7 @@ def count_words(path: Path) -> int: # Coverage/test-artefact dirs — generated, never architecturally meaningful "coverage", "lcov-report", # Vitest/Istanbul/nyc HTML reports (#870) "visual-tests", "visual-test", # Playwright/visual-regression bundles (#869) - "__snapshots__", "snapshots", # Jest/Vitest snapshot dirs + "__snapshots__", # Jest/Vitest snapshot dir (unambiguous) "storybook-static", # Storybook production build output "dist-protected", # Protected dist variants (same noise as dist) # Framework cache/build dirs — generated, never architecturally meaningful (#873) @@ -694,10 +714,31 @@ def count_words(path: Path) -> int: "composer.lock", "go.sum", "go.work.sum", } +# A bare "snapshots" dir is a Jest/Vitest artifact only when it actually holds +# snapshot files or lives directly under a JS test root. Elsewhere it is often a +# real code namespace (e.g. Rails app/services/snapshots/), so pruning it by name +# silently dropped legitimate source from the graph (#1666). "__snapshots__" stays +# unconditionally pruned above; only the ambiguous bare name is gated here. +_JS_SNAPSHOT_TEST_ROOTS = frozenset({"__tests__", "__test__"}) + + def _is_noise_dir(part: str, parent: "Path | None" = None) -> bool: """Return True if this directory name looks like a venv, cache, or dep dir.""" if part in _SKIP_DIRS: return True + if part == "snapshots": + # Prune only when it looks like an actual JS/Vitest snapshot dir. + if parent is None: + return False # cannot verify; keep a possibly-real code dir + snap_dir = parent / part + if parent.name in _JS_SNAPSHOT_TEST_ROOTS: + return True + try: + if next(snap_dir.glob("*.snap"), None) is not None: + return True + except OSError: + pass + return False # Catch *_venv, *_repo/site-packages patterns if part.endswith("_venv") or part.endswith("_env"): return True diff --git a/tests/test_detect.py b/tests/test_detect.py index 458bb6a84..0c07e24fd 100644 --- a/tests/test_detect.py +++ b/tests/test_detect.py @@ -1,3 +1,4 @@ +import os import unicodedata from pathlib import Path from graphify.detect import classify_file, count_words, detect, detect_incremental, save_manifest, FileType, _looks_like_paper, _is_ignored, _load_graphifyignore, _is_sensitive @@ -467,16 +468,35 @@ def test_detect_skips_visual_tests_dir(tmp_path): def test_detect_skips_snapshots_dir(tmp_path): - """__snapshots__/ and snapshots/ are jest/vitest artefacts — must be excluded.""" + """__snapshots__/ and real jest/vitest snapshots/ dirs are artefacts — excluded.""" (tmp_path / "__snapshots__").mkdir() (tmp_path / "__snapshots__" / "app.test.ts.snap").write_text("// Jest Snapshot\nexports[`test 1`] = `
`") + # a bare snapshots/ dir that actually holds .snap files is still a JS artefact + snap = tmp_path / "snapshots" + snap.mkdir() + (snap / "component.test.tsx.snap").write_text("exports[`renders`] = ``") (tmp_path / "app.ts").write_text("export function greet() { return 'hi'; }") result = detect(tmp_path) all_files = [f for files in result["files"].values() for f in files] assert not any("__snapshots__" in f for f in all_files) + assert not any(f"{os.sep}snapshots{os.sep}" in f for f in all_files) assert any("app.ts" in f for f in all_files) +def test_detect_keeps_snapshots_code_namespace(tmp_path): + """#1666: a bare snapshots/ dir with no .snap files is a legit code namespace + (e.g. Rails app/services/snapshots/) and must NOT be pruned as a JS artefact.""" + svc = tmp_path / "app" / "services" / "snapshots" + svc.mkdir(parents=True) + (svc / "round_reader.rb").write_text("class RoundReader\n def call; end\nend\n") + (svc / "backfill_marker.rb").write_text("class BackfillMarker\n def run; end\nend\n") + (tmp_path / "app.rb").write_text("class App; end\n") + result = detect(tmp_path) + all_files = [f for files in result["files"].values() for f in files] + assert any("round_reader.rb" in f for f in all_files) + assert any("backfill_marker.rb" in f for f in all_files) + + def test_detect_skips_storybook_static_dir(tmp_path): """storybook-static/ is a build artefact — must be excluded.""" sb = tmp_path / "storybook-static" @@ -810,9 +830,26 @@ def test_sensitive_does_not_flag_tokenizer_py(): def test_sensitive_does_not_flag_tokenize_py(): assert not _is_sensitive(Path("tokenize.py")) -def test_sensitive_flags_passwords_py(): - # passwords.py is just as likely a secret store as passwords.txt — code ext is no excuse - assert _is_sensitive(Path("passwords.py")) +def test_sensitive_does_not_flag_passwords_py(): + # #1666: a programming-language source file named after a domain noun is a + # module, not a secret store. Silently dropping it hid real code from the graph. + # Genuine secret stores are .env/.pem/credentials.json etc. (still flagged below). + assert not _is_sensitive(Path("passwords.py")) + + +def test_sensitive_does_not_flag_ruby_code_modules(): + # #1666 exact cases: Rails source modules with keyword-ish names must survive. + assert not _is_sensitive(Path("app/models/device_token.rb")) + assert not _is_sensitive(Path("app/controllers/api/v1/passwords_controller.rb")) + + +def test_sensitive_still_flags_data_secret_stores(): + # #1666 guard: the exemption is ONLY for real source code, not data/config + # formats — credentials.json / oauth_token.json / secrets.yaml are the secret + # stores Stage 3 must keep catching (even though .json routes through CODE). + assert _is_sensitive(Path("credentials.json")) + assert _is_sensitive(Path("oauth_token.json")) + assert _is_sensitive(Path("app_secret.yaml")) def test_sensitive_flags_ssh_dir(): assert _is_sensitive(Path("/home/user/.ssh/id_rsa")) From 5d0137388e06c5a4428602ae1355ad5e7a57e881 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 6 Jul 2026 16:02:45 +0100 Subject: [PATCH 009/292] Feat: opt-in GRAPHIFY_DISABLE_THINKING; correct deepseek thinking default (#1621) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @sub4biz verified against the live DeepSeek API that deepseek-v4-flash (and v4-pro) have thinking ENABLED by default, contradicting the built-in config's stale "non-thinking" comment (now corrected). The naive fix (mirror the kimi branch and force thinking off) is the wrong call: @sub4biz's production testing on real corpora found that disabling thinking removes a rare reasoning-leak failure — which the adaptive extraction/labeling retry already recovers from — but trades it for far more frequent benign truncation AND measurably lower extraction quality and file coverage, confirmed by a blind second reviewer. So thinking stays ON by default (quality/coverage), with a documented opt-in `GRAPHIFY_DISABLE_THINKING=1` for users who prefer run-to-run stability. Applies to reasoning-capable OpenAI-compatible backends at both extra_body sites (extraction + labeling). An explicit providers.json extra_body still wins, and the moonshot/kimi branch is unchanged (it must disable thinking or content is empty). Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 1 + graphify/llm.py | 28 +++++++++++++++++++++-- tests/test_llm_backends.py | 47 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 74 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 34200b835..de1f58978 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ Full release notes with details on each version: [GitHub Releases](https://githu - Fix: the Ollama backend no longer multiplies a hang by the retry count (#1686, thanks @Kyzcreig). A stalled local model would wedge for `timeout * (max_retries + 1)`, which with the default 6 retries turned one long stall into a very long one. Ollama now defaults to zero client-side retries (a local model that stalls will not un-stall on retry); set `GRAPHIFY_MAX_RETRIES` to opt back in. Other backends are unchanged. Note: the underlying stall is non-deterministic and driven by the model server, so this bounds the wait rather than eliminating the hang. - Fix: a truncated or slightly malformed community-labeling reply no longer discards the whole batch (#1690, thanks @vdgbcrypto). `_parse_label_response` now salvages the complete `"id": "name"` pairs from a reply that failed a strict `json.loads` (e.g. a reply truncated mid-object), raising only when no pairs can be recovered. The per-batch token budget was also raised (`256 + 48*n`, was `64 + 24*n`) to give models that prepend a short preamble enough headroom to finish the JSON. The exact provider truncation in the report could not be reproduced without a live key; the parser and budget fixes address the mechanism. - Fix: cluster-only mode now reports the real token cost of community labeling instead of a hardcoded zero (#1694, thanks @sub4biz). The labeling LLM calls were never accounted for, so `GRAPH_REPORT.md`'s "Token cost" line always read `0 input · 0 output` in cluster-only runs. `_call_llm` now accumulates per-response usage into an optional accumulator that is threaded through the labeling path and surfaced in the report. Backends that do not return usage (the Claude Code CLI) still contribute nothing, which is honest rather than estimated. +- Docs/Feat: `deepseek-v4-flash` (and `v4-pro`) have thinking ENABLED by default; graphify no longer implies otherwise and adds an opt-in `GRAPHIFY_DISABLE_THINKING=1` toggle (#1621, thanks @sub4biz for the empirical testing). Disabling thinking removes a rare reasoning-leak failure mode (which the adaptive extraction/labeling retry already recovers from) but, measured on real corpora, trades it for more frequent benign truncation and measurably lower extraction quality and file coverage — so it stays a documented user choice rather than a forced default. The stale "non-thinking" comment on the built-in deepseek config is corrected. The moonshot (kimi) branch is unchanged (it must disable thinking or content comes back empty). - Fix: source files are no longer silently dropped during discovery by two over-broad filters (#1666, thanks @krishnateja7 for the precise root-cause). (a) A bare `snapshots/` directory was pruned as a Jest/Vitest artifact, which killed legitimate code namespaces like a Rails `app/services/snapshots/`; it is now pruned only when it actually contains `.snap` files or sits directly under a JS test root (`__snapshots__` stays unconditionally pruned). (b) `_is_sensitive` dropped files on a bare name-keyword hit (`device_token.rb`, `passwords_controller.rb`) even when `classify_file` had already resolved them to source code; a genuine programming-language source file is now exempt from the weak keyword heuristic, while real secret stores in data/config formats (`credentials.json`, `secrets.yaml`, `.env`, `.pem`, ...) are still caught. This is the discovery-layer fix; the 0.9.7 no-cache-on-empty change could not surface these because the files never reached extraction. ## 0.9.7 (2026-07-06) diff --git a/graphify/llm.py b/graphify/llm.py index a0c073afa..f00ed0c84 100644 --- a/graphify/llm.py +++ b/graphify/llm.py @@ -131,8 +131,10 @@ def _get_tokenizer(): "env_key": "DEEPSEEK_API_KEY", "model_env_key": "GRAPHIFY_DEEPSEEK_MODEL", "pricing": {"input": 0.14, "output": 0.28}, # USD per 1M tokens (v4-flash) - # deepseek-reasoner / thinking-mode models silently ignore temperature; - # deepseek-chat / v4-flash (non-thinking) accept 0-2. Safe to send 0. + # deepseek-reasoner silently ignores temperature; deepseek-chat / v4-flash + # accept 0-2, so sending 0 is safe. Note: deepseek-v4-flash (and v4-pro) have + # thinking ENABLED by default (verified against the live API, #1621) — set + # GRAPHIFY_DISABLE_THINKING=1 to turn it off (tradeoff documented on the flag). "temperature": 0, "max_tokens": 16384, }, @@ -388,6 +390,22 @@ def _resolve_max_retries(default: int = 6) -> int: pass return default + +def _thinking_disabled_via_env() -> bool: + """Opt-in (GRAPHIFY_DISABLE_THINKING) to send ``{"thinking": {"type": "disabled"}}`` + to reasoning-capable OpenAI-compatible models such as ``deepseek-v4-flash``. + + Off by default and deliberately so (#1621): a thinking-on model can occasionally + leak reasoning prose instead of JSON, but that response is caught and re-tried by + the adaptive extraction/labeling retry, so it is a rare, recoverable failure. + Disabling thinking removes that failure mode but, measured on real corpora, trades + it for far more frequent (benign) truncation AND measurably lower extraction + quality and file coverage. So this stays a user choice for those who value + run-to-run stability over extraction quality, not a forced default. The moonshot + (kimi) branch keeps disabling thinking unconditionally because that model returns + empty content otherwise.""" + return os.environ.get("GRAPHIFY_DISABLE_THINKING", "").strip().lower() in ("1", "true", "yes", "on") + _EXTRACTION_SYSTEM = """\ You are a graphify semantic extraction agent. Extract a knowledge graph fragment from the files provided. Output ONLY valid JSON — no explanation, no markdown fences, no preamble. @@ -996,6 +1014,10 @@ def _call_openai_compat( # Kimi-k2.6 is a reasoning model — disable thinking so content isn't empty elif "moonshot" in base_url: kwargs["extra_body"] = {"thinking": {"type": "disabled"}} + # Opt-in only: disable thinking for reasoning models like deepseek-v4-flash + # (#1621). Not a default — see _thinking_disabled_via_env for the tradeoff. + elif _thinking_disabled_via_env(): + kwargs["extra_body"] = {"thinking": {"type": "disabled"}} # Ollama defaults num_ctx to 2048 and silently truncates prompts larger # than that — the symptom is hollow 200 OK responses after the first few # chunks (#798). We derive num_ctx from the actual prompt size so we don't @@ -2113,6 +2135,8 @@ def _rec(inp, out) -> None: kwargs["extra_body"] = cfg["extra_body"] elif "moonshot" in cfg["base_url"]: kwargs["extra_body"] = {"thinking": {"type": "disabled"}} + elif _thinking_disabled_via_env(): + kwargs["extra_body"] = {"thinking": {"type": "disabled"}} resp = client.chat.completions.create(**kwargs) if not resp.choices or resp.choices[0].message is None: raise ValueError("LLM returned empty or filtered response") diff --git a/tests/test_llm_backends.py b/tests/test_llm_backends.py index 79f64027e..e6556ba4a 100644 --- a/tests/test_llm_backends.py +++ b/tests/test_llm_backends.py @@ -578,6 +578,53 @@ def test_call_openai_compat_extra_body_wins_over_moonshot_default(monkeypatch): assert captured["extra_body"] == {"thinking": {"type": "enabled"}} +# --------------------------------------------------------------------------- +# GRAPHIFY_DISABLE_THINKING: opt-in disable-thinking for reasoning models like +# deepseek-v4-flash. Off by default — disabling thinking trades a rare reasoning +# leak for lower extraction quality/coverage, so it must not be forced (#1621). +# --------------------------------------------------------------------------- + + +def test_deepseek_thinking_on_by_default(monkeypatch): + monkeypatch.delenv("GRAPHIFY_DISABLE_THINKING", raising=False) + captured = _install_capturing_openai(monkeypatch) + + llm._call_openai_compat( + "https://api.deepseek.com", "sk", "deepseek-v4-flash", + "u", temperature=0, max_completion_tokens=8192, backend="deepseek", + ) + + eb = captured.get("extra_body") + assert eb is None or "thinking" not in eb, "thinking must NOT be disabled by default" + + +def test_deepseek_thinking_disabled_via_env(monkeypatch): + monkeypatch.setenv("GRAPHIFY_DISABLE_THINKING", "1") + captured = _install_capturing_openai(monkeypatch) + + llm._call_openai_compat( + "https://api.deepseek.com", "sk", "deepseek-v4-flash", + "u", temperature=0, max_completion_tokens=8192, backend="deepseek", + ) + + assert captured["extra_body"] == {"thinking": {"type": "disabled"}} + + +def test_explicit_extra_body_wins_over_thinking_env(monkeypatch): + # A provider-supplied extra_body is an explicit request-shape choice and must + # take precedence over the env toggle. + monkeypatch.setenv("GRAPHIFY_DISABLE_THINKING", "1") + captured = _install_capturing_openai(monkeypatch) + + llm._call_openai_compat( + "https://api.deepseek.com", "sk", "deepseek-v4-flash", + "u", temperature=0, max_completion_tokens=8192, backend="deepseek", + extra_body={"thinking": {"type": "enabled"}}, + ) + + assert captured["extra_body"] == {"thinking": {"type": "enabled"}} + + def test_call_openai_compat_explicit_extra_body_skips_ollama_auto_derive(monkeypatch): # An explicit extra_body means "I own this request shape" — Ollama's # num_ctx auto-derive (a default) must step aside or we'd clobber it. From f7911fd1b3e8362bde0019472b4c0d873d61087b Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 6 Jul 2026 16:38:42 +0100 Subject: [PATCH 010/292] Fix: make Claude Code / Codebuddy PreToolUse graph-nudge hooks work on Windows (#522) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The hooks were inline POSIX bash (case/esac, [ -f ], single-quoted echo), which Windows cmd.exe/PowerShell cannot parse. On Windows the hook failed silently, so the "run `graphify query` before grepping/reading raw files" nudge was never injected and users fell back to manual /graphify. The detection logic (grep-command match; source/doc extension match; skip if the target is under the output dir; require graph.json to exist) moved into a shell-agnostic `graphify hook-guard ` subcommand, invoked via the absolute exe path resolved by _resolve_graphify_exe() — the exact pattern the codex hook already uses. A single console-script invocation has no shell syntax, so it parses identically under sh, cmd.exe and PowerShell. Behavior on macOS/Linux is unchanged: the nudge payload is byte-identical (compact JSON, same additionalContext text), matchers stay "Bash"/"Read|Glob" so install/uninstall still find and replace old hooks, and the command still contains "graphify". The graph-exists check now honors GRAPHIFY_OUT instead of the hardcoded graphify-out/ path. Codex stays a no-op there (hook-check) because Codex Desktop rejects additionalContext. Detection is fully unit-tested (ported test_read_hook.py + new test_search_hook.py, byte-identical output on POSIX); Windows execution itself is not testable in CI here, but the mechanism is now shell-independent by construction. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 1 + graphify/__main__.py | 179 ++++++++++++++++++++++------------ tests/test_install.py | 35 +++++++ tests/test_install_strings.py | 8 +- tests/test_install_upgrade.py | 9 +- tests/test_read_hook.py | 46 ++++++--- tests/test_search_hook.py | 111 +++++++++++++++++++++ 7 files changed, 311 insertions(+), 78 deletions(-) create mode 100644 tests/test_search_hook.py diff --git a/CHANGELOG.md b/CHANGELOG.md index de1f58978..1fa09a737 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ Full release notes with details on each version: [GitHub Releases](https://githu ## Unreleased +- Fix: the Claude Code / Codebuddy `PreToolUse` graph-nudge hooks now work on Windows (#522). The hooks were inline POSIX bash (`case/esac`, `[ -f ]`, single-quoted `echo`), which Windows cmd.exe/PowerShell cannot parse — so on Windows the hook failed silently, no "run `graphify query` before grepping/reading raw files" context was injected, and users had to invoke `/graphify` by hand. The detection logic (grep-command match, source-file extension match, skip-if-under-output-dir, graph-exists check) moved into a shell-agnostic `graphify hook-guard ` subcommand invoked via the absolute exe path (the same pattern the codex hook already uses), so the hook parses and runs identically on Windows, macOS, and Linux. Behavior on macOS/Linux is unchanged (byte-identical nudge payload); the graph path now also honors `GRAPHIFY_OUT`. Codex stays a no-op there because Codex Desktop rejects `additionalContext`. - Fix: `--update`-style section writes to `CLAUDE.md`/`AGENTS.md` no longer corrupt or drop content (#1688, thanks @bdfinst). `_replace_or_append_section` located its managed block by substring (`marker in content`) and `next(... if marker in line)`, so a heading that appeared as a substring of another line (or duplicate headings) matched the wrong offset and the rewrite could truncate the file. It now matches the section heading exactly (`line.strip() == marker`), appends when absent, and prefers the last exact match when several exist, so unrelated content is preserved. - Fix: token estimation no longer crashes on files containing tiktoken special-token text like `<|endoftext|>` (#1685, thanks @Kyzcreig). `_TOKENIZER.encode(content)` raises `ValueError` by default when the text contains a special token, which aborted packing on docs/corpora that merely mention these strings. Both `encode` sites now pass `disallowed_special=()` so such text is tokenized as ordinary bytes. - Fix: the Ollama backend no longer multiplies a hang by the retry count (#1686, thanks @Kyzcreig). A stalled local model would wedge for `timeout * (max_retries + 1)`, which with the default 6 retries turned one long stall into a very long one. Ollama now defaults to zero client-side retries (a local model that stalls will not un-stall on retry); set `GRAPHIFY_MAX_RETRIES` to opt back in. Other backends are unchanged. Note: the underlying stall is non-deterministic and driven by the model server, so this bounds the wait rather than eliminating the hang. diff --git a/graphify/__main__.py b/graphify/__main__.py index 3a8113587..4b502341f 100644 --- a/graphify/__main__.py +++ b/graphify/__main__.py @@ -434,62 +434,70 @@ def _print_project_git_add_hint(paths: list[Path]) -> None: print("Project-scoped install. Add to version control:") print(f" git add {' '.join(unique)}") -_SETTINGS_HOOK = { - # Claude Code v2.1.117+ removed dedicated Grep/Glob tools; searches now go through Bash. - # We match on Bash and inspect the command string to avoid firing on every shell call. - "matcher": "Bash", - "hooks": [ - { - "type": "command", - "command": ( - "CMD=$(python3 -c \"" - "import json,sys; d=json.load(sys.stdin); " - "print(d.get('tool_input',d).get('command',''))\" 2>/dev/null || true); " - "case \"$CMD\" in " - r"*grep*|*rg\ *|*ripgrep*|*find\ *|*fd\ *|*ack\ *|*ag\ *) " - " [ -f graphify-out/graph.json ] && " - r""" echo '{"hookSpecificOutput":{"hookEventName":"PreToolUse","additionalContext":"MANDATORY: graphify-out/graph.json exists. You MUST run `graphify query \"\"` before grepping raw files. Only grep after graphify has oriented you, or to modify/debug specific lines."}}' """ - " || true ;; " - "esac" - ), - } - ], -} - -_READ_SETTINGS_HOOK = { - # The Bash hook above never sees a file read through the native Read tool or a - # Glob, which is the most common way an agent skips the graph: answering a - # codebase question by Read-ing many source files one by one (issue #1114). - # Match Read|Glob, inspect the target path, and nudge (never block) only for a - # source/doc file outside graphify-out/ when a graph exists. The parser is - # python3 (already a graphify dependency), the shell is POSIX, and every branch - # fails open, so a legitimate read always goes through. Reading the graph's own - # report under graphify-out/ is suppressed so it never starts a feedback loop. - # The extension test compares each value's real trailing extension (segment - # after the last '/' then after the last '.') against exts -- not a substring - # scan, which both missed framework files like .astro and false-matched .json - # against .js (the substring '.js' is inside '.json'). - "matcher": "Read|Glob", - "hooks": [ - { - "type": "command", - "command": ( - "HIT=$(python3 -c \"" - "import json,sys;" - "d=json.load(sys.stdin);" - "t=d.get('tool_input',d);" - "exts=('.py','.js','.ts','.tsx','.jsx','.astro','.vue','.svelte','.go','.rs','.java','.rb','.c','.h','.cpp','.hpp','.cc','.cs','.kt','.swift','.php','.scala','.lua','.sh','.md','.rst','.txt','.mdx');" - "vals=[str(t.get('file_path') or ''),str(t.get('pattern') or ''),str(t.get('path') or '')];" - "j=' '.join(vals).lower().replace(chr(92),'/');" - "tails=[('.'+x.rsplit('.',1)[-1]) for v in vals if v for x in [v.lower().replace(chr(92),'/').rsplit('/',1)[-1]] if '.' in x];" - "sys.stdout.write('1' if 'graphify-out/' not in j and any(tl in exts for tl in tails) else '')\" 2>/dev/null || true); " - "if [ \"$HIT\" = 1 ] && [ -f graphify-out/graph.json ]; then " - r"""echo '{"hookSpecificOutput":{"hookEventName":"PreToolUse","additionalContext":"MANDATORY: graphify-out/graph.json exists. You MUST run graphify before reading source files. Use: `graphify query \"\"` (scoped subgraph), `graphify explain \"\"`, or `graphify path \"\" \"\"`. Only read raw files after graphify has oriented you, or to modify/debug specific lines. This rule applies to subagents too — include it in every subagent prompt involving code exploration."}}'; """ - "fi || true" - ), - } - ], -} +# PreToolUse nudge payloads, emitted verbatim by the shell-agnostic +# `graphify hook-guard` subcommand (see _run_hook_guard). The previous hooks +# inlined POSIX bash (case/esac, [ -f ], single-quoted echo) which Windows +# cmd.exe/PowerShell cannot parse, so on Windows the hook failed and the nudge +# silently vanished — users had to invoke /graphify by hand (#522). Moving the +# logic into a Python subcommand invoked via an absolute exe path makes the hook +# parse identically under sh, cmd.exe and PowerShell. Claude Code accepts +# additionalContext on PreToolUse (Codex Desktop does not — that path stays a +# no-op via `hook-check`). Compact separators keep the payload byte-for-byte the +# same JSON the old `echo` emitted. +_SEARCH_NUDGE = json.dumps({ + "hookSpecificOutput": { + "hookEventName": "PreToolUse", + "additionalContext": ( + 'MANDATORY: graphify-out/graph.json exists. You MUST run ' + '`graphify query ""` before grepping raw files. Only grep ' + 'after graphify has oriented you, or to modify/debug specific lines.' + ), + } +}, ensure_ascii=False, separators=(",", ":")) + "\n" + +_READ_NUDGE = json.dumps({ + "hookSpecificOutput": { + "hookEventName": "PreToolUse", + "additionalContext": ( + 'MANDATORY: graphify-out/graph.json exists. You MUST run graphify ' + 'before reading source files. Use: `graphify query ""` ' + '(scoped subgraph), `graphify explain ""`, or ' + '`graphify path "" ""`. Only read raw files after graphify has ' + 'oriented you, or to modify/debug specific lines. This rule applies to ' + 'subagents too — include it in every subagent prompt involving code ' + 'exploration.' + ), + } +}, ensure_ascii=False, separators=(",", ":")) + "\n" + +# Source/doc extensions the Read|Glob guard nudges on (verbatim from the old hook). +# The trailing-extension test (real final path segment, then its last '.') means +# '.json' never false-matches '.js', and framework files like '.astro' are kept. +_HOOK_SOURCE_EXTS = ( + '.py', '.js', '.ts', '.tsx', '.jsx', '.astro', '.vue', '.svelte', '.go', + '.rs', '.java', '.rb', '.c', '.h', '.cpp', '.hpp', '.cc', '.cs', '.kt', + '.swift', '.php', '.scala', '.lua', '.sh', '.md', '.rst', '.txt', '.mdx', +) + + +def _claude_pretooluse_hooks() -> "list[dict]": + """graphify's Claude/Codebuddy PreToolUse hooks, resolved at install time. + + The command invokes `graphify hook-guard ` via the absolute exe + path (`_resolve_graphify_exe`), so it parses under sh, cmd.exe and PowerShell + alike — this is the #522 fix, and mirrors the codex hook. Matchers stay "Bash" + and "Read|Glob" and the command always contains "graphify", so the existing + install/uninstall filters find and replace both old bash hooks and these. + """ + exe = _resolve_graphify_exe() + if " " in exe and not exe.startswith('"'): + exe = f'"{exe}"' + return [ + {"matcher": "Bash", + "hooks": [{"type": "command", "command": f"{exe} hook-guard search"}]}, + {"matcher": "Read|Glob", + "hooks": [{"type": "command", "command": f"{exe} hook-guard read"}]}, + ] def _skill_registration(skill_path: str = "~/.claude/skills/graphify/SKILL.md") -> str: return ( @@ -1618,6 +1626,51 @@ def _resolve_graphify_exe() -> str: return "graphify" +def _run_hook_guard(kind: str) -> None: + """Shell-agnostic PreToolUse guard (#522). + + Reads the tool-call JSON from stdin and, when a knowledge graph exists in the + current output dir, prints a nudge (`additionalContext`) telling the agent to + use graphify instead of grepping/reading raw files. Replaces the old inline + bash hooks that failed to parse on Windows. Always fails open: any error, or a + non-matching tool call, prints nothing and the caller exits 0, so a legitimate + tool call is never blocked. Detection mirrors the previous hooks exactly. + """ + from graphify.paths import out_path, GRAPHIFY_OUT_NAME + try: + d = json.loads(sys.stdin.buffer.read().decode("utf-8", "replace")) + except Exception: + return + if not isinstance(d, dict): + return + t = d.get("tool_input", d) + if not isinstance(t, dict): + return + try: + if kind == "search": + cmd_str = str(t.get("command", "") or "") + # Same set the old `case` matched: *grep*, *ripgrep*, and rg/find/fd/ + # ack/ag as a token (name followed by a space). + if any(tok in cmd_str for tok in ("grep", "ripgrep", "rg ", "find ", "fd ", "ack ", "ag ")) \ + and out_path("graph.json").is_file(): + sys.stdout.write(_SEARCH_NUDGE) + elif kind == "read": + vals = [str(t.get("file_path") or ""), str(t.get("pattern") or ""), str(t.get("path") or "")] + j = " ".join(vals).lower().replace("\\", "/") + tails = [ + "." + seg.rsplit(".", 1)[-1] + for v in vals if v + for seg in [v.lower().replace("\\", "/").rsplit("/", 1)[-1]] + if "." in seg + ] + under_out = "graphify-out/" in j or (GRAPHIFY_OUT_NAME.lower() + "/") in j + if not under_out and any(tl in _HOOK_SOURCE_EXTS for tl in tails) \ + and out_path("graph.json").is_file(): + sys.stdout.write(_READ_NUDGE) + except Exception: + pass + + def _install_codex_hook(project_dir: Path) -> None: """Add graphify PreToolUse hook to .codex/hooks.json.""" hooks_path = project_dir / ".codex" / "hooks.json" @@ -1975,8 +2028,7 @@ def _install_claude_hook(project_dir: Path) -> None: pre_tool = hooks.setdefault("PreToolUse", []) hooks["PreToolUse"] = [h for h in pre_tool if not (h.get("matcher") in ("Glob|Grep", "Bash", "Read|Glob") and "graphify" in str(h))] - hooks["PreToolUse"].append(_SETTINGS_HOOK) - hooks["PreToolUse"].append(_READ_SETTINGS_HOOK) + hooks["PreToolUse"].extend(_claude_pretooluse_hooks()) settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") print(f" .claude/settings.json -> PreToolUse hooks registered (Bash search + Read/Glob)") @@ -2125,8 +2177,7 @@ def _install_codebuddy_hook(project_dir: Path) -> None: pre_tool = hooks.setdefault("PreToolUse", []) hooks["PreToolUse"] = [h for h in pre_tool if not (h.get("matcher") in ("Glob|Grep", "Bash", "Read|Glob") and "graphify" in str(h))] - hooks["PreToolUse"].append(_SETTINGS_HOOK) - hooks["PreToolUse"].append(_READ_SETTINGS_HOOK) + hooks["PreToolUse"].extend(_claude_pretooluse_hooks()) settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") print(f" .codebuddy/settings.json -> PreToolUse hooks registered") @@ -2250,7 +2301,7 @@ def main() -> None: # Skip during install/uninstall (hook writes trigger a fresh check anyway). # Skip during hook-check — it runs on every editor tool use and must be silent. # Deduplicate paths so platforms sharing the same install dir don't warn twice. - _silent_cmds = {"install", "uninstall", "hook-check"} + _silent_cmds = {"install", "uninstall", "hook-check", "hook-guard"} if not any(arg in _silent_cmds for arg in sys.argv): # Resolve each platform's real user-scope destination so per-platform # overrides (gemini, opencode, devin, antigravity, amp) check the dir @@ -3777,6 +3828,12 @@ def main() -> None: # Keep this as a cross-platform no-op so installed hooks never break Bash # tool calls. Graph guidance reaches the agent via AGENTS.md / skill instead. sys.exit(0) + elif cmd == "hook-guard": + # Shell-agnostic Claude/Codebuddy PreToolUse guard (#522). Replaces the old + # inline-bash hooks that failed on Windows. Prints an additionalContext nudge + # toward graphify when a graph exists; always exits 0 (never blocks a tool). + _run_hook_guard(sys.argv[2] if len(sys.argv) > 2 else "") + sys.exit(0) elif cmd == "check-update": if len(sys.argv) < 3: print("Usage: graphify check-update ", file=sys.stderr) diff --git a/tests/test_install.py b/tests/test_install.py index fc1a4e90b..b1bcaa780 100644 --- a/tests/test_install.py +++ b/tests/test_install.py @@ -359,6 +359,41 @@ def test_codebuddy_install_writes_hook(tmp_path): assert any("graphify" in str(h) for h in hooks) +def test_claude_hook_is_shell_agnostic(tmp_path): + # #522: the installed PreToolUse hooks must be plain exe invocations, not + # POSIX bash (which fails on Windows cmd.exe/PowerShell). + import json as _json + from graphify.__main__ import _install_claude_hook + _install_claude_hook(tmp_path) + hooks = _json.loads((tmp_path / ".claude" / "settings.json").read_text())["hooks"]["PreToolUse"] + matchers = {h["matcher"] for h in hooks} + assert {"Bash", "Read|Glob"} <= matchers + for h in hooks: + cmd = h["hooks"][0]["command"] + for token in ("$(", "case ", "[ -f", "&&", "||", ";;", "echo '"): + assert token not in cmd, f"shell syntax {token!r} in {cmd!r}" + assert "graphify" in cmd and "hook-guard" in cmd + + +def test_claude_hook_install_idempotent_and_replaces_old_bash_hook(tmp_path): + import json as _json + from graphify.__main__ import _install_claude_hook + settings_path = tmp_path / ".claude" / "settings.json" + settings_path.parent.mkdir(parents=True) + # Pre-seed a legacy bash-style graphify hook (the thing #522 shipped before). + settings_path.write_text(_json.dumps({"hooks": {"PreToolUse": [ + {"matcher": "Bash", "hooks": [{"type": "command", + "command": "[ -f graphify-out/graph.json ] && echo '{...}' || true"}]}, + ]}}), encoding="utf-8") + _install_claude_hook(tmp_path) + _install_claude_hook(tmp_path) # second install must not duplicate + hooks = _json.loads(settings_path.read_text())["hooks"]["PreToolUse"] + graphify_hooks = [h for h in hooks if "graphify" in str(h)] + assert len(graphify_hooks) == 2, "exactly the Bash + Read|Glob guards, no dupes" + # the legacy bash payload must be gone + assert not any("[ -f graphify-out" in h["hooks"][0]["command"] for h in graphify_hooks) + + def test_codebuddy_install_idempotent(tmp_path): from graphify.__main__ import codebuddy_install codebuddy_install(tmp_path) diff --git a/tests/test_install_strings.py b/tests/test_install_strings.py index 38bec1e02..36c5f56ec 100644 --- a/tests/test_install_strings.py +++ b/tests/test_install_strings.py @@ -12,8 +12,8 @@ import json from graphify.__main__ import ( - _SETTINGS_HOOK, - _READ_SETTINGS_HOOK, + _SEARCH_NUDGE, + _READ_NUDGE, _skill_registration, _CLAUDE_MD_SECTION, _AGENTS_MD_SECTION, @@ -32,8 +32,8 @@ # Hook constants are dicts/JSON; serialize them so we can do substring checks # against the actual payload text the assistant will receive. _INSTALL_TEXTS: dict[str, str] = { - "_SETTINGS_HOOK": json.dumps(_SETTINGS_HOOK), - "_READ_SETTINGS_HOOK": json.dumps(_READ_SETTINGS_HOOK), + "_SEARCH_NUDGE": _SEARCH_NUDGE, + "_READ_NUDGE": _READ_NUDGE, "_CLAUDE_MD_SECTION": _CLAUDE_MD_SECTION, "_AGENTS_MD_SECTION": _AGENTS_MD_SECTION, "_GEMINI_MD_SECTION": _GEMINI_MD_SECTION, diff --git a/tests/test_install_upgrade.py b/tests/test_install_upgrade.py index 13e7c55a5..e85add452 100644 --- a/tests/test_install_upgrade.py +++ b/tests/test_install_upgrade.py @@ -145,9 +145,14 @@ def test_claude_install_upgrades_stale_hook_payload(tmp_path, monkeypatch): assert _OLD_HOOK_PAYLOAD_SNIPPET not in new_settings_text, ( "stale hook payload survived upgrade" ) - assert "graphify query" in new_settings_text, ( - "new hook payload should route to `graphify query`" + # Since #522 the nudge text lives in the `graphify hook-guard` subcommand, not + # inline in settings.json (so the command parses on Windows). The upgraded hook + # must therefore route to that shell-agnostic subcommand, and the old bash + # pipeline must be gone. + assert "hook-guard" in new_settings_text, ( + "new hook payload should route to the `graphify hook-guard` subcommand" ) + assert "case x in" not in new_settings_text, "stale bash pipeline survived upgrade" def test_agents_install_upgrades_stale_section(tmp_path, monkeypatch): diff --git a/tests/test_read_hook.py b/tests/test_read_hook.py index d3bf58a31..82a514d92 100644 --- a/tests/test_read_hook.py +++ b/tests/test_read_hook.py @@ -1,17 +1,31 @@ -"""The Read|Glob PreToolUse hook nudges toward the graph instead of raw reads. +"""The Read|Glob PreToolUse guard nudges toward the graph instead of raw reads. Closes the issue #1114 gap: the Bash search hook never sees a file read through -the native Read tool or a Glob. These tests run the hook command the way Claude -Code does - via `sh -c` with crafted stdin JSON - and assert it nudges only for -a source/doc file outside graphify-out/ when a graph exists, and otherwise stays -silent and fails open. +the native Read tool or a Glob. Since #522 the guard runs as the shell-agnostic +`graphify hook-guard read` subcommand (not inline bash), so it works on Windows +too. These tests invoke that subcommand with crafted stdin JSON and assert it +nudges only for a source/doc file outside graphify-out/ when a graph exists, and +otherwise stays silent and fails open. """ import json +import os import subprocess +import sys -from graphify.__main__ import _READ_SETTINGS_HOOK +from graphify.__main__ import _claude_pretooluse_hooks -CMD = _READ_SETTINGS_HOOK["hooks"][0]["command"] + +def _read_matcher(): + hooks = _claude_pretooluse_hooks() + return next(h for h in hooks if h["matcher"] == "Read|Glob") + + +def _env(): + # The guard resolves the graph via GRAPHIFY_OUT (default "graphify-out", + # relative to cwd). Drop any inherited override so the tmp_path graph is found. + e = dict(os.environ) + e.pop("GRAPHIFY_OUT", None) + return e def _run(tool_input, cwd, *, graph: bool): @@ -20,12 +34,21 @@ def _run(tool_input, cwd, *, graph: bool): (cwd / "graphify-out" / "graph.json").write_text("{}", encoding="utf-8") stdin = json.dumps({"tool_input": tool_input}) return subprocess.run( - ["sh", "-c", CMD], input=stdin, capture_output=True, text=True, cwd=cwd + [sys.executable, "-m", "graphify", "hook-guard", "read"], + input=stdin, capture_output=True, text=True, cwd=cwd, env=_env(), ) def test_matcher_targets_read_and_glob(): - assert _READ_SETTINGS_HOOK["matcher"] == "Read|Glob" + assert _read_matcher()["matcher"] == "Read|Glob" + + +def test_command_has_no_shell_syntax(): + # #522: the command must be a plain exe invocation, not POSIX bash. + cmd = _read_matcher()["hooks"][0]["command"] + for token in ("$(", "case ", "[ -f", "&&", "||", ";;", "echo '"): + assert token not in cmd, f"shell syntax {token!r} leaked into the hook" + assert "graphify" in cmd and "hook-guard read" in cmd def test_silent_without_graph(tmp_path): @@ -106,14 +129,15 @@ def test_fails_open_on_malformed_stdin(tmp_path): (tmp_path / "graphify-out").mkdir() (tmp_path / "graphify-out" / "graph.json").write_text("{}", encoding="utf-8") r = subprocess.run( - ["sh", "-c", CMD], input="this is not json", capture_output=True, text=True, cwd=tmp_path + [sys.executable, "-m", "graphify", "hook-guard", "read"], + input="this is not json", capture_output=True, text=True, cwd=tmp_path, env=_env(), ) assert r.returncode == 0 assert r.stdout.strip() == "" def test_never_blocks(tmp_path): - """A nudge is additionalContext only - the hook must exit 0, never deny.""" + """A nudge is additionalContext only - the guard must exit 0, never deny.""" r = _run({"file_path": "src/app.py"}, tmp_path, graph=True) assert r.returncode == 0 assert '"permissionDecision"' not in r.stdout diff --git a/tests/test_search_hook.py b/tests/test_search_hook.py new file mode 100644 index 000000000..4c34fcea1 --- /dev/null +++ b/tests/test_search_hook.py @@ -0,0 +1,111 @@ +"""The Bash PreToolUse guard nudges toward the graph before grep/find searches. + +Since #522 it runs as the shell-agnostic `graphify hook-guard search` subcommand +(not inline bash), so it works on Windows too. These tests invoke the subcommand +with crafted stdin JSON and assert it nudges only for a search command when a +graph exists, and otherwise stays silent and fails open. +""" +import json +import os +import subprocess +import sys + +from graphify.__main__ import _claude_pretooluse_hooks + + +def _search_matcher(): + hooks = _claude_pretooluse_hooks() + return next(h for h in hooks if h["matcher"] == "Bash") + + +def _env(): + e = dict(os.environ) + e.pop("GRAPHIFY_OUT", None) + return e + + +def _run(command, cwd, *, graph: bool): + if graph: + (cwd / "graphify-out").mkdir(parents=True, exist_ok=True) + (cwd / "graphify-out" / "graph.json").write_text("{}", encoding="utf-8") + stdin = json.dumps({"tool_input": {"command": command}}) + return subprocess.run( + [sys.executable, "-m", "graphify", "hook-guard", "search"], + input=stdin, capture_output=True, text=True, cwd=cwd, env=_env(), + ) + + +def test_matcher_targets_bash(): + assert _search_matcher()["matcher"] == "Bash" + + +def test_command_has_no_shell_syntax(): + # #522: no POSIX bash that Windows cmd.exe/PowerShell can't parse. + cmd = _search_matcher()["hooks"][0]["command"] + for token in ("$(", "case ", "[ -f", "&&", "||", ";;", "echo '"): + assert token not in cmd, f"shell syntax {token!r} leaked into the hook" + assert "graphify" in cmd and "hook-guard search" in cmd + + +def test_nudges_on_search_commands_with_graph(tmp_path): + for command in ( + "grep -rn foo .", + "rg pattern src/", + "ripgrep thing", + "find . -name '*.py'", + "fd bar", + "ack needle", + "ag needle", + ): + out = _run(command, tmp_path, graph=True).stdout + assert "graphify query" in out, f"{command!r} should nudge" + + +def test_silent_without_graph(tmp_path): + out = _run("grep -rn foo .", tmp_path, graph=False).stdout + assert out.strip() == "" + + +def test_silent_on_non_search_commands(tmp_path): + for command in ("ls -la", "git status", "cat README.md", "python app.py"): + out = _run(command, tmp_path, graph=True).stdout + assert out.strip() == "", f"{command!r} should not nudge" + + +def test_nudge_payload_is_valid_pretooluse_json(tmp_path): + out = _run("grep -rn foo .", tmp_path, graph=True).stdout + payload = json.loads(out) + assert payload["hookSpecificOutput"]["hookEventName"] == "PreToolUse" + assert "graphify query" in payload["hookSpecificOutput"]["additionalContext"] + + +def test_fails_open_on_malformed_stdin(tmp_path): + (tmp_path / "graphify-out").mkdir() + (tmp_path / "graphify-out" / "graph.json").write_text("{}", encoding="utf-8") + r = subprocess.run( + [sys.executable, "-m", "graphify", "hook-guard", "search"], + input="not json", capture_output=True, text=True, cwd=tmp_path, env=_env(), + ) + assert r.returncode == 0 + assert r.stdout.strip() == "" + + +def test_never_blocks(tmp_path): + r = _run("grep -rn foo .", tmp_path, graph=True) + assert r.returncode == 0 + assert '"permissionDecision"' not in r.stdout + assert '"deny"' not in r.stdout + + +def test_honors_graphify_out_override(tmp_path): + """The guard resolves the graph via GRAPHIFY_OUT, not a hardcoded path.""" + custom = tmp_path / "custom-out" + custom.mkdir() + (custom / "graph.json").write_text("{}", encoding="utf-8") + env = dict(os.environ, GRAPHIFY_OUT=str(custom)) + stdin = json.dumps({"tool_input": {"command": "grep -rn foo ."}}) + r = subprocess.run( + [sys.executable, "-m", "graphify", "hook-guard", "search"], + input=stdin, capture_output=True, text=True, cwd=tmp_path, env=env, + ) + assert "graphify query" in r.stdout From 85f9fac7f70d9bc8760170031e809d719f867fd8 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 6 Jul 2026 16:43:21 +0100 Subject: [PATCH 011/292] Fix: port the Gemini CLI BeforeTool graph-nudge hook to be shell-agnostic too (#522) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Gemini hook was a `python -c "..."` one-liner that (a) depended on a bare `python` being on PATH (frequently `python`/`py` or absent on Windows) and (b) embedded backticks + escaped quotes that Windows PowerShell mangles. Same fix as the Claude/Codebuddy hooks: it now invokes `graphify hook-guard gemini` via the absolute exe path. The gemini mode always returns {"decision":"allow"} (never blocks a tool) and appends the graph nudge as additionalContext only when graph.json exists — the BeforeTool contract Gemini expects, byte-identical to the old payload. It takes no stdin, honors GRAPHIFY_OUT, and the matcher stays "read_file|list_directory" so install/uninstall find and replace old hooks unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 2 +- graphify/__main__.py | 55 ++++++++++++++++++--------- tests/test_gemini_hook.py | 71 +++++++++++++++++++++++++++++++++++ tests/test_install_strings.py | 4 +- 4 files changed, 111 insertions(+), 21 deletions(-) create mode 100644 tests/test_gemini_hook.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 1fa09a737..0c95333a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ Full release notes with details on each version: [GitHub Releases](https://githu ## Unreleased -- Fix: the Claude Code / Codebuddy `PreToolUse` graph-nudge hooks now work on Windows (#522). The hooks were inline POSIX bash (`case/esac`, `[ -f ]`, single-quoted `echo`), which Windows cmd.exe/PowerShell cannot parse — so on Windows the hook failed silently, no "run `graphify query` before grepping/reading raw files" context was injected, and users had to invoke `/graphify` by hand. The detection logic (grep-command match, source-file extension match, skip-if-under-output-dir, graph-exists check) moved into a shell-agnostic `graphify hook-guard ` subcommand invoked via the absolute exe path (the same pattern the codex hook already uses), so the hook parses and runs identically on Windows, macOS, and Linux. Behavior on macOS/Linux is unchanged (byte-identical nudge payload); the graph path now also honors `GRAPHIFY_OUT`. Codex stays a no-op there because Codex Desktop rejects `additionalContext`. +- Fix: the Claude Code / Codebuddy `PreToolUse` and Gemini CLI `BeforeTool` graph-nudge hooks now work on Windows (#522). The hooks were inline POSIX bash (`case/esac`, `[ -f ]`, single-quoted `echo`), which Windows cmd.exe/PowerShell cannot parse — so on Windows the hook failed silently, no "run `graphify query` before grepping/reading raw files" context was injected, and users had to invoke `/graphify` by hand. The detection logic (grep-command match, source-file extension match, skip-if-under-output-dir, graph-exists check) moved into a shell-agnostic `graphify hook-guard ` subcommand invoked via the absolute exe path (the same pattern the codex hook already uses), so the hook parses and runs identically on Windows, macOS, and Linux. Behavior on macOS/Linux is unchanged (byte-identical nudge payload); the graph path now also honors `GRAPHIFY_OUT`. The Gemini `BeforeTool` hook got the same treatment (`graphify hook-guard gemini`), which also removes its dependency on a bare `python` being on PATH. Codex stays a no-op there because Codex Desktop rejects `additionalContext`. - Fix: `--update`-style section writes to `CLAUDE.md`/`AGENTS.md` no longer corrupt or drop content (#1688, thanks @bdfinst). `_replace_or_append_section` located its managed block by substring (`marker in content`) and `next(... if marker in line)`, so a heading that appeared as a substring of another line (or duplicate headings) matched the wrong offset and the rewrite could truncate the file. It now matches the section heading exactly (`line.strip() == marker`), appends when absent, and prefers the last exact match when several exist, so unrelated content is preserved. - Fix: token estimation no longer crashes on files containing tiktoken special-token text like `<|endoftext|>` (#1685, thanks @Kyzcreig). `_TOKENIZER.encode(content)` raises `ValueError` by default when the text contains a special token, which aborted packing on docs/corpora that merely mention these strings. Both `encode` sites now pass `disallowed_special=()` so such text is tokenized as ordinary bytes. - Fix: the Ollama backend no longer multiplies a hang by the retry count (#1686, thanks @Kyzcreig). A stalled local model would wedge for `timeout * (max_retries + 1)`, which with the default 6 retries turned one long stall into a very long one. Ollama now defaults to zero client-side retries (a local model that stalls will not un-stall on retry); set `GRAPHIFY_MAX_RETRIES` to opt back in. Other backends are unchanged. Note: the underlying stall is non-deterministic and driven by the model server, so this bounds the wait rather than eliminating the hang. diff --git a/graphify/__main__.py b/graphify/__main__.py index 4b502341f..8d37c911e 100644 --- a/graphify/__main__.py +++ b/graphify/__main__.py @@ -852,23 +852,30 @@ def _print_install_usage() -> None: _GEMINI_MD_MARKER = "## graphify" -_GEMINI_HOOK = { - "matcher": "read_file|list_directory", - "hooks": [ - { - "type": "command", - "command": ( - 'python -c "' - "import sys,pathlib,json;" - "e=pathlib.Path('graphify-out/graph.json').exists();" - "d={'decision':'allow'};" - "e and d.update({'additionalContext':'graphify: knowledge graph at graphify-out/. For focused questions, run `graphify query \"\"` (scoped subgraph, usually much smaller than GRAPH_REPORT.md) instead of grepping raw files. Read GRAPH_REPORT.md only for broad architecture context.'});" - "sys.stdout.write(json.dumps(d))" - '"' - ), - } - ], -} +# Gemini CLI BeforeTool hook nudge text. The hook always returns +# {"decision":"allow"} (never blocks a tool) and appends this as additionalContext +# when a graph exists. Emitted by `graphify hook-guard gemini`. The old hook was a +# `python -c "..."` one-liner that depended on a bare `python` on PATH (often +# `python`/`py` or absent on Windows) and embedded backticks + escaped quotes that +# Windows PowerShell mangles (#522 follow-up); the subcommand form has no such +# dependency and parses under every shell. +_GEMINI_NUDGE_TEXT = ( + 'graphify: knowledge graph at graphify-out/. For focused questions, run ' + '`graphify query ""` (scoped subgraph, usually much smaller than ' + 'GRAPH_REPORT.md) instead of grepping raw files. Read GRAPH_REPORT.md only ' + 'for broad architecture context.' +) + + +def _gemini_hook() -> dict: + """Gemini CLI BeforeTool hook, resolved to a shell-agnostic `graphify` call.""" + exe = _resolve_graphify_exe() + if " " in exe and not exe.startswith('"'): + exe = f'"{exe}"' + return { + "matcher": "read_file|list_directory", + "hooks": [{"type": "command", "command": f"{exe} hook-guard gemini"}], + } def gemini_install(project_dir: Path | None = None, *, project: bool = False) -> None: @@ -917,7 +924,7 @@ def _install_gemini_hook(project_dir: Path) -> None: settings["hooks"]["BeforeTool"] = [ h for h in before_tool if "graphify" not in str(h) ] - settings["hooks"]["BeforeTool"].append(_GEMINI_HOOK) + settings["hooks"]["BeforeTool"].append(_gemini_hook()) settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") print(" .gemini/settings.json -> BeforeTool hook registered") @@ -1637,6 +1644,18 @@ def _run_hook_guard(kind: str) -> None: tool call is never blocked. Detection mirrors the previous hooks exactly. """ from graphify.paths import out_path, GRAPHIFY_OUT_NAME + # Gemini's BeforeTool hook takes no stdin and must ALWAYS return a decision so + # the tool is never blocked; the graph nudge is appended only when a graph + # exists. Handled before the stdin read below (which the search/read guards need). + if kind == "gemini": + payload = {"decision": "allow"} + try: + if out_path("graph.json").is_file(): + payload["additionalContext"] = _GEMINI_NUDGE_TEXT + except Exception: + pass + sys.stdout.write(json.dumps(payload, ensure_ascii=False, separators=(",", ":"))) + return try: d = json.loads(sys.stdin.buffer.read().decode("utf-8", "replace")) except Exception: diff --git a/tests/test_gemini_hook.py b/tests/test_gemini_hook.py new file mode 100644 index 000000000..aa84117dd --- /dev/null +++ b/tests/test_gemini_hook.py @@ -0,0 +1,71 @@ +"""The Gemini CLI BeforeTool guard nudges toward the graph, shell-agnostically. + +Since #522 it runs as `graphify hook-guard gemini` (not a `python -c` one-liner +that depended on a bare `python` on PATH and embedded PowerShell-hostile +backticks). It always returns {"decision":"allow"} so a tool is never blocked, +and appends additionalContext only when a graph exists. +""" +import json +import os +import subprocess +import sys + +from graphify.__main__ import _gemini_hook + + +def _env(): + e = dict(os.environ) + e.pop("GRAPHIFY_OUT", None) + return e + + +def _run(cwd, *, graph: bool): + if graph: + (cwd / "graphify-out").mkdir(parents=True, exist_ok=True) + (cwd / "graphify-out" / "graph.json").write_text("{}", encoding="utf-8") + return subprocess.run( + [sys.executable, "-m", "graphify", "hook-guard", "gemini"], + input="", capture_output=True, text=True, cwd=cwd, env=_env(), + ) + + +def test_matcher_and_command_shape(): + h = _gemini_hook() + assert h["matcher"] == "read_file|list_directory" + cmd = h["hooks"][0]["command"] + # #522: no bare `python` dependency, no embedded quote/backtick soup. + assert "python -c" not in cmd + assert "graphify" in cmd and "hook-guard gemini" in cmd + + +def test_allows_and_nudges_with_graph(tmp_path): + out = _run(tmp_path, graph=True).stdout + payload = json.loads(out) + assert payload["decision"] == "allow" + assert "graphify query" in payload["additionalContext"] + + +def test_allows_without_nudge_when_no_graph(tmp_path): + out = _run(tmp_path, graph=False).stdout + payload = json.loads(out) + assert payload["decision"] == "allow" + assert "additionalContext" not in payload + + +def test_never_blocks(tmp_path): + r = _run(tmp_path, graph=True) + assert r.returncode == 0 + payload = json.loads(r.stdout) + assert payload["decision"] == "allow" + + +def test_honors_graphify_out_override(tmp_path): + custom = tmp_path / "custom-out" + custom.mkdir() + (custom / "graph.json").write_text("{}", encoding="utf-8") + env = dict(os.environ, GRAPHIFY_OUT=str(custom)) + r = subprocess.run( + [sys.executable, "-m", "graphify", "hook-guard", "gemini"], + input="", capture_output=True, text=True, cwd=tmp_path, env=env, + ) + assert "graphify query" in json.loads(r.stdout).get("additionalContext", "") diff --git a/tests/test_install_strings.py b/tests/test_install_strings.py index 36c5f56ec..7e65b768e 100644 --- a/tests/test_install_strings.py +++ b/tests/test_install_strings.py @@ -18,7 +18,7 @@ _CLAUDE_MD_SECTION, _AGENTS_MD_SECTION, _GEMINI_MD_SECTION, - _GEMINI_HOOK, + _GEMINI_NUDGE_TEXT, _VSCODE_INSTRUCTIONS_SECTION, _ANTIGRAVITY_RULES, _KIRO_STEERING, @@ -37,7 +37,7 @@ "_CLAUDE_MD_SECTION": _CLAUDE_MD_SECTION, "_AGENTS_MD_SECTION": _AGENTS_MD_SECTION, "_GEMINI_MD_SECTION": _GEMINI_MD_SECTION, - "_GEMINI_HOOK": json.dumps(_GEMINI_HOOK), + "_GEMINI_NUDGE_TEXT": _GEMINI_NUDGE_TEXT, "_VSCODE_INSTRUCTIONS_SECTION": _VSCODE_INSTRUCTIONS_SECTION, "_ANTIGRAVITY_RULES": _ANTIGRAVITY_RULES, "_KIRO_STEERING": _KIRO_STEERING, From 737a0b84540f6037ffe936b4a51eb51e57c53e52 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 6 Jul 2026 16:49:31 +0100 Subject: [PATCH 012/292] test: rigorous edge cases for the hook-guard subcommand (#522) Adds a wide matrix over _run_hook_guard covering the search/read detection boundaries and the gemini BeforeTool contract: - search: grep-family variants (grep/pgrep/egrep/fgrep/ripgrep), token matches (rg/find/fd/ack/ag with trailing space), piped commands; and silence for non-search commands, empty/missing/non-string command, 'find' without a trailing space, 'ag' mid-word, top-level vs nested tool_input, non-dict tool_input, and no-graph. - read: source/framework extensions, uppercase and multi-dot names, Windows backslash paths, glob patterns; and silence for .json (not .js), lockfiles, images, extensionless files, an extension on a directory segment, targets under the (default and custom-named) output dir, and no-graph. - fail-open: malformed/empty/binary stdin and a throwing graph-existence check never crash or block. - gemini: always returns {"decision":"allow"}, nudges only with a graph, and stays "allow" even if the check throws. - dispatch/exit/encoding via real subprocess: missing/unknown mode exits 0 silently, every mode exits 0 (never blocks), and the read nudge's em dash round-trips as valid UTF-8. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/test_hook_guard.py | 280 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 280 insertions(+) create mode 100644 tests/test_hook_guard.py diff --git a/tests/test_hook_guard.py b/tests/test_hook_guard.py new file mode 100644 index 000000000..5146ef250 --- /dev/null +++ b/tests/test_hook_guard.py @@ -0,0 +1,280 @@ +"""Rigorous edge-case coverage for the `graphify hook-guard` subcommand (#522). + +Covers the shell-agnostic PreToolUse/BeforeTool guard that replaced the inline +bash hooks: the search/read detection matrix, the gemini BeforeTool contract, +fail-open behavior, output-dir overrides, subcommand dispatch, exit codes, and +UTF-8 (em dash) byte fidelity. Detection is exercised by calling _run_hook_guard +directly (hermetic, fast); dispatch/exit/encoding go through a real subprocess. +""" +import io +import json +import os +import subprocess +import sys + +import pytest + +from graphify import __main__ as m + + +# --------------------------------------------------------------------------- # +# Direct-call harness: hermetic w.r.t. the ambient GRAPHIFY_OUT env. +# --------------------------------------------------------------------------- # +def _invoke(kind, payload, tmp_path, monkeypatch, *, graph=True, out_name="graphify-out"): + monkeypatch.setattr("graphify.paths.GRAPHIFY_OUT", out_name) + monkeypatch.setattr("graphify.paths.GRAPHIFY_OUT_NAME", out_name) + monkeypatch.chdir(tmp_path) + if graph: + (tmp_path / out_name).mkdir(parents=True, exist_ok=True) + (tmp_path / out_name / "graph.json").write_text("{}", encoding="utf-8") + + if isinstance(payload, (bytes, bytearray)): + data = bytes(payload) + elif payload is None: + data = b"" + else: + data = json.dumps(payload).encode("utf-8") + + class _Stdin: + def __init__(self, b): + self.buffer = io.BytesIO(b) + + monkeypatch.setattr(sys, "stdin", _Stdin(data)) + buf = io.StringIO() + monkeypatch.setattr(sys, "stdout", buf) + m._run_hook_guard(kind) + return buf.getvalue() + + +# --------------------------------------------------------------------------- # +# search: commands that MUST nudge (mirror the old *grep*/token globs) +# --------------------------------------------------------------------------- # +@pytest.mark.parametrize("command", [ + "grep -rn foo .", + "pgrep -f server", # contains 'grep' + "egrep pattern file", # contains 'grep' + "fgrep lit file", # contains 'grep' + "ls -la | grep foo", # piped + "ripgrep thing", + "rg pattern src/", + "find . -name '*.py'", + "fd bar", + "ack needle", + "ag needle", +]) +def test_search_nudges(command, tmp_path, monkeypatch): + out = _invoke("search", {"tool_input": {"command": command}}, tmp_path, monkeypatch) + assert "graphify query" in out, f"{command!r} should nudge" + assert json.loads(out)["hookSpecificOutput"]["hookEventName"] == "PreToolUse" + + +# --------------------------------------------------------------------------- # +# search: commands / inputs that MUST stay silent +# --------------------------------------------------------------------------- # +@pytest.mark.parametrize("command", [ + "", # empty + "ls -la", + "git status", + "cat README.md", + "python app.py", + "cd findings && ls", # 'find' without a trailing space is not a match + "manage db migrate", # 'ag' mid-word, no 'ag ' token + "echo hello", +]) +def test_search_silent(command, tmp_path, monkeypatch): + out = _invoke("search", {"tool_input": {"command": command}}, tmp_path, monkeypatch) + assert out.strip() == "", f"{command!r} should be silent" + + +def test_search_silent_without_graph(tmp_path, monkeypatch): + out = _invoke("search", {"tool_input": {"command": "grep x"}}, tmp_path, monkeypatch, graph=False) + assert out.strip() == "" + + +def test_search_missing_command_key(tmp_path, monkeypatch): + out = _invoke("search", {"tool_input": {}}, tmp_path, monkeypatch) + assert out.strip() == "" + + +def test_search_non_string_command_is_silent(tmp_path, monkeypatch): + out = _invoke("search", {"tool_input": {"command": 123}}, tmp_path, monkeypatch) + assert out.strip() == "" + + +def test_search_top_level_command_without_tool_input(tmp_path, monkeypatch): + # Some hosts pass the tool payload flat (no "tool_input" wrapper). + out = _invoke("search", {"command": "grep x"}, tmp_path, monkeypatch) + assert "graphify query" in out + + +def test_search_non_dict_tool_input_is_silent(tmp_path, monkeypatch): + out = _invoke("search", {"tool_input": "grep foo"}, tmp_path, monkeypatch) + assert out.strip() == "" + + +# --------------------------------------------------------------------------- # +# read: file targets that MUST nudge +# --------------------------------------------------------------------------- # +@pytest.mark.parametrize("tool_input", [ + {"file_path": "src/app.py"}, + {"file_path": "pkg/mod.ts"}, + {"file_path": "src/App.vue"}, + {"file_path": "src/Hero.astro"}, + {"file_path": "src/Card.svelte"}, + {"file_path": "SRC/APP.PY"}, # uppercase extension + {"file_path": "src/a.test.tsx"}, # multi-dot -> .tsx + {"file_path": "lib/foo.min.js"}, # multi-dot -> .js + {"file_path": r"src\components\app.py"}, # windows backslashes + {"pattern": "**/*.py", "path": "src"}, # glob pattern + {"pattern": "**/*.astro"}, +]) +def test_read_nudges(tool_input, tmp_path, monkeypatch): + out = _invoke("read", {"tool_input": tool_input}, tmp_path, monkeypatch) + assert "graphify query" in out, f"{tool_input!r} should nudge" + + +# --------------------------------------------------------------------------- # +# read: targets that MUST stay silent +# --------------------------------------------------------------------------- # +@pytest.mark.parametrize("tool_input", [ + {"file_path": "package.json"}, # .json must not match .js + {"file_path": "tsconfig.json"}, + {"file_path": "data.geojson"}, + {"file_path": "uv.lock"}, + {"file_path": "logo.png"}, + {"file_path": "data.bin"}, + {"file_path": ".gitignore"}, + {"file_path": "Makefile"}, # no extension + {"file_path": "my.ts/file"}, # extension on a directory segment + {"file_path": "graphify-out/GRAPH_REPORT.md"}, # the graph's own output + {"file_path": ""}, + {}, # nothing at all +]) +def test_read_silent(tool_input, tmp_path, monkeypatch): + out = _invoke("read", {"tool_input": tool_input}, tmp_path, monkeypatch) + assert out.strip() == "", f"{tool_input!r} should be silent" + + +def test_read_silent_without_graph(tmp_path, monkeypatch): + out = _invoke("read", {"tool_input": {"file_path": "src/app.py"}}, tmp_path, monkeypatch, graph=False) + assert out.strip() == "" + + +def test_read_non_dict_tool_input_is_silent(tmp_path, monkeypatch): + out = _invoke("read", {"tool_input": ["src/app.py"]}, tmp_path, monkeypatch) + assert out.strip() == "" + + +def test_read_respects_custom_output_dir_name(tmp_path, monkeypatch): + # A source file living under a CUSTOM output dir name must be suppressed too, + # not just the literal 'graphify-out/'. + out = _invoke("read", {"tool_input": {"file_path": "build-out/report.py"}}, + tmp_path, monkeypatch, graph=True, out_name="build-out") + assert out.strip() == "" + + +def test_read_nudges_source_outside_custom_output_dir(tmp_path, monkeypatch): + out = _invoke("read", {"tool_input": {"file_path": "src/app.py"}}, + tmp_path, monkeypatch, graph=True, out_name="build-out") + assert "graphify query" in out + + +# --------------------------------------------------------------------------- # +# fail-open: malformed / empty stdin never crashes or blocks +# --------------------------------------------------------------------------- # +@pytest.mark.parametrize("kind", ["search", "read"]) +@pytest.mark.parametrize("raw", [b"not json at all", b"", b"[1,2,3]", b"\xff\xfe\x00bad"]) +def test_fail_open_on_bad_stdin(kind, raw, tmp_path, monkeypatch): + out = _invoke(kind, raw, tmp_path, monkeypatch) + assert out.strip() == "" + + +def test_search_out_path_error_is_swallowed(tmp_path, monkeypatch): + # If the graph-existence check itself throws, the guard stays silent (never + # blocks the tool). + def _boom(*a, **k): + raise OSError("boom") + monkeypatch.setattr("graphify.paths.out_path", _boom) + out = _invoke("search", {"tool_input": {"command": "grep x"}}, tmp_path, monkeypatch) + assert out.strip() == "" + + +# --------------------------------------------------------------------------- # +# gemini: BeforeTool contract (always allow; nudge only when a graph exists) +# --------------------------------------------------------------------------- # +def test_gemini_allow_with_nudge(tmp_path, monkeypatch): + out = _invoke("gemini", None, tmp_path, monkeypatch, graph=True) + payload = json.loads(out) + assert payload["decision"] == "allow" + assert "graphify query" in payload["additionalContext"] + + +def test_gemini_allow_without_graph(tmp_path, monkeypatch): + out = _invoke("gemini", None, tmp_path, monkeypatch, graph=False) + payload = json.loads(out) + assert payload == {"decision": "allow"} + + +def test_gemini_always_allows_even_when_check_throws(tmp_path, monkeypatch): + def _boom(*a, **k): + raise OSError("boom") + monkeypatch.setattr("graphify.paths.out_path", _boom) + out = _invoke("gemini", None, tmp_path, monkeypatch, graph=True) + assert json.loads(out) == {"decision": "allow"} + + +# --------------------------------------------------------------------------- # +# subcommand dispatch, exit codes, and UTF-8 fidelity (real subprocess) +# --------------------------------------------------------------------------- # +def _env(): + e = dict(os.environ) + e.pop("GRAPHIFY_OUT", None) + return e + + +def _cli(args, tmp_path, stdin=""): + return subprocess.run( + [sys.executable, "-m", "graphify", *args], + input=stdin, capture_output=True, text=True, cwd=tmp_path, env=_env(), + ) + + +def test_dispatch_missing_mode_exits_zero_silent(tmp_path): + r = _cli(["hook-guard"], tmp_path, stdin="{}") + assert r.returncode == 0 + assert r.stdout.strip() == "" + + +def test_dispatch_unknown_mode_exits_zero_silent(tmp_path): + r = _cli(["hook-guard", "bogus"], tmp_path, stdin="{}") + assert r.returncode == 0 + assert r.stdout.strip() == "" + + +@pytest.mark.parametrize("args,stdin", [ + (["hook-guard", "search"], '{"tool_input":{"command":"grep x"}}'), + (["hook-guard", "read"], '{"tool_input":{"file_path":"a.py"}}'), + (["hook-guard", "gemini"], ""), +]) +def test_dispatch_always_exits_zero(args, stdin, tmp_path): + # even with a graph present (nudge path), exit code must be 0 (never blocks) + (tmp_path / "graphify-out").mkdir() + (tmp_path / "graphify-out" / "graph.json").write_text("{}", encoding="utf-8") + r = _cli(args, tmp_path, stdin=stdin) + assert r.returncode == 0 + + +def test_read_nudge_em_dash_survives_utf8(tmp_path): + # The read nudge contains an em dash; the emitted bytes must be valid UTF-8 + # and parse back cleanly (guards the ensure_ascii=False + stdout reconfigure). + (tmp_path / "graphify-out").mkdir() + (tmp_path / "graphify-out" / "graph.json").write_text("{}", encoding="utf-8") + r = subprocess.run( + [sys.executable, "-m", "graphify", "hook-guard", "read"], + input=b'{"tool_input":{"file_path":"src/app.py"}}', + capture_output=True, cwd=tmp_path, env=_env(), + ) + assert r.returncode == 0 + text = r.stdout.decode("utf-8") # raises if not valid UTF-8 + payload = json.loads(text) + assert "—" in payload["hookSpecificOutput"]["additionalContext"] # em dash preserved From 80e199defa4a63781f7e846fe7d4f24bfae732b9 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 6 Jul 2026 16:52:10 +0100 Subject: [PATCH 013/292] Release 0.9.8 Fixes: Windows PreToolUse/BeforeTool hooks for Claude Code, Codebuddy and Gemini CLI (#522); CLAUDE.md/AGENTS.md section-write data loss (#1688); tiktoken special-token crash (#1685); Ollama hang retry-multiplication (#1686); truncated community-label reply salvage (#1690); cluster-only labeling token/cost accounting (#1694); discovery-layer file drops from snapshots/ and name-keyword filters (#1666); deepseek thinking default + GRAPHIFY_DISABLE_THINKING opt-in (#1621). Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 2 +- pyproject.toml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0c95333a5..2fc06fbcf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ Full release notes with details on each version: [GitHub Releases](https://github.com/safishamsi/graphify/releases) -## Unreleased +## 0.9.8 (2026-07-06) - Fix: the Claude Code / Codebuddy `PreToolUse` and Gemini CLI `BeforeTool` graph-nudge hooks now work on Windows (#522). The hooks were inline POSIX bash (`case/esac`, `[ -f ]`, single-quoted `echo`), which Windows cmd.exe/PowerShell cannot parse — so on Windows the hook failed silently, no "run `graphify query` before grepping/reading raw files" context was injected, and users had to invoke `/graphify` by hand. The detection logic (grep-command match, source-file extension match, skip-if-under-output-dir, graph-exists check) moved into a shell-agnostic `graphify hook-guard ` subcommand invoked via the absolute exe path (the same pattern the codex hook already uses), so the hook parses and runs identically on Windows, macOS, and Linux. Behavior on macOS/Linux is unchanged (byte-identical nudge payload); the graph path now also honors `GRAPHIFY_OUT`. The Gemini `BeforeTool` hook got the same treatment (`graphify hook-guard gemini`), which also removes its dependency on a bare `python` being on PATH. Codex stays a no-op there because Codex Desktop rejects `additionalContext`. - Fix: `--update`-style section writes to `CLAUDE.md`/`AGENTS.md` no longer corrupt or drop content (#1688, thanks @bdfinst). `_replace_or_append_section` located its managed block by substring (`marker in content`) and `next(... if marker in line)`, so a heading that appeared as a substring of another line (or duplicate headings) matched the wrong offset and the rewrite could truncate the file. It now matches the section heading exactly (`line.strip() == marker`), appends when absent, and prefers the last exact match when several exist, so unrelated content is preserved. diff --git a/pyproject.toml b/pyproject.toml index 9f3f48881..aa154a49b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "graphifyy" -version = "0.9.7" +version = "0.9.8" description = "AI coding assistant skill (Claude Code, CodeBuddy, Codex, OpenCode, Kilo Code, Cursor, Gemini CLI, Aider, OpenClaw, Factory Droid, Trae, Hermes, Kiro, Pi, Devin CLI, Google Antigravity) - turn any folder of code, docs, papers, images, or videos into a queryable knowledge graph" readme = "README.md" license = { file = "LICENSE" } From aba1232b669024f43a6ad2731933a27e88b1b656 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 6 Jul 2026 17:38:03 +0100 Subject: [PATCH 014/292] =?UTF-8?q?docs(readme):=20stronger=20front=20door?= =?UTF-8?q?=20=E2=80=94=20badge=20reorder,=20accurate=20hero,=2030-sec=20q?= =?UTF-8?q?uickstart?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Badges reranked for click-through and split: lean top strip (PyPI, Downloads, Discord, LinkedIn, YC S26) with trust signals leading and social/company links at the tail; a "Community and links" footer (Discord, X, Sponsor, Book); CI badge dropped, book also linked from "Learn more". - Hero leads with three differentiator bullets, corrected to match the codebase: code is tree-sitter AST (deterministic, no LLM, local), while docs/PDFs/images/ video use your assistant's model or a configured API key for a semantic pass. Removed the inaccurate "local embedder" claim (there is no embedder; dedup is MinHash/LSH and there is no vector store) and qualified "zero LLM tokens" to the code path. "Not a vector index" is accurate. - Added a 30-second quickstart (install + graphify install + /graphify .) above the fold, moved "See it in action" above "What it does" so differentiation is shown before it is enumerated, and compressed the "works in" list to one line. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 65 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 43 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 1952acbc2..bd1dff423 100644 --- a/README.md +++ b/README.md @@ -15,19 +15,19 @@

- YC S26 - Discord - The Memory Layer - CI PyPI Downloads - Sponsor + Discord LinkedIn - X + YC S26

Type `/graphify` in your AI coding assistant and it maps your entire project (code, docs, PDFs, images, videos) into a **knowledge graph** you can **query instead of grepping** through files. +- **Code maps for free, fully local.** Code is parsed with tree-sitter AST: deterministic, no LLM, nothing leaves your machine. (Docs, PDFs, images and video use your assistant's model, or a configured API key, for a semantic pass.) +- **Every edge is explained.** Each connection is tagged `EXTRACTED` (explicit in the source) or `INFERRED` (resolved by graphify), so you can tell what was read directly from what was inferred. +- **Not a vector index.** No embeddings, no vector store: a real graph you traverse. Ask a question, trace the path between two things, or explain one concept. +

graphify's interactive graph.html showing the FastAPI codebase as a force-directed knowledge graph with a legend of detected communities

@@ -35,7 +35,14 @@ Type `/graphify` in your AI coding assistant and it maps your entire project (co The FastAPI codebase mapped by graphify. Every node is a concept, colors are detected communities, and the whole thing is clickable in graph.html.

-**Works in:** Claude Code, Codex, OpenCode, Kilo Code, Cursor, Gemini CLI, GitHub Copilot CLI, VS Code Copilot Chat, Aider, Amp, OpenClaw, Factory Droid, Trae, Hermes, Kimi Code, Kiro, Pi, Devin CLI, and Google Antigravity. +**Get started** (30 seconds): + +```bash +uv tool install graphifyy # install the CLI (or: pipx install graphifyy) +graphify install # register the skill with your AI assistant +``` + +Then, in your AI assistant: ``` /graphify . @@ -50,21 +57,7 @@ graphify-out/ └── graph.json the full graph — query it anytime without re-reading your files ``` ---- - -## What it does - -A knowledge graph is not a vector index: it is **deterministic**, **every edge is explained**, and it costs **zero tokens to build**. - -| Capability | What you get | -|---|---| -| **God nodes** | The most-connected concepts, so you see what everything flows through | -| **Communities** | The graph split into subsystems (Leiden), with LLM-free labels | -| **Cross-file links** | `calls` / `imports` / `inherits` / `mixes_in` resolved across ~40 languages via tree-sitter AST | -| **Query, path, explain** | Ask a question, trace the path between two things, or explain one concept, all against `graph.json` | -| **Rationale + doc refs** | `# NOTE:` / `# WHY:` comments and ADR/RFC citations become first-class nodes linked to the code | -| **Beyond code** | Docs, PDFs, images, and video/audio all map into the same graph | -| **Local-first** | AST plus a local embedder build the graph with zero LLM credits; your code never leaves your machine | +**Works in** Claude Code, Cursor, Codex, Gemini CLI, GitHub Copilot, and 15+ more — [pick your platform](#install). --- @@ -99,6 +92,22 @@ Every edge carries a **confidence tag** (`EXTRACTED` = explicit in the source, ` --- +## What it does + +What you get out of the box: + +| Capability | What you get | +|---|---| +| **God nodes** | The most-connected concepts, so you see what everything flows through | +| **Communities** | The graph split into subsystems (Leiden), with LLM-free labels | +| **Cross-file links** | `calls` / `imports` / `inherits` / `mixes_in` resolved across ~40 languages via tree-sitter AST | +| **Query, path, explain** | Ask a question, trace the path between two things, or explain one concept, all against `graph.json` | +| **Rationale + doc refs** | `# NOTE:` / `# WHY:` comments and ADR/RFC citations become first-class nodes linked to the code | +| **Beyond code** | Docs, PDFs, images, and video/audio all map into the same graph | +| **Local-first** | Code is parsed locally with tree-sitter (no LLM, nothing leaves your machine); only the semantic pass over docs/media calls a backend, and only if you configure one | + +--- + ## Benchmarks | Benchmark | Metric | graphify | Field | @@ -757,6 +766,7 @@ graphify label ./my-project --backend=openai --model gpt-4o # force a specific - [How it works](docs/how-it-works.md) — the extraction pipeline, community detection, confidence scoring, benchmarks - [ARCHITECTURE.md](ARCHITECTURE.md) — module breakdown, how to add a language - [Optional integrations](docs/docker-mcp-sqlite.md) — Docker MCP Toolkit + SQLite +- [The Memory Layer](https://safishamsi.gumroad.com/l/qetvlo) — the book on the ideas behind graphify, the architecture end to end --- @@ -828,3 +838,14 @@ See [ARCHITECTURE.md](ARCHITECTURE.md) for module responsibilities and how to ad Star History Chart

+ +--- + +## Community and links + +

+ Discord + X + Sponsor + The Memory Layer +

From d1d1f412b2826fbce216fb957c417049dd61fde7 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 6 Jul 2026 23:28:32 +0100 Subject: [PATCH 015/292] fix(serve): match a punctuated label against norm_label symmetrically in _find_node (#1704) `_find_node` built its search term with `_search_tokens` (\w+ tokenization), so "blockStream.ts" became "blockstream ts" (space where the '.' was) while the node's stored `norm_label` keeps punctuation ("blockstream.ts"). The verbatim case is already rescued by the `term == label_tokens` tier (the node label tokenizes the same way), but that is a coincidence: if `label` and `norm_label` diverge, an exactly-typed punctuated label fails to resolve through `explain` even though `path`/`query` find it. Add a punctuation-preserving `norm_query` (`_strip_diacritics(label).lower()`) matched against `norm_label`/`bare_label` across the exact/prefix/substring tiers (and fed to the trigram prefilter so candidates are not missed). Purely additive, symmetric with how norm_label is stored. Regression tests cover the verbatim file-label case and the label/norm_label divergence case that only norm_query resolves. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/serve.py | 18 +++++++++++++++--- tests/test_serve.py | 24 ++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 3 deletions(-) diff --git a/graphify/serve.py b/graphify/serve.py index 28cb46616..6d8b369df 100644 --- a/graphify/serve.py +++ b/graphify/serve.py @@ -664,13 +664,20 @@ def _find_node(G: nx.Graph, label: str) -> list[str]: term = " ".join(_search_tokens(label)) if not term: return [] + # Punctuation-preserving normalized query. `term` tokenizes on \w+ (so + # "blockStream.ts" -> "blockstream ts", space where the '.' was), but a node's + # stored `norm_label` keeps punctuation ("blockstream.ts"). Matching only via + # `term`/`label_tokens` works when the node label tokenizes the same way, but is + # fragile if `label` and `norm_label` diverge. `norm_query` matches `norm_label` + # symmetrically so an exactly-typed punctuated label always resolves (#1704). + norm_query = _strip_diacritics(str(label)).lower().strip() source_exact: list[str] = [] exact: list[str] = [] prefix: list[str] = [] substring: list[str] = [] # Trigram prefilter (graph-iteration order preserved so exact/prefix/substring # ordering — and thus matches[0] — is byte-identical to the full scan). - candidate_ids = _trigram_candidates(G, [term]) + candidate_ids = _trigram_candidates(G, [term, norm_query]) node_iter = ( G.nodes(data=True) if candidate_ids is None else ((nid, G.nodes[nid]) for nid in candidate_ids) @@ -683,16 +690,21 @@ def _find_node(G: nx.Graph, label: str) -> list[str]: nid_lower = nid.lower() if term == source_tokens: source_exact.append(nid) - elif term == norm_label or term == bare_label or term == label_tokens or term == nid_lower: + elif ( + term == norm_label or term == bare_label or term == label_tokens or term == nid_lower + or norm_query == norm_label or norm_query == bare_label + ): exact.append(nid) elif ( norm_label.startswith(term) or bare_label.startswith(term) or label_tokens.startswith(term) or nid_lower.startswith(term) + or norm_label.startswith(norm_query) + or bare_label.startswith(norm_query) ): prefix.append(nid) - elif term in norm_label or term in label_tokens: + elif term in norm_label or term in label_tokens or norm_query in norm_label: substring.append(nid) if source_exact: diff --git a/tests/test_serve.py b/tests/test_serve.py index 2647aa1a8..9bf9aa6a8 100644 --- a/tests/test_serve.py +++ b/tests/test_serve.py @@ -135,6 +135,30 @@ def test_find_node_matches_full_punctuated_unicode_label(): assert _find_node(G, "Skill /auditar — Auditoría inquisitiva de enlaces") == ["n1"] +def test_find_node_matches_punctuated_file_label_exactly(): + # #1704: an exactly-typed punctuated file label must resolve through explain, + # just like it does through path/query. + G = nx.Graph() + G.add_node("f1", label="blockStream.ts", norm_label="blockstream.ts", + source_file="lib/blockStream.ts", source_location="L1") + G.add_node("f2", label="blockStream.test.ts", norm_label="blockstream.test.ts", + source_file="lib/blockStream.test.ts", source_location="L1") + assert _find_node(G, "blockStream.ts")[0] == "f1" + assert _find_node(G, "blockStream.test.ts")[0] == "f2" + + +def test_find_node_resolves_when_label_and_norm_label_diverge(): + # #1704 hardening: the tokenized-label tier only rescues the match by + # coincidence (label tokenizes the same as the query). When `label` and + # `norm_label` diverge, only the symmetric `norm_query == norm_label` match + # resolves it. Here label tokenizes to "blockstream" but norm_label is + # "blockstream.ts" — this fails without the norm_query path. + G = nx.Graph() + G.add_node("n1", label="BlockStream", norm_label="blockstream.ts", + source_file="lib/x.ts", source_location="L1") + assert _find_node(G, "blockStream.ts") == ["n1"] + + # --- trigram candidate prefilter (the trigram index that shrinks the O(N) scan) --- From 377dc7f38403a56a70f406d9a6224742c87e9a01 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Tue, 7 Jul 2026 00:06:49 +0100 Subject: [PATCH 016/292] fix(extract): warn when code files have no AST extractor instead of silently dropping them (#1689) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extensions like .r/.R (also .ejs, .ets) are in CODE_EXTENSIONS, so those files are classified as code and counted in the scan, but there is no entry for them in the extractor dispatch — so they produce zero nodes and are silently absent from the graph while the CLI still reports success. The #1666 zero-node warning deliberately skips them (it only fires when an extractor exists), so nothing surfaced the gap. extract() now emits a warning listing the offending extensions and counts ("N file(s) are classified as code but graphify has no AST extractor ...: .r (2)") so a primarily-R (or .ejs/.ets) corpus no longer looks fully mapped when it is not. Grouped by extension, fires only for files actually present with no extractor (today: .ejs, .ets, .r). Adding real grammars for these remains the follow-up; this removes the silent-data-loss now. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/extract.py | 23 +++++++++++++++++++++++ tests/test_extract.py | 26 ++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/graphify/extract.py b/graphify/extract.py index be7e99448..3d73ee7d8 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -16403,6 +16403,29 @@ def extract( file=sys.stderr, flush=True, ) + # #1689: a file counted as code (extension in CODE_EXTENSIONS) but with no AST + # extractor wired up (e.g. .r/.R — there is no tree-sitter-r dispatch) silently + # contributes zero nodes. The #1666 warning above deliberately skips these (it + # only fires when an extractor exists), so surface them explicitly, grouped by + # extension, rather than reporting success as if the language were mapped. + from graphify.detect import CODE_EXTENSIONS as _CODE_EXTS + _no_extractor: dict[str, int] = {} + for _p in paths: + _ext = _p.suffix.lower() + if _ext in _CODE_EXTS and _get_extractor(_p) is None: + _no_extractor[_ext] = _no_extractor.get(_ext, 0) + 1 + if _no_extractor: + _by_count = ", ".join( + f"{ext} ({n})" for ext, n in sorted(_no_extractor.items(), key=lambda kv: (-kv[1], kv[0])) + ) + _tot = sum(_no_extractor.values()) + print( + f" warning: {_tot} file(s) are classified as code but graphify has no AST " + f"extractor for their language, so they contributed nothing to the graph: " + f"{_by_count}. Please open an issue to request support for these (#1689).", + file=sys.stderr, flush=True, + ) + all_nodes: list[dict] = [] all_edges: list[dict] = [] all_raw_calls: list[dict] = [] diff --git a/tests/test_extract.py b/tests/test_extract.py index 4e76ad120..d67a763f2 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -1769,3 +1769,29 @@ def test_case_insensitive_suffix_filtering(tmp_path): assert "myJSFunction()" in labels assert "MyTSClass" in labels + + +def test_extract_warns_on_code_files_with_no_ast_extractor(tmp_path, capsys): + # #1689: .r/.R is in CODE_EXTENSIONS (counted as code) but has no AST extractor, + # so R files silently contribute nothing. extract() must surface that instead of + # reporting success as if the language were mapped. + r1 = tmp_path / "analysis.R"; r1.write_text("f <- function(x) x + 1\n") + r2 = tmp_path / "helper.r"; r2.write_text("g <- function(y) y * 2\n") + py = tmp_path / "main.py"; py.write_text("def main():\n return 1\n") + + result = extract([r1, r2, py], cache_root=tmp_path) + err = capsys.readouterr().err + + assert "no AST extractor" in err + assert ".r (2)" in err # both R files grouped under the lowercased ext + assert "#1689" in err + # the Python file still extracts normally + labels = [n.get("label") for n in result["nodes"]] + assert any(str(l).startswith("main") for l in labels) + + +def test_extract_no_warning_when_all_code_has_extractors(tmp_path, capsys): + py = tmp_path / "a.py"; py.write_text("def a():\n return 1\n") + extract([py], cache_root=tmp_path) + err = capsys.readouterr().err + assert "no AST extractor" not in err From f5d50adbd003dba310f4111c24e3d67c3a5ba141 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Tue, 7 Jul 2026 00:12:36 +0100 Subject: [PATCH 017/292] fix(extract): keep AST progress denominator consistent to the end (#1693) Intermediate progress lines count against len(uncached_work) ("X/Y uncached files"), but the final line switched to total_files ("Y/Y files"), which includes cached hits and files with no extractor that never entered uncached_work. On a large corpus with unsupported-language files, the total jumped upward right after 99% with no explanation. Both the parallel and sequential final lines now report the same uncached_work denominator, so the count no longer appears to change mid-run. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/extract.py | 11 +++++++++-- tests/test_extract.py | 18 ++++++++++++++++++ 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/graphify/extract.py b/graphify/extract.py index 3d73ee7d8..dea5bd0b3 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -16254,8 +16254,13 @@ def _extract_parallel( ) return False if total_files >= _PROGRESS_INTERVAL: + # Report the same denominator the intermediate lines used (uncached files + # actually processed this run), not total_files — switching to the full + # corpus made the count jump upward at the end (cached hits + files with no + # extractor never entered uncached_work), which read as inconsistent (#1693). + _done = len(uncached_work) print( - f" AST extraction: {total_files}/{total_files} files (100%) [{max_workers} workers]", + f" AST extraction: {_done}/{_done} uncached files (100%) [{max_workers} workers]", flush=True, ) return True @@ -16290,7 +16295,9 @@ def _extract_sequential( save_cached(path, result, effective_root) per_file[idx] = result if total_files >= _PROGRESS_INTERVAL: - print(f" AST extraction: {total_files}/{total_files} files (100%)", flush=True) + # Consistent denominator with the intermediate lines (#1693). + _done = len(uncached_work) + print(f" AST extraction: {_done}/{_done} uncached files (100%)", flush=True) _PARALLEL_THRESHOLD = 20 diff --git a/tests/test_extract.py b/tests/test_extract.py index d67a763f2..7cae30776 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -1795,3 +1795,21 @@ def test_extract_no_warning_when_all_code_has_extractors(tmp_path, capsys): extract([py], cache_root=tmp_path) err = capsys.readouterr().err assert "no AST extractor" not in err + + +def test_extract_progress_final_line_uses_consistent_denominator(tmp_path, capsys): + # #1693: intermediate progress lines count against uncached_work; the final + # "100%" line must NOT switch to total_files (which includes cached hits and + # files with no extractor), or the count appears to jump upward at the end. + for i in range(100): + (tmp_path / f"m{i}.py").write_text(f"def f{i}():\n return {i}\n") + for i in range(5): + (tmp_path / f"s{i}.r").write_text(f"g{i} <- function(x) x\n") # no extractor + paths = sorted(tmp_path.glob("*.py")) + sorted(tmp_path.glob("*.r")) # total 105 + + extract(paths, cache_root=tmp_path, parallel=False) + out = capsys.readouterr().out + + # final progress line reports the uncached count (100), not the total (105) + assert "100/100 uncached files (100%)" in out + assert "105/105 files" not in out, "final line must not switch to total_files (#1693)" From f84a0af1bf7970d69c5b46fceceb55d444b5407e Mon Sep 17 00:00:00 2001 From: safishamsi Date: Tue, 7 Jul 2026 10:57:52 +0100 Subject: [PATCH 018/292] fix(report): don't emit dangling [[_COMMUNITY_*]] wikilinks by default (#1712) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GRAPH_REPORT.md rendered the Community Hubs section as Obsidian wikilinks, but the `_COMMUNITY_*.md` notes they target are only created by the opt-in `--obsidian` export — and the report is written at build time, before any export runs. So on a default run every link dangled: inside an Obsidian vault they spawned phantom `_COMMUNITY_*` nodes in the graph view, and outside Obsidian they rendered as literal brackets that navigate nowhere. `generate()` now takes `obsidian: bool = False`; by default the hubs render as a plain list, and the wikilink form is emitted only when a caller opts in. The Obsidian export's own community notes already cross-link each other, so the vault stays navigable without the report's links. Mirrors the #1444/#1465 portability fix that was applied to `export wiki`. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/report.py | 16 ++++++++++++---- tests/test_report.py | 19 +++++++++++++++++++ 2 files changed, 31 insertions(+), 4 deletions(-) diff --git a/graphify/report.py b/graphify/report.py index 5bac06d05..248bce9a1 100644 --- a/graphify/report.py +++ b/graphify/report.py @@ -82,6 +82,7 @@ def generate( min_community_size: int = 3, built_at_commit: str | None = None, learning: dict | None = None, + obsidian: bool = False, ) -> str: today = date.today().isoformat() @@ -140,14 +141,21 @@ def generate( "- Run `graphify update .` after code changes (no API cost).", ] - # Community hub navigation - links to _COMMUNITY_*.md files in the Obsidian vault. - # Without these, GRAPH_REPORT.md is a dead-end and the vault splits into disconnected components. + # Community hub navigation. The `_COMMUNITY_*.md` notes these wikilinks target + # are only created by the opt-in `--obsidian` export, and the report is written + # at build time (before any export runs), so emitting wikilinks by default left + # every link dangling — polluting an Obsidian vault's graph view and rendering as + # literal brackets everywhere else (#1712). Emit wikilinks only when the caller + # signals Obsidian output; otherwise a plain list, which navigates nowhere-to-break. if non_empty: lines += ["", "## Community Hubs (Navigation)"] for cid in non_empty: label = community_labels.get(cid, f"Community {cid}") - safe = _safe_community_name(label) - lines.append(f"- [[_COMMUNITY_{safe}|{label}]]") + if obsidian: + safe = _safe_community_name(label) + lines.append(f"- [[_COMMUNITY_{safe}|{label}]]") + else: + lines.append(f"- {label}") lines += [ "", diff --git a/tests/test_report.py b/tests/test_report.py index 00be0f36d..767e2ba34 100644 --- a/tests/test_report.py +++ b/tests/test_report.py @@ -135,3 +135,22 @@ def test_import_cycles_section_absent_for_documents_only_corpus(): tokens = {"input": 0, "output": 0} report = generate(G, communities, cohesion, labels, gods, surprises, detection, tokens, "./project") assert "## Import Cycles" not in report + + +def test_report_hubs_are_plain_text_by_default(): + # #1712: without --obsidian the _COMMUNITY_*.md notes don't exist, so wikilinks + # would dangle (and pollute an Obsidian vault's graph view). Default to plain text. + G, communities, cohesion, labels, gods, surprises, detection, tokens = make_inputs() + labels = {cid: f"Widget {cid}" for cid in communities} + report = generate(G, communities, cohesion, labels, gods, surprises, detection, tokens, "./project", min_community_size=1) + assert "## Community Hubs (Navigation)" in report + assert "[[_COMMUNITY_" not in report, "must not emit dangling Obsidian wikilinks by default (#1712)" + assert any(f"- Widget {cid}" in report for cid in communities) + + +def test_report_hubs_use_wikilinks_when_obsidian(): + # The opt-in path keeps the vault-navigable wikilink form. + G, communities, cohesion, labels, gods, surprises, detection, tokens = make_inputs() + labels = {cid: f"Widget {cid}" for cid in communities} + report = generate(G, communities, cohesion, labels, gods, surprises, detection, tokens, "./project", min_community_size=1, obsidian=True) + assert "[[_COMMUNITY_" in report From 23457d11ff3c01fd51413a2c9c22912ab7d295aa Mon Sep 17 00:00:00 2001 From: safishamsi Date: Tue, 7 Jul 2026 11:19:18 +0100 Subject: [PATCH 019/292] feat(detect): surface unclassified files instead of dropping them silently (#1692) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When classify_file() returned None — an extensionless, non-shebang file (Dockerfile, Gemfile, Makefile, Rakefile, LICENSE, ...) or an unsupported extension — the file left no trace at all: not counted, not listed, nothing. A user had no way to tell from graphify's output that those files were even considered. detect() now collects these into an "unclassified" list in its result, and `graphify extract` prints a one-line summary after the scan counts: "N file(s) not classified (no supported extension or shebang), skipped: Dockerfile, Makefile, ...". Real code/docs are unaffected. This is the visibility half of the issue; wiring up extractors/manifest handling for Dockerfile/Makefile-style files remains a separate feature. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/__main__.py | 11 +++++++++++ graphify/detect.py | 9 +++++++++ tests/test_detect.py | 22 ++++++++++++++++++++++ 3 files changed, 42 insertions(+) diff --git a/graphify/__main__.py b/graphify/__main__.py index 8d37c911e..285d19c1b 100644 --- a/graphify/__main__.py +++ b/graphify/__main__.py @@ -4673,6 +4673,17 @@ def _parse_float(name: str, raw: str) -> float: f"{len(doc_files)} docs, {len(paper_files)} papers, " f"{len(image_files)} images" ) + # Surface files that were seen but not classified (extensionless non-shebang + # project files like Dockerfile/Makefile, or unsupported extensions), so they + # are no longer invisible in graphify's own output (#1692). + _unclassified = detection.get("unclassified", []) if isinstance(detection, dict) else [] + if _unclassified: + _names = ", ".join(sorted({Path(p).name for p in _unclassified})[:6]) + _more = f" (+{len(_unclassified) - 6} more)" if len(_unclassified) > 6 else "" + print( + f"[graphify extract] {len(_unclassified)} file(s) not classified " + f"(no supported extension or shebang), skipped: {_names}{_more}" + ) stages.mark("detect") # Resolve the LLM backend only now that we know whether the corpus diff --git a/graphify/detect.py b/graphify/detect.py index c9d7792c7..85f4b8b8b 100644 --- a/graphify/detect.py +++ b/graphify/detect.py @@ -1086,6 +1086,7 @@ def _wc(path: Path) -> int: return _cache.cached_word_count(path, root, count_words) skipped_sensitive: list[str] = [] + unclassified: list[str] = [] ignore_patterns = _load_graphifyignore(root) ignore_cache: dict[Path, bool] = {} # shared across all _is_ignored calls in this scan # CLI --exclude patterns are anchored at the scan root and appended last @@ -1171,6 +1172,13 @@ def _wc(path: Path) -> int: skipped_sensitive.append(str(p)) continue ftype = classify_file(p) + if not ftype: + # Considered but unclassifiable: an extension not in any supported set, + # or an extensionless, non-shebang file (Dockerfile, Gemfile, Makefile, + # Rakefile, LICENSE, ...). Previously these left no trace at all — not + # counted, not listed — so a user couldn't tell they were seen (#1692). + unclassified.append(str(p)) + continue if ftype: if p.suffix.lower() in GOOGLE_WORKSPACE_EXTENSIONS: if not google_workspace: @@ -1236,6 +1244,7 @@ def _wc(path: Path) -> int: "needs_graph": needs_graph, "warning": warning, "skipped_sensitive": skipped_sensitive, + "unclassified": sorted(unclassified), "graphifyignore_patterns": len(ignore_patterns), "scan_root": str(root.resolve()), } diff --git a/tests/test_detect.py b/tests/test_detect.py index 0c07e24fd..b26d37ec5 100644 --- a/tests/test_detect.py +++ b/tests/test_detect.py @@ -1575,3 +1575,25 @@ def test_convert_office_file_does_not_rewrite_existing_sidecar(tmp_path, monkeyp second = detect_mod.convert_office_file(src, out_dir) assert second == first assert second.stat().st_mtime_ns == mtime_before + + +def test_detect_records_unclassified_extensionless_files(tmp_path): + # #1692: extensionless, non-shebang project files (Dockerfile, Makefile, ...) + # were considered but left no trace. detect() now lists them under + # "unclassified" so they can be surfaced instead of silently vanishing. + (tmp_path / "app.py").write_text("def f():\n return 1\n") + (tmp_path / "Dockerfile").write_text("FROM python:3.12\nRUN pip install x\n") + (tmp_path / "Makefile").write_text("build:\n\techo hi\n") + (tmp_path / "LICENSE").write_text("MIT License\n") + res = detect(tmp_path) + unclassified = sorted(Path(p).name for p in res.get("unclassified", [])) + assert unclassified == ["Dockerfile", "LICENSE", "Makefile"] + # real code is still classified, not swept into unclassified + assert any("app.py" in f for f in res["files"].get("code", [])) + + +def test_detect_unclassified_empty_when_all_supported(tmp_path): + (tmp_path / "a.py").write_text("x = 1\n") + (tmp_path / "README.md").write_text("# hi\n") + res = detect(tmp_path) + assert res.get("unclassified", []) == [] From ac6bb274a17d37a7ea20678ab722ae0aea747a67 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Tue, 7 Jul 2026 11:34:54 +0100 Subject: [PATCH 020/292] docs(skill): fix stale /graphify usage comment that claimed an Obsidian vault by default (#1681) The Usage block's bare `/graphify` comment read "full pipeline on current directory -> Obsidian vault", contradicting Step 6 (HTML viz always; Obsidian vault only when --obsidian is explicitly given). The comment predated the opt-in change and told agents a bare /graphify produces a vault. It now reads "full pipeline on current directory (HTML viz; add --obsidian for a vault)". Fixed at the skillgen source (fragments/core/core.md + the aider monolith override), added a sanctioned-diff predicate so the monolith round-trip guard allows the change, and re-blessed the expected/ snapshots. Every generated skill-*.md now carries the corrected comment; the only content change across all rendered files is this one line. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/skill-agents.md | 2 +- graphify/skill-aider.md | 2 +- graphify/skill-amp.md | 2 +- graphify/skill-claw.md | 2 +- graphify/skill-codex.md | 2 +- graphify/skill-copilot.md | 2 +- graphify/skill-droid.md | 2 +- graphify/skill-kilo.md | 2 +- graphify/skill-kiro.md | 2 +- graphify/skill-opencode.md | 2 +- graphify/skill-pi.md | 2 +- graphify/skill-trae.md | 2 +- graphify/skill-vscode.md | 2 +- graphify/skill-windows.md | 2 +- graphify/skill.md | 2 +- tools/skillgen/expected/graphify__skill-agents.md | 2 +- tools/skillgen/expected/graphify__skill-aider.md | 2 +- tools/skillgen/expected/graphify__skill-amp.md | 2 +- tools/skillgen/expected/graphify__skill-claw.md | 2 +- tools/skillgen/expected/graphify__skill-codex.md | 2 +- tools/skillgen/expected/graphify__skill-copilot.md | 2 +- tools/skillgen/expected/graphify__skill-droid.md | 2 +- tools/skillgen/expected/graphify__skill-kilo.md | 2 +- tools/skillgen/expected/graphify__skill-kiro.md | 2 +- tools/skillgen/expected/graphify__skill-opencode.md | 2 +- tools/skillgen/expected/graphify__skill-pi.md | 2 +- tools/skillgen/expected/graphify__skill-trae.md | 2 +- tools/skillgen/expected/graphify__skill-vscode.md | 2 +- tools/skillgen/expected/graphify__skill-windows.md | 2 +- tools/skillgen/expected/graphify__skill.md | 2 +- tools/skillgen/fragments/core/aider.md | 2 +- tools/skillgen/fragments/core/core.md | 2 +- tools/skillgen/gen.py | 12 ++++++++++++ 33 files changed, 44 insertions(+), 32 deletions(-) diff --git a/graphify/skill-agents.md b/graphify/skill-agents.md index 2e19931fa..1b3da7c56 100644 --- a/graphify/skill-agents.md +++ b/graphify/skill-agents.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/graphify/skill-aider.md b/graphify/skill-aider.md index 2ccf5326c..3559b4802 100644 --- a/graphify/skill-aider.md +++ b/graphify/skill-aider.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify --mode deep # thorough extraction, richer INFERRED edges /graphify --update # incremental - re-extract only new/changed files diff --git a/graphify/skill-amp.md b/graphify/skill-amp.md index 2e19931fa..1b3da7c56 100644 --- a/graphify/skill-amp.md +++ b/graphify/skill-amp.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/graphify/skill-claw.md b/graphify/skill-claw.md index b3542438d..a53809df6 100644 --- a/graphify/skill-claw.md +++ b/graphify/skill-claw.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/graphify/skill-codex.md b/graphify/skill-codex.md index e6e411ea0..36cadce59 100644 --- a/graphify/skill-codex.md +++ b/graphify/skill-codex.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/graphify/skill-copilot.md b/graphify/skill-copilot.md index b3542438d..a53809df6 100644 --- a/graphify/skill-copilot.md +++ b/graphify/skill-copilot.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/graphify/skill-droid.md b/graphify/skill-droid.md index d1147b903..2bef076dc 100644 --- a/graphify/skill-droid.md +++ b/graphify/skill-droid.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/graphify/skill-kilo.md b/graphify/skill-kilo.md index c45578165..d3aca08c2 100644 --- a/graphify/skill-kilo.md +++ b/graphify/skill-kilo.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/graphify/skill-kiro.md b/graphify/skill-kiro.md index b3542438d..a53809df6 100644 --- a/graphify/skill-kiro.md +++ b/graphify/skill-kiro.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/graphify/skill-opencode.md b/graphify/skill-opencode.md index 684e0e223..2aa66f5b3 100644 --- a/graphify/skill-opencode.md +++ b/graphify/skill-opencode.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/graphify/skill-pi.md b/graphify/skill-pi.md index b3542438d..a53809df6 100644 --- a/graphify/skill-pi.md +++ b/graphify/skill-pi.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/graphify/skill-trae.md b/graphify/skill-trae.md index ac16b925d..c1635cde7 100644 --- a/graphify/skill-trae.md +++ b/graphify/skill-trae.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/graphify/skill-vscode.md b/graphify/skill-vscode.md index 31a3352f7..30247498e 100644 --- a/graphify/skill-vscode.md +++ b/graphify/skill-vscode.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/graphify/skill-windows.md b/graphify/skill-windows.md index 5a55c577b..b298f29ab 100644 --- a/graphify/skill-windows.md +++ b/graphify/skill-windows.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/graphify/skill.md b/graphify/skill.md index b3542438d..a53809df6 100644 --- a/graphify/skill.md +++ b/graphify/skill.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/expected/graphify__skill-agents.md b/tools/skillgen/expected/graphify__skill-agents.md index 2e19931fa..1b3da7c56 100644 --- a/tools/skillgen/expected/graphify__skill-agents.md +++ b/tools/skillgen/expected/graphify__skill-agents.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/expected/graphify__skill-aider.md b/tools/skillgen/expected/graphify__skill-aider.md index 2ccf5326c..3559b4802 100644 --- a/tools/skillgen/expected/graphify__skill-aider.md +++ b/tools/skillgen/expected/graphify__skill-aider.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify --mode deep # thorough extraction, richer INFERRED edges /graphify --update # incremental - re-extract only new/changed files diff --git a/tools/skillgen/expected/graphify__skill-amp.md b/tools/skillgen/expected/graphify__skill-amp.md index 2e19931fa..1b3da7c56 100644 --- a/tools/skillgen/expected/graphify__skill-amp.md +++ b/tools/skillgen/expected/graphify__skill-amp.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/expected/graphify__skill-claw.md b/tools/skillgen/expected/graphify__skill-claw.md index b3542438d..a53809df6 100644 --- a/tools/skillgen/expected/graphify__skill-claw.md +++ b/tools/skillgen/expected/graphify__skill-claw.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/expected/graphify__skill-codex.md b/tools/skillgen/expected/graphify__skill-codex.md index e6e411ea0..36cadce59 100644 --- a/tools/skillgen/expected/graphify__skill-codex.md +++ b/tools/skillgen/expected/graphify__skill-codex.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/expected/graphify__skill-copilot.md b/tools/skillgen/expected/graphify__skill-copilot.md index b3542438d..a53809df6 100644 --- a/tools/skillgen/expected/graphify__skill-copilot.md +++ b/tools/skillgen/expected/graphify__skill-copilot.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/expected/graphify__skill-droid.md b/tools/skillgen/expected/graphify__skill-droid.md index d1147b903..2bef076dc 100644 --- a/tools/skillgen/expected/graphify__skill-droid.md +++ b/tools/skillgen/expected/graphify__skill-droid.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/expected/graphify__skill-kilo.md b/tools/skillgen/expected/graphify__skill-kilo.md index c45578165..d3aca08c2 100644 --- a/tools/skillgen/expected/graphify__skill-kilo.md +++ b/tools/skillgen/expected/graphify__skill-kilo.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/expected/graphify__skill-kiro.md b/tools/skillgen/expected/graphify__skill-kiro.md index b3542438d..a53809df6 100644 --- a/tools/skillgen/expected/graphify__skill-kiro.md +++ b/tools/skillgen/expected/graphify__skill-kiro.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/expected/graphify__skill-opencode.md b/tools/skillgen/expected/graphify__skill-opencode.md index 684e0e223..2aa66f5b3 100644 --- a/tools/skillgen/expected/graphify__skill-opencode.md +++ b/tools/skillgen/expected/graphify__skill-opencode.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/expected/graphify__skill-pi.md b/tools/skillgen/expected/graphify__skill-pi.md index b3542438d..a53809df6 100644 --- a/tools/skillgen/expected/graphify__skill-pi.md +++ b/tools/skillgen/expected/graphify__skill-pi.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/expected/graphify__skill-trae.md b/tools/skillgen/expected/graphify__skill-trae.md index ac16b925d..c1635cde7 100644 --- a/tools/skillgen/expected/graphify__skill-trae.md +++ b/tools/skillgen/expected/graphify__skill-trae.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/expected/graphify__skill-vscode.md b/tools/skillgen/expected/graphify__skill-vscode.md index 31a3352f7..30247498e 100644 --- a/tools/skillgen/expected/graphify__skill-vscode.md +++ b/tools/skillgen/expected/graphify__skill-vscode.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/expected/graphify__skill-windows.md b/tools/skillgen/expected/graphify__skill-windows.md index 5a55c577b..b298f29ab 100644 --- a/tools/skillgen/expected/graphify__skill-windows.md +++ b/tools/skillgen/expected/graphify__skill-windows.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/expected/graphify__skill.md b/tools/skillgen/expected/graphify__skill.md index b3542438d..a53809df6 100644 --- a/tools/skillgen/expected/graphify__skill.md +++ b/tools/skillgen/expected/graphify__skill.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/fragments/core/aider.md b/tools/skillgen/fragments/core/aider.md index 2ccf5326c..3559b4802 100644 --- a/tools/skillgen/fragments/core/aider.md +++ b/tools/skillgen/fragments/core/aider.md @@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify --mode deep # thorough extraction, richer INFERRED edges /graphify --update # incremental - re-extract only new/changed files diff --git a/tools/skillgen/fragments/core/core.md b/tools/skillgen/fragments/core/core.md index 0516b3a6e..8d8aab7aa 100644 --- a/tools/skillgen/fragments/core/core.md +++ b/tools/skillgen/fragments/core/core.md @@ -7,7 +7,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti ## Usage ``` -/graphify # full pipeline on current directory → Obsidian vault +/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault) /graphify # full pipeline on specific path /graphify https://github.com// # clone repo then run full pipeline on it /graphify https://github.com// --branch # clone a specific branch diff --git a/tools/skillgen/gen.py b/tools/skillgen/gen.py index 0082e9e51..e742a62c2 100644 --- a/tools/skillgen/gen.py +++ b/tools/skillgen/gen.py @@ -877,6 +877,17 @@ def _is_shebang_allowlist_fix_line(line: str) -> bool: return "[!a-zA-Z0-9/_." in line +def _is_obsidian_usage_comment_line(line: str) -> bool: + """Whether a line is part of the ``/graphify`` usage-comment fix (#1681). + + The Usage block's bare ``/graphify`` comment said "full pipeline on current + directory -> Obsidian vault", contradicting Step 6 (HTML always; Obsidian vault + only when ``--obsidian`` is explicitly given). The comment now describes the + real default. Both the old (removed) and new (added) comment forms match here. + """ + return "# full pipeline on current directory" in line + + # Every line that may differ between a rendered monolith and its pristine v8 # baseline. Each predicate documents one sanctioned change-class; a blank line is # allowed because the multi-line fix blocks insert spacing. Anything else failing @@ -892,6 +903,7 @@ def _is_shebang_allowlist_fix_line(line: str) -> bool: _is_manifest_root_fix_line, _is_no_api_key_fix_line, _is_shebang_allowlist_fix_line, + _is_obsidian_usage_comment_line, ) From 733ad08852267c02a08ffadc82f6de31d09096f1 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Tue, 7 Jul 2026 11:46:51 +0100 Subject: [PATCH 021/292] fix(extract): don't force-parse MATLAB .m through the Objective-C grammar (#1702) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `.m` is shared by Objective-C implementation files and MATLAB, but the extractor dispatch routed every `.m` to extract_objc unconditionally. Feeding real MATLAB to the Objective-C tree-sitter grammar yields root.has_error and garbage nodes/edges — worse than skipping, because it pollutes the graph with wrong data. _get_extractor now content-sniffs `.m` the same way `.h` already is: a genuine Objective-C `.m` carries an ObjC directive (@implementation/@interface/@import/ #import) and still routes to extract_objc; a `.m` without one (MATLAB, Octave) gets no extractor, so it is surfaced by the no-AST-extractor warning (#1689) instead of mis-parsed. `.mm` is unambiguously Objective-C++ and is left untouched. This stops the wrong-by-omission garbage; wiring a real tree-sitter-matlab extractor (the issue's primary ask) remains a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/extract.py | 21 +++++++++++++++++++++ tests/test_extract.py | 31 +++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/graphify/extract.py b/graphify/extract.py index dea5bd0b3..652261ba6 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -16063,6 +16063,20 @@ def _is_objc_header(path: Path) -> bool: ) +def _is_objc_source(path: Path) -> bool: + """Whether a `.m` file is Objective-C rather than MATLAB/Octave (#1702). + + `.m` is shared by Objective-C implementation files and MATLAB (also Octave). + The suffix map routes `.m` to extract_objc unconditionally, which force-parses + MATLAB through the Objective-C tree-sitter grammar and emits garbage nodes/edges + (worse than skipping). A genuine ObjC `.m` always carries an ObjC directive + (@implementation/@interface/@import/#import); MATLAB has none of them. Reuses + the same marker set as the `.h` sniff. `.mm` is unambiguously Objective-C++ and + is not sniffed. + """ + return _is_objc_header(path) + + def _is_cpp_header(path: Path) -> bool: """Whether a `.h` file is C++ rather than plain C (#1547). @@ -16106,6 +16120,13 @@ def _get_extractor(path: Path) -> Any | None: # grammar has no class_specifier). Reroute to extract_cpp (#1547). if _is_cpp_header(path): return extract_cpp + # `.m` is Objective-C OR MATLAB. extract_objc unconditionally would force-parse + # MATLAB through the ObjC grammar into garbage (#1702). Route to extract_objc + # only when the file actually looks like Objective-C; otherwise leave it without + # an extractor (surfaced by the no-AST-extractor warning, #1689) rather than + # mis-parsed. `.mm` is unambiguously Objective-C++ and stays on extract_objc. + if suffix == ".m" and not _is_objc_source(path): + return None # Extensionless files: resolve by shebang, mirroring detect.classify_file. # Without this, detect labels e.g. `#!/usr/bin/env bash` CLIs as code but # extraction returns no extractor and the file silently contributes nothing. diff --git a/tests/test_extract.py b/tests/test_extract.py index 7cae30776..04baf3926 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -1813,3 +1813,34 @@ def test_extract_progress_final_line_uses_consistent_denominator(tmp_path, capsy # final progress line reports the uncached count (100), not the total (105) assert "100/100 uncached files (100%)" in out assert "105/105 files" not in out, "final line must not switch to total_files (#1693)" + + +def test_get_extractor_routes_matlab_m_away_from_objc(tmp_path): + # #1702: .m is shared by Objective-C and MATLAB. A real ObjC .m still routes to + # extract_objc, but a MATLAB .m must NOT be force-parsed by the ObjC grammar + # (which produces garbage) — it gets no extractor instead. + from graphify.extract import _get_extractor, extract_objc + + objc = tmp_path / "Foo.m" + objc.write_text('#import "Foo.h"\n@implementation Foo\n- (void)bar {}\n@end\n') + matlab_fn = tmp_path / "solver.m" + matlab_fn.write_text("function y = solver(x)\n y = x + 1;\nend\n") + matlab_cls = tmp_path / "Model.m" + matlab_cls.write_text("classdef Model\n methods\n function run(obj); end\n end\nend\n") + mm = tmp_path / "x.mm" + mm.write_text("#import \n@implementation X\n@end\n") + + assert _get_extractor(objc) is extract_objc # real ObjC .m -> objc + assert _get_extractor(matlab_fn) is None # MATLAB function -> no garbage + assert _get_extractor(matlab_cls) is None # MATLAB classdef -> no garbage + assert _get_extractor(mm) is extract_objc # .mm is unambiguously ObjC++ + + +def test_matlab_m_not_extracted_as_garbage(tmp_path, capsys): + # End to end: a MATLAB .m produces no (garbage) nodes and is surfaced by the + # no-AST-extractor warning (#1702 + #1689), rather than mis-parsed as ObjC. + m = tmp_path / "controller.m" + m.write_text("function u = controller(x)\n u = -x;\nend\n") + result = extract([m], cache_root=tmp_path) + assert result["nodes"] == [] # no garbage ObjC nodes + assert "no AST extractor" in capsys.readouterr().err # surfaced, not silent From 53efaf89b68190d367feb73f9ef5dba15899377c Mon Sep 17 00:00:00 2001 From: safishamsi Date: Tue, 7 Jul 2026 12:36:16 +0100 Subject: [PATCH 022/292] Release 0.9.9 Fixes since 0.9.8: explain punctuated-label matching (#1704); surface code files with no AST extractor instead of dropping them silently (#1689); consistent AST-extraction progress denominator (#1693); no dangling Obsidian wikilinks in GRAPH_REPORT.md by default (#1712); MATLAB .m no longer force-parsed by the Objective-C grammar (#1702); corrected the /graphify usage comment in the skill files (#1681); surface unclassified files (Dockerfile/Makefile/...) instead of vanishing (#1692). Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 10 ++++++++++ pyproject.toml | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2fc06fbcf..ad4860419 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,16 @@ Full release notes with details on each version: [GitHub Releases](https://github.com/safishamsi/graphify/releases) +## 0.9.9 (2026-07-07) + +- Fix: `graphify explain` resolves an exactly-typed punctuated label symmetrically against `norm_label` (#1704). The search term tokenized on `\w+` ("blockStream.ts" -> "blockstream ts", space where the '.' was) while a node's stored `norm_label` keeps punctuation ("blockstream.ts"). The verbatim case was already rescued by the tokenized-label tier, but that broke if a node's `label` and `norm_label` diverged; a punctuation-preserving `norm_query` is now matched against `norm_label` across the exact/prefix/substring tiers (and fed to the trigram prefilter), so it is robust by construction. +- Fix: code files with no AST extractor are surfaced instead of silently dropped (#1689, thanks for the precise root-cause). `.r`/`.R` (also `.ejs`, `.ets`) are in `CODE_EXTENSIONS` so they are counted as code, but there is no extractor for them, so they produced zero nodes with no warning. `extract` now prints a grouped warning ("N file(s) are classified as code but graphify has no AST extractor ...: .r (17)"). Adding a real `tree-sitter-r` extractor remains a follow-up. +- Fix: the AST-extraction progress line keeps a consistent denominator to the end (#1693). Intermediate lines counted against `len(uncached_work)` but the final line switched to `total_files` (which includes cached hits and no-extractor files), so on a large corpus the count appeared to jump upward right after 99%. Both the parallel and sequential final lines now use the `uncached_work` denominator. +- Fix: `GRAPH_REPORT.md` no longer emits dangling `[[_COMMUNITY_*]]` Obsidian wikilinks by default (#1712). The `_COMMUNITY_*.md` notes those links target are only created by the opt-in `--obsidian` export, and the report is written at build time before any export, so on a default run every link dangled (spawning phantom nodes in a vault's graph view, literal brackets elsewhere). The Community Hubs section now renders as plain text by default; the wikilink form is behind an `obsidian=True` opt-in. +- Fix: `.m` files are no longer force-parsed by the Objective-C grammar when they are MATLAB (#1702, thanks @catalystdream for the diagnosis). `.m` is shared by Objective-C and MATLAB, but the dispatch routed every `.m` to `extract_objc`, which turned real MATLAB into garbage nodes/edges. `.m` is now content-sniffed like `.h`: a genuine Objective-C `.m` (with `@implementation`/`@interface`/`@import`/`#import`) still routes to `extract_objc`; a MATLAB `.m` gets no extractor and is surfaced by the #1689 warning rather than mis-parsed. `.mm` is unchanged (unambiguously Objective-C++). A real `tree-sitter-matlab` extractor remains a follow-up. +- Fix: the `/graphify` usage comment in the skill files no longer claims a bare `/graphify` produces an Obsidian vault by default (#1681, thanks for the audit). It now reads "full pipeline on current directory (HTML viz; add `--obsidian` for a vault)", matching Step 6. Fixed at the skillgen source so every generated `skill-*.md` variant carries the corrected comment. +- Feat: files graphify sees but cannot classify are surfaced instead of vanishing (#1692). Extensionless, non-shebang project files (Dockerfile, Gemfile, Makefile, Rakefile, LICENSE, ...) and unsupported extensions previously left no trace at all. `detect` now collects them into an `unclassified` list, and `graphify extract` reports "N file(s) not classified (no supported extension or shebang), skipped: ...". Actually extracting Dockerfile/Makefile-style content remains a follow-up. + ## 0.9.8 (2026-07-06) - Fix: the Claude Code / Codebuddy `PreToolUse` and Gemini CLI `BeforeTool` graph-nudge hooks now work on Windows (#522). The hooks were inline POSIX bash (`case/esac`, `[ -f ]`, single-quoted `echo`), which Windows cmd.exe/PowerShell cannot parse — so on Windows the hook failed silently, no "run `graphify query` before grepping/reading raw files" context was injected, and users had to invoke `/graphify` by hand. The detection logic (grep-command match, source-file extension match, skip-if-under-output-dir, graph-exists check) moved into a shell-agnostic `graphify hook-guard ` subcommand invoked via the absolute exe path (the same pattern the codex hook already uses), so the hook parses and runs identically on Windows, macOS, and Linux. Behavior on macOS/Linux is unchanged (byte-identical nudge payload); the graph path now also honors `GRAPHIFY_OUT`. The Gemini `BeforeTool` hook got the same treatment (`graphify hook-guard gemini`), which also removes its dependency on a bare `python` being on PATH. Codex stays a no-op there because Codex Desktop rejects `additionalContext`. diff --git a/pyproject.toml b/pyproject.toml index aa154a49b..2f3aa24ee 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "graphifyy" -version = "0.9.8" +version = "0.9.9" description = "AI coding assistant skill (Claude Code, CodeBuddy, Codex, OpenCode, Kilo Code, Cursor, Gemini CLI, Aider, OpenClaw, Factory Droid, Trae, Hermes, Kiro, Pi, Devin CLI, Google Antigravity) - turn any folder of code, docs, papers, images, or videos into a queryable knowledge graph" readme = "README.md" license = { file = "LICENSE" } From cf36d102924ff97db655f1389c7c5b44912b47a4 Mon Sep 17 00:00:00 2001 From: ivanzhilovich Date: Tue, 7 Jul 2026 18:04:53 +0200 Subject: [PATCH 023/292] fix(extract): emit Java enum constants as nodes with case_of edges --- graphify/extract.py | 32 ++++++++++++++++++++++++++++++++ tests/fixtures/sample.java | 11 +++++++++++ tests/test_languages.py | 9 +++++++++ 3 files changed, 52 insertions(+) diff --git a/graphify/extract.py b/graphify/extract.py index 652261ba6..b81245992 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -2933,6 +2933,32 @@ def _swift_extra_walk(node, source: bytes, file_nid: str, stem: str, str_path: s return False +# ── Java extra walk for enum constants ─────────────────────────────────────── + +def _java_extra_walk(node, source: bytes, file_nid: str, stem: str, str_path: str, + nodes: list, edges: list, seen_ids: set, function_bodies: list, + parent_class_nid: str | None, add_node_fn, add_edge_fn, + walk_fn) -> bool: + """Handle enum_constant for Java. Returns True if handled.""" + if node.type == "enum_constant" and parent_class_nid: + name_node = node.child_by_field_name("name") + if name_node is None: + return True + const_name = _read_text(name_node, source) + line = node.start_point[0] + 1 + const_nid = _make_id(parent_class_nid, const_name) + add_node_fn(const_nid, const_name, line) + add_edge_fn(parent_class_nid, const_nid, "case_of", line) + # Anonymous-body constants (`MONDAY { void greet(){} }`): descend so the + # body's methods aren't dropped; const_nid attaches them to the constant. + for child in node.children: + if child.type == "class_body": + for member in child.children: + walk_fn(member, parent_class_nid=const_nid) + return True + return False + + # ── Language configs ────────────────────────────────────────────────────────── _PYTHON_CONFIG = LanguageConfig( @@ -4855,6 +4881,12 @@ def _emit_java_parent_type(type_node, rel: str, at_line: int) -> None: ensure_named_node): return + if config.ts_module == "tree_sitter_java": + if _java_extra_walk(node, source, file_nid, stem, str_path, + nodes, edges, seen_ids, function_bodies, + parent_class_nid, add_node, add_edge, walk): + return + if config.ts_module == "tree_sitter_ruby": if _ruby_extra_walk(node, source, file_nid, stem, str_path, nodes, edges, seen_ids, function_bodies, diff --git a/tests/fixtures/sample.java b/tests/fixtures/sample.java index 826cfaae1..792fdf945 100644 --- a/tests/fixtures/sample.java +++ b/tests/fixtures/sample.java @@ -39,3 +39,14 @@ private List validate(List data) { interface Processor { List process(); } + +enum ErrorCode { + OK(0), + GAME_DONE(1001); + + private final int code; + + ErrorCode(int code) { + this.code = code; + } +} diff --git a/tests/test_languages.py b/tests/test_languages.py index 9aa64b526..565bf0a29 100644 --- a/tests/test_languages.py +++ b/tests/test_languages.py @@ -115,6 +115,15 @@ def test_java_no_dangling_edges(): assert e["source"] in node_ids +def test_java_enum_constants_have_case_of_edge(): + r = extract_java(FIXTURES / "sample.java") + labels = _labels(r) + assert "OK" in labels + assert "GAME_DONE" in labels + assert ("ErrorCode", "OK") in _edge_labels(r, "case_of") + assert ("ErrorCode", "GAME_DONE") in _edge_labels(r, "case_of") + + # ── C ──────────────────────────────────────────────────────────────────────── def test_c_no_error(): From 0ce8ea00e7bbc01672bbc2eb57407e6818486337 Mon Sep 17 00:00:00 2001 From: Thizeidler Date: Tue, 7 Jul 2026 09:41:26 -0300 Subject: [PATCH 024/292] docs(security): clarify stdio-only claim now that HTTP transport exists SECURITY.md said graphify never runs a network listener and only communicates over stdio. README.md documents an opt-in --transport http mode (with --host/--api-key flags) for sharing one server across a team, which does open a network listener. Update the claim to describe the default (stdio, no listener) and the documented opt-in exception. --- SECURITY.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SECURITY.md b/SECURITY.md index 297b7d8d3..795454b21 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -42,7 +42,7 @@ graphify is a **local development tool**. It runs as a Claude Code skill and opt ### What graphify does NOT do -- Does not run a network listener (MCP server communicates over stdio only) +- Does not run a network listener by default (stdio transport); `--transport http` is opt-in, documented in the README, and binds to `127.0.0.1` unless `--host 0.0.0.0` is passed - Does not execute code from source files (tree-sitter parses ASTs - no eval/exec) - Does not use `shell=True` in any subprocess call - Does not store credentials or API keys From 6c33e3aad73c6d37e789489c3dea691810247982 Mon Sep 17 00:00:00 2001 From: kaanrumin Date: Wed, 8 Jul 2026 00:39:07 +0300 Subject: [PATCH 025/292] test(security): cover GRAPHIFY_MAX_GRAPH_BYTES parsing; clarify unit docstring MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit _max_graph_file_bytes() resolves the graph-load memory-bomb cap and parses the GRAPHIFY_MAX_GRAPH_BYTES env var (plain bytes, MB/GB suffix, fallbacks), but had no behavioral tests — only the default constant was asserted. - Add tests for: unset/blank -> default, plain integer, MB/GB suffix (binary), case-insensitive and space-tolerant suffixes, unparseable -> default, and non-positive -> default. - Fix the docstring, which described the suffix as "decimal multipliers of 1024" (self-contradictory). MB/GB are treated as binary (MiB/GiB); the tests lock that behavior in. No behavior change. --- graphify/security.py | 3 ++- tests/test_security.py | 52 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+), 1 deletion(-) diff --git a/graphify/security.py b/graphify/security.py index d371e53c2..2dbe5bd77 100644 --- a/graphify/security.py +++ b/graphify/security.py @@ -38,7 +38,8 @@ def _max_graph_file_bytes() -> int: Honors the ``GRAPHIFY_MAX_GRAPH_BYTES`` environment variable so users with large codebases can raise the limit without editing source. The value may be plain bytes (``671088640``) or carry an ``MB`` / ``GB`` suffix - (``640MB``, ``2GB`` — case-insensitive, decimal multipliers of 1024). + (``640MB``, ``2GB`` — case-insensitive, binary multipliers: ``MB`` is + 1024*1024 and ``GB`` is 1024*1024*1024, i.e. MiB / GiB). Falls back to ``_MAX_GRAPH_FILE_BYTES`` (512 MiB) when the env var is unset, blank, or unparseable. diff --git a/tests/test_security.py b/tests/test_security.py index d2e08c0d0..2f69be6b4 100644 --- a/tests/test_security.py +++ b/tests/test_security.py @@ -20,6 +20,7 @@ _MAX_FETCH_BYTES, _MAX_GRAPH_FILE_BYTES, _MAX_TEXT_BYTES, + _max_graph_file_bytes, _METADATA_MAX_LIST_ITEMS, _METADATA_MAX_VALUE_LEN, _sanitize_metadata_string, @@ -227,6 +228,57 @@ def test_graph_size_cap_default_is_512_mib(): assert _MAX_GRAPH_FILE_BYTES == 512 * 1024 * 1024 +# --------------------------------------------------------------------------- +# _max_graph_file_bytes — GRAPHIFY_MAX_GRAPH_BYTES env-var parsing +# --------------------------------------------------------------------------- + +def test_max_graph_bytes_default_when_unset(monkeypatch): + monkeypatch.delenv("GRAPHIFY_MAX_GRAPH_BYTES", raising=False) + assert _max_graph_file_bytes() == _MAX_GRAPH_FILE_BYTES + + +def test_max_graph_bytes_default_when_blank(monkeypatch): + monkeypatch.setenv("GRAPHIFY_MAX_GRAPH_BYTES", " ") + assert _max_graph_file_bytes() == _MAX_GRAPH_FILE_BYTES + + +def test_max_graph_bytes_plain_integer(monkeypatch): + monkeypatch.setenv("GRAPHIFY_MAX_GRAPH_BYTES", "671088640") + assert _max_graph_file_bytes() == 671088640 + + +def test_max_graph_bytes_mb_suffix_is_binary(monkeypatch): + monkeypatch.setenv("GRAPHIFY_MAX_GRAPH_BYTES", "640MB") + assert _max_graph_file_bytes() == 640 * 1024 * 1024 + + +def test_max_graph_bytes_gb_suffix_is_binary(monkeypatch): + monkeypatch.setenv("GRAPHIFY_MAX_GRAPH_BYTES", "2GB") + assert _max_graph_file_bytes() == 2 * 1024 * 1024 * 1024 + + +def test_max_graph_bytes_suffix_is_case_insensitive(monkeypatch): + monkeypatch.setenv("GRAPHIFY_MAX_GRAPH_BYTES", "3gb") + assert _max_graph_file_bytes() == 3 * 1024 * 1024 * 1024 + + +def test_max_graph_bytes_tolerates_space_before_suffix(monkeypatch): + monkeypatch.setenv("GRAPHIFY_MAX_GRAPH_BYTES", "5 GB") + assert _max_graph_file_bytes() == 5 * 1024 * 1024 * 1024 + + +@pytest.mark.parametrize("bad", ["not-a-number", "1.5GB", "0x10", "640KB"]) +def test_max_graph_bytes_unparseable_falls_back(monkeypatch, bad): + monkeypatch.setenv("GRAPHIFY_MAX_GRAPH_BYTES", bad) + assert _max_graph_file_bytes() == _MAX_GRAPH_FILE_BYTES + + +@pytest.mark.parametrize("nonpositive", ["0", "-1", "-4GB"]) +def test_max_graph_bytes_nonpositive_falls_back(monkeypatch, nonpositive): + monkeypatch.setenv("GRAPHIFY_MAX_GRAPH_BYTES", nonpositive) + assert _max_graph_file_bytes() == _MAX_GRAPH_FILE_BYTES + + def test_graph_size_cap_under_limit_returns_none(tmp_path): p = tmp_path / "graph.json" p.write_text('{"nodes": [], "links": []}', encoding="utf-8") From e5044c32532e49edbbf14bea1daeb4f0a418aa72 Mon Sep 17 00:00:00 2001 From: Francisco Javier Serna Barragan <148145893+FranciscoJSBarragan@users.noreply.github.com> Date: Mon, 6 Jul 2026 09:11:28 -0600 Subject: [PATCH 026/292] Fix rebuilds from deleted hook cwd What changed - Stabilize relative rebuild execution before graphify-out queue/lock setup. - Recover from deleted transient hook CWDs when GRAPHIFY_REPO_ROOT points at the repository root. - Fail cleanly when the current working directory is gone and no repo root fallback is available. - Add regression coverage for both fallback and clean-failure paths. Why - Detached post-commit/post-checkout rebuilds can inherit a transient CWD that is deleted before the background process starts. - _rebuild_code previously used relative graphify-out paths before Path.cwd()/watch_path resolution could be handled, causing FileNotFoundError: [Errno 2] No such file or directory. Validation - .venv/bin/pytest tests/test_watch.py -q: 54 passed, 2 skipped. - .venv/bin/ruff check graphify/watch.py tests/test_watch.py: passed. - .venv/bin/python -m py_compile graphify/watch.py tests/test_watch.py: passed. - env -u PYTHONPATH -u PYTHONHOME PYTHONHASHSEED=0 .venv/bin/pytest -q: 2904 passed, 30 skipped. Notes - Full suite without PYTHONHASHSEED=0 exposed an unrelated ordering-sensitive labeling test; with the deterministic hash seed used by graphify hooks, the suite passes. --- graphify/watch.py | 34 ++++++++++++++++++++++++++++++ tests/test_watch.py | 50 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+) diff --git a/graphify/watch.py b/graphify/watch.py index ba3f09b17..94d4e8b20 100644 --- a/graphify/watch.py +++ b/graphify/watch.py @@ -657,6 +657,37 @@ def _json_text(data: dict) -> str: return json.dumps(data, indent=2, ensure_ascii=False) + "\n" +def _stabilize_rebuild_cwd(watch_path: Path) -> bool: + """Ensure relative rebuild paths have a usable CWD before queue/lock setup. + + Detached git hooks can inherit a transient working directory that is deleted + before the background rebuild starts. In that state Path.cwd(), + Path('.').resolve(), and relative graphify-out mkdirs raise FileNotFoundError + before the normal rebuild error handling can run. Hooks that know the repo + root export GRAPHIFY_REPO_ROOT so the rebuild can recover by chdir'ing there. + """ + if watch_path.is_absolute(): + return True + + repo_root = os.environ.get("GRAPHIFY_REPO_ROOT", "").strip() + if repo_root and Path(repo_root).is_dir(): + try: + os.chdir(repo_root) + return True + except OSError: + pass + + try: + Path.cwd() + return True + except FileNotFoundError: + print( + "[graphify watch] Rebuild failed: current working directory " + "no longer exists and GRAPHIFY_REPO_ROOT is not set." + ) + return False + + def _rebuild_code( watch_path: Path, *, @@ -690,6 +721,9 @@ def _rebuild_code( Returns True on success, False on error or skipped-due-to-lock. """ + if not _stabilize_rebuild_cwd(watch_path): + return False + out = watch_path / _GRAPHIFY_OUT if acquire_lock: # #1059: incremental (changed_paths is not None) hooks must not drop diff --git a/tests/test_watch.py b/tests/test_watch.py index fb9868677..f0a54690f 100644 --- a/tests/test_watch.py +++ b/tests/test_watch.py @@ -176,6 +176,56 @@ def test_graphify_root_preserves_absolute_when_user_supplied(tmp_path): ) +def test_rebuild_code_deleted_cwd_without_repo_root_returns_false(tmp_path, monkeypatch, capsys): + """Detached hooks can inherit a CWD that no longer exists. + + Without GRAPHIFY_REPO_ROOT, the rebuild should fail cleanly before creating + relative graphify-out queue/lock files. + """ + from graphify.watch import _rebuild_code + + old_cwd = Path.cwd() + gone = tmp_path / "gone" + gone.mkdir() + monkeypatch.delenv("GRAPHIFY_REPO_ROOT", raising=False) + + os.chdir(gone) + gone.rmdir() + try: + assert _rebuild_code(Path("."), changed_paths=[Path("lib.py")]) is False + finally: + os.chdir(old_cwd) + + out = capsys.readouterr().out + assert "current working directory no longer exists" in out + + +def test_rebuild_code_deleted_cwd_uses_graphify_repo_root(tmp_path, monkeypatch): + """GRAPHIFY_REPO_ROOT lets detached hook rebuilds recover from a deleted CWD.""" + from graphify.watch import _rebuild_code + + old_cwd = Path.cwd() + corpus = tmp_path / "corpus" + corpus.mkdir() + (corpus / "lib.py").write_text("def f(): pass\n", encoding="utf-8") + gone = tmp_path / "gone" + gone.mkdir() + monkeypatch.setenv("GRAPHIFY_REPO_ROOT", str(corpus)) + + os.chdir(gone) + gone.rmdir() + try: + assert _rebuild_code( + Path("."), + changed_paths=[Path("lib.py")], + no_cluster=True, + ) is True + assert Path.cwd().resolve() == corpus.resolve() + assert (corpus / "graphify-out" / "graph.json").exists() + finally: + os.chdir(old_cwd) + + def test_rebuild_code_evicts_nodes_from_deleted_files(tmp_path): """#1007: graphify update (_rebuild_code with no changed_paths) must remove nodes and edges from files deleted since the last run.""" From 2ea37734c0f9adff317bb62cfadd942fb82024e5 Mon Sep 17 00:00:00 2001 From: "A.Levin" Date: Tue, 7 Jul 2026 16:30:16 +0300 Subject: [PATCH 027/292] Checkpoint semantic cache per chunk so interrupted runs resume MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Semantic extraction only wrote to the cache once, at the very end of the run (save_semantic_cache in __main__ after extract_corpus_parallel returns). A run interrupted partway — a crash, a kill, or a claude-cli/API run that exits when it hits a rate limit — therefore lost every completed chunk and restarted from scratch. On a large corpus with a slow local backend this can throw away many hours of work. Persist each chunk's results to the semantic cache as soon as it completes, in both the serial and threaded paths of extract_corpus_parallel. Add a merge_existing option to save_semantic_cache so a file split into slices across several chunks accumulates its slices instead of the later chunk overwriting the earlier one. The checkpoint is best-effort (a cache write error never aborts extraction) and can be disabled with GRAPHIFY_NO_INCREMENTAL_CACHE. Default behaviour of save_semantic_cache is unchanged (merge_existing defaults to False). --- graphify/cache.py | 14 ++++++++++++++ graphify/llm.py | 23 +++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/graphify/cache.py b/graphify/cache.py index bb3f9d593..35dc7cc11 100644 --- a/graphify/cache.py +++ b/graphify/cache.py @@ -533,12 +533,18 @@ def save_semantic_cache( edges: list[dict], hyperedges: list[dict] | None = None, root: Path = Path("."), + merge_existing: bool = False, ) -> int: """Save semantic extraction results to cache, keyed by source_file. Groups nodes and edges by source_file, then saves one cache entry per file under cache/semantic/ (separate from AST entries in cache/ast/) to prevent hash-key collisions (#582). + + When ``merge_existing`` is True, any already-cached entry for a file is + unioned with the new results before saving instead of being overwritten. + This lets callers checkpoint incrementally (e.g. once per chunk) without + dropping a prior slice of a large file that was split across chunks. Returns the number of files cached. """ from collections import defaultdict @@ -563,6 +569,14 @@ def save_semantic_cache( if not p.is_absolute(): p = Path(root) / p if p.is_file(): + if merge_existing: + prev = load_cached(p, root, kind="semantic") + if prev: + result = { + "nodes": (prev.get("nodes", []) or []) + result["nodes"], + "edges": (prev.get("edges", []) or []) + result["edges"], + "hyperedges": (prev.get("hyperedges", []) or []) + result["hyperedges"], + } save_cached(p, result, root, kind="semantic") saved += 1 return saved diff --git a/graphify/llm.py b/graphify/llm.py index f00ed0c84..560bbcb88 100644 --- a/graphify/llm.py +++ b/graphify/llm.py @@ -1903,6 +1903,27 @@ def _run_one(idx: int, chunk: list[Path]) -> tuple[int, dict | None, Exception | # over session state. Force serial unless the user explicitly opts in. if backend == "claude-cli" and os.environ.get("GRAPHIFY_CLAUDE_CLI_PARALLEL", "").strip() != "1": max_concurrency = 1 + def _checkpoint_chunk(result: dict) -> None: + # Persist each chunk's semantic results to the cache as soon as it + # completes. Without this, the semantic cache is only written once, at + # the very end of the run (in __main__), so a run interrupted partway + # — a crash, a kill, or a claude-cli/API run that exits on a rate + # limit — loses every completed chunk and restarts from scratch. This + # is best-effort: a cache write failure must never abort extraction. + if os.environ.get("GRAPHIFY_NO_INCREMENTAL_CACHE"): + return + try: + from .cache import save_semantic_cache as _scs + _scs( + result.get("nodes", []), + result.get("edges", []), + result.get("hyperedges", []), + root=root, + merge_existing=True, + ) + except Exception as _exc: # noqa: BLE001 — checkpoint is best-effort + print(f"[graphify] incremental cache checkpoint failed: {_exc}", file=sys.stderr) + workers = max(1, min(max_concurrency, total)) if workers == 1: # Avoid thread pool overhead for single-worker runs (and keep @@ -1915,6 +1936,7 @@ def _run_one(idx: int, chunk: list[Path]) -> tuple[int, dict | None, Exception | continue assert result is not None _merge_into(merged, result) + _checkpoint_chunk(result) if callable(on_chunk_done): on_chunk_done(idx, total, result) else: @@ -1939,6 +1961,7 @@ def _run_one(idx: int, chunk: list[Path]) -> tuple[int, dict | None, Exception | continue assert result is not None results_by_idx[idx] = result + _checkpoint_chunk(result) if callable(on_chunk_done): on_chunk_done(idx, total, result) for idx in sorted(results_by_idx): From 27b523e76981aa70f89400a300e39d8a0a721caf Mon Sep 17 00:00:00 2001 From: safishamsi Date: Wed, 8 Jul 2026 01:14:12 +0100 Subject: [PATCH 028/292] test(cache): cover save_semantic_cache merge_existing union + default overwrite (#1715) The per-chunk checkpoint feature added merge_existing without test coverage. Add tests asserting merge_existing=True unions a file's slices across chunks and the default still overwrites (the authoritative final write in extract). Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/test_cache.py | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/tests/test_cache.py b/tests/test_cache.py index 730265be4..9bf2cdc82 100644 --- a/tests/test_cache.py +++ b/tests/test_cache.py @@ -510,3 +510,33 @@ def test_semantic_prune_ignores_ast_and_tmp(tmp_path): assert not (semantic_dir / "deadbeef.json").exists() assert tmp_entry.exists(), "*.tmp temporaries must not be swept" assert len(list(ast_dir.glob("*.json"))) == 1, "AST entries must not be touched" + + +def test_save_semantic_cache_overwrites_by_default(tmp_path): + """Default save_semantic_cache replaces a file's cached entry (the final, + authoritative write in the extract pipeline).""" + from graphify.cache import save_semantic_cache + f = tmp_path / "doc.md"; f.write_text("# Doc\n") + save_semantic_cache([{"id": "a", "source_file": "doc.md"}], [], root=tmp_path) + save_semantic_cache([{"id": "b", "source_file": "doc.md"}], [], root=tmp_path) + cached = load_cached(f, root=tmp_path, kind="semantic") + ids = {n["id"] for n in cached["nodes"]} + assert ids == {"b"}, "default must overwrite, not accumulate" + + +def test_save_semantic_cache_merge_existing_unions(tmp_path): + """#1715: merge_existing=True unions with the prior entry so a file split + across chunks (checkpointed per chunk) keeps every slice.""" + from graphify.cache import save_semantic_cache + f = tmp_path / "big.md"; f.write_text("# Big\n") + # chunk 1 slice + save_semantic_cache([{"id": "a", "source_file": "big.md"}], + [{"source": "a", "target": "x", "source_file": "big.md"}], + root=tmp_path, merge_existing=True) + # chunk 2 slice for the same file + save_semantic_cache([{"id": "b", "source_file": "big.md"}], [], + root=tmp_path, merge_existing=True) + cached = load_cached(f, root=tmp_path, kind="semantic") + ids = {n["id"] for n in cached["nodes"]} + assert ids == {"a", "b"}, "merge_existing must union both chunk slices" + assert len(cached["edges"]) == 1 From bf7fa50f38180454b37e578f30629dcd03bdf628 Mon Sep 17 00:00:00 2001 From: edinaldoof Date: Tue, 7 Jul 2026 12:48:35 -0300 Subject: [PATCH 029/292] fix(extract): never bind a cross-file call to a definition in another language family MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The cross-file call resolver matches raw-call callees against a repo-wide label index with no language check. In a repo that mixes a web app with a native Android app, a TSX callback passed by name (register(refreshHeading)) resolved to a same-named Kotlin method and shipped as an INFERRED indirect_call edge — a phantom the extraction spec itself forbids ('calls edges MUST stay within one language'). Direct calls from non-JS/TS callers had the same hole with no gate at all: a bare Python call bound to the lone same-named Kotlin fun. Found on a production Next.js + Android codebase, where the phantom edge also inflated the Kotlin node's betweenness enough to surface it as a top suggested question in GRAPH_REPORT.md. Resolution candidates are now filtered by language interop family before the single-candidate/import-evidence logic runs. Families are grouped by REAL interop so legitimate cross-language resolution keeps working: Kotlin/Java/Scala/Groovy share the JVM, C/C++/Objective-C/CUDA share headers (Swift bridges to ObjC), and JS/TS variants plus Vue/Svelte/Astro SFCs compile into one module graph. Candidates whose family is unknown (no source_file, non-code nodes) are never filtered, preserving the previous permissive behavior, and callers with an unmapped extension skip the guard entirely. --- graphify/extract.py | 63 ++++++++++++ tests/test_cross_language_call_resolution.py | 103 +++++++++++++++++++ 2 files changed, 166 insertions(+) create mode 100644 tests/test_cross_language_call_resolution.py diff --git a/graphify/extract.py b/graphify/extract.py index b81245992..40b9c8b26 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -9860,6 +9860,52 @@ def _lang_is_case_insensitive(source_file: object) -> bool: return Path(str(source_file)).suffix.lower() in _CASE_INSENSITIVE_EXTS +# Language interop families for cross-file call resolution. A call in one language +# can never bind by name to a definition in another family — a TSX component does +# not invoke a Kotlin method, and a Python function does not invoke a Java one. +# Families are grouped by REAL interop so legitimate cross-language resolution +# keeps working: Kotlin/Java/Scala/Groovy share the JVM, C/C++/Objective-C/CUDA +# share headers and symbols (Swift bridges to Objective-C), and JS/TS variants +# (plus Vue/Svelte/Astro SFC script blocks) compile into one module graph. +# Extensions absent from this map (docs, configs, unknown languages) resolve to +# no family and are never filtered — same permissive default as before. +_LANG_FAMILY_BY_EXT: dict[str, str] = { + # JS/TS module graph (SFCs embed JS/TS) + ".js": "jsts", ".jsx": "jsts", ".mjs": "jsts", ".cjs": "jsts", + ".ts": "jsts", ".tsx": "jsts", ".mts": "jsts", ".cts": "jsts", + ".vue": "jsts", ".svelte": "jsts", ".astro": "jsts", + # JVM interop + ".java": "jvm", ".kt": "jvm", ".kts": "jvm", + ".scala": "jvm", ".groovy": "jvm", ".gradle": "jvm", + # C-family: shared headers, Objective-C/C++ mix, Swift↔ObjC bridging + ".c": "native", ".h": "native", ".cpp": "native", ".cc": "native", + ".cxx": "native", ".hpp": "native", ".cu": "native", ".cuh": "native", + ".metal": "native", ".m": "native", ".mm": "native", ".swift": "native", + # Single-language families + ".py": "python", + ".go": "go", + ".rs": "rust", + ".rb": "ruby", + ".php": "php", ".phtml": "php", ".php3": "php", ".php4": "php", + ".php5": "php", ".php7": "php", ".phps": "php", + ".cs": "dotnet", ".razor": "dotnet", ".cshtml": "dotnet", ".xaml": "dotnet", + ".lua": "lua", ".luau": "lua", + ".zig": "zig", + ".ex": "elixir", ".exs": "elixir", + ".jl": "julia", + ".dart": "dart", + ".sh": "shell", ".bash": "shell", + ".ps1": "powershell", ".psm1": "powershell", ".psd1": "powershell", +} + + +def _lang_family(source_file: object) -> str | None: + """Interop family of the file's language, or None when unknown/not code.""" + if not source_file: + return None + return _LANG_FAMILY_BY_EXT.get(Path(str(source_file)).suffix.lower()) + + def _node_label_key(node: dict, fold: bool = False) -> str: label = str(node.get("label", "")).strip() key = re.sub(r"[^a-zA-Z0-9]+", "", label) @@ -16780,6 +16826,23 @@ def extract( candidates = global_label_to_nids_ci.get(callee.lower(), []) if not candidates: continue + # Cross-language guard: never bind a call to a definition in a different + # language family. Name-only matching was resolving a TSX callback passed + # by name to a same-named Kotlin method in the Android half of the repo + # (and a Python call to a Kotlin fun) — phantom edges the extraction spec + # explicitly forbids. Candidates whose family is unknown (no source_file, + # non-code nodes) are kept, preserving the previous permissive behavior; + # real interop pairs (Kotlin↔Java, C↔C++↔ObjC, JS↔TS) share a family and + # still resolve. + caller_family = _lang_family(rc.get("source_file")) + if caller_family is not None: + candidates = [ + c for c in candidates + if (candidate_family := _lang_family(nid_to_source_file.get(c))) is None + or candidate_family == caller_family + ] + if not candidates: + continue caller = rc["caller_nid"] # Resolve the caller's file via the raw_call's own source_file string, # which is stable regardless of any caller_nid remap. An indirect diff --git a/tests/test_cross_language_call_resolution.py b/tests/test_cross_language_call_resolution.py new file mode 100644 index 000000000..62fe4c60f --- /dev/null +++ b/tests/test_cross_language_call_resolution.py @@ -0,0 +1,103 @@ +"""Cross-language call resolution — a call in one language must never bind by +name to a definition in another language family. + +The cross-file resolver matched raw-call callees against a repo-wide label +index with no language check, so in a repo that mixes a web app with a native +Android app a TSX callback passed by name (``register(refreshHeading)``) +resolved to a same-named Kotlin method and shipped as an INFERRED +``indirect_call`` edge — a phantom the extraction spec explicitly forbids +("calls edges MUST stay within one language"). Direct calls from non-JS/TS +languages had the same hole: a Python call bound to a Kotlin ``fun``. + +The fix filters resolution candidates by language interop family. Families are +grouped by REAL interop so legitimate cross-language resolution keeps working: +Kotlin/Java share the JVM, C/C++/Objective-C share headers, JS/TS variants +compile into one module graph. Candidates with no known family (non-code +nodes) are never filtered, preserving the previous permissive behavior. +""" +from __future__ import annotations + +from pathlib import Path + +from graphify.extract import extract + + +def _write(path: Path, text: str) -> Path: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text, encoding="utf-8") + return path + + +def _call_edges(files: list[Path], base: Path) -> set[tuple[str, str, str, str]]: + r = extract(files, cache_root=base, parallel=False) + lbl = {n["id"]: n["label"] for n in r["nodes"]} + return { + (lbl.get(e["source"], ""), lbl.get(e["target"], ""), e["relation"], e.get("confidence")) + for e in r["edges"] if e["relation"] in ("calls", "indirect_call") + } + + +def test_tsx_callback_does_not_bind_to_kotlin_method(tmp_path: Path) -> None: + # The real-world symptom: a repo with a web app and a native Android app. + # A TSX component passes a callback by name; the only same-named definition + # repo-wide is a Kotlin method. No edge must be emitted. + _write(tmp_path / "web/Upcoming.tsx", + "declare function register(cb: () => void): void;\n" + "export function UpcomingPanel() {\n" + " register(refreshHeading);\n" + " return null;\n" + "}\n") + _write(tmp_path / "android/HeadingSensorBridge.kt", + "class HeadingSensorBridge {\n" + " fun refreshHeading() {\n" + " println(\"native sensor\")\n" + " }\n" + "}\n") + edges = _call_edges(sorted(tmp_path.rglob("*.tsx")) + sorted(tmp_path.rglob("*.kt")), tmp_path) + assert not any("refreshHeading" in t for _s, t, _r, _c in edges), edges + + +def test_python_call_does_not_bind_to_kotlin_function(tmp_path: Path) -> None: + # Direct-call path (non-JS/TS callers have no import-evidence gate): a bare + # Python call must not resolve to the lone same-named Kotlin definition. + _write(tmp_path / "py/worker.py", + "def process():\n" + " return refreshHeading()\n") + _write(tmp_path / "android/HeadingSensorBridge.kt", + "class HeadingSensorBridge {\n" + " fun refreshHeading() {\n" + " println(\"native sensor\")\n" + " }\n" + "}\n") + edges = _call_edges(sorted(tmp_path.rglob("*.py")) + sorted(tmp_path.rglob("*.kt")), tmp_path) + assert not any("refreshHeading" in t for _s, t, _r, _c in edges), edges + + +def test_same_language_callback_still_resolves(tmp_path: Path) -> None: + # Positive control: a TS callback passed by name with a same-language + # definition and import evidence keeps resolving as INFERRED indirect_call. + _write(tmp_path / "a.ts", + 'import { refreshHeading } from "./b";\n' + "declare function register(cb: () => void): void;\n" + "export function run() { register(refreshHeading); }\n") + _write(tmp_path / "b.ts", + "export function refreshHeading(): void {}\n") + edges = _call_edges([tmp_path / "a.ts", tmp_path / "b.ts"], tmp_path) + resolved = [e for e in edges if "refreshHeading" in e[1] and e[2] == "indirect_call"] + assert resolved, edges + assert resolved[0][3] == "INFERRED" + + +def test_jvm_interop_kotlin_call_to_java_still_resolves(tmp_path: Path) -> None: + # Kotlin and Java share the JVM — same interop family, so a Kotlin call to a + # Java method must keep resolving exactly as it did before the guard. + _write(tmp_path / "Alarm.java", + "public class Alarm {\n" + " public static void ring() { System.out.println(\"ring\"); }\n" + "}\n") + _write(tmp_path / "Scheduler.kt", + "fun schedule() {\n" + " ring()\n" + "}\n") + edges = _call_edges([tmp_path / "Alarm.java", tmp_path / "Scheduler.kt"], tmp_path) + assert any("ring" in t for _s, t, _r, _c in edges), edges From 9557bf67338d993c96b9e885e4508bbe9baee401 Mon Sep 17 00:00:00 2001 From: mallyskies Date: Mon, 6 Jul 2026 19:54:22 -0600 Subject: [PATCH 030/292] fix(extract): tag inline base-class stubs with origin_file ensure_named_node() tags the sourceless stub it creates for an unresolved reference with origin_file, so _disambiguate_colliding_node_ids can tell one file's unresolved reference apart from another's instead of merging every file's same-named reference onto one shared bare id (which can then collide with an unrelated same-named real definition anywhere else in the corpus, since ids are case-normalized and global). Five call sites duplicated that stub-creation logic inline instead of calling ensure_named_node() -- Ruby's `Class.new(Super)` and `class Foo < Base` inheritance, Python inheritance, Kotlin delegation -specifier inheritance/conformance, and C++ base_class_clause inheritance -- and none of them were updated when origin_file was added, so all five still produce the un-disambiguated bare-id stub the fix was meant to eliminate. Four of the five sit directly inside _extract_generic, where ensure_named_node() is already in scope as a closure, so they're switched to call it directly. The fifth (Ruby's `Class.new(Super)`) is handled by a separate helper, _ruby_extra_walk(), which doesn't have that closure in scope; that one gets the same origin_file tag added directly to its own inline stub dict, matching what ensure_named_node() already does, without changing the helper's signature. (A sixth occurrence of the same inline pattern exists in extract_apex(), a fully separate regex-based extractor with no ensure_named_node() equivalent of its own -- left out of this fix, which is scoped to the shared _extract_generic path and its one directly-affiliated helper.) Added a regression test: two different C++ files each inheriting from the same undefined base class must produce two distinct stub nodes, not one shared one. Fails on main (one shared 'base' id for both files), passes with this fix. --- graphify/extract.py | 60 ++++++++----------------------------------- tests/test_extract.py | 31 ++++++++++++++++++++++ 2 files changed, 42 insertions(+), 49 deletions(-) diff --git a/graphify/extract.py b/graphify/extract.py index 40b9c8b26..ed0550432 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -3450,10 +3450,16 @@ def _ruby_extra_walk(node, source: bytes, file_nid: str, stem: str, str_path: st if base_nid not in seen_ids: base_nid = _make_id(base) if base_nid not in seen_ids: + # origin_file lets _disambiguate_colliding_node_ids + # tell this file's unresolved reference apart from + # another file's same-named one, instead of every + # file's stub collapsing onto one shared bare id + # (see ensure_named_node(), which sets the same + # field for this exact reason). nodes.append({ "id": base_nid, "label": base, "file_type": "code", "source_file": "", - "source_location": "", + "source_location": "", "origin_file": str_path, }) seen_ids.add(base_nid) add_edge(class_nid, base_nid, "inherits", line) @@ -3707,18 +3713,7 @@ def walk(node, parent_class_nid: str | None = None) -> None: for arg in args.children: if arg.type == "identifier": base = _read_text(arg, source) - base_nid = _make_id(stem, base) - if base_nid not in seen_ids: - base_nid = _make_id(base) - if base_nid not in seen_ids: - nodes.append({ - "id": base_nid, - "label": base, - "file_type": "code", - "source_file": "", - "source_location": "", - }) - seen_ids.add(base_nid) + base_nid = ensure_named_node(base, line) add_edge(class_nid, base_nid, "inherits", line) # Swift-specific: conformance / inheritance @@ -3857,18 +3852,7 @@ def _php_emit_base(base_name: str, rel: str, at_line: int) -> None: base = _kotlin_user_type_name(user_type_node, source) if not base: continue - base_nid = _make_id(stem, base) - if base_nid not in seen_ids: - base_nid = _make_id(base) - if base_nid not in seen_ids: - nodes.append({ - "id": base_nid, - "label": base, - "file_type": "code", - "source_file": "", - "source_location": "", - }) - seen_ids.add(base_nid) + base_nid = ensure_named_node(base, line) add_edge(class_nid, base_nid, relation, line) for arg_child in user_type_node.children: if arg_child.type != "type_arguments": @@ -3902,18 +3886,7 @@ def _php_emit_base(base_name: str, rel: str, at_line: int) -> None: base = _read_text(consts[-1], source) break if base: - base_nid = _make_id(stem, base) - if base_nid not in seen_ids: - base_nid = _make_id(base) - if base_nid not in seen_ids: - nodes.append({ - "id": base_nid, - "label": base, - "file_type": "code", - "source_file": "", - "source_location": "", - }) - seen_ids.add(base_nid) + base_nid = ensure_named_node(base, line) add_edge(class_nid, base_nid, "inherits", line) # `include`/`extend`/`prepend ` in the class/module body -> @@ -4179,18 +4152,7 @@ def _emit_java_parent_type(type_node, rel: str, at_line: int) -> None: continue if not base: continue - base_nid = _make_id(stem, base) - if base_nid not in seen_ids: - base_nid = _make_id(base) - if base_nid not in seen_ids: - nodes.append({ - "id": base_nid, - "label": base, - "file_type": "code", - "source_file": "", - "source_location": "", - }) - seen_ids.add(base_nid) + base_nid = ensure_named_node(base, line) add_edge(class_nid, base_nid, "inherits", line) # Emit a generic_arg reference for each type argument on the # base (Base -> Car references Dep). _cpp_collect_type_refs diff --git a/tests/test_extract.py b/tests/test_extract.py index 04baf3926..9254a25f1 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -100,6 +100,37 @@ def test_extract_disambiguates_duplicate_symbol_ids_by_source_path(tmp_path): assert edge["target"] in node_ids, f"Dangling structural target: {edge}" +def test_cpp_unresolved_base_class_stubs_stay_disambiguated_by_file(tmp_path): + """Two different files' same-named, otherwise-undefined base class must not + collapse onto one shared stub node. + + The C++ base_class_clause handler used to build its stub inline instead of + calling ensure_named_node(), so it never tagged the stub with origin_file. + Without that tag, _disambiguate_colliding_node_ids couldn't tell file A's + reference to unresolved `Base` apart from file B's, and every file's + unresolved base class merged onto one bare id -- which could then collide + with an unrelated same-named real definition anywhere else in the corpus. + """ + first = tmp_path / "a" / "Foo.cpp" + second = tmp_path / "b" / "Bar.cpp" + first.parent.mkdir(parents=True) + second.parent.mkdir(parents=True) + first.write_text("class Foo : public Base {};\n", encoding="utf-8") + second.write_text("class Bar : public Base {};\n", encoding="utf-8") + + result = extract([first, second], cache_root=tmp_path) + base_stubs = [ + node for node in result["nodes"] + if node["label"] == "Base" and not node.get("source_file") + ] + assert len(base_stubs) == 2 + assert len({node["id"] for node in base_stubs}) == 2 + + inherits_edges = [e for e in result["edges"] if e["relation"] == "inherits"] + assert len(inherits_edges) == 2 + assert len({e["target"] for e in inherits_edges}) == 2 + + def test_cross_file_type_annotation_refs_resolve_to_single_node(tmp_path): """#1402: a class defined once but referenced via type annotations in N other files must NOT create 1+N phantom duplicate nodes (with the referencing file's From 3b2ca2e55fa2cd4f12fafaf051c7cd594787590f Mon Sep 17 00:00:00 2001 From: mallyskies Date: Mon, 6 Jul 2026 23:14:45 -0600 Subject: [PATCH 031/292] fix(build): don't let an ambiguous legacy-stem alias silently merge files build_from_json's "pre-migration alias index" (#1504) registers each real file's OLD-style bare-stem id (extension dropped) as an alias so a stale cached fragment referencing that old form still resolves after an id-scheme migration. It never checked for collisions: two unrelated real files easily compute the same bare alias (e.g. "ping.h" and "ping.php" in different directories both alias to "ping"), and dict.setdefault let whichever file happened to iterate first in a Python set win, arbitrarily. A dangling edge with a bare, deliberately-unscoped fallback target (e.g. the C/C++ extractor's last-resort id for an #include it couldn't resolve to a real path) would ride that alias onto whatever unrelated file won the collision -- silently wiring, say, a C++ server file to an unrelated PHP script, purely because both files happen to share a common basename somewhere in the corpus. Now every candidate for an alias is collected before any of them are committed to norm_to_id, and the alias is only trusted when exactly one real file claims it. Ambiguous aliases are dropped entirely, so the dangling edge correctly stays dangling (and gets discarded) instead of merging two unrelated files -- same "don't guess through ambiguity" principle already applied to stub-node disambiguation and cross-file call resolution elsewhere in this codebase. Found via graphify-practice round 2: a `path` query between two unrelated symbols routed through exactly this kind of bogus edge, traced back to Dev/Poker/TDServer/server.cpp's unresolved `#include "ping.h"` / `#include "utility.h"` landing on unrelated www.masque.com PHP scripts of the same bare name. --- graphify/build.py | 20 +++++++++++++-- tests/test_build.py | 59 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+), 2 deletions(-) diff --git a/graphify/build.py b/graphify/build.py index 279eccbf4..077db58ca 100644 --- a/graphify/build.py +++ b/graphify/build.py @@ -507,7 +507,20 @@ def build_from_json(extraction: dict, *, directed: bool = False, root: str | Pat # fragment (e.g. an incremental update whose fragment references a symbol in a # file that was NOT re-extracted) still resolves to the migrated node instead # of dangling. Only fills gaps — never overrides a real node id. + # + # The old-stem form drops the extension and (for the file node itself) every + # directory but the immediate parent, so it collapses easily: "ping.h" and + # "ping.php" in different directories both alias to bare "ping". Collecting + # every candidate for an alias BEFORE committing any of them — and only + # committing when exactly one candidate claims it — keeps this a precise + # re-keying aid instead of a silent cross-file (and cross-language) merge. + # Without this, a dangling edge to a bare, deliberately-unscoped fallback id + # (e.g. the C/C++ extractor's last-resort target for an #include it couldn't + # resolve to a real path) could ride this alias onto whichever unrelated + # same-stem file happened to be inserted first into ``node_set`` — a Python + # set, so "first" is hash-order, not anything meaningful. from graphify.extractors.base import _file_stem as _fs + _alias_candidates: dict[str, set[str]] = {} for nid in node_set: attrs = G.nodes[nid] sf = attrs.get("source_file") @@ -524,8 +537,11 @@ def build_from_json(extraction: dict, *, directed: bool = False, root: str | Pat if old_stem == new_stem: continue alias = old_stem + suffix - norm_to_id.setdefault(_normalize_id(alias), nid) - norm_to_id.setdefault(alias, nid) + _alias_candidates.setdefault(_normalize_id(alias), set()).add(nid) + _alias_candidates.setdefault(alias, set()).add(nid) + for alias_key, candidates in _alias_candidates.items(): + if len(candidates) == 1: + norm_to_id.setdefault(alias_key, next(iter(candidates))) # Iterate edges in a deterministic order. The graph is undirected and stores # direction in _src/_tgt; when two edges collapse onto the same node pair the # last write wins, so an unstable iteration order flips _src/_tgt run-to-run diff --git a/tests/test_build.py b/tests/test_build.py index 2d8bfdd60..1e1d5b56e 100644 --- a/tests/test_build.py +++ b/tests/test_build.py @@ -505,6 +505,65 @@ def test_build_relativizes_absolute_source_file(tmp_path): assert sf == "src/main.py" +def test_build_from_json_ambiguous_old_stem_alias_stays_dangling(tmp_path): + """The #1504 old-stem alias (e.g. "ping.h" -> bare "ping") is meant to let a + stale-id edge from an un-re-extracted fragment still find its own file after + a rekey. But the old-stem form drops the extension and most of the path, so + two unrelated real files easily collapse onto the same bare alias (a C header + and a PHP script both named "ping", in different directories). A dangling + edge produced by an unrelated third file's own unscoped fallback id (e.g. the + C/C++ extractor's last-resort target for an #include it couldn't resolve to + a real path) must not silently ride that alias onto an arbitrary one of them + — it should stay dangling and get dropped, same as any other unresolvable + edge, rather than wire two unrelated files/languages together by accident.""" + root = tmp_path / "repo" + root.mkdir() + extraction = { + "nodes": [ + # Ids given in their canonical (post-extract.py, extension-stripped) + # form, matching what a real graphify update run would already have + # produced before build_from_json assembles the final graph. + {"id": "dev_monitoring_ping", "label": "ping.h", "file_type": "code", + "source_file": "Dev/monitoring/ping.h"}, + {"id": "www_pages_api_ping", "label": "ping.php", "file_type": "code", + "source_file": "www/pages/api/ping.php"}, + {"id": "dev_poker_server", "label": "server.cpp", "file_type": "code", + "source_file": "Dev/poker/server.cpp"}, + ], + "edges": [ + # The unscoped, deliberately-unresolved fallback edge a C/C++ #include + # resolver leaves behind when it can't find the header on disk. + {"source": "dev_poker_server", "target": "ping", "relation": "imports", + "confidence": "EXTRACTED", "source_file": "Dev/poker/server.cpp"}, + ], + } + G = build_from_json(extraction, root=root) + assert not G.has_edge("dev_poker_server", "dev_monitoring_ping") + assert not G.has_edge("dev_poker_server", "www_pages_api_ping") + + +def test_build_from_json_unambiguous_old_stem_alias_still_resolves(tmp_path): + """Companion to the ambiguous case above: when exactly one real file claims + an old-stem alias, a dangling edge to that bare alias should still resolve + to it — the #1504 migration-compat behavior this index exists for.""" + root = tmp_path / "repo" + root.mkdir() + extraction = { + "nodes": [ + {"id": "dev_monitoring_utility", "label": "utility.h", "file_type": "code", + "source_file": "Dev/monitoring/utility.h"}, + {"id": "dev_poker_server", "label": "server.cpp", "file_type": "code", + "source_file": "Dev/poker/server.cpp"}, + ], + "edges": [ + {"source": "dev_poker_server", "target": "utility", "relation": "imports", + "confidence": "EXTRACTED", "source_file": "Dev/poker/server.cpp"}, + ], + } + G = build_from_json(extraction, root=root) + assert G.has_edge("dev_poker_server", "dev_monitoring_utility") + + def test_build_from_json_relative_source_file_unchanged(tmp_path): """Already-relative source_file paths must not be modified.""" extraction = { From 96db75c8c29b7c9a7a5a4275dbb07f6e4593714b Mon Sep 17 00:00:00 2001 From: mallyskies Date: Mon, 6 Jul 2026 23:55:30 -0600 Subject: [PATCH 032/292] fix(build): recognize a salted file node as its own alias claimant Follow-up to the previous commit on this branch. That fix's ambiguity check missed a case: it detects "is this node the file itself" by checking whether the node's id starts with the file's plain new_stem, but a same-directory .h/.cpp pair that collides on their shared pre-extension id gets salted apart by _disambiguate_colliding_node_ids into ids like "tools_aolserver_utility_h_tools_aolserver_utility" -- no longer a clean new_stem prefix. That salted header silently failed to compute an empty suffix, so it never entered the bare "utility" alias race at all, leaving an unrelated wwwapi.masque.com/pages/utility.php as the lone (wrong) "unambiguous" winner -- reproduced exactly against the real depot's Tools/aolserver/utility.h and .cpp. Detect "this node IS the file" by label instead: every file node's label is its own basename regardless of what its id looks like after salting. That keeps a salted file node in the alias competition, so the real collision between the C header and the PHP file is correctly caught as ambiguous. --- graphify/build.py | 21 ++++++++++++++++++--- tests/test_build.py | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+), 3 deletions(-) diff --git a/graphify/build.py b/graphify/build.py index 077db58ca..b26ddcb61 100644 --- a/graphify/build.py +++ b/graphify/build.py @@ -519,6 +519,18 @@ def build_from_json(extraction: dict, *, directed: bool = False, root: str | Pat # resolve to a real path) could ride this alias onto whichever unrelated # same-stem file happened to be inserted first into ``node_set`` — a Python # set, so "first" is hash-order, not anything meaningful. + # + # A file node's OWN id is not always a clean ``new_stem`` prefix: when a + # same-directory ``.h``/``.cpp`` pair collides on their shared pre-extension + # id, _disambiguate_colliding_node_ids salts both apart into ids like + # ``tools_aolserver_utility_h_tools_aolserver_utility`` — which no longer + # string-prefixes cleanly for the suffix math below. Detecting "this IS the + # file node" by label (every file node's label is its own basename, + # regardless of id mangling) instead of by id shape keeps a salted file node + # in the alias competition, so a genuine collision (a C header AND an + # unrelated same-named PHP script) is still caught as ambiguous instead of + # the header silently dropping out of the race and leaving the PHP file as + # the lone (wrong) "unambiguous" winner. from graphify.extractors.base import _file_stem as _fs _alias_candidates: dict[str, set[str]] = {} for nid in node_set: @@ -530,9 +542,12 @@ def build_from_json(extraction: dict, *, directed: bool = False, root: str | Pat if rel.is_absolute(): continue new_stem = make_id(_fs(rel)) - suffix = "" - if _normalize_id(nid).startswith(new_stem): - suffix = _normalize_id(nid)[len(new_stem):] # leading "_entity" or "" + if str(attrs.get("label", "")) == rel.name: + suffix = "" # this node IS the file, whatever its (possibly salted) id + else: + suffix = "" + if _normalize_id(nid).startswith(new_stem): + suffix = _normalize_id(nid)[len(new_stem):] # leading "_entity" or "" for old_stem in _old_file_stems(rel): if old_stem == new_stem: continue diff --git a/tests/test_build.py b/tests/test_build.py index 1e1d5b56e..e8c0309a9 100644 --- a/tests/test_build.py +++ b/tests/test_build.py @@ -542,6 +542,40 @@ def test_build_from_json_ambiguous_old_stem_alias_stays_dangling(tmp_path): assert not G.has_edge("dev_poker_server", "www_pages_api_ping") +def test_build_from_json_ambiguous_alias_detected_despite_header_impl_salting(tmp_path): + """A same-directory .h/.cpp pair collides on their shared pre-extension id + and gets salted apart into ids like "tools_aolserver_utility_h_..." — no + longer a clean new_stem prefix. The ambiguity check must still recognize + the salted header as a legitimate claimant for the bare old-stem alias (by + label, not id shape), so a real collision with an unrelated same-named PHP + file is still caught instead of the header silently dropping out of the + race and leaving the PHP file as the lone "unambiguous" winner (this + reproduced against the real depot: Tools/aolserver/utility.h and .cpp, + salted apart, let wwwapi.masque.com/pages/utility.php win the bare + "utility" alias uncontested).""" + root = tmp_path / "repo" + root.mkdir() + extraction = { + "nodes": [ + {"id": "tools_aolserver_utility_h_tools_aolserver_utility", "label": "utility.h", + "file_type": "code", "source_file": "Tools/aolserver/utility.h"}, + {"id": "tools_aolserver_utility_cpp_tools_aolserver_utility", "label": "utility.cpp", + "file_type": "code", "source_file": "Tools/aolserver/utility.cpp"}, + {"id": "wwwapi_masque_com_pages_utility", "label": "utility.php", + "file_type": "code", "source_file": "wwwapi.masque.com/pages/utility.php"}, + {"id": "dev_poker_server", "label": "server.cpp", "file_type": "code", + "source_file": "Dev/poker/server.cpp"}, + ], + "edges": [ + {"source": "dev_poker_server", "target": "utility", "relation": "imports", + "confidence": "EXTRACTED", "source_file": "Dev/poker/server.cpp"}, + ], + } + G = build_from_json(extraction, root=root) + assert not G.has_edge("dev_poker_server", "wwwapi_masque_com_pages_utility") + assert not G.has_edge("dev_poker_server", "tools_aolserver_utility_h_tools_aolserver_utility") + + def test_build_from_json_unambiguous_old_stem_alias_still_resolves(tmp_path): """Companion to the ambiguous case above: when exactly one real file claims an old-stem alias, a dangling edge to that bare alias should still resolve From 67f4f835f7634cf31ae4bdc2d859a3561efee1b9 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Wed, 8 Jul 2026 11:25:28 +0100 Subject: [PATCH 033/292] fix(extract): skip builtin-global receiver types in TS/JS member-call resolution (#1726) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `_resolve_typescript_member_calls` resolves a member call's receiver to a type definition by casefolded label. For a builtin-typed receiver (`x: Date; x.getTime()`), `_key("Date")` == "date" matched a same-named user `class DATE` / `const DATE` in another file, binding the caller to it as a phantom `references[call]` edge. In a real 3,368-file repo one module-local `const DATE` accumulated 469 false cross-file edges (degree 472, betweenness 0.435) — a false god node distorting path/god-node results. Skip the resolution when the receiver type is an ECMAScript/Python builtin global (Date, Promise, Map, ...), mirroring the guard the cross-file CALL resolver already applies (#726). The guard is a strict no-op for genuine user types, so legitimate constructor-injection member-call resolution (#1316) is unaffected. Verified across new Date().method(), return-type, and var-decl-type shapes: zero phantom edges, user DATE degree back to 1. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/extract.py | 7 ++++ tests/test_builtin_global_type_refs.py | 52 ++++++++++++++++++++++++++ 2 files changed, 59 insertions(+) create mode 100644 tests/test_builtin_global_type_refs.py diff --git a/graphify/extract.py b/graphify/extract.py index ed0550432..16a3f753a 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -12026,6 +12026,13 @@ def _key(label: str) -> str: type_name = type_table_by_file.get(rc.get("source_file", ""), {}).get(receiver) if not type_name: continue + # A builtin global receiver type (Date, Promise, Map, ...) must not resolve + # to a user symbol. _key() casefolds, so `x: Date; x.getTime()` would bind + # the caller to a same-named user `class DATE` in another file, inventing + # phantom `references[call]` edges and a false god node (#1726). The + # cross-file CALL resolver already skips these globals; do the same here. + if type_name in _LANGUAGE_BUILTIN_GLOBALS: + continue type_defs = type_def_nids.get(_key(type_name), []) if len(type_defs) != 1: continue diff --git a/tests/test_builtin_global_type_refs.py b/tests/test_builtin_global_type_refs.py new file mode 100644 index 000000000..0fe1fd72c --- /dev/null +++ b/tests/test_builtin_global_type_refs.py @@ -0,0 +1,52 @@ +"""Builtin-global receiver types must not resolve to same-named user symbols. + +#1726: `x: Date; x.getTime()` had its caller bound (by casefolded label) to a +user `class DATE` / `const DATE` in another file, inventing phantom +`references[call]` edges and a false god node. The cross-file CALL resolver +already skips ECMAScript/Python builtins; `_resolve_typescript_member_calls` +must do the same. +""" +from pathlib import Path +from graphify.extract import extract + + +def _labels_by_id(r): + return {n["id"]: n.get("label") for n in r["nodes"]} + + +def test_builtin_date_type_ref_does_not_bind_to_user_DATE(tmp_path): + (tmp_path / "model.ts").write_text('export class DATE {\n value: string = "";\n}\n') + (tmp_path / "a.ts").write_text('export function parse(x: Date): number { return x.getTime(); }\n') + (tmp_path / "b.ts").write_text('export function fmt(w: Date): string { return w.toISOString(); }\n') + r = extract(sorted(tmp_path.glob("*.ts")), cache_root=tmp_path, parallel=False) + lbl = _labels_by_id(r) + date_ids = [n["id"] for n in r["nodes"] if n.get("label") == "DATE"] + assert date_ids, "the user class DATE must still exist as a node" + for e in r["edges"]: + if e.get("relation") == "references" and e.get("target") in date_ids: + src = lbl.get(e["source"]) + assert False, f"phantom builtin-Date reference bound to user DATE from {src!r}" + # the user DATE node accumulates no phantom references — degree is just its file + deg = sum(1 for e in r["edges"] if date_ids[0] in (e["source"], e["target"])) + assert deg <= 1, f"user DATE should not be a god node; degree={deg}" + + +def test_nonbuiltin_receiver_type_still_resolves(tmp_path): + # Guard must be a no-op for a genuine user type: a member call on a user-typed + # field still resolves cross-file (constructor-injection type table, #1316). + (tmp_path / "svc.ts").write_text( + "export class PaymentClient {\n charge(n: number): boolean { return true; }\n}\n") + (tmp_path / "order.ts").write_text( + 'import { PaymentClient } from "./svc";\n' + "export class Order {\n" + " constructor(private client: PaymentClient) {}\n" + " pay(): boolean { return this.client.charge(1); }\n" + "}\n") + r = extract(sorted(tmp_path.glob("*.ts")), cache_root=tmp_path, parallel=False) + lbl = _labels_by_id(r) + resolved = { + (lbl.get(e["source"]), lbl.get(e["target"]), e["relation"]) + for e in r["edges"] if "charge" in str(e.get("target", "")).lower() + } + assert any(t and "charge" in str(t).lower() for _, t, _ in resolved), \ + f"user member-call must still resolve; got {resolved}" From 781d1cd8ec5a0197aa8f3912af8d645eedba5413 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Wed, 8 Jul 2026 11:39:39 +0100 Subject: [PATCH 034/292] Release 0.9.10 Correctness batch since 0.9.9: TS/JS builtin-typed receiver no longer collapses onto a same-named user symbol (#1726); no cross-language calls edges (#1718); build_merge ambiguous-alias no longer merges unrelated files (#1713); base-class stubs tagged with origin_file (#1707); Java enum constants as nodes (#1719); rebuild recovers from a deleted hook cwd (#1703); per-chunk semantic-cache checkpoint (#1715); SECURITY.md http-transport doc + GRAPHIFY_MAX_GRAPH_BYTES tests (#1714, #1722). Nine merged PRs plus #1726. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 11 +++++++++++ pyproject.toml | 2 +- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ad4860419..554bb7981 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,17 @@ Full release notes with details on each version: [GitHub Releases](https://github.com/safishamsi/graphify/releases) +## 0.9.10 (2026-07-08) + +- Fix: TS/JS member calls on a builtin-typed receiver no longer collapse onto a same-named user symbol (#1726). `_resolve_typescript_member_calls` matched a receiver's type to a definition by casefolded label, so `x: Date; x.getTime()` bound the caller to a user `class DATE`/`const DATE` in another file — inventing hundreds of phantom `references` edges and a false god node. Builtin-global receiver types (`Date`, `Promise`, `Map`, ...) are now skipped, mirroring the cross-file call guard; genuine user types are unaffected. +- Fix: never bind a cross-file `calls` edge to a definition in a different language family (#1718, thanks @edinaldoof). Name-only matching resolved a TSX callback passed by name to a same-named Kotlin method (and a Python call to a Kotlin fun) — phantom edges the spec forbids. Candidates are now filtered by interop family (JVM, native C-family, JS/TS module graph, ...); unknown families stay permissive. +- Fix: an ambiguous legacy-stem alias in `build_merge` no longer silently merges two unrelated files (#1713, thanks @mallyskies). The `#1504` old-stem alias (`ping.h`/`ping.php` → bare `ping`) resolved by hash-order, riding a dangling edge onto an arbitrary same-named file. Aliases are now committed only when exactly one file claims them; a salted `.h`/`.cpp` file node is recognized as its own claimant so a genuine collision stays ambiguous (and dropped) instead of picking a wrong winner. +- Fix: inline base-class stubs are tagged with `origin_file` (#1707, thanks @mallyskies). Five inheritance handlers built cross-file base-class stubs without `origin_file`, so same-named bases across files collapsed onto one shared stub that could then merge with an unrelated real class (218 wrong `inherits` edges observed). They now route through `ensure_named_node`, which sets the tag. +- Fix: Java enum constants are extracted as nodes with `case_of` edges to their enum (#1719, thanks @ivanzhl). Closes the Java half of #1700; `affected ErrorCode` / "where is ErrorCode.X used" now works for Java. +- Fix: `graphify` rebuilds recover from a deleted hook working directory instead of crashing (#1703, thanks @FranciscoJSBarragan). A detached git hook can inherit a CWD that no longer exists; the rebuild now recovers via `GRAPHIFY_REPO_ROOT` or fails cleanly instead of raising `FileNotFoundError`. +- Feat: the semantic cache is checkpointed per chunk so an interrupted extraction resumes instead of restarting (#1715, thanks @A-Levin). Each completed chunk is unioned into the cache immediately (opt out with `GRAPHIFY_NO_INCREMENTAL_CACHE`); the final write still overwrites authoritatively. +- Docs: `SECURITY.md` no longer claims stdio-only now that an opt-in `--transport http` (binds `127.0.0.1` by default) exists (#1714, thanks @Thizeidler); added tests for `GRAPHIFY_MAX_GRAPH_BYTES` parsing and corrected its unit docstring to binary MiB/GiB (#1722, thanks @Cekaru). + ## 0.9.9 (2026-07-07) - Fix: `graphify explain` resolves an exactly-typed punctuated label symmetrically against `norm_label` (#1704). The search term tokenized on `\w+` ("blockStream.ts" -> "blockstream ts", space where the '.' was) while a node's stored `norm_label` keeps punctuation ("blockstream.ts"). The verbatim case was already rescued by the tokenized-label tier, but that broke if a node's `label` and `norm_label` diverged; a punctuation-preserving `norm_query` is now matched against `norm_label` across the exact/prefix/substring tiers (and fed to the trigram prefilter), so it is robust by construction. diff --git a/pyproject.toml b/pyproject.toml index 2f3aa24ee..7fbde83e0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "graphifyy" -version = "0.9.9" +version = "0.9.10" description = "AI coding assistant skill (Claude Code, CodeBuddy, Codex, OpenCode, Kilo Code, Cursor, Gemini CLI, Aider, OpenClaw, Factory Droid, Trae, Hermes, Kiro, Pi, Devin CLI, Google Antigravity) - turn any folder of code, docs, papers, images, or videos into a queryable knowledge graph" readme = "README.md" license = { file = "LICENSE" } From 5c6f7272cd7d50c0528bd9b7f44da437c87d97c3 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Wed, 8 Jul 2026 12:09:40 +0100 Subject: [PATCH 035/292] test(extract): cover builtin static-call shape (Date.now()) for #1726 Adds a regression test for the Date.now()/static-call shape (credit PR #1727 / @2loch-ness6, who independently found the same fix): a capitalized builtin receiver must not bind cross-file to a same-spelled user const/class. The same guard added in 67f4f83 already covers it (verified); this locks the static-call shape in, while allowing the legitimate same-file const reference. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/test_builtin_global_type_refs.py | 31 ++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/tests/test_builtin_global_type_refs.py b/tests/test_builtin_global_type_refs.py index 0fe1fd72c..25ecb151d 100644 --- a/tests/test_builtin_global_type_refs.py +++ b/tests/test_builtin_global_type_refs.py @@ -50,3 +50,34 @@ def test_nonbuiltin_receiver_type_still_resolves(tmp_path): } assert any(t and "charge" in str(t).lower() for _, t, _ in resolved), \ f"user member-call must still resolve; got {resolved}" + + +def test_builtin_static_call_does_not_bind_to_user_symbol(tmp_path): + # #1726 (static-call shape, credit PR #1727 / @2loch-ness6): `Date.now()` treats + # the capitalized receiver `Date` as a type name; without the builtin guard it + # binds to a same-spelled user `const DATE`, a false god node. A typed param in + # the same class arms the cross-file member-call resolver (the real service shape). + (tmp_path / "format.ts").write_text( + "const DATE = new Intl.DateTimeFormat('en-US', {});\n" + "export function fmt(x: number): string { return DATE.format(x); }\n") + (tmp_path / "svc.ts").write_text( + "export class Svc {\n" + " expiry(d: Date): Date { return d; }\n" + " stamp(): number { return Date.now(); }\n" + " when(): string { return new Date().toISOString(); }\n" + "}\n") + r = extract(sorted(tmp_path.glob("*.ts")), cache_root=tmp_path, parallel=False) + lbl = _labels_by_id(r) + by_id = {n["id"]: n for n in r["nodes"]} + date_ids = [n["id"] for n in r["nodes"] if n.get("label") == "DATE"] + date_sf = {str(by_id[i].get("source_file", "")) for i in date_ids} + # A same-file reference to the real const DATE (fmt() -> DATE in format.ts) is + # legitimate. The bug is a CROSS-FILE bind: svc.ts's Date.now()/new Date() + # resolving to format.ts's const DATE. Assert no such cross-file phantom. + for e in r["edges"]: + if e.get("target") in date_ids and e.get("relation") == "references": + src_sf = str(by_id.get(e["source"], {}).get("source_file", "")) + assert src_sf in date_sf, ( + f"cross-file builtin phantom: {lbl.get(e['source'])!r} in {src_sf} " + f"bound to user DATE" + ) From 4be0f7b3bfe9ac32f6a8caca5bbd6cac632d20c8 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Wed, 8 Jul 2026 12:45:08 +0100 Subject: [PATCH 036/292] fix(merge-graphs): give each input a distinct repo tag so same-stem nodes don't collapse (#1729) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit merge-graphs prefixed each graph's node ids with `::` where repo was `gp.parent.parent.name` — the graphify-out parent dir name. That tag is not unique across inputs: `src/graphify-out` and `frontend/src/graphify-out` both yield `src`, so a bare `app` node from a backend `src/app.js` and a frontend `App.jsx` both became `src::app` and nx.compose silently merged them into one node (one graph's label/source_file, both graphs' edges) — inventing false cross-runtime `path` results with no warning. New `distinct_repo_tags` guarantees a unique prefix per graph: colliding tags are widened with their own parent dir (`frontend_src`), then an index suffix backstops any residual duplicate. The handler prints a note when it disambiguates. Verified end to end: the two `app` nodes now survive as `..._src::app` and `frontend_src::app` instead of collapsing. Regression tests cover the merge case and the tag uniquifier (pass-through, widening, and triple-collision fallback). Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/__main__.py | 13 +++++++--- graphify/build.py | 28 +++++++++++++++++++++ tests/test_merge_graphs_cli.py | 46 ++++++++++++++++++++++++++++++++++ 3 files changed, 84 insertions(+), 3 deletions(-) diff --git a/graphify/__main__.py b/graphify/__main__.py index 285d19c1b..b869e8653 100644 --- a/graphify/__main__.py +++ b/graphify/__main__.py @@ -3993,7 +3993,7 @@ def _load_graph(p: str): sys.exit(1) import networkx as _nx from networkx.readwrite import json_graph as _jg - from graphify.build import prefix_graph_for_global as _prefix + from graphify.build import prefix_graph_for_global as _prefix, distinct_repo_tags as _repo_tags graphs = [] for gp in graph_paths: if not gp.exists(): @@ -4025,9 +4025,16 @@ def _to_simple(g: "_nx.Graph") -> "_nx.Graph": if type(g) is not _nx.Graph: return _nx.Graph(g) return g + # Unique repo tag per graph. The bare `graphify-out/..` dir name is not + # unique across inputs (src/graphify-out and frontend/src/graphify-out both + # → "src"), which collides same-stem node ids and silently merges unrelated + # entities (#1729). distinct_repo_tags guarantees a distinct prefix per graph. + repo_tags = _repo_tags(graph_paths) + naive_tags = [gp.parent.parent.name for gp in graph_paths] + if len(set(naive_tags)) != len(naive_tags): + print(f" note: repo dir names collide; using distinct tags: {', '.join(repo_tags)}") merged = _nx.Graph() - for G, gp in zip(graphs, graph_paths): - repo_tag = gp.parent.parent.name # graphify-out/../ → repo dir name + for G, repo_tag in zip(graphs, repo_tags): prefixed = _to_simple(_prefix(G, repo_tag)) merged = _nx.compose(merged, prefixed) try: diff --git a/graphify/build.py b/graphify/build.py index b26ddcb61..f0676a8d5 100644 --- a/graphify/build.py +++ b/graphify/build.py @@ -956,6 +956,34 @@ def prefix_graph_for_global(G: nx.Graph, repo_tag: str) -> nx.Graph: return H +def distinct_repo_tags(graph_paths: "list[Path]") -> "list[str]": + """Return a unique, human-meaningful repo tag per input graph for merge-graphs. + + The naive tag (the ``graphify-out`` parent dir name) is NOT unique across + inputs: ``src/graphify-out`` and ``frontend/src/graphify-out`` both yield + ``src``. Prefixing both node sets with ``src::`` then makes same-stem nodes + (a backend ``src/app.js`` and a frontend ``App.jsx``, both bare ``app``) + collide, so ``nx.compose`` silently merges two unrelated entities and invents + cross-runtime edges (#1729). Colliding tags are widened with their own parent + dir (``frontend_src``), then an index suffix guarantees uniqueness so no two + graphs ever share a prefix. + """ + repo_dirs = [p.parent.parent for p in graph_paths] # graphify-out/.. → repo dir + tags = [d.name or "repo" for d in repo_dirs] + if len(set(tags)) != len(tags): + widened: list[str] = [] + for d in repo_dirs: + parent = d.parent.name + widened.append(f"{parent}_{d.name}" if parent and d.name else (d.name or "repo")) + tags = widened + seen: dict[str, int] = {} + unique: list[str] = [] + for t in tags: + seen[t] = seen.get(t, 0) + 1 + unique.append(t if seen[t] == 1 else f"{t}-{seen[t]}") + return unique + + def prune_repo_from_graph(G: nx.Graph, repo_tag: str) -> int: """Remove all nodes tagged with repo_tag from G in-place. Returns count removed.""" to_remove = [n for n, d in G.nodes(data=True) if d.get("repo") == repo_tag] diff --git a/tests/test_merge_graphs_cli.py b/tests/test_merge_graphs_cli.py index f3e0e05d2..5ff06b391 100644 --- a/tests/test_merge_graphs_cli.py +++ b/tests/test_merge_graphs_cli.py @@ -46,3 +46,49 @@ def test_merge_graphs_mixed_directed_and_multigraph(tmp_path): assert {"r1::x", "r2::y", "r3::z"} <= ids or len(ids) == 3 assert data.get("directed") is False assert data.get("multigraph") is False + + +def test_merge_graphs_same_named_repo_dirs_do_not_collapse(tmp_path): + # #1729: two graphs under a same-named repo dir (src/graphify-out and + # frontend/src/graphify-out both → tag "src") share the `src::` prefix, so a + # bare `app` node from each collapsed into one — silently merging unrelated + # entities and inventing cross-runtime edges. Distinct tags must keep them apart. + a = tmp_path / "src" / "graphify-out" / "graph.json" + b = tmp_path / "frontend" / "src" / "graphify-out" / "graph.json" + a.parent.mkdir(parents=True, exist_ok=True) + b.parent.mkdir(parents=True, exist_ok=True) + a.write_text(json.dumps({"directed": False, "multigraph": False, "nodes": [ + {"id": "app", "label": "app.js", "source_file": "app.js"}], "links": []})) + b.write_text(json.dumps({"directed": False, "multigraph": False, "nodes": [ + {"id": "app", "label": "App.jsx", "source_file": "App.jsx"}], "links": []})) + out = tmp_path / "merged.json" + + r = _run(["merge-graphs", str(a), str(b), "--out", str(out)], tmp_path) + assert r.returncode == 0, r.stderr + data = json.loads(out.read_text()) + app_nodes = [n for n in data["nodes"] if n["id"].endswith("::app")] + assert len(app_nodes) == 2, f"both app nodes must survive; got {[n['id'] for n in app_nodes]}" + labels = {n.get("label") for n in app_nodes} + assert labels == {"app.js", "App.jsx"}, f"both entities preserved; got {labels}" + + +def test_distinct_repo_tags_unit(tmp_path): + from graphify.build import distinct_repo_tags + # distinct repo dirs pass through unchanged + assert distinct_repo_tags([ + Path("backend/graphify-out/graph.json"), + Path("web/graphify-out/graph.json"), + ]) == ["backend", "web"] + # same-named repo dirs are widened to stay distinct + tags = distinct_repo_tags([ + Path("proj/src/graphify-out/graph.json"), + Path("proj/frontend/src/graphify-out/graph.json"), + ]) + assert len(set(tags)) == 2, tags + # a repeated dir name triple still yields all-distinct tags (index fallback) + tags3 = distinct_repo_tags([ + Path("a/src/graphify-out/graph.json"), + Path("b/src/graphify-out/graph.json"), + Path("c/src/graphify-out/graph.json"), + ]) + assert len(set(tags3)) == 3, tags3 From 0d206cbe111be4316978fadb08a023f15e615f42 Mon Sep 17 00:00:00 2001 From: tpateeq Date: Wed, 8 Jul 2026 19:05:18 +0530 Subject: [PATCH 037/292] fix(uninstall): remove graphify hook/section from Claude local-only files (#1731) `graphify uninstall` (and `graphify claude uninstall`) only looked at `.claude/settings.json` and root `CLAUDE.md`. A user who relocates the PreToolUse hook into `.claude/settings.local.json` and the `## graphify` instructions into `CLAUDE.local.md` / `.claude/CLAUDE.local.md` - so they are not committed to a shared repo - was left with both behind after uninstall, which reported "nothing to do". - `_uninstall_claude_hook` now strips the hook from both `settings.json` and `settings.local.json` (factored into `_strip_graphify_hook`). - `claude_uninstall` now strips the `## graphify` section from `CLAUDE.md`, root `CLAUDE.local.md`, and `.claude/CLAUDE.local.md` (factored into `_strip_graphify_md_section`), cleaning every present file rather than the first, and always runs the hook cleanup. - `_strip_graphify_md_section` reads the two newly-covered files defensively so a non-UTF-8 or otherwise unreadable file can't abort uninstall. Unrelated local-only files (no graphify content) are left byte-for-byte untouched. The `--local` install flag suggested in the issue is left out of scope; this change makes uninstall symmetric with wherever the config lives. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/__main__.py | 67 ++++++++++++++++++++++++++++++--------- tests/test_claude_md.py | 69 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+), 14 deletions(-) diff --git a/graphify/__main__.py b/graphify/__main__.py index b869e8653..9aa2cf806 100644 --- a/graphify/__main__.py +++ b/graphify/__main__.py @@ -2053,8 +2053,19 @@ def _install_claude_hook(project_dir: Path) -> None: def _uninstall_claude_hook(project_dir: Path) -> None: - """Remove graphify PreToolUse hook from .claude/settings.json.""" - settings_path = project_dir / ".claude" / "settings.json" + """Remove the graphify PreToolUse hook from .claude/settings.json and its + local-only sibling .claude/settings.local.json. + + A user may relocate the hook into settings.local.json so it is not committed + to a shared repo, so uninstall has to clean whichever file holds it (#1731). + """ + claude_dir = project_dir / ".claude" + for name in ("settings.json", "settings.local.json"): + _strip_graphify_hook(claude_dir / name) + + +def _strip_graphify_hook(settings_path: Path) -> None: + """Drop graphify PreToolUse hooks from a single Claude settings file, if present.""" if not settings_path.exists(): return try: @@ -2067,7 +2078,7 @@ def _uninstall_claude_hook(project_dir: Path) -> None: return settings["hooks"]["PreToolUse"] = filtered settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") - print(f" .claude/settings.json -> PreToolUse hook removed") + print(f" .claude/{settings_path.name} -> PreToolUse hook removed") def uninstall_all(project_dir: Path | None = None, purge: bool = False) -> None: @@ -2116,25 +2127,54 @@ def uninstall_all(project_dir: Path | None = None, purge: bool = False) -> None: def claude_uninstall(project_dir: Path | None = None, *, project: bool = False) -> None: - """Remove the graphify skill tree (SKILL.md + references/) and the CLAUDE.md section. + """Remove the graphify skill tree (SKILL.md + references/) and the graphify + section from CLAUDE.md and its local-only variants, plus the PreToolUse hook. Mirrors gemini_uninstall: the bare `graphify uninstall` and `graphify claude uninstall` must remove the installed skill, not just strip CLAUDE.md, or the progressive-disclosure tree (SKILL.md + references/) is orphaned (#1121). + + A user may relocate the section/hook into the local-only files Claude Code + supports so they are not committed to a shared repo, so uninstall also cleans + CLAUDE.local.md, .claude/CLAUDE.local.md and .claude/settings.local.json (#1731). """ project_dir = project_dir or Path(".") _remove_skill_file("claude", project=project, project_dir=project_dir) - target = project_dir / "CLAUDE.md" - if not target.exists(): + md_targets = [ + project_dir / "CLAUDE.md", + project_dir / "CLAUDE.local.md", + project_dir / ".claude" / "CLAUDE.local.md", + ] + existing = [t for t in md_targets if t.exists()] + removed_any = False + for target in existing: + # Not short-circuited: every present file must be cleaned, not just the first. + if _strip_graphify_md_section(target): + removed_any = True + + if not existing: print("No CLAUDE.md found in current directory - nothing to do") - return - - content = target.read_text(encoding="utf-8") - if _CLAUDE_MD_MARKER not in content: + elif not removed_any: print("graphify section not found in CLAUDE.md - nothing to do") - return + _uninstall_claude_hook(project_dir) + + +def _strip_graphify_md_section(target: Path) -> bool: + """Strip the ## graphify section from one CLAUDE.md-style file. + + Returns True if a section was removed. Deletes the file if nothing else + remains after removal. + """ + try: + content = target.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError): + # An unreadable/undecodable CLAUDE.md-style file (e.g. non-UTF-8, or a + # directory of that name) must not abort uninstall - nothing to strip. + return False + if _CLAUDE_MD_MARKER not in content: + return False # Remove the ## graphify section: from the marker to the next ## heading or EOF cleaned = re.sub( r"\n*## graphify\n.*?(?=\n## |\Z)", @@ -2147,9 +2187,8 @@ def claude_uninstall(project_dir: Path | None = None, *, project: bool = False) print(f"graphify section removed from {target.resolve()}") else: target.unlink() - print(f"CLAUDE.md was empty after removal - deleted {target.resolve()}") - - _uninstall_claude_hook(project_dir or Path(".")) + print(f"{target.name} was empty after removal - deleted {target.resolve()}") + return True def codebuddy_install(project_dir: Path | None = None) -> None: diff --git a/tests/test_claude_md.py b/tests/test_claude_md.py index f81f10dd3..55b13e5ee 100644 --- a/tests/test_claude_md.py +++ b/tests/test_claude_md.py @@ -134,3 +134,72 @@ def test_uninstall_removes_settings_hook(tmp_path): settings = json.loads(settings_path.read_text()) hooks = settings.get("hooks", {}).get("PreToolUse", []) assert not any(h.get("matcher") == "Bash" and "graphify" in str(h) for h in hooks) + + +# --------------------------------------------------------------------------- +# local-only variants: settings.local.json / CLAUDE.local.md (#1731) +# --------------------------------------------------------------------------- + +def test_uninstall_removes_hook_from_settings_local_json(tmp_path): + """A hook relocated to .claude/settings.local.json is removed on uninstall.""" + import json + claude_install(tmp_path) + # User moved the hook out of the committed settings.json into the local-only file. + (tmp_path / ".claude" / "settings.json").rename(tmp_path / ".claude" / "settings.local.json") + claude_uninstall(tmp_path) + local = tmp_path / ".claude" / "settings.local.json" + hooks = json.loads(local.read_text()).get("hooks", {}).get("PreToolUse", []) + assert not any("graphify" in str(h) for h in hooks) + + +def test_uninstall_removes_section_from_dot_claude_local_md(tmp_path): + """Instructions relocated to .claude/CLAUDE.local.md are removed on uninstall.""" + claude_install(tmp_path) + local_md = tmp_path / ".claude" / "CLAUDE.local.md" + local_md.write_text((tmp_path / "CLAUDE.md").read_text()) + (tmp_path / "CLAUDE.md").unlink() + claude_uninstall(tmp_path) + assert not local_md.exists() or _CLAUDE_MD_MARKER not in local_md.read_text() + + +def test_uninstall_removes_section_from_root_claude_local_md(tmp_path): + """Instructions relocated to root CLAUDE.local.md are removed on uninstall.""" + claude_install(tmp_path) + local_md = tmp_path / "CLAUDE.local.md" + local_md.write_text((tmp_path / "CLAUDE.md").read_text()) + (tmp_path / "CLAUDE.md").unlink() + claude_uninstall(tmp_path) + assert not local_md.exists() or _CLAUDE_MD_MARKER not in local_md.read_text() + + +def test_uninstall_cleans_both_standard_and_local(tmp_path): + """When the section lives in both CLAUDE.md and a local variant, both are cleaned.""" + claude_install(tmp_path) + claude_md = tmp_path / "CLAUDE.md" + local_md = tmp_path / ".claude" / "CLAUDE.local.md" + local_md.write_text(claude_md.read_text()) # duplicated into the local file too + claude_uninstall(tmp_path) + for f in (claude_md, local_md): + assert not f.exists() or _CLAUDE_MD_MARKER not in f.read_text() + + +def test_uninstall_preserves_other_content_in_local_md(tmp_path): + """Uninstall keeps non-graphify content in CLAUDE.local.md.""" + claude_install(tmp_path) + local_md = tmp_path / ".claude" / "CLAUDE.local.md" + local_md.write_text("# Local notes\n\nkeep me\n\n" + (tmp_path / "CLAUDE.md").read_text()) + claude_uninstall(tmp_path) + assert local_md.exists() + content = local_md.read_text() + assert "Local notes" in content + assert "keep me" in content + assert _CLAUDE_MD_MARKER not in content + + +def test_uninstall_tolerates_unreadable_local_md(tmp_path): + """A non-UTF-8 CLAUDE.local.md must not abort uninstall (it has no marker to strip).""" + claude_install(tmp_path) + local_md = tmp_path / ".claude" / "CLAUDE.local.md" + local_md.write_bytes(b"\xff\xfe not valid utf-8 \x80\x81") + claude_uninstall(tmp_path) # must not raise + assert local_md.read_bytes() == b"\xff\xfe not valid utf-8 \x80\x81" # left untouched From 20bfdf60ac7187edc2f8594252222dc8e9b96399 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Wed, 8 Jul 2026 17:07:31 +0100 Subject: [PATCH 038/292] feat(extract): add --code-only to index code without an LLM key on a mixed repo (#1734) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `graphify extract` on a repo containing docs/papers/images hard-failed when no LLM backend was configured — even for a user who only wants the code graph. The only workaround was hand-building a .graphifyignore of everything non-code, which is onerous (the "not code" set is far larger than the code set). `--code-only` skips the semantic (doc/paper/image) pass entirely: it indexes the code via pure local AST (no key required) and reports what it skipped ("skipping N non-code file(s) ...") rather than silently dropping it. The no-key error on a mixed repo now also points users at the flag. Code-only was always keyless; this just makes a *mixed* repo usable without a key instead of failing outright. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/__main__.py | 24 ++++++++++++- tests/test_extract_code_only_cli.py | 55 +++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+), 1 deletion(-) create mode 100644 tests/test_extract_code_only_cli.py diff --git a/graphify/__main__.py b/graphify/__main__.py index 9aa2cf806..2e0a71c75 100644 --- a/graphify/__main__.py +++ b/graphify/__main__.py @@ -2472,6 +2472,7 @@ def main() -> None: print(" --out DIR output dir (default: ); writes /graphify-out/") print(" --google-workspace export .gdoc/.gsheet/.gslides shortcuts via gws before extraction") print(" --no-cluster skip clustering, write raw extraction only") + print(" --code-only index code (local AST, no API key) and skip doc/paper/image files") print(" --postgres DSN extract schema from a live PostgreSQL database") print(" maps tables, views, functions + FK relationships;") print(" column-level detail is not represented in the graph") @@ -4526,6 +4527,7 @@ def _to_simple(g: "_nx.Graph") -> "_nx.Graph": dedup_llm = False google_workspace = False global_merge = False + code_only = False global_repo_tag: str | None = None # Performance/tuning knobs (issue #792). None means "use library default". cli_max_workers: int | None = None @@ -4584,6 +4586,8 @@ def _parse_float(name: str, raw: str) -> float: no_cluster = True; i += 1 elif a == "--dedup-llm": dedup_llm = True; i += 1 + elif a == "--code-only": + code_only = True; i += 1 elif a == "--google-workspace": google_workspace = True; i += 1 elif a == "--global": @@ -4707,6 +4711,20 @@ def _parse_float(name: str, raw: str) -> float: unchanged_total = 0 semantic_files = doc_files + paper_files + image_files + # --code-only: index code (pure local AST, no key) and skip the semantic + # (doc/paper/image) pass entirely, so a mixed repo doesn't hard-fail when no + # LLM backend is configured (#1734). Report what was skipped rather than + # silently dropping it. + if code_only and semantic_files: + print( + f"[graphify extract] --code-only: skipping {len(semantic_files)} " + f"non-code file(s) ({len(doc_files)} docs, {len(paper_files)} papers, " + f"{len(image_files)} images) — no LLM extraction" + ) + semantic_files = [] + doc_files = [] + paper_files = [] + image_files = [] if incremental_mode: print( f"[graphify extract] {len(code_files)} code, {len(doc_files)} docs, " @@ -4762,12 +4780,16 @@ def _parse_float(name: str, raw: str) -> float: ) if dedup_llm: reasons.append("--dedup-llm was passed") + hint = "" + if semantic_files: + hint = (" Or pass --code-only to index just the code " + "(local AST, no key) and skip the non-code files.") print( "error: no LLM API key found (" + "; ".join(reasons) + "). " "Set GEMINI_API_KEY or GOOGLE_API_KEY (gemini), MOONSHOT_API_KEY " "(kimi), ANTHROPIC_API_KEY (claude), OPENAI_API_KEY (openai), " "DEEPSEEK_API_KEY (deepseek), or pass --backend. A code-only " - "corpus needs no key.", + "corpus needs no key." + hint, file=sys.stderr, ) sys.exit(1) diff --git a/tests/test_extract_code_only_cli.py b/tests/test_extract_code_only_cli.py new file mode 100644 index 000000000..2fcc8a7e3 --- /dev/null +++ b/tests/test_extract_code_only_cli.py @@ -0,0 +1,55 @@ +"""`graphify extract --code-only` indexes code without an LLM key (#1734). + +A mixed repo (code + docs) with no API key configured used to hard-fail on the +doc/paper/image files. `--code-only` skips the semantic pass so the code graph +still builds, and the no-key error now points users at the flag. +""" +from __future__ import annotations + +import os +import subprocess +import sys +from pathlib import Path + +PYTHON = sys.executable +_KEY_VARS = ("GEMINI_API_KEY", "GOOGLE_API_KEY", "OPENAI_API_KEY", "OPENAI_BASE_URL", + "ANTHROPIC_API_KEY", "MOONSHOT_API_KEY", "DEEPSEEK_API_KEY") + + +def _mixed_repo(tmp_path: Path) -> Path: + repo = tmp_path / "repo" + repo.mkdir() + (repo / "app.py").write_text("def hello():\n return 1\n") + (repo / "README.md").write_text("# Design\n\nHow it works.\n") + (repo / "NOTES.txt").write_text("Architecture notes and rationale.\n") + return repo + + +def _run(repo: Path, *extra: str): + env = {k: v for k, v in os.environ.items() if k not in _KEY_VARS} + env["GRAPHIFY_OUT"] = str(repo / "graphify-out") + return subprocess.run( + [PYTHON, "-m", "graphify", "extract", ".", *extra], + cwd=repo, capture_output=True, text=True, env=env, + ) + + +def test_code_only_succeeds_without_key(tmp_path): + repo = _mixed_repo(tmp_path) + r = _run(repo, "--code-only") + assert r.returncode == 0, f"--code-only should succeed with no key: {r.stderr}" + out = r.stdout + r.stderr + assert "--code-only: skipping" in out + graph = repo / "graphify-out" / "graph.json" + assert graph.exists(), "code graph must still be written" + import json + g = json.loads(graph.read_text()) + labels = [n.get("label") for n in g["nodes"]] + assert any(str(l).startswith("hello") for l in labels), "code was indexed" + + +def test_mixed_repo_without_key_errors_and_points_at_code_only(tmp_path): + repo = _mixed_repo(tmp_path) + r = _run(repo) # no --code-only, no key + assert r.returncode != 0, "mixed repo with no key should still error without the flag" + assert "--code-only" in r.stderr, "the no-key error must point users at --code-only" From cc2d3c13a62870bca95f14f27652daa83bcba894 Mon Sep 17 00:00:00 2001 From: tpateeq Date: Thu, 9 Jul 2026 00:20:48 +0530 Subject: [PATCH 039/292] refactor(main): extract install/uninstall subsystem into graphify/install.py MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit __main__.py was 5,368 LOC, more than half of it per-platform install/uninstall machinery interleaved with the CLI dispatcher. Move that subsystem — 68 functions (all *_install/*_uninstall, _copy_skill_file, _platform_skill_destination, _always_on, etc.) plus 21 module constants (_PLATFORM_CONFIG and the platform skill/hook payload constants) — into a new graphify/install.py, extracted verbatim. Behavior-preserving: - install.py lives in the same package dir as __main__ so packaged-asset lookups via Path(__file__).parent ("always_on"/"skills") resolve unchanged. - __main__ re-exports all 89 moved names, so `from graphify.__main__ import claude_install` (and every other import, incl. private helpers used by tests) keeps working, with object identity preserved. - The install cluster was verified (AST) to have zero back-references into __main__, so no circular import; __main__ imports _PLATFORM_CONFIG et al. one-way. __main__.py drops 5,368 -> 3,642 LOC. Full suite unchanged: 3036 passed, 29 skipped (excluding the env-only openai tests). ruff check clean; skillgen --check OK. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/__main__.py | 1917 +++--------------------------------------- graphify/install.py | 1853 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 1948 insertions(+), 1822 deletions(-) create mode 100644 graphify/install.py diff --git a/graphify/__main__.py b/graphify/__main__.py index 2e0a71c75..dc4f855a7 100644 --- a/graphify/__main__.py +++ b/graphify/__main__.py @@ -23,30 +23,101 @@ # same override (#1423). from graphify.paths import GRAPHIFY_OUT as _GRAPHIFY_OUT +# Install/uninstall subsystem moved to graphify/install.py; re-exported here so +# `from graphify.__main__ import ` keeps working unchanged. +from graphify.install import ( # noqa: E402,F401 + _agents_install, + _agents_platform_install, + _agents_platform_uninstall, + _agents_uninstall, + _always_on, + _amp_install, + _amp_legacy_cleanup, + _amp_uninstall, + _antigravity_finalize, + _antigravity_install, + _antigravity_uninstall, + _canonical_platform, + _claude_pretooluse_hooks, + _copy_skill_file, + _cursor_install, + _cursor_uninstall, + _devin_rules_install, + _devin_rules_uninstall, + _gemini_hook, + _install_claude_hook, + _install_codebuddy_hook, + _install_codex_hook, + _install_gemini_hook, + _install_kilo_plugin, + _install_opencode_plugin, + _install_skill_references, + _kilo_config_path, + _kilo_config_write_path, + _kilo_install, + _kilo_uninstall, + _kilo_uninstall_global, + _kiro_install, + _kiro_uninstall, + _load_json_like, + _packaged_skill_refs_dir, + _platform_skill_destination, + _print_banner, + _print_install_usage, + _print_project_git_add_hint, + _project_install, + _project_scope_root, + _project_uninstall, + _project_uninstall_all, + _refresh_all_version_stamps, + _remove_claude_skill_registration, + _remove_skill_file, + _replace_or_append_section, + _resolve_graphify_exe, + _skill_registration, + _strip_graphify_hook, + _strip_graphify_md_section, + _strip_json_comments, + _uninstall_claude_hook, + _uninstall_codebuddy_hook, + _uninstall_codex_hook, + _uninstall_gemini_hook, + _uninstall_kilo_plugin, + _uninstall_opencode_plugin, + claude_install, + claude_uninstall, + codebuddy_install, + codebuddy_uninstall, + gemini_install, + gemini_uninstall, + install, + uninstall_all, + vscode_install, + vscode_uninstall, + _PLATFORM_ALIASES, + _CLAUDE_MD_MARKER, + _CODEBUDDY_MD_MARKER, + _AGENTS_MD_MARKER, + _GEMINI_MD_MARKER, + _VSCODE_INSTRUCTIONS_MARKER, + _ANTIGRAVITY_RULES_PATH, + _ANTIGRAVITY_WORKFLOW_PATH, + _ANTIGRAVITY_WORKFLOW, + _CURSOR_RULE_PATH, + _CURSOR_RULE, + _DEVIN_RULES_PATH, + _DEVIN_RULES, + _KILO_PLUGIN_JS, + _KILO_PLUGIN_PATH, + _KILO_CONFIG_JSON_PATH, + _KILO_CONFIG_JSONC_PATH, + _OPENCODE_PLUGIN_JS, + _OPENCODE_PLUGIN_PATH, + _OPENCODE_CONFIG_PATH, + _PLATFORM_CONFIG, +) + -@functools.lru_cache(maxsize=None) -def _always_on(basename: str) -> str: - """Read a packaged always-on instruction block from graphify/always_on/. - - The six always-on blocks (CLAUDE.md / AGENTS.md / GEMINI.md / VS Code - Copilot instructions / Antigravity rules / Kiro steering) live as committed - markdown next to this module, generated by tools/skillgen from a single - human-edited fragment and guarded against drift by ``skillgen --check``. The - installer injects them verbatim via ``_replace_or_append_section``, so the - bytes here must match the former triple-quoted constant exactly — the - always-on-roundtrip validator proves that. - """ - path = Path(__file__).parent / "always_on" / f"{basename}.md" - try: - return path.read_text(encoding="utf-8") - except OSError as exc: - # Defer to use-time so a missing/corrupt packaged block can't crash module - # import (which would brick every CLI command, not just install). Reached - # only by an install/integration path that actually needs this block. - raise RuntimeError( - f"graphify install is incomplete: missing always-on block '{basename}' " - f"at {path}. Reinstall graphifyy (e.g. `uv tool install --reinstall graphifyy`)." - ) from exc _ALWAYS_ON_ALIASES = { @@ -185,254 +256,22 @@ def _version_tuple(version: str) -> tuple[int, ...]: return tuple(parts) -def _refresh_all_version_stamps() -> None: - """After a successful install, update .graphify_version in all other known skill dirs. - Prevents stale-version warnings from platforms that were installed previously - but not explicitly re-installed during this upgrade. - """ - for name in _PLATFORM_CONFIG: - skill_dst = _platform_skill_destination(name) - vf = skill_dst.parent / ".graphify_version" - if skill_dst.exists(): - vf.write_text(__version__, encoding="utf-8") - - -def _platform_skill_destination(platform_name: str, *, project: bool = False, project_dir: Path | None = None) -> Path: - """Return the skill destination for a platform and scope.""" - if platform_name == "gemini": - if project: - return (project_dir or Path(".")) / ".gemini" / "skills" / "graphify" / "SKILL.md" - if platform.system() == "Windows": - return Path.home() / ".agents" / "skills" / "graphify" / "SKILL.md" - return Path.home() / ".gemini" / "skills" / "graphify" / "SKILL.md" - - if platform_name == "opencode": - if project: - return (project_dir or Path(".")) / ".opencode" / "skills" / "graphify" / "SKILL.md" - return Path.home() / ".config" / "opencode" / "skills" / "graphify" / "SKILL.md" - - if platform_name == "hermes": - if project: - return (project_dir or Path(".")) / ".hermes" / "skills" / "graphify" / "SKILL.md" - # On Windows, Hermes scans %LOCALAPPDATA%\hermes\skills, not ~/.hermes (#1403). - if platform.system() == "Windows": - local_appdata = Path(os.environ.get("LOCALAPPDATA") or (Path.home() / "AppData" / "Local")) - return local_appdata / "hermes" / "skills" / "graphify" / "SKILL.md" - return Path.home() / ".hermes" / "skills" / "graphify" / "SKILL.md" - - if platform_name == "devin": - if project: - return (project_dir or Path(".")) / ".devin" / "skills" / "graphify" / "SKILL.md" - return Path.home() / ".config" / "devin" / "skills" / "graphify" / "SKILL.md" - - if platform_name == "amp": - if project: - return (project_dir or Path(".")) / ".agents" / "skills" / "graphify" / "SKILL.md" - return Path.home() / ".config" / "agents" / "skills" / "graphify" / "SKILL.md" - - if platform_name == "agents": - # The generic Agent-Skills target: project ./.agents/skills, global the - # spec's user-global ~/.agents/skills (read by `npx skills` and compliant - # frameworks), NOT amp's ~/.config/agents/skills. - if project: - return (project_dir or Path(".")) / ".agents" / "skills" / "graphify" / "SKILL.md" - return Path.home() / ".agents" / "skills" / "graphify" / "SKILL.md" - - if platform_name in ("antigravity", "antigravity-windows"): - if project: - return (project_dir or Path(".")) / ".agents" / "skills" / "graphify" / "SKILL.md" - # Global Antigravity skill dir (all workspaces): ~/.gemini/config/skills/ - return Path.home() / ".gemini" / "config" / "skills" / "graphify" / "SKILL.md" - - cfg = _PLATFORM_CONFIG[platform_name] - if project: - return (project_dir or Path(".")) / cfg["skill_dst"] - - if platform_name in ("claude", "windows") and os.environ.get("CLAUDE_CONFIG_DIR"): - return Path(os.environ["CLAUDE_CONFIG_DIR"]) / "skills" / "graphify" / "SKILL.md" - return Path.home() / cfg["skill_dst"] - - -def _packaged_skill_refs_dir(platform_name: str) -> Path | None: - """Return the packaged references source dir for a progressive platform, else None. - - A platform opts into progressive disclosure by setting ``skill_refs`` in its - ``_PLATFORM_CONFIG`` entry. The value names a bundle under - ``graphify/skills//references/``. Reuse keys (e.g. trae-cn) point at - their twin's bundle. - - ``gemini`` has no ``_PLATFORM_CONFIG`` entry: it installs claude's - ``skill.md`` body verbatim (see ``_copy_skill_file``). Since that body is the - lean progressive core that links to ``references/``, gemini needs claude's - references/ sidecar too, or its SKILL.md ships with dead pointers. So gemini - resolves to the claude bundle rather than opting out. - - Bundles ship one platform-group at a time. A host whose bundle directory - ``graphify/skills//`` is not in this build has not gone progressive - yet, so this returns None and the host installs today's monolithic SKILL.md - with no references/ sidecar. Only when the bundle directory IS present does - this return the references path; if that directory then lacks its - ``references/`` subdir, ``_copy_skill_file`` hard-fails (a malformed bundle, - the empty-sidecar regression the wheel-content test also guards). - """ - if platform_name == "gemini": - bundle = "claude" - else: - bundle = _PLATFORM_CONFIG[platform_name].get("skill_refs") - if not bundle: - return None - bundle_dir = Path(__file__).parent / "skills" / bundle - if not bundle_dir.is_dir(): - return None - return bundle_dir / "references" - - -def _install_skill_references(skill_dst: Path, refs_src: Path) -> None: - """Atomically install a packaged references/ sidecar next to SKILL.md. - - Stages the packaged dir into ``references.tmp`` (copytree), drops any stale - ``references/`` already on disk, then ``os.replace``-renames the staged dir - into place. The rename is atomic on the same filesystem, so an interrupted - install never leaves a half-written references/ visible to the agent. - """ - refs_dst = skill_dst.parent / "references" - refs_staged = skill_dst.parent / "references.tmp" - if refs_staged.exists(): - shutil.rmtree(refs_staged) - try: - shutil.copytree(refs_src, refs_staged) - if refs_dst.exists(): - shutil.rmtree(refs_dst) - os.replace(refs_staged, refs_dst) - except Exception: - if refs_staged.exists(): - shutil.rmtree(refs_staged, ignore_errors=True) - raise -def _copy_skill_file(platform_name: str, *, project: bool = False, project_dir: Path | None = None) -> Path: - """Copy a packaged skill file and write its version stamp. - For progressive platforms (those with ``skill_refs`` set), the packaged - ``references/`` sidecar is installed alongside SKILL.md and the single - ``.graphify_version`` stamp covers both. For monolith platforms (no - ``skill_refs``), any orphan ``references/`` left by a prior progressive - install is removed so the on-disk layout matches the package. - """ - skill_file = "skill.md" if platform_name == "gemini" else _PLATFORM_CONFIG[platform_name]["skill_file"] - skill_src = Path(__file__).parent / skill_file - if not skill_src.exists(): - print(f"error: {skill_file} not found in package - reinstall graphify", file=sys.stderr) - sys.exit(1) - refs_src = _packaged_skill_refs_dir(platform_name) - if refs_src is not None and not refs_src.exists(): - # Progressive platform declared a references bundle that is missing from - # the package. Fail loud rather than silently shipping an empty sidecar. - print( - f"error: references for '{platform_name}' not found in package " - f"({refs_src}) - reinstall graphify", - file=sys.stderr, - ) - sys.exit(1) - skill_dst = _platform_skill_destination(platform_name, project=project, project_dir=project_dir) - skill_dst.parent.mkdir(parents=True, exist_ok=True) - # Install the references/ sidecar (or clear an orphan one) BEFORE writing - # SKILL.md, so SKILL.md is the last artifact laid down. An install that is - # interrupted partway then leaves no SKILL.md rather than a SKILL.md that - # points at an absent references/ dir. - if refs_src is not None: - _install_skill_references(skill_dst, refs_src) - print(f" references -> {skill_dst.parent / 'references'}") - else: - # Monolith (or progressive-with-no-refs): clear any orphan references/. - orphan_refs = skill_dst.parent / "references" - if orphan_refs.exists(): - shutil.rmtree(orphan_refs) - # SKILL.md last (crash-safety), via an atomic temp + rename. - tmp_dst = skill_dst.with_suffix(skill_dst.suffix + ".tmp") - try: - shutil.copy(skill_src, tmp_dst) - os.replace(tmp_dst, skill_dst) - except Exception: - try: - tmp_dst.unlink(missing_ok=True) - except OSError: - pass - raise - (skill_dst.parent / ".graphify_version").write_text(__version__, encoding="utf-8") - print(f" skill installed -> {skill_dst}") - return skill_dst -def _remove_skill_file(platform_name: str, *, project: bool = False, project_dir: Path | None = None) -> bool: - """Remove a platform skill file and its version stamp without touching other scopes.""" - skill_dst = _platform_skill_destination(platform_name, project=project, project_dir=project_dir) - removed = False - if skill_dst.exists(): - skill_dst.unlink() - print(f" skill removed -> {skill_dst}") - removed = True - version_file = skill_dst.parent / ".graphify_version" - if version_file.exists(): - version_file.unlink() - removed = True - refs_dir = skill_dst.parent / "references" - if refs_dir.exists(): - shutil.rmtree(refs_dir) - removed = True - for d in (skill_dst.parent, skill_dst.parent.parent, skill_dst.parent.parent.parent): - try: - d.rmdir() - except OSError: - break - return removed -def _project_scope_root(path: Path, project_dir: Path) -> Path: - """Return the top-level project artifact for a project-scoped skill path.""" - try: - rel = path.relative_to(project_dir) - except ValueError: - return path - return project_dir / rel.parts[0] if rel.parts else path -def _remove_claude_skill_registration(project_dir: Path) -> None: - """Remove the project-scoped Claude skill registration file/section.""" - claude_md = project_dir / ".claude" / "CLAUDE.md" - if not claude_md.exists(): - return - content = claude_md.read_text(encoding="utf-8") - if "# graphify" not in content: - return - cleaned = re.sub(r"\n*# graphify\n.*?(?=\n# |\Z)", "", content, flags=re.DOTALL).rstrip() - if cleaned: - claude_md.write_text(cleaned + "\n", encoding="utf-8") - print(f" CLAUDE.md -> graphify skill registration removed from {claude_md}") - else: - claude_md.unlink() - print(f" CLAUDE.md -> deleted {claude_md}") - - -def _print_project_git_add_hint(paths: list[Path]) -> None: - unique: list[str] = [] - for path in paths: - text = path.as_posix().rstrip("/") - if path.exists() and path.is_dir(): - text += "/" - if text not in unique: - unique.append(text) - if not unique: - return - print() - print("Project-scoped install. Add to version control:") - print(f" git add {' '.join(unique)}") + # PreToolUse nudge payloads, emitted verbatim by the shell-agnostic # `graphify hook-guard` subcommand (see _run_hook_guard). The previous hooks @@ -480,359 +319,20 @@ def _print_project_git_add_hint(paths: list[Path]) -> None: ) -def _claude_pretooluse_hooks() -> "list[dict]": - """graphify's Claude/Codebuddy PreToolUse hooks, resolved at install time. - The command invokes `graphify hook-guard ` via the absolute exe - path (`_resolve_graphify_exe`), so it parses under sh, cmd.exe and PowerShell - alike — this is the #522 fix, and mirrors the codex hook. Matchers stay "Bash" - and "Read|Glob" and the command always contains "graphify", so the existing - install/uninstall filters find and replace both old bash hooks and these. - """ - exe = _resolve_graphify_exe() - if " " in exe and not exe.startswith('"'): - exe = f'"{exe}"' - return [ - {"matcher": "Bash", - "hooks": [{"type": "command", "command": f"{exe} hook-guard search"}]}, - {"matcher": "Read|Glob", - "hooks": [{"type": "command", "command": f"{exe} hook-guard read"}]}, - ] - -def _skill_registration(skill_path: str = "~/.claude/skills/graphify/SKILL.md") -> str: - return ( - "\n# graphify\n" - f"- **graphify** (`{skill_path}`) " - "- any input to knowledge graph. Trigger: `/graphify`\n" - "When the user types `/graphify`, use the installed graphify skill " - "or instructions before doing anything else.\n" - ) - - -_PLATFORM_CONFIG: dict[str, dict] = { - "claude": { - "skill_file": "skill.md", - "skill_dst": Path(".claude") / "skills" / "graphify" / "SKILL.md", - "claude_md": True, - "skill_refs": "claude", - }, - "codex": { - "skill_file": "skill-codex.md", - "skill_dst": Path(".codex") / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "codex", - }, - "opencode": { - "skill_file": "skill-opencode.md", - "skill_dst": Path(".config") / "opencode" / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "opencode", - }, - "kilo": { - "skill_file": "skill-kilo.md", - "skill_dst": Path(".config") / "kilo" / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "kilo", - }, - "aider": { - # Monolith: aider ships the full SKILL.md inline, no references/ sidecar. - "skill_file": "skill-aider.md", - "skill_dst": Path(".aider") / "graphify" / "SKILL.md", - "claude_md": False, - }, - "copilot": { - "skill_file": "skill-copilot.md", - "skill_dst": Path(".copilot") / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "copilot", - }, - "claw": { - "skill_file": "skill-claw.md", - "skill_dst": Path(".openclaw") / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "claw", - }, - "droid": { - "skill_file": "skill-droid.md", - "skill_dst": Path(".factory") / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "droid", - }, - "trae": { - "skill_file": "skill-trae.md", - "skill_dst": Path(".trae") / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "trae", - }, - "trae-cn": { - # Reuses trae's split bundle (same skill body + references). - "skill_file": "skill-trae.md", - "skill_dst": Path(".trae-cn") / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "trae", - }, - "hermes": { - # Reuses claw's split bundle. - "skill_file": "skill-claw.md", - "skill_dst": Path(".hermes") / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "claw", - }, - "kiro": { - "skill_file": "skill-kiro.md", - "skill_dst": Path(".kiro") / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "kiro", - }, - "pi": { - "skill_file": "skill-pi.md", - "skill_dst": Path(".pi") / "agent" / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "pi", - }, - "codebuddy": { - # Reuses claude's split bundle (shares skill.md). - "skill_file": "skill.md", - "skill_dst": Path(".codebuddy") / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "claude", - }, - "antigravity": { - # Rides claude's split bundle (shares skill.md). - "skill_file": "skill.md", - "skill_dst": Path(".agents") / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "claude", - }, - "antigravity-windows": { - # Rides windows' split bundle. - "skill_file": "skill-windows.md", - "skill_dst": Path(".agents") / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "windows", - }, - "windows": { - "skill_file": "skill-windows.md", - "skill_dst": Path(".claude") / "skills" / "graphify" / "SKILL.md", - "claude_md": True, - "skill_refs": "windows", - }, - "kimi": { - # Reuses claude's split bundle (shares skill.md). - "skill_file": "skill.md", - "skill_dst": Path(".kimi") / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "claude", - }, - "amp": { - # Amp searches .agents/skills (project) and ~/.config/agents/skills (user), - # not .amp/skills. The user-scope path is set in _platform_skill_destination. - "skill_file": "skill-amp.md", - "skill_dst": Path(".agents") / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "amp", - }, - "agents": { - # The generic cross-framework Agent-Skills target. Global: ~/.agents/skills - # (the spec's user-global location, read by `npx skills` and compliant - # frameworks); project: ./.agents/skills. The CLI accepts `skills` as an - # alias (see _canonical_platform). Ships its own rendered bundle. - "skill_file": "skill-agents.md", - "skill_dst": Path(".agents") / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - "skill_refs": "agents", - }, - "devin": { - # Monolith: devin ships the full SKILL.md inline, no references/ sidecar. - "skill_file": "skill-devin.md", - # User scope: ~/.config/devin/skills/graphify/SKILL.md - # Project scope: .devin/skills/graphify/SKILL.md (overridden in _platform_skill_destination) - "skill_dst": Path(".config") / "devin" / "skills" / "graphify" / "SKILL.md", - "claude_md": False, - }, -} -# CLI-only platform aliases, resolved to a real _PLATFORM_CONFIG key before -# dispatch. `skills` is the friendly alias for the generic `agents` platform -# (the Agent-Skills ecosystem calls them "skills"). -_PLATFORM_ALIASES: dict[str, str] = {"skills": "agents"} -def _canonical_platform(platform_name: str) -> str: - """Resolve a CLI platform alias to its real _PLATFORM_CONFIG key.""" - return _PLATFORM_ALIASES.get(platform_name, platform_name) -def _replace_or_append_section(content: str, marker: str, new_section: str) -> str: - """Idempotently update or append a graphify-owned section in shared files. - If no line is exactly ``marker`` (the heading, at column 0), append - ``new_section`` to the end (with a blank-line separator if there's existing - content). - If a real ``marker`` heading exists, replace the existing section in place. - The section runs from that heading to the line before the next H2 heading - (``## `` at line start), or to EOF if no later H2 exists. This lets older - installs receive the updated copy without users having to uninstall and - reinstall (issue #580). - The heading is matched only when a line *is* exactly ``marker`` (after - stripping surrounding whitespace), never as a substring. Matching ``## - graphify`` inside a bullet or an inline reference used to anchor the replace - on that mention and delete every line from there to the next heading, - silently destroying hand-curated content (#1688). When several exact - headings exist, the last one is used, since graphify's section is appended. - """ - lines = content.split("\n") - starts = [i for i, line in enumerate(lines) if line.strip() == marker] - if not starts: - if content.strip(): - return content.rstrip() + "\n\n" + new_section.lstrip() - return new_section.lstrip() - - start = starts[-1] - end = len(lines) - for j in range(start + 1, len(lines)): - if lines[j].startswith("## "): - end = j - break - - head = "\n".join(lines[:start]).rstrip() - tail = "\n".join(lines[end:]).lstrip() - section = new_section.strip() - - parts: list[str] = [] - if head: - parts.append(head) - parts.append(section) - if tail: - parts.append(tail) - out = "\n\n".join(parts) - if not out.endswith("\n"): - out += "\n" - return out - - -def _print_banner() -> None: - """Amber brain banner on graphify install. TTY-only, never raises.""" - if not sys.stdout.isatty(): - return - try: - if sys.platform == "win32": - import ctypes - ctypes.windll.kernel32.SetConsoleMode( - ctypes.windll.kernel32.GetStdHandle(-11), 7 - ) - A = "\033[38;5;214m" - D = "\033[38;5;130m" - R = "\033[0m" - print(f"""{A} - ╭──◉──╮ ╭──◉──╮ - ╱ ◉ ◉ ╲ ╱ ◉ ◉ ╲ -│ ◉─◉─◉ ◉ ◉─◉─◉ │ -│ ◉ ◉ │ ◉ ◉ │ -│ ◉─◉─◉ ◉ ◉─◉─◉ │ - ╲ ◉ ◉ ╱ ╲ ◉ ◉ ╱ - ╰──◉──╯ ╰──◉──╯ - ◉ - - █▀▀ █▀█ ▄▀█ █▀█ █ █ █ █▀▀ █▄█ - █▄█ █▀▄ █▀█ █▀▀ █▀█ █ █▀ █{D} {__version__}{R} -""") - except Exception: - pass -def install(platform: str = "claude", *, project: bool = False, project_dir: Path | None = None) -> None: - _print_banner() - platform = _canonical_platform(platform) - if platform == "gemini": - gemini_install(project_dir=project_dir, project=project) - return - if platform == "cursor": - _cursor_install(Path(".")) - return - # On Windows, antigravity needs the PowerShell skill, not the bash one - if platform == "antigravity" and sys.platform == "win32": - platform = "antigravity-windows" - if platform not in _PLATFORM_CONFIG: - print( - f"error: unknown platform '{platform}'. Choose from: {', '.join(_PLATFORM_CONFIG)}, gemini, cursor", - file=sys.stderr, - ) - sys.exit(1) - cfg = _PLATFORM_CONFIG[platform] - project_dir = project_dir or Path(".") - skill_dst = _copy_skill_file(platform, project=project, project_dir=project_dir) - if platform == "kilo": - # Kilo Code also supports a native /graphify command file. - command_src = Path(__file__).parent / "command-kilo.md" - if not command_src.exists(): - print( - f"error: command-kilo.md not found in package - reinstall graphify", - file=sys.stderr, - ) - sys.exit(1) - command_dst = Path.home() / ".config" / "kilo" / "command" / "graphify.md" - command_dst.parent.mkdir(parents=True, exist_ok=True) - shutil.copy(command_src, command_dst) - print(f" command installed -> {command_dst}") - - if cfg["claude_md"]: - # Register in the matching Claude Code scope. - claude_md = (project_dir / ".claude" / "CLAUDE.md") if project else Path.home() / ".claude" / "CLAUDE.md" - registration = _skill_registration(".claude/skills/graphify/SKILL.md" if project else "~/.claude/skills/graphify/SKILL.md") - if claude_md.exists(): - content = claude_md.read_text(encoding="utf-8") - if "graphify" in content: - print(f" CLAUDE.md -> already registered (no change)") - else: - claude_md.write_text(content.rstrip() + registration, encoding="utf-8") - print(f" CLAUDE.md -> skill registered in {claude_md}") - else: - claude_md.parent.mkdir(parents=True, exist_ok=True) - claude_md.write_text(registration.lstrip(), encoding="utf-8") - print(f" CLAUDE.md -> created at {claude_md}") - - if platform == "codebuddy": - # Register in ~/.codebuddy/CODEBUDDY.md (CodeBuddy only) - codebuddy_md = Path.home() / ".codebuddy" / "CODEBUDDY.md" - registration = _skill_registration("~/.codebuddy/skills/graphify/SKILL.md") - if codebuddy_md.exists(): - content = codebuddy_md.read_text(encoding="utf-8") - if "graphify" in content: - print(f" CODEBUDDY.md -> already registered (no change)") - else: - codebuddy_md.write_text(content.rstrip() + registration, encoding="utf-8") - print(f" CODEBUDDY.md -> skill registered in {codebuddy_md}") - else: - codebuddy_md.parent.mkdir(parents=True, exist_ok=True) - codebuddy_md.write_text(registration.lstrip(), encoding="utf-8") - print(f" CODEBUDDY.md -> created at {codebuddy_md}") - if platform == "opencode": - _install_opencode_plugin(project_dir if project else Path(".")) - - # Refresh version stamps in all other previously-installed skill dirs so - # stale-version warnings don't fire for platforms not explicitly re-installed. - if project: - _print_project_git_add_hint([_project_scope_root(skill_dst, project_dir)]) - else: - _refresh_all_version_stamps() - - print() - print("Done. Open your AI coding assistant and type:") - print() - print(" /graphify .") - print() - - -def _print_install_usage() -> None: - platforms = ", ".join([*_PLATFORM_CONFIG, "gemini", "cursor"]) - print("Usage: graphify install [--project] [--platform P|P]") - print(f"Platforms: {platforms}") # The always-on instruction blocks are packaged markdown under graphify/always_on/, @@ -840,17 +340,13 @@ def _print_install_usage() -> None: # load keeps the install-string / issue-#580 contract byte-for-byte while letting # a human edit one fragment instead of a triple-quoted literal here. -_CLAUDE_MD_MARKER = "## graphify" -_CODEBUDDY_MD_MARKER = "## graphify" # AGENTS.md section for Codex, OpenCode, and OpenClaw. # All three platforms read AGENTS.md in the project root for persistent instructions. -_AGENTS_MD_MARKER = "## graphify" -_GEMINI_MD_MARKER = "## graphify" # Gemini CLI BeforeTool hook nudge text. The hook always returns # {"decision":"allow"} (never blocks a tool) and appends this as additionalContext @@ -867,729 +363,72 @@ def _print_install_usage() -> None: ) -def _gemini_hook() -> dict: - """Gemini CLI BeforeTool hook, resolved to a shell-agnostic `graphify` call.""" - exe = _resolve_graphify_exe() - if " " in exe and not exe.startswith('"'): - exe = f'"{exe}"' - return { - "matcher": "read_file|list_directory", - "hooks": [{"type": "command", "command": f"{exe} hook-guard gemini"}], - } - - -def gemini_install(project_dir: Path | None = None, *, project: bool = False) -> None: - """Copy skill file, write GEMINI.md section, and install BeforeTool hook.""" - project_dir = project_dir or Path(".") - skill_dst = _copy_skill_file("gemini", project=project, project_dir=project_dir) - target = project_dir / "GEMINI.md" - if target.exists(): - content = target.read_text(encoding="utf-8") - new_content = _replace_or_append_section( - content, _GEMINI_MD_MARKER, _always_on("gemini-md") - ) - else: - new_content = _always_on("gemini-md") - if target.exists() and new_content == target.read_text(encoding="utf-8"): - print(f"graphify already configured in {target.resolve()} (no change)") - else: - target.write_text(new_content, encoding="utf-8") - print(f"graphify section written to {target.resolve()}") - - # Always re-install the Gemini hook so an older payload (e.g. pre-issue-#580 - # wording) is replaced on upgrade. - _install_gemini_hook(project_dir) - if project: - _print_project_git_add_hint([_project_scope_root(skill_dst, project_dir), project_dir / "GEMINI.md", project_dir / ".gemini"]) - print() - print("Gemini CLI will now check the knowledge graph before answering") - print("codebase questions and rebuild it after code changes.") - - -def _install_gemini_hook(project_dir: Path) -> None: - settings_path = project_dir / ".gemini" / "settings.json" - settings_path.parent.mkdir(parents=True, exist_ok=True) - try: - settings = ( - json.loads(settings_path.read_text(encoding="utf-8")) - if settings_path.exists() - else {} - ) - except json.JSONDecodeError: - settings = {} - before_tool = settings.setdefault("hooks", {}).setdefault("BeforeTool", []) - settings["hooks"]["BeforeTool"] = [ - h for h in before_tool if "graphify" not in str(h) - ] - settings["hooks"]["BeforeTool"].append(_gemini_hook()) - settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") - print(" .gemini/settings.json -> BeforeTool hook registered") - - -def _uninstall_gemini_hook(project_dir: Path) -> None: - settings_path = project_dir / ".gemini" / "settings.json" - if not settings_path.exists(): - return - try: - settings = json.loads(settings_path.read_text(encoding="utf-8")) - except json.JSONDecodeError: - return - before_tool = settings.get("hooks", {}).get("BeforeTool", []) - filtered = [h for h in before_tool if "graphify" not in str(h)] - if len(filtered) == len(before_tool): - return - settings["hooks"]["BeforeTool"] = filtered - settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") - print(" .gemini/settings.json -> BeforeTool hook removed") -def gemini_uninstall(project_dir: Path | None = None, *, project: bool = False) -> None: - """Remove the graphify section from GEMINI.md, uninstall hook, and remove skill file.""" - project_dir = project_dir or Path(".") - _remove_skill_file("gemini", project=project, project_dir=project_dir) - target = project_dir / "GEMINI.md" - if not target.exists(): - print("No GEMINI.md found in current directory - nothing to do") - return - content = target.read_text(encoding="utf-8") - if _GEMINI_MD_MARKER not in content: - print("graphify section not found in GEMINI.md - nothing to do") - return - cleaned = re.sub( - r"\n*## graphify\n.*?(?=\n## |\Z)", "", content, flags=re.DOTALL - ).rstrip() - if cleaned: - target.write_text(cleaned + "\n", encoding="utf-8") - print(f"graphify section removed from {target.resolve()}") - else: - target.unlink() - print(f"GEMINI.md was empty after removal - deleted {target.resolve()}") - _uninstall_gemini_hook(project_dir) -_VSCODE_INSTRUCTIONS_MARKER = "## graphify" -def vscode_install(project_dir: Path | None = None) -> None: - """Install graphify skill for VS Code Copilot Chat + write .github/copilot-instructions.md.""" - skill_src = Path(__file__).parent / "skill-vscode.md" - refs_bundle = "vscode" - if not skill_src.exists(): - skill_src = Path(__file__).parent / "skill-copilot.md" - refs_bundle = "copilot" - skill_dst = Path.home() / ".copilot" / "skills" / "graphify" / "SKILL.md" - skill_dst.parent.mkdir(parents=True, exist_ok=True) - tmp_dst = skill_dst.with_suffix(skill_dst.suffix + ".tmp") - try: - shutil.copy(skill_src, tmp_dst) - os.replace(tmp_dst, skill_dst) - except Exception: - try: - tmp_dst.unlink(missing_ok=True) - except OSError: - pass - raise - # Progressive-capable: install the packaged references/ sidecar when present. - refs_src = Path(__file__).parent / "skills" / refs_bundle / "references" - if refs_src.exists(): - _install_skill_references(skill_dst, refs_src) - print(f" references -> {skill_dst.parent / 'references'}") - else: - orphan_refs = skill_dst.parent / "references" - if orphan_refs.exists(): - shutil.rmtree(orphan_refs) - (skill_dst.parent / ".graphify_version").write_text(__version__, encoding="utf-8") - print(f" skill installed -> {skill_dst}") - - instructions = (project_dir or Path(".")) / ".github" / "copilot-instructions.md" - instructions.parent.mkdir(parents=True, exist_ok=True) - if instructions.exists(): - content = instructions.read_text(encoding="utf-8") - new_content = _replace_or_append_section( - content, _VSCODE_INSTRUCTIONS_MARKER, _always_on("vscode-instructions") - ) - if new_content == content: - print(f" {instructions} -> already configured (no change)") - else: - instructions.write_text(new_content, encoding="utf-8") - print(f" {instructions} -> graphify section {'updated' if _VSCODE_INSTRUCTIONS_MARKER in content else 'added'}") - else: - instructions.write_text(_always_on("vscode-instructions"), encoding="utf-8") - print(f" {instructions} -> created") - - print() - print( - "VS Code Copilot Chat configured. Type /graphify in the chat panel to build the graph." - ) - print("Note: for GitHub Copilot CLI (terminal), use: graphify copilot install") - - -def vscode_uninstall(project_dir: Path | None = None) -> None: - """Remove graphify VS Code Copilot Chat skill and .github/copilot-instructions.md section.""" - skill_dst = Path.home() / ".copilot" / "skills" / "graphify" / "SKILL.md" - if skill_dst.exists(): - skill_dst.unlink() - print(f" skill removed -> {skill_dst}") - version_file = skill_dst.parent / ".graphify_version" - if version_file.exists(): - version_file.unlink() - refs_dir = skill_dst.parent / "references" - if refs_dir.exists(): - shutil.rmtree(refs_dir) - for d in ( - skill_dst.parent, - skill_dst.parent.parent, - skill_dst.parent.parent.parent, - ): - try: - d.rmdir() - except OSError: - break - instructions = (project_dir or Path(".")) / ".github" / "copilot-instructions.md" - if not instructions.exists(): - return - content = instructions.read_text(encoding="utf-8") - if _VSCODE_INSTRUCTIONS_MARKER not in content: - return - cleaned = re.sub( - r"\n*## graphify\n.*?(?=\n## |\Z)", "", content, flags=re.DOTALL - ).rstrip() - if cleaned: - instructions.write_text(cleaned + "\n", encoding="utf-8") - print(f" graphify section removed from {instructions}") - else: - instructions.unlink() - print(f" {instructions} -> deleted (was empty after removal)") -_ANTIGRAVITY_RULES_PATH = Path(".agents") / "rules" / "graphify.md" -_ANTIGRAVITY_WORKFLOW_PATH = Path(".agents") / "workflows" / "graphify.md" -_ANTIGRAVITY_WORKFLOW = """\ ---- -name: graphify -description: Turn any folder of files into a navigable knowledge graph ---- -# Workflow: graphify -Follow the graphify skill installed at ~/.gemini/config/skills/graphify/SKILL.md to run the full pipeline. -If no path argument is given, use `.` (current directory). -""" _KIRO_STEERING_MARKER = "graphify: A knowledge graph of this project" -def _kiro_install(project_dir: Path) -> None: - """Write graphify skill + steering file for Kiro IDE/CLI.""" - project_dir = project_dir or Path(".") - # Skill file + references/ sidecar + .graphify_version stamp via the shared - # progressive-disclosure helper. Previously this used a bare write_text that - # bypassed _copy_skill_file, so the references/ dir and version stamp were - # never written even though kiro declares skill_refs: "kiro" (#1142). - _copy_skill_file("kiro", project=True, project_dir=project_dir) - # Steering file → .kiro/steering/graphify.md (always-on) - steering_dir = project_dir / ".kiro" / "steering" - steering_dir.mkdir(parents=True, exist_ok=True) - steering_dst = steering_dir / "graphify.md" - if steering_dst.exists() and steering_dst.read_text(encoding="utf-8") == _always_on("kiro-steering"): - print(f" .kiro/steering/graphify.md -> already configured (no change)") - else: - # File is wholly graphify-owned. Overwrite on upgrade so older - # report-first wording does not silently linger (issue #580). - action = "updated" if steering_dst.exists() else "written" - steering_dst.write_text(_always_on("kiro-steering"), encoding="utf-8") - print(f" .kiro/steering/graphify.md -> always-on steering {action}") - print() - print("Kiro will now read the knowledge graph before every conversation.") - print("Use /graphify to build or update the graph.") -def _kiro_uninstall(project_dir: Path) -> None: - """Remove graphify skill + steering file for Kiro.""" - project_dir = project_dir or Path(".") - removed = [] - # Skill + .graphify_version + references/ sidecar + empty-dir walk. - skill_dst = _platform_skill_destination("kiro", project=True, project_dir=project_dir) - if _remove_skill_file("kiro", project=True, project_dir=project_dir): - removed.append(str(skill_dst.relative_to(project_dir))) - steering_dst = project_dir / ".kiro" / "steering" / "graphify.md" - if steering_dst.exists(): - steering_dst.unlink() - removed.append(str(steering_dst.relative_to(project_dir))) - print("Removed: " + (", ".join(removed) if removed else "nothing to remove")) -def _antigravity_finalize(skill_dst: Path, project_dir: Path) -> None: - """Write Antigravity's always-on layer next to an installed skill. - Injects the native tool-discovery YAML frontmatter into *skill_dst*, then - writes ``.agents/rules/graphify.md`` and ``.agents/workflows/graphify.md`` - under *project_dir*. Shared by the global ``antigravity install`` and the - project-scoped ``install --project --platform antigravity`` paths, so both lay - down the rules/workflows that the uninstall path already expects to remove. - """ - # Inject YAML frontmatter for native Antigravity tool discovery. - if skill_dst.exists(): - content = skill_dst.read_text(encoding="utf-8") - if not content.startswith("---\n"): - frontmatter = "---\nname: graphify-manager\ndescription: Rebuild the code graph or perform manual CLI queries when MCP server is offline.\n---\n\n" - skill_dst.write_text(frontmatter + content, encoding="utf-8") - - # .agents/rules/graphify.md - rules_path = project_dir / _ANTIGRAVITY_RULES_PATH - rules_path.parent.mkdir(parents=True, exist_ok=True) - if rules_path.exists(): - existing = rules_path.read_text(encoding="utf-8") - if _always_on("antigravity-rules").strip() != existing.strip(): - rules_path.write_text(_always_on("antigravity-rules"), encoding="utf-8") - print(f"graphify rule updated at {rules_path.resolve()}") - else: - print(f"graphify rule already configured at {rules_path.resolve()} (no change)") - else: - rules_path.write_text(_always_on("antigravity-rules"), encoding="utf-8") - print(f"graphify rule written to {rules_path.resolve()}") - - # .agents/workflows/graphify.md - wf_path = project_dir / _ANTIGRAVITY_WORKFLOW_PATH - wf_path.parent.mkdir(parents=True, exist_ok=True) - if wf_path.exists(): - existing = wf_path.read_text(encoding="utf-8") - if _ANTIGRAVITY_WORKFLOW.strip() != existing.strip(): - wf_path.write_text(_ANTIGRAVITY_WORKFLOW, encoding="utf-8") - print(f"graphify workflow updated at {wf_path.resolve()}") - else: - print(f"graphify workflow already configured at {wf_path.resolve()} (no change)") - else: - wf_path.write_text(_ANTIGRAVITY_WORKFLOW, encoding="utf-8") - print(f"graphify workflow written to {wf_path.resolve()}") - - -def _antigravity_install(project_dir: Path) -> None: - """Install graphify for Google Antigravity (global skill + .agents/rules + .agents/workflows).""" - # Copy the skill to ~/.gemini/config/skills/graphify/SKILL.md (global), then - # lay down the always-on rules/workflows under the project dir. - install(platform="antigravity") - _antigravity_finalize(_platform_skill_destination("antigravity"), project_dir) - - print() - print("Antigravity will now check the knowledge graph before answering") - print("codebase questions. Run /graphify first to build the graph.") - print() - print( - "To enable full MCP architecture navigation, add this to ~/.gemini/antigravity/mcp_config.json:" - ) - print(' "graphify": {') - print(' "command": "uv",') - print( - ' "args": ["run", "--with", "graphifyy", "--with", "mcp", "-m", "graphify.serve", "${workspace.path}/graphify-out/graph.json"]' - ) - print(" }") - - -def _antigravity_uninstall(project_dir: Path, *, project: bool = False) -> None: - """Remove graphify Antigravity rules, workflow, and skill files.""" - # Remove rules file - rules_path = project_dir / _ANTIGRAVITY_RULES_PATH - if rules_path.exists(): - rules_path.unlink() - print(f"graphify rule removed from {rules_path.resolve()}") - else: - print("No graphify Antigravity rule found - nothing to do") - - # Remove workflow file - wf_path = project_dir / _ANTIGRAVITY_WORKFLOW_PATH - if wf_path.exists(): - wf_path.unlink() - print(f"graphify workflow removed from {wf_path.resolve()}") - - # Remove skill file - skill_dst = _platform_skill_destination("antigravity", project=project, project_dir=project_dir) - if skill_dst.exists(): - skill_dst.unlink() - print(f"graphify skill removed from {skill_dst}") - version_file = skill_dst.parent / ".graphify_version" - if version_file.exists(): - version_file.unlink() - refs_dir = skill_dst.parent / "references" - if refs_dir.exists(): - shutil.rmtree(refs_dir) - for d in ( - skill_dst.parent, - skill_dst.parent.parent, - skill_dst.parent.parent.parent, - ): - try: - d.rmdir() - except OSError: - break -_CURSOR_RULE_PATH = Path(".cursor") / "rules" / "graphify.mdc" -_CURSOR_RULE = """\ ---- -description: graphify knowledge graph context -alwaysApply: true ---- -This project has a graphify knowledge graph at graphify-out/. -**MANDATORY: Before using Read, Grep, Glob, or Bash to explore the codebase, you MUST run graphify first:** -- `graphify query ""` — scoped subgraph for any codebase or architecture question -- `graphify path "" ""` — dependency path between two symbols -- `graphify explain ""` — all nodes related to a concept -This applies to YOU and to every subagent you spawn. Include this rule explicitly in every subagent prompt that involves code exploration. Do not skip graphify because files are "already known" or because you are executing a plan — the graph surfaces cross-file dependencies and INFERRED edges that grep and Read cannot find. -Only use Read/Grep/Glob directly when: -1. graphify has already oriented you and you need to modify or debug specific lines -2. `graphify-out/graph.json` does not exist yet -- If `graphify-out/wiki/index.md` exists, navigate it instead of reading raw files -- Read `graphify-out/GRAPH_REPORT.md` only for broad architecture review when query/path/explain do not surface enough context -- After modifying code files, run `graphify update .` to keep the graph current (AST-only, no API cost) -""" -def _cursor_install(project_dir: Path) -> None: - """Write .cursor/rules/graphify.mdc with alwaysApply: true.""" - rule_path = (project_dir or Path(".")) / _CURSOR_RULE_PATH - rule_path.parent.mkdir(parents=True, exist_ok=True) - if rule_path.exists() and rule_path.read_text(encoding="utf-8") == _CURSOR_RULE: - print(f"graphify rule at {rule_path} already configured (no change)") - return - # File is wholly graphify-owned. Overwrite on upgrade so older - # report-first wording does not silently linger (issue #580). - action = "updated" if rule_path.exists() else "written" - rule_path.write_text(_CURSOR_RULE, encoding="utf-8") - print(f"graphify rule {action} at {rule_path.resolve()}") - print() - print("Cursor will now always include the knowledge graph context.") - print("Run /graphify . first to build the graph if you haven't already.") - - -def _cursor_uninstall(project_dir: Path) -> None: - """Remove .cursor/rules/graphify.mdc.""" - rule_path = (project_dir or Path(".")) / _CURSOR_RULE_PATH - if not rule_path.exists(): - print("No graphify Cursor rule found - nothing to do") - return - rule_path.unlink() - print(f"graphify Cursor rule removed from {rule_path.resolve()}") -# Devin CLI — .windsurf/rules/graphify.md (always-on context) -# Devin reads .windsurf/rules/*.md files the same way Windsurf IDE does. -_DEVIN_RULES_PATH = Path(".windsurf") / "rules" / "graphify.md" -_DEVIN_RULES = """\ -## graphify -This project has a graphify knowledge graph at graphify-out/. -Rules: -- For codebase or architecture questions, when `graphify-out/graph.json` exists, first run `graphify query ""` (or `graphify path "" ""` / `graphify explain ""`). These return a scoped subgraph, usually much smaller than `GRAPH_REPORT.md` or raw grep output. -- If graphify-out/wiki/index.md exists, navigate it instead of reading raw files -- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain do not surface enough context -- After modifying code files in this session, run `graphify update .` to keep the graph current (AST-only, no API cost) -""" -def _devin_rules_install(project_dir: Path) -> None: - """Write .windsurf/rules/graphify.md for always-on Devin context.""" - rules_path = (project_dir or Path(".")) / _DEVIN_RULES_PATH - rules_path.parent.mkdir(parents=True, exist_ok=True) - if rules_path.exists() and rules_path.read_text(encoding="utf-8") == _DEVIN_RULES: - print(f" {rules_path} -> already configured (no change)") - return - action = "updated" if rules_path.exists() else "written" - rules_path.write_text(_DEVIN_RULES, encoding="utf-8") - print(f" rules {action} -> {rules_path}") -def _devin_rules_uninstall(project_dir: Path) -> None: - """Remove .windsurf/rules/graphify.md.""" - rules_path = (project_dir or Path(".")) / _DEVIN_RULES_PATH - if not rules_path.exists(): - return - rules_path.unlink() - print(f" rules removed -> {rules_path}") - - -_KILO_PLUGIN_JS = """\ -// graphify Kilo plugin -// Injects a knowledge graph reminder before bash tool calls when the graph exists. -import { existsSync } from "fs"; -import { join } from "path"; - -export const GraphifyPlugin = async ({ directory }) => { - let reminded = false; - - return { - "tool.execute.before": async (input, output) => { - if (reminded) return; - if (!existsSync(join(directory, "graphify-out", "graph.json"))) return; - - if (input.tool === "bash") { - // Separate with ';' not '&&' — Windows PowerShell 5.1 rejects '&&' as a - // statement separator ("not a valid statement separator"), which broke - // the first bash command in every OpenCode session on Windows (#1646). - // ';' works in PowerShell 5.1, Bash, and POSIX shells alike. - output.args.command = - 'echo "[graphify] Knowledge graph available. Read graphify-out/GRAPH_REPORT.md for god nodes and architecture context before searching files." ; ' + - output.args.command; - reminded = true; - } - }, - }; -}; -""" - -_KILO_PLUGIN_PATH = Path(".kilo") / "plugins" / "graphify.js" -_KILO_CONFIG_JSON_PATH = Path(".kilo") / "kilo.json" -_KILO_CONFIG_JSONC_PATH = Path(".kilo") / "kilo.jsonc" - - -def _strip_json_comments(raw: str) -> str: - """Remove JSONC-style comments while leaving string content intact.""" - result: list[str] = [] - in_string = False - escaped = False - line_comment = False - block_comment = False - i = 0 - - while i < len(raw): - ch = raw[i] - nxt = raw[i + 1] if i + 1 < len(raw) else "" - - if line_comment: - if ch == "\n": - line_comment = False - result.append(ch) - i += 1 - continue - if block_comment: - if ch == "*" and nxt == "/": - block_comment = False - i += 2 - else: - i += 1 - continue - - if in_string: - result.append(ch) - if escaped: - escaped = False - elif ch == "\\": - escaped = True - elif ch == '"': - in_string = False - i += 1 - continue - if ch == "/" and nxt == "/": - line_comment = True - i += 2 - continue - if ch == "/" and nxt == "*": - block_comment = True - i += 2 - continue - result.append(ch) - if ch == '"': - in_string = True - i += 1 - return re.sub(r",(\s*[}\]])", r"\1", "".join(result)) -def _load_json_like(config_file: Path) -> dict: - if not config_file.exists(): - return {} - try: - raw = config_file.read_text(encoding="utf-8") - if config_file.suffix == ".jsonc": - raw = _strip_json_comments(raw) - loaded = json.loads(raw) - except (OSError, json.JSONDecodeError): - return {} - return loaded if isinstance(loaded, dict) else {} - - -def _kilo_config_path(project_dir: Path) -> Path: - kilo_dir = (project_dir or Path(".")) / ".kilo" - json_path = kilo_dir / _KILO_CONFIG_JSON_PATH.name - if json_path.exists(): - return json_path - jsonc_path = kilo_dir / _KILO_CONFIG_JSONC_PATH.name - if jsonc_path.exists(): - return jsonc_path - return json_path - - -def _kilo_config_write_path(project_dir: Path) -> Path: - """Write automated Kilo edits to kilo.json so existing JSONC stays untouched.""" - kilo_dir = (project_dir or Path(".")) / ".kilo" - return kilo_dir / _KILO_CONFIG_JSON_PATH.name - - -def _install_kilo_plugin(project_dir: Path) -> None: - """Write graphify.js plugin and register it without rewriting user JSONC.""" - plugin_file = project_dir / _KILO_PLUGIN_PATH - plugin_file.parent.mkdir(parents=True, exist_ok=True) - plugin_file.write_text(_KILO_PLUGIN_JS, encoding="utf-8") - print(f" {_KILO_PLUGIN_PATH} -> tool.execute.before hook written") - - config_file = _kilo_config_path(project_dir) - write_config_file = _kilo_config_write_path(project_dir) - write_config_file.parent.mkdir(parents=True, exist_ok=True) - config = _load_json_like(config_file) - plugins = config.get("plugin") - if not isinstance(plugins, list): - plugins = [] - config["plugin"] = plugins - entry = plugin_file.resolve().as_uri() - if entry not in plugins: - plugins.append(entry) - write_config_file.write_text(json.dumps(config, indent=2), encoding="utf-8") - print(f" {write_config_file.relative_to(project_dir)} -> plugin registered") - else: - print( - f" {config_file.relative_to(project_dir)} -> plugin already registered (no change)" - ) -def _uninstall_kilo_plugin(project_dir: Path) -> None: - """Remove graphify.js plugin and deregister it without rewriting user JSONC.""" - plugin_file = project_dir / _KILO_PLUGIN_PATH - if plugin_file.exists(): - plugin_file.unlink() - print(f" {_KILO_PLUGIN_PATH} -> removed") - config_file = _kilo_config_path(project_dir) - if not config_file.exists(): - return - write_config_file = _kilo_config_write_path(project_dir) - config = _load_json_like(config_file) - plugins = config.get("plugin", []) - if not isinstance(plugins, list): - plugins = [] - entry = plugin_file.resolve().as_uri() - if entry in plugins: - config["plugin"] = [plugin for plugin in plugins if plugin != entry] - if not config["plugin"]: - config.pop("plugin") - write_config_file.parent.mkdir(parents=True, exist_ok=True) - write_config_file.write_text(json.dumps(config, indent=2), encoding="utf-8") - print( - f" {write_config_file.relative_to(project_dir)} -> plugin deregistered" - ) -# OpenCode tool.execute.before plugin — fires before every tool call. -# Injects a graph reminder into bash command output when graph.json exists. -_OPENCODE_PLUGIN_JS = """\ -// graphify OpenCode plugin -// Injects a knowledge graph reminder before bash tool calls when the graph exists. -// -// IMPORTANT: keep the reminder string free of backticks and $(...) constructs. -// The hook prepends `echo "" && ` to the user's bash command; -// backticks inside the double-quoted echo trigger bash command substitution, -// which both corrupts tool output and silently executes the very graphify -// command we are only suggesting. Plain words render fine in opencode's TUI. -import { existsSync } from "fs"; -import { join } from "path"; - -export const GraphifyPlugin = async ({ directory }) => { - let reminded = false; - - return { - "tool.execute.before": async (input, output) => { - if (reminded) return; - if (!existsSync(join(directory, "graphify-out", "graph.json"))) return; - - if (input.tool === "bash") { - // ';' not '&&' — Windows PowerShell 5.1 rejects '&&' as a statement - // separator, breaking the first bash command of the session (#1646). - output.args.command = - 'echo "[graphify] knowledge graph at graphify-out/. For focused questions, run graphify query with your question (scoped subgraph, usually much smaller than GRAPH_REPORT.md) instead of grepping raw files. Read GRAPH_REPORT.md only for broad architecture context." ; ' + - output.args.command; - reminded = true; - } - }, - }; -}; -""" - -_OPENCODE_PLUGIN_PATH = Path(".opencode") / "plugins" / "graphify.js" -_OPENCODE_CONFIG_PATH = Path(".opencode") / "opencode.json" - - -def _install_opencode_plugin(project_dir: Path) -> None: - """Write graphify.js plugin and register it in opencode.json.""" - plugin_file = project_dir / _OPENCODE_PLUGIN_PATH - plugin_file.parent.mkdir(parents=True, exist_ok=True) - plugin_file.write_text(_OPENCODE_PLUGIN_JS, encoding="utf-8") - print(f" {_OPENCODE_PLUGIN_PATH} -> tool.execute.before hook written") - - config_file = project_dir / _OPENCODE_CONFIG_PATH - if config_file.exists(): - try: - config = json.loads(config_file.read_text(encoding="utf-8")) - except json.JSONDecodeError: - config = {} - else: - config = {} - - plugins = config.setdefault("plugin", []) - entry = _OPENCODE_PLUGIN_PATH.as_posix() - if entry not in plugins: - plugins.append(entry) - config_file.write_text(json.dumps(config, indent=2), encoding="utf-8") - print(f" {_OPENCODE_CONFIG_PATH} -> plugin registered") - else: - print(f" {_OPENCODE_CONFIG_PATH} -> plugin already registered (no change)") -def _uninstall_opencode_plugin(project_dir: Path) -> None: - """Remove graphify.js plugin and deregister from opencode.json.""" - plugin_file = project_dir / _OPENCODE_PLUGIN_PATH - if plugin_file.exists(): - plugin_file.unlink() - print(f" {_OPENCODE_PLUGIN_PATH} -> removed") - config_file = project_dir / _OPENCODE_CONFIG_PATH - if not config_file.exists(): - return - try: - config = json.loads(config_file.read_text(encoding="utf-8")) - except json.JSONDecodeError: - return - plugins = config.get("plugin", []) - entry = _OPENCODE_PLUGIN_PATH.as_posix() - if entry in plugins: - plugins.remove(entry) - if not plugins: - config.pop("plugin") - config_file.write_text(json.dumps(config, indent=2), encoding="utf-8") - print(f" {_OPENCODE_CONFIG_PATH} -> plugin deregistered") _CODEX_HOOK = { @@ -1613,24 +452,6 @@ def _uninstall_opencode_plugin(project_dir: Path) -> None: } -def _resolve_graphify_exe() -> str: - """Return the absolute path to the graphify executable. - - Falls back to bare 'graphify' if resolution fails. Using an absolute path - ensures the hook works in environments where the venv Scripts/ directory is - not on PATH (e.g. VS Code Codex extension on Windows). - """ - import shutil - found = shutil.which("graphify") - if found: - return found - # Derive from sys.executable: same Scripts/ (Windows) or bin/ (Unix) dir - scripts_dir = Path(sys.executable).parent - for name in ("graphify.exe", "graphify"): - candidate = scripts_dir / name - if candidate.exists(): - return str(candidate) - return "graphify" def _run_hook_guard(kind: str) -> None: @@ -1690,604 +511,56 @@ def _run_hook_guard(kind: str) -> None: pass -def _install_codex_hook(project_dir: Path) -> None: - """Add graphify PreToolUse hook to .codex/hooks.json.""" - hooks_path = project_dir / ".codex" / "hooks.json" - hooks_path.parent.mkdir(parents=True, exist_ok=True) - - if hooks_path.exists(): - try: - existing = json.loads(hooks_path.read_text(encoding="utf-8")) - except json.JSONDecodeError: - existing = {} - else: - existing = {} - - graphify_exe = _resolve_graphify_exe() - hook_entry = { - "hooks": { - "PreToolUse": [ - { - "matcher": "Bash", - "hooks": [{"type": "command", "command": f"{graphify_exe} hook-check"}], - } - ] - } - } - - pre_tool = existing.setdefault("hooks", {}).setdefault("PreToolUse", []) - existing["hooks"]["PreToolUse"] = [h for h in pre_tool if "graphify" not in str(h)] - existing["hooks"]["PreToolUse"].extend(hook_entry["hooks"]["PreToolUse"]) - hooks_path.write_text(json.dumps(existing, indent=2), encoding="utf-8") - print(f" .codex/hooks.json -> PreToolUse hook registered ({graphify_exe} hook-check)") - - -def _uninstall_codex_hook(project_dir: Path) -> None: - """Remove graphify PreToolUse hook from .codex/hooks.json.""" - hooks_path = project_dir / ".codex" / "hooks.json" - if not hooks_path.exists(): - return - try: - existing = json.loads(hooks_path.read_text(encoding="utf-8")) - except json.JSONDecodeError: - return - pre_tool = existing.get("hooks", {}).get("PreToolUse", []) - filtered = [h for h in pre_tool if "graphify" not in str(h)] - existing["hooks"]["PreToolUse"] = filtered - hooks_path.write_text(json.dumps(existing, indent=2), encoding="utf-8") - print(f" .codex/hooks.json -> PreToolUse hook removed") - - -def _agents_install(project_dir: Path, platform: str) -> None: - """Write the graphify section to the local AGENTS.md for always-on platforms.""" - target = (project_dir or Path(".")) / "AGENTS.md" - - if target.exists(): - content = target.read_text(encoding="utf-8") - new_content = _replace_or_append_section( - content, _AGENTS_MD_MARKER, _always_on("agents-md") - ) - else: - new_content = _always_on("agents-md") - - if target.exists() and new_content == target.read_text(encoding="utf-8"): - print(f"graphify already configured in {target.resolve()} (no change)") - else: - target.write_text(new_content, encoding="utf-8") - print(f"graphify section written to {target.resolve()}") - - if platform == "codex": - _install_codex_hook(project_dir or Path(".")) - elif platform == "opencode": - _install_opencode_plugin(project_dir or Path(".")) - elif platform == "kilo": - _install_kilo_plugin(project_dir or Path(".")) - - print() - print( - f"{platform.capitalize()} will now check the knowledge graph before answering" - ) - print("codebase questions and rebuild it after code changes.") - if platform not in ("codex", "opencode", "kilo"): - print() - print("Note: unlike Claude Code, there is no PreToolUse hook equivalent for") - print( - f"{platform.capitalize()} — the AGENTS.md rules are the always-on mechanism." - ) - - -def _amp_legacy_cleanup() -> None: - """Best-effort removal of the pre-fix ~/.amp/skills/graphify install dir. - - Older graphify versions wrote the Amp skill to ~/.amp/skills, which Amp does - not search. Clean it up on install so a stale, never-loaded copy does not - linger. Failures are ignored (the new path is what matters). - """ - legacy = Path.home() / ".amp" / "skills" / "graphify" - if legacy.exists(): - shutil.rmtree(legacy, ignore_errors=True) - if not legacy.exists(): - print(f" legacy removed -> {legacy}") - - -def _amp_install(project_dir: Path | None = None) -> None: - """User-scope Amp install: skill into ~/.config/agents/skills + AGENTS.md.""" - _amp_legacy_cleanup() - _copy_skill_file("amp") - _agents_install(project_dir or Path("."), "amp") -def _amp_uninstall(project_dir: Path | None = None) -> None: - """User-scope Amp uninstall: remove the skill and the AGENTS.md section.""" - removed = _remove_skill_file("amp") - if removed: - print("skill removed") - _agents_uninstall(project_dir or Path("."), platform="amp") -def _agents_platform_install(project_dir: Path | None = None) -> None: - """`graphify agents install`: skill into ~/.agents/skills + AGENTS.md. - The amp-twin of the generic Agent-Skills target. Mirrors _amp_install but - lands the skill at the spec's user-global ~/.agents/skills (set in - _platform_skill_destination). Wiring AGENTS.md keeps it honest with the - rendered hooks reference, which points at `graphify agents install`. The bare - `graphify install --platform agents` path stays skill-only (via install()), - exactly as amp's `--platform amp` does. - """ - _copy_skill_file("agents") - _agents_install(project_dir or Path("."), "agents") - - -def _agents_platform_uninstall(project_dir: Path | None = None) -> None: - """`graphify agents uninstall`: remove the skill and the AGENTS.md section.""" - removed = _remove_skill_file("agents") - if removed: - print("skill removed") - _agents_uninstall(project_dir or Path("."), platform="agents") - - -def _project_install(platform_name: str, project_dir: Path | None = None) -> None: - """Install platform skill/config files in the current project.""" - project_dir = project_dir or Path(".") - platform_name = _canonical_platform(platform_name) - if platform_name in ("claude", "windows"): - install(platform=platform_name, project=True, project_dir=project_dir) - claude_install(project_dir) - _print_project_git_add_hint([project_dir / ".claude", project_dir / "CLAUDE.md"]) - elif platform_name == "gemini": - gemini_install(project_dir, project=True) - elif platform_name == "cursor": - _cursor_install(project_dir) - _print_project_git_add_hint([project_dir / ".cursor"]) - elif platform_name == "kiro": - _kiro_install(project_dir) - _print_project_git_add_hint([project_dir / ".kiro"]) - elif platform_name in ("aider", "amp", "codex", "opencode", "claw", "droid", "trae", "trae-cn", "hermes"): - skill_dst = _copy_skill_file(platform_name, project=True, project_dir=project_dir) - _agents_install(project_dir, platform_name) - hint_paths = [_project_scope_root(skill_dst, project_dir), project_dir / "AGENTS.md"] - if platform_name == "opencode": - hint_paths.append(project_dir / ".opencode") - elif platform_name == "codex": - hint_paths.append(project_dir / ".codex") - _print_project_git_add_hint(hint_paths) - elif platform_name == "devin": - skill_dst = _copy_skill_file("devin", project=True, project_dir=project_dir) - _devin_rules_install(project_dir) - _print_project_git_add_hint([_project_scope_root(skill_dst, project_dir), project_dir / ".windsurf"]) - elif platform_name == "antigravity": - # Project-scoped: skill in .agents/skills/ PLUS the .agents/rules + - # .agents/workflows always-on layer (previously this path wrote only the - # skill, leaving the rules/workflows the uninstall path removes unset). - skill_dst = _copy_skill_file("antigravity", project=True, project_dir=project_dir) - _antigravity_finalize(skill_dst, project_dir) - _print_project_git_add_hint([_project_scope_root(skill_dst, project_dir), project_dir / ".agents"]) - elif platform_name in ("copilot", "pi", "kimi", "agents"): - # Skill-only project install: drop SKILL.md (+ references) at the scope - # root. `agents` -> ./.agents/skills/graphify/SKILL.md. - skill_dst = _copy_skill_file(platform_name, project=True, project_dir=project_dir) - _print_project_git_add_hint([_project_scope_root(skill_dst, project_dir)]) - else: - install(platform=platform_name, project=True, project_dir=project_dir) - - -def _project_uninstall(platform_name: str, project_dir: Path | None = None) -> None: - """Remove project-scoped platform skill/config files only.""" - project_dir = project_dir or Path(".") - platform_name = _canonical_platform(platform_name) - if platform_name in ("claude", "windows"): - _remove_skill_file(platform_name, project=True, project_dir=project_dir) - _remove_claude_skill_registration(project_dir) - claude_uninstall(project_dir, project=True) - elif platform_name == "gemini": - gemini_uninstall(project_dir, project=True) - elif platform_name == "cursor": - _cursor_uninstall(project_dir) - elif platform_name == "kiro": - _kiro_uninstall(project_dir) - elif platform_name in ("aider", "amp", "codex", "opencode", "claw", "droid", "trae", "trae-cn", "hermes"): - _remove_skill_file(platform_name, project=True, project_dir=project_dir) - _agents_uninstall(project_dir, platform=platform_name) - if platform_name == "codex": - _uninstall_codex_hook(project_dir) - elif platform_name == "antigravity": - _antigravity_uninstall(project_dir, project=True) - elif platform_name == "devin": - removed = _remove_skill_file("devin", project=True, project_dir=project_dir) - _devin_rules_uninstall(project_dir) - if not removed: - print("nothing to remove") - elif platform_name in ("copilot", "pi", "kimi", "agents"): - removed = _remove_skill_file(platform_name, project=True, project_dir=project_dir) - if not removed: - print("nothing to remove") - elif platform_name == "codebuddy": - codebuddy_uninstall(project_dir) - else: - _remove_skill_file(platform_name, project=True, project_dir=project_dir) - - -def _project_uninstall_all(project_dir: Path | None = None) -> None: - """Remove project-scoped install files without touching user-scope installs.""" - project_dir = project_dir or Path(".") - print("Uninstalling project-scoped graphify files...\n") - for platform_name in _PLATFORM_CONFIG: - _project_uninstall(platform_name, project_dir) - for platform_name in ("gemini", "cursor"): - _project_uninstall(platform_name, project_dir) - print("\nDone.") - - -def _agents_uninstall(project_dir: Path, platform: str = "") -> None: - """Remove the graphify section from the local AGENTS.md.""" - target = (project_dir or Path(".")) / "AGENTS.md" - - if not target.exists(): - print("No AGENTS.md found in current directory - nothing to do") - if platform == "opencode": - _uninstall_opencode_plugin(project_dir or Path(".")) - elif platform == "kilo": - _uninstall_kilo_plugin(project_dir or Path(".")) - return - - content = target.read_text(encoding="utf-8") - if _AGENTS_MD_MARKER not in content: - print("graphify section not found in AGENTS.md - nothing to do") - if platform == "opencode": - _uninstall_opencode_plugin(project_dir or Path(".")) - elif platform == "kilo": - _uninstall_kilo_plugin(project_dir or Path(".")) - return - - cleaned = re.sub( - r"\n*## graphify\n.*?(?=\n## |\Z)", - "", - content, - flags=re.DOTALL, - ).rstrip() - if cleaned: - target.write_text(cleaned + "\n", encoding="utf-8") - print(f"graphify section removed from {target.resolve()}") - else: - target.unlink() - print(f"AGENTS.md was empty after removal - deleted {target.resolve()}") - - if platform == "opencode": - _uninstall_opencode_plugin(project_dir or Path(".")) - elif platform == "kilo": - _uninstall_kilo_plugin(project_dir or Path(".")) - - -def _kilo_uninstall_global() -> list[str]: - removed = [] - command_dst = Path.home() / ".config" / "kilo" / "command" / "graphify.md" - if command_dst.exists(): - command_dst.unlink() - removed.append(f"command removed: {command_dst}") - try: - command_dst.parent.rmdir() - except OSError: - pass - - skill_dst = Path.home() / _PLATFORM_CONFIG["kilo"]["skill_dst"] - if skill_dst.exists(): - skill_dst.unlink() - removed.append(f"skill removed: {skill_dst}") - version_file = skill_dst.parent / ".graphify_version" - if version_file.exists(): - version_file.unlink() - for d in ( - skill_dst.parent, - skill_dst.parent.parent, - skill_dst.parent.parent.parent, - ): - try: - d.rmdir() - except OSError: - break - - return removed -def _kilo_install(project_dir: Path) -> None: - """Install native Kilo skill + command globally and always-on project wiring locally.""" - install(platform="kilo") - _agents_install(project_dir or Path("."), "kilo") -def _kilo_uninstall(project_dir: Path) -> None: - """Remove Kilo always-on project wiring and global skill/command files.""" - _agents_uninstall(project_dir or Path("."), platform="kilo") - removed = _kilo_uninstall_global() - print("; ".join(removed) if removed else "nothing to remove") -def claude_install(project_dir: Path | None = None) -> None: - """Write the graphify section to the local CLAUDE.md.""" - target = (project_dir or Path(".")) / "CLAUDE.md" - if target.exists(): - content = target.read_text(encoding="utf-8") - new_content = _replace_or_append_section( - content, _CLAUDE_MD_MARKER, _always_on("claude-md") - ) - else: - new_content = _always_on("claude-md") - if target.exists() and new_content == target.read_text(encoding="utf-8"): - print(f"graphify already configured in {target.resolve()} (no change)") - else: - target.write_text(new_content, encoding="utf-8") - print(f"graphify section written to {target.resolve()}") - # Always re-install the Claude Code PreToolUse hook so an old hook - # payload (e.g. pre-issue-#580 wording) is replaced on upgrade. - _install_claude_hook(project_dir or Path(".")) - print() - print("Claude Code will now check the knowledge graph before answering") - print("codebase questions and rebuild it after code changes.") -def _install_claude_hook(project_dir: Path) -> None: - """Add graphify PreToolUse hook to .claude/settings.json.""" - settings_path = project_dir / ".claude" / "settings.json" - settings_path.parent.mkdir(parents=True, exist_ok=True) - if settings_path.exists(): - try: - settings = json.loads(settings_path.read_text(encoding="utf-8")) - except json.JSONDecodeError: - settings = {} - else: - settings = {} - hooks = settings.setdefault("hooks", {}) - pre_tool = hooks.setdefault("PreToolUse", []) - hooks["PreToolUse"] = [h for h in pre_tool if not (h.get("matcher") in ("Glob|Grep", "Bash", "Read|Glob") and "graphify" in str(h))] - hooks["PreToolUse"].extend(_claude_pretooluse_hooks()) - settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") - print(f" .claude/settings.json -> PreToolUse hooks registered (Bash search + Read/Glob)") -def _uninstall_claude_hook(project_dir: Path) -> None: - """Remove the graphify PreToolUse hook from .claude/settings.json and its - local-only sibling .claude/settings.local.json. - A user may relocate the hook into settings.local.json so it is not committed - to a shared repo, so uninstall has to clean whichever file holds it (#1731). - """ - claude_dir = project_dir / ".claude" - for name in ("settings.json", "settings.local.json"): - _strip_graphify_hook(claude_dir / name) -def _strip_graphify_hook(settings_path: Path) -> None: - """Drop graphify PreToolUse hooks from a single Claude settings file, if present.""" - if not settings_path.exists(): - return - try: - settings = json.loads(settings_path.read_text(encoding="utf-8")) - except json.JSONDecodeError: - return - pre_tool = settings.get("hooks", {}).get("PreToolUse", []) - filtered = [h for h in pre_tool if not (h.get("matcher") in ("Glob|Grep", "Bash", "Read|Glob") and "graphify" in str(h))] - if len(filtered) == len(pre_tool): - return - settings["hooks"]["PreToolUse"] = filtered - settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") - print(f" .claude/{settings_path.name} -> PreToolUse hook removed") - - -def uninstall_all(project_dir: Path | None = None, purge: bool = False) -> None: - """Remove graphify from every platform detected in the current project.""" - pd = project_dir or Path(".") - print("Uninstalling graphify from all detected platforms...\n") - - # Skill-file / config-section uninstallers - claude_uninstall(pd) - codebuddy_uninstall(pd) - gemini_uninstall(pd) - vscode_uninstall(pd) - _cursor_uninstall(pd) - _kiro_uninstall(pd) - _antigravity_uninstall(pd) - # AGENTS.md covers: codex, aider, opencode, claw, droid, trae, trae-cn, hermes, copilot - _agents_uninstall(pd) - # Amp also drops a user-scope skill at ~/.config/agents/skills, which the - # AGENTS.md cleanup above does not touch. - _remove_skill_file("amp") - # The generic agents platform's user-scope skill lives at ~/.agents/skills, - # which neither the AGENTS.md cleanup nor amp's removal reaches. - _remove_skill_file("agents") - _uninstall_opencode_plugin(pd) - _uninstall_codex_hook(pd) - - # Git hook - try: - from graphify.hooks import uninstall as hook_uninstall - result = hook_uninstall(pd) - if result: - print(result) - except Exception: - pass - if purge: - import shutil as _shutil - out = pd / _GRAPHIFY_OUT - if out.exists(): - _shutil.rmtree(out) - print(f"\n {_GRAPHIFY_OUT}/ -> deleted (--purge)") - else: - print(f"\n {_GRAPHIFY_OUT}/ -> not found (nothing to purge)") - print("\nDone. Run 'pip uninstall graphifyy' to remove the package itself.") -def claude_uninstall(project_dir: Path | None = None, *, project: bool = False) -> None: - """Remove the graphify skill tree (SKILL.md + references/) and the graphify - section from CLAUDE.md and its local-only variants, plus the PreToolUse hook. - Mirrors gemini_uninstall: the bare `graphify uninstall` and `graphify claude - uninstall` must remove the installed skill, not just strip CLAUDE.md, or the - progressive-disclosure tree (SKILL.md + references/) is orphaned (#1121). - A user may relocate the section/hook into the local-only files Claude Code - supports so they are not committed to a shared repo, so uninstall also cleans - CLAUDE.local.md, .claude/CLAUDE.local.md and .claude/settings.local.json (#1731). - """ - project_dir = project_dir or Path(".") - _remove_skill_file("claude", project=project, project_dir=project_dir) - - md_targets = [ - project_dir / "CLAUDE.md", - project_dir / "CLAUDE.local.md", - project_dir / ".claude" / "CLAUDE.local.md", - ] - existing = [t for t in md_targets if t.exists()] - removed_any = False - for target in existing: - # Not short-circuited: every present file must be cleaned, not just the first. - if _strip_graphify_md_section(target): - removed_any = True - - if not existing: - print("No CLAUDE.md found in current directory - nothing to do") - elif not removed_any: - print("graphify section not found in CLAUDE.md - nothing to do") - - _uninstall_claude_hook(project_dir) - - -def _strip_graphify_md_section(target: Path) -> bool: - """Strip the ## graphify section from one CLAUDE.md-style file. - - Returns True if a section was removed. Deletes the file if nothing else - remains after removal. - """ - try: - content = target.read_text(encoding="utf-8") - except (OSError, UnicodeDecodeError): - # An unreadable/undecodable CLAUDE.md-style file (e.g. non-UTF-8, or a - # directory of that name) must not abort uninstall - nothing to strip. - return False - if _CLAUDE_MD_MARKER not in content: - return False - # Remove the ## graphify section: from the marker to the next ## heading or EOF - cleaned = re.sub( - r"\n*## graphify\n.*?(?=\n## |\Z)", - "", - content, - flags=re.DOTALL, - ).rstrip() - if cleaned: - target.write_text(cleaned + "\n", encoding="utf-8") - print(f"graphify section removed from {target.resolve()}") - else: - target.unlink() - print(f"{target.name} was empty after removal - deleted {target.resolve()}") - return True -def codebuddy_install(project_dir: Path | None = None) -> None: - """Install the graphify skill and CODEBUDDY.md section for CodeBuddy.""" - _copy_skill_file("codebuddy", project=bool(project_dir), project_dir=project_dir) - target = (project_dir or Path(".")) / "CODEBUDDY.md" - if target.exists(): - content = target.read_text(encoding="utf-8") - new_content = _replace_or_append_section( - content, _CODEBUDDY_MD_MARKER, _always_on("claude-md") - ) - else: - new_content = _always_on("claude-md") - if target.exists() and new_content == target.read_text(encoding="utf-8"): - print(f"graphify already configured in {target.resolve()} (no change)") - else: - target.write_text(new_content, encoding="utf-8") - print(f"graphify section written to {target.resolve()}") - # Also write CodeBuddy PreToolUse hook to .codebuddy/settings.json - _install_codebuddy_hook(project_dir or Path(".")) - print() - print("CodeBuddy will now check the knowledge graph before answering") - print("codebase questions and rebuild it after code changes.") -def _install_codebuddy_hook(project_dir: Path) -> None: - """Add graphify PreToolUse hook to .codebuddy/settings.json.""" - settings_path = project_dir / ".codebuddy" / "settings.json" - settings_path.parent.mkdir(parents=True, exist_ok=True) - if settings_path.exists(): - try: - settings = json.loads(settings_path.read_text(encoding="utf-8")) - except json.JSONDecodeError: - settings = {} - else: - settings = {} - hooks = settings.setdefault("hooks", {}) - pre_tool = hooks.setdefault("PreToolUse", []) - hooks["PreToolUse"] = [h for h in pre_tool if not (h.get("matcher") in ("Glob|Grep", "Bash", "Read|Glob") and "graphify" in str(h))] - hooks["PreToolUse"].extend(_claude_pretooluse_hooks()) - settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") - print(f" .codebuddy/settings.json -> PreToolUse hooks registered") -def _uninstall_codebuddy_hook(project_dir: Path) -> None: - """Remove graphify PreToolUse hook from .codebuddy/settings.json.""" - settings_path = project_dir / ".codebuddy" / "settings.json" - if not settings_path.exists(): - return - try: - settings = json.loads(settings_path.read_text(encoding="utf-8")) - except json.JSONDecodeError: - return - pre_tool = settings.get("hooks", {}).get("PreToolUse", []) - filtered = [h for h in pre_tool if not (h.get("matcher") in ("Glob|Grep", "Bash", "Read|Glob") and "graphify" in str(h))] - if len(filtered) == len(pre_tool): - return - settings["hooks"]["PreToolUse"] = filtered - settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") - print(f" .codebuddy/settings.json -> PreToolUse hook removed") -def codebuddy_uninstall(project_dir: Path | None = None, *, project: bool = False) -> None: - """Remove the graphify skill tree (SKILL.md + references/) and the CODEBUDDY.md section.""" - project_dir = project_dir or Path(".") - _remove_skill_file("codebuddy", project=project, project_dir=project_dir) - target = project_dir / "CODEBUDDY.md" - if not target.exists(): - print("No CODEBUDDY.md found in current directory - nothing to do") - return - content = target.read_text(encoding="utf-8") - if _CODEBUDDY_MD_MARKER not in content: - print("graphify section not found in CODEBUDDY.md - nothing to do") - return - # Remove the ## graphify section: from the marker to the next ## heading or EOF - cleaned = re.sub( - r"\n*## graphify\n.*?(?=\n## |\Z)", - "", - content, - flags=re.DOTALL, - ).rstrip() - if cleaned: - target.write_text(cleaned + "\n", encoding="utf-8") - print(f"graphify section removed from {target.resolve()}") - else: - target.unlink() - print(f"CODEBUDDY.md was empty after removal - deleted {target.resolve()}") - _uninstall_codebuddy_hook(project_dir or Path(".")) def _clone_repo( url: str, branch: str | None = None, out_dir: Path | None = None diff --git a/graphify/install.py b/graphify/install.py new file mode 100644 index 000000000..a5362ac53 --- /dev/null +++ b/graphify/install.py @@ -0,0 +1,1853 @@ +"""graphify install/uninstall subsystem. + +The per-platform skill/hook installers and uninstallers, extracted verbatim from +graphify/__main__.py so the CLI dispatcher (`main`) stays readable. Import path is +unchanged: `graphify.__main__` re-exports every name defined here, so +`from graphify.__main__ import claude_install` (etc.) keeps working. + +Lives at graphify/install.py (same package dir as __main__) on purpose: the +installers resolve packaged assets via `Path(__file__).parent / "always_on"` and +`/ "skills"`, which only works from within the graphify/ directory. +""" + +from __future__ import annotations +import functools +import json +import os +import platform +import re +import shutil +import sys +from pathlib import Path + +try: + from importlib.metadata import version as _pkg_version + + __version__ = _pkg_version("graphifyy") +except Exception: + __version__ = "unknown" + +from graphify.paths import GRAPHIFY_OUT as _GRAPHIFY_OUT + + +@functools.lru_cache(maxsize=None) +def _always_on(basename: str) -> str: + """Read a packaged always-on instruction block from graphify/always_on/. + + The six always-on blocks (CLAUDE.md / AGENTS.md / GEMINI.md / VS Code + Copilot instructions / Antigravity rules / Kiro steering) live as committed + markdown next to this module, generated by tools/skillgen from a single + human-edited fragment and guarded against drift by ``skillgen --check``. The + installer injects them verbatim via ``_replace_or_append_section``, so the + bytes here must match the former triple-quoted constant exactly — the + always-on-roundtrip validator proves that. + """ + path = Path(__file__).parent / "always_on" / f"{basename}.md" + try: + return path.read_text(encoding="utf-8") + except OSError as exc: + # Defer to use-time so a missing/corrupt packaged block can't crash module + # import (which would brick every CLI command, not just install). Reached + # only by an install/integration path that actually needs this block. + raise RuntimeError( + f"graphify install is incomplete: missing always-on block '{basename}' " + f"at {path}. Reinstall graphifyy (e.g. `uv tool install --reinstall graphifyy`)." + ) from exc +def _refresh_all_version_stamps() -> None: + """After a successful install, update .graphify_version in all other known skill dirs. + + Prevents stale-version warnings from platforms that were installed previously + but not explicitly re-installed during this upgrade. + """ + for name in _PLATFORM_CONFIG: + skill_dst = _platform_skill_destination(name) + vf = skill_dst.parent / ".graphify_version" + if skill_dst.exists(): + vf.write_text(__version__, encoding="utf-8") +def _platform_skill_destination(platform_name: str, *, project: bool = False, project_dir: Path | None = None) -> Path: + """Return the skill destination for a platform and scope.""" + if platform_name == "gemini": + if project: + return (project_dir or Path(".")) / ".gemini" / "skills" / "graphify" / "SKILL.md" + if platform.system() == "Windows": + return Path.home() / ".agents" / "skills" / "graphify" / "SKILL.md" + return Path.home() / ".gemini" / "skills" / "graphify" / "SKILL.md" + + if platform_name == "opencode": + if project: + return (project_dir or Path(".")) / ".opencode" / "skills" / "graphify" / "SKILL.md" + return Path.home() / ".config" / "opencode" / "skills" / "graphify" / "SKILL.md" + + if platform_name == "hermes": + if project: + return (project_dir or Path(".")) / ".hermes" / "skills" / "graphify" / "SKILL.md" + # On Windows, Hermes scans %LOCALAPPDATA%\hermes\skills, not ~/.hermes (#1403). + if platform.system() == "Windows": + local_appdata = Path(os.environ.get("LOCALAPPDATA") or (Path.home() / "AppData" / "Local")) + return local_appdata / "hermes" / "skills" / "graphify" / "SKILL.md" + return Path.home() / ".hermes" / "skills" / "graphify" / "SKILL.md" + + if platform_name == "devin": + if project: + return (project_dir or Path(".")) / ".devin" / "skills" / "graphify" / "SKILL.md" + return Path.home() / ".config" / "devin" / "skills" / "graphify" / "SKILL.md" + + if platform_name == "amp": + if project: + return (project_dir or Path(".")) / ".agents" / "skills" / "graphify" / "SKILL.md" + return Path.home() / ".config" / "agents" / "skills" / "graphify" / "SKILL.md" + + if platform_name == "agents": + # The generic Agent-Skills target: project ./.agents/skills, global the + # spec's user-global ~/.agents/skills (read by `npx skills` and compliant + # frameworks), NOT amp's ~/.config/agents/skills. + if project: + return (project_dir or Path(".")) / ".agents" / "skills" / "graphify" / "SKILL.md" + return Path.home() / ".agents" / "skills" / "graphify" / "SKILL.md" + + if platform_name in ("antigravity", "antigravity-windows"): + if project: + return (project_dir or Path(".")) / ".agents" / "skills" / "graphify" / "SKILL.md" + # Global Antigravity skill dir (all workspaces): ~/.gemini/config/skills/ + return Path.home() / ".gemini" / "config" / "skills" / "graphify" / "SKILL.md" + + cfg = _PLATFORM_CONFIG[platform_name] + if project: + return (project_dir or Path(".")) / cfg["skill_dst"] + + if platform_name in ("claude", "windows") and os.environ.get("CLAUDE_CONFIG_DIR"): + return Path(os.environ["CLAUDE_CONFIG_DIR"]) / "skills" / "graphify" / "SKILL.md" + return Path.home() / cfg["skill_dst"] +def _packaged_skill_refs_dir(platform_name: str) -> Path | None: + """Return the packaged references source dir for a progressive platform, else None. + + A platform opts into progressive disclosure by setting ``skill_refs`` in its + ``_PLATFORM_CONFIG`` entry. The value names a bundle under + ``graphify/skills//references/``. Reuse keys (e.g. trae-cn) point at + their twin's bundle. + + ``gemini`` has no ``_PLATFORM_CONFIG`` entry: it installs claude's + ``skill.md`` body verbatim (see ``_copy_skill_file``). Since that body is the + lean progressive core that links to ``references/``, gemini needs claude's + references/ sidecar too, or its SKILL.md ships with dead pointers. So gemini + resolves to the claude bundle rather than opting out. + + Bundles ship one platform-group at a time. A host whose bundle directory + ``graphify/skills//`` is not in this build has not gone progressive + yet, so this returns None and the host installs today's monolithic SKILL.md + with no references/ sidecar. Only when the bundle directory IS present does + this return the references path; if that directory then lacks its + ``references/`` subdir, ``_copy_skill_file`` hard-fails (a malformed bundle, + the empty-sidecar regression the wheel-content test also guards). + """ + if platform_name == "gemini": + bundle = "claude" + else: + bundle = _PLATFORM_CONFIG[platform_name].get("skill_refs") + if not bundle: + return None + bundle_dir = Path(__file__).parent / "skills" / bundle + if not bundle_dir.is_dir(): + return None + return bundle_dir / "references" +def _install_skill_references(skill_dst: Path, refs_src: Path) -> None: + """Atomically install a packaged references/ sidecar next to SKILL.md. + + Stages the packaged dir into ``references.tmp`` (copytree), drops any stale + ``references/`` already on disk, then ``os.replace``-renames the staged dir + into place. The rename is atomic on the same filesystem, so an interrupted + install never leaves a half-written references/ visible to the agent. + """ + refs_dst = skill_dst.parent / "references" + refs_staged = skill_dst.parent / "references.tmp" + if refs_staged.exists(): + shutil.rmtree(refs_staged) + try: + shutil.copytree(refs_src, refs_staged) + if refs_dst.exists(): + shutil.rmtree(refs_dst) + os.replace(refs_staged, refs_dst) + except Exception: + if refs_staged.exists(): + shutil.rmtree(refs_staged, ignore_errors=True) + raise +def _copy_skill_file(platform_name: str, *, project: bool = False, project_dir: Path | None = None) -> Path: + """Copy a packaged skill file and write its version stamp. + + For progressive platforms (those with ``skill_refs`` set), the packaged + ``references/`` sidecar is installed alongside SKILL.md and the single + ``.graphify_version`` stamp covers both. For monolith platforms (no + ``skill_refs``), any orphan ``references/`` left by a prior progressive + install is removed so the on-disk layout matches the package. + """ + skill_file = "skill.md" if platform_name == "gemini" else _PLATFORM_CONFIG[platform_name]["skill_file"] + skill_src = Path(__file__).parent / skill_file + if not skill_src.exists(): + print(f"error: {skill_file} not found in package - reinstall graphify", file=sys.stderr) + sys.exit(1) + + refs_src = _packaged_skill_refs_dir(platform_name) + if refs_src is not None and not refs_src.exists(): + # Progressive platform declared a references bundle that is missing from + # the package. Fail loud rather than silently shipping an empty sidecar. + print( + f"error: references for '{platform_name}' not found in package " + f"({refs_src}) - reinstall graphify", + file=sys.stderr, + ) + sys.exit(1) + + skill_dst = _platform_skill_destination(platform_name, project=project, project_dir=project_dir) + skill_dst.parent.mkdir(parents=True, exist_ok=True) + + # Install the references/ sidecar (or clear an orphan one) BEFORE writing + # SKILL.md, so SKILL.md is the last artifact laid down. An install that is + # interrupted partway then leaves no SKILL.md rather than a SKILL.md that + # points at an absent references/ dir. + if refs_src is not None: + _install_skill_references(skill_dst, refs_src) + print(f" references -> {skill_dst.parent / 'references'}") + else: + # Monolith (or progressive-with-no-refs): clear any orphan references/. + orphan_refs = skill_dst.parent / "references" + if orphan_refs.exists(): + shutil.rmtree(orphan_refs) + + # SKILL.md last (crash-safety), via an atomic temp + rename. + tmp_dst = skill_dst.with_suffix(skill_dst.suffix + ".tmp") + try: + shutil.copy(skill_src, tmp_dst) + os.replace(tmp_dst, skill_dst) + except Exception: + try: + tmp_dst.unlink(missing_ok=True) + except OSError: + pass + raise + + (skill_dst.parent / ".graphify_version").write_text(__version__, encoding="utf-8") + print(f" skill installed -> {skill_dst}") + return skill_dst +def _remove_skill_file(platform_name: str, *, project: bool = False, project_dir: Path | None = None) -> bool: + """Remove a platform skill file and its version stamp without touching other scopes.""" + skill_dst = _platform_skill_destination(platform_name, project=project, project_dir=project_dir) + removed = False + if skill_dst.exists(): + skill_dst.unlink() + print(f" skill removed -> {skill_dst}") + removed = True + version_file = skill_dst.parent / ".graphify_version" + if version_file.exists(): + version_file.unlink() + removed = True + refs_dir = skill_dst.parent / "references" + if refs_dir.exists(): + shutil.rmtree(refs_dir) + removed = True + for d in (skill_dst.parent, skill_dst.parent.parent, skill_dst.parent.parent.parent): + try: + d.rmdir() + except OSError: + break + return removed +def _project_scope_root(path: Path, project_dir: Path) -> Path: + """Return the top-level project artifact for a project-scoped skill path.""" + try: + rel = path.relative_to(project_dir) + except ValueError: + return path + return project_dir / rel.parts[0] if rel.parts else path +def _remove_claude_skill_registration(project_dir: Path) -> None: + """Remove the project-scoped Claude skill registration file/section.""" + claude_md = project_dir / ".claude" / "CLAUDE.md" + if not claude_md.exists(): + return + content = claude_md.read_text(encoding="utf-8") + if "# graphify" not in content: + return + cleaned = re.sub(r"\n*# graphify\n.*?(?=\n# |\Z)", "", content, flags=re.DOTALL).rstrip() + if cleaned: + claude_md.write_text(cleaned + "\n", encoding="utf-8") + print(f" CLAUDE.md -> graphify skill registration removed from {claude_md}") + else: + claude_md.unlink() + print(f" CLAUDE.md -> deleted {claude_md}") +def _print_project_git_add_hint(paths: list[Path]) -> None: + unique: list[str] = [] + for path in paths: + text = path.as_posix().rstrip("/") + if path.exists() and path.is_dir(): + text += "/" + if text not in unique: + unique.append(text) + if not unique: + return + print() + print("Project-scoped install. Add to version control:") + print(f" git add {' '.join(unique)}") +def _claude_pretooluse_hooks() -> "list[dict]": + """graphify's Claude/Codebuddy PreToolUse hooks, resolved at install time. + + The command invokes `graphify hook-guard ` via the absolute exe + path (`_resolve_graphify_exe`), so it parses under sh, cmd.exe and PowerShell + alike — this is the #522 fix, and mirrors the codex hook. Matchers stay "Bash" + and "Read|Glob" and the command always contains "graphify", so the existing + install/uninstall filters find and replace both old bash hooks and these. + """ + exe = _resolve_graphify_exe() + if " " in exe and not exe.startswith('"'): + exe = f'"{exe}"' + return [ + {"matcher": "Bash", + "hooks": [{"type": "command", "command": f"{exe} hook-guard search"}]}, + {"matcher": "Read|Glob", + "hooks": [{"type": "command", "command": f"{exe} hook-guard read"}]}, + ] +def _skill_registration(skill_path: str = "~/.claude/skills/graphify/SKILL.md") -> str: + return ( + "\n# graphify\n" + f"- **graphify** (`{skill_path}`) " + "- any input to knowledge graph. Trigger: `/graphify`\n" + "When the user types `/graphify`, use the installed graphify skill " + "or instructions before doing anything else.\n" + ) +_PLATFORM_CONFIG: dict[str, dict] = { + "claude": { + "skill_file": "skill.md", + "skill_dst": Path(".claude") / "skills" / "graphify" / "SKILL.md", + "claude_md": True, + "skill_refs": "claude", + }, + "codex": { + "skill_file": "skill-codex.md", + "skill_dst": Path(".codex") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "codex", + }, + "opencode": { + "skill_file": "skill-opencode.md", + "skill_dst": Path(".config") / "opencode" / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "opencode", + }, + "kilo": { + "skill_file": "skill-kilo.md", + "skill_dst": Path(".config") / "kilo" / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "kilo", + }, + "aider": { + # Monolith: aider ships the full SKILL.md inline, no references/ sidecar. + "skill_file": "skill-aider.md", + "skill_dst": Path(".aider") / "graphify" / "SKILL.md", + "claude_md": False, + }, + "copilot": { + "skill_file": "skill-copilot.md", + "skill_dst": Path(".copilot") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "copilot", + }, + "claw": { + "skill_file": "skill-claw.md", + "skill_dst": Path(".openclaw") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "claw", + }, + "droid": { + "skill_file": "skill-droid.md", + "skill_dst": Path(".factory") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "droid", + }, + "trae": { + "skill_file": "skill-trae.md", + "skill_dst": Path(".trae") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "trae", + }, + "trae-cn": { + # Reuses trae's split bundle (same skill body + references). + "skill_file": "skill-trae.md", + "skill_dst": Path(".trae-cn") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "trae", + }, + "hermes": { + # Reuses claw's split bundle. + "skill_file": "skill-claw.md", + "skill_dst": Path(".hermes") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "claw", + }, + "kiro": { + "skill_file": "skill-kiro.md", + "skill_dst": Path(".kiro") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "kiro", + }, + "pi": { + "skill_file": "skill-pi.md", + "skill_dst": Path(".pi") / "agent" / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "pi", + }, + "codebuddy": { + # Reuses claude's split bundle (shares skill.md). + "skill_file": "skill.md", + "skill_dst": Path(".codebuddy") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "claude", + }, + "antigravity": { + # Rides claude's split bundle (shares skill.md). + "skill_file": "skill.md", + "skill_dst": Path(".agents") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "claude", + }, + "antigravity-windows": { + # Rides windows' split bundle. + "skill_file": "skill-windows.md", + "skill_dst": Path(".agents") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "windows", + }, + "windows": { + "skill_file": "skill-windows.md", + "skill_dst": Path(".claude") / "skills" / "graphify" / "SKILL.md", + "claude_md": True, + "skill_refs": "windows", + }, + "kimi": { + # Reuses claude's split bundle (shares skill.md). + "skill_file": "skill.md", + "skill_dst": Path(".kimi") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "claude", + }, + "amp": { + # Amp searches .agents/skills (project) and ~/.config/agents/skills (user), + # not .amp/skills. The user-scope path is set in _platform_skill_destination. + "skill_file": "skill-amp.md", + "skill_dst": Path(".agents") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "amp", + }, + "agents": { + # The generic cross-framework Agent-Skills target. Global: ~/.agents/skills + # (the spec's user-global location, read by `npx skills` and compliant + # frameworks); project: ./.agents/skills. The CLI accepts `skills` as an + # alias (see _canonical_platform). Ships its own rendered bundle. + "skill_file": "skill-agents.md", + "skill_dst": Path(".agents") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "agents", + }, + "devin": { + # Monolith: devin ships the full SKILL.md inline, no references/ sidecar. + "skill_file": "skill-devin.md", + # User scope: ~/.config/devin/skills/graphify/SKILL.md + # Project scope: .devin/skills/graphify/SKILL.md (overridden in _platform_skill_destination) + "skill_dst": Path(".config") / "devin" / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + }, +} +# CLI-only platform aliases, resolved to a real _PLATFORM_CONFIG key before +# dispatch. `skills` is the friendly alias for the generic `agents` platform +# (the Agent-Skills ecosystem calls them "skills"). +_PLATFORM_ALIASES: dict[str, str] = {"skills": "agents"} +def _canonical_platform(platform_name: str) -> str: + """Resolve a CLI platform alias to its real _PLATFORM_CONFIG key.""" + return _PLATFORM_ALIASES.get(platform_name, platform_name) +def _replace_or_append_section(content: str, marker: str, new_section: str) -> str: + """Idempotently update or append a graphify-owned section in shared files. + + If no line is exactly ``marker`` (the heading, at column 0), append + ``new_section`` to the end (with a blank-line separator if there's existing + content). + + If a real ``marker`` heading exists, replace the existing section in place. + The section runs from that heading to the line before the next H2 heading + (``## `` at line start), or to EOF if no later H2 exists. This lets older + installs receive the updated copy without users having to uninstall and + reinstall (issue #580). + + The heading is matched only when a line *is* exactly ``marker`` (after + stripping surrounding whitespace), never as a substring. Matching ``## + graphify`` inside a bullet or an inline reference used to anchor the replace + on that mention and delete every line from there to the next heading, + silently destroying hand-curated content (#1688). When several exact + headings exist, the last one is used, since graphify's section is appended. + """ + lines = content.split("\n") + starts = [i for i, line in enumerate(lines) if line.strip() == marker] + if not starts: + if content.strip(): + return content.rstrip() + "\n\n" + new_section.lstrip() + return new_section.lstrip() + + start = starts[-1] + end = len(lines) + for j in range(start + 1, len(lines)): + if lines[j].startswith("## "): + end = j + break + + head = "\n".join(lines[:start]).rstrip() + tail = "\n".join(lines[end:]).lstrip() + section = new_section.strip() + + parts: list[str] = [] + if head: + parts.append(head) + parts.append(section) + if tail: + parts.append(tail) + out = "\n\n".join(parts) + if not out.endswith("\n"): + out += "\n" + return out +def _print_banner() -> None: + """Amber brain banner on graphify install. TTY-only, never raises.""" + if not sys.stdout.isatty(): + return + try: + if sys.platform == "win32": + import ctypes + ctypes.windll.kernel32.SetConsoleMode( + ctypes.windll.kernel32.GetStdHandle(-11), 7 + ) + A = "\033[38;5;214m" + D = "\033[38;5;130m" + R = "\033[0m" + print(f"""{A} + ╭──◉──╮ ╭──◉──╮ + ╱ ◉ ◉ ╲ ╱ ◉ ◉ ╲ +│ ◉─◉─◉ ◉ ◉─◉─◉ │ +│ ◉ ◉ │ ◉ ◉ │ +│ ◉─◉─◉ ◉ ◉─◉─◉ │ + ╲ ◉ ◉ ╱ ╲ ◉ ◉ ╱ + ╰──◉──╯ ╰──◉──╯ + ◉ + + █▀▀ █▀█ ▄▀█ █▀█ █ █ █ █▀▀ █▄█ + █▄█ █▀▄ █▀█ █▀▀ █▀█ █ █▀ █{D} {__version__}{R} +""") + except Exception: + pass +def install(platform: str = "claude", *, project: bool = False, project_dir: Path | None = None) -> None: + _print_banner() + platform = _canonical_platform(platform) + if platform == "gemini": + gemini_install(project_dir=project_dir, project=project) + return + if platform == "cursor": + _cursor_install(Path(".")) + return + # On Windows, antigravity needs the PowerShell skill, not the bash one + if platform == "antigravity" and sys.platform == "win32": + platform = "antigravity-windows" + if platform not in _PLATFORM_CONFIG: + print( + f"error: unknown platform '{platform}'. Choose from: {', '.join(_PLATFORM_CONFIG)}, gemini, cursor", + file=sys.stderr, + ) + sys.exit(1) + + cfg = _PLATFORM_CONFIG[platform] + project_dir = project_dir or Path(".") + skill_dst = _copy_skill_file(platform, project=project, project_dir=project_dir) + + if platform == "kilo": + # Kilo Code also supports a native /graphify command file. + command_src = Path(__file__).parent / "command-kilo.md" + if not command_src.exists(): + print( + f"error: command-kilo.md not found in package - reinstall graphify", + file=sys.stderr, + ) + sys.exit(1) + command_dst = Path.home() / ".config" / "kilo" / "command" / "graphify.md" + command_dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copy(command_src, command_dst) + print(f" command installed -> {command_dst}") + + if cfg["claude_md"]: + # Register in the matching Claude Code scope. + claude_md = (project_dir / ".claude" / "CLAUDE.md") if project else Path.home() / ".claude" / "CLAUDE.md" + registration = _skill_registration(".claude/skills/graphify/SKILL.md" if project else "~/.claude/skills/graphify/SKILL.md") + if claude_md.exists(): + content = claude_md.read_text(encoding="utf-8") + if "graphify" in content: + print(f" CLAUDE.md -> already registered (no change)") + else: + claude_md.write_text(content.rstrip() + registration, encoding="utf-8") + print(f" CLAUDE.md -> skill registered in {claude_md}") + else: + claude_md.parent.mkdir(parents=True, exist_ok=True) + claude_md.write_text(registration.lstrip(), encoding="utf-8") + print(f" CLAUDE.md -> created at {claude_md}") + + if platform == "codebuddy": + # Register in ~/.codebuddy/CODEBUDDY.md (CodeBuddy only) + codebuddy_md = Path.home() / ".codebuddy" / "CODEBUDDY.md" + registration = _skill_registration("~/.codebuddy/skills/graphify/SKILL.md") + if codebuddy_md.exists(): + content = codebuddy_md.read_text(encoding="utf-8") + if "graphify" in content: + print(f" CODEBUDDY.md -> already registered (no change)") + else: + codebuddy_md.write_text(content.rstrip() + registration, encoding="utf-8") + print(f" CODEBUDDY.md -> skill registered in {codebuddy_md}") + else: + codebuddy_md.parent.mkdir(parents=True, exist_ok=True) + codebuddy_md.write_text(registration.lstrip(), encoding="utf-8") + print(f" CODEBUDDY.md -> created at {codebuddy_md}") + + if platform == "opencode": + _install_opencode_plugin(project_dir if project else Path(".")) + + # Refresh version stamps in all other previously-installed skill dirs so + # stale-version warnings don't fire for platforms not explicitly re-installed. + if project: + _print_project_git_add_hint([_project_scope_root(skill_dst, project_dir)]) + else: + _refresh_all_version_stamps() + + print() + print("Done. Open your AI coding assistant and type:") + print() + print(" /graphify .") + print() +def _print_install_usage() -> None: + platforms = ", ".join([*_PLATFORM_CONFIG, "gemini", "cursor"]) + print("Usage: graphify install [--project] [--platform P|P]") + print(f"Platforms: {platforms}") +_CLAUDE_MD_MARKER = "## graphify" +_CODEBUDDY_MD_MARKER = "## graphify" +_AGENTS_MD_MARKER = "## graphify" +_GEMINI_MD_MARKER = "## graphify" +def _gemini_hook() -> dict: + """Gemini CLI BeforeTool hook, resolved to a shell-agnostic `graphify` call.""" + exe = _resolve_graphify_exe() + if " " in exe and not exe.startswith('"'): + exe = f'"{exe}"' + return { + "matcher": "read_file|list_directory", + "hooks": [{"type": "command", "command": f"{exe} hook-guard gemini"}], + } +def gemini_install(project_dir: Path | None = None, *, project: bool = False) -> None: + """Copy skill file, write GEMINI.md section, and install BeforeTool hook.""" + project_dir = project_dir or Path(".") + skill_dst = _copy_skill_file("gemini", project=project, project_dir=project_dir) + + target = project_dir / "GEMINI.md" + + if target.exists(): + content = target.read_text(encoding="utf-8") + new_content = _replace_or_append_section( + content, _GEMINI_MD_MARKER, _always_on("gemini-md") + ) + else: + new_content = _always_on("gemini-md") + + if target.exists() and new_content == target.read_text(encoding="utf-8"): + print(f"graphify already configured in {target.resolve()} (no change)") + else: + target.write_text(new_content, encoding="utf-8") + print(f"graphify section written to {target.resolve()}") + + # Always re-install the Gemini hook so an older payload (e.g. pre-issue-#580 + # wording) is replaced on upgrade. + _install_gemini_hook(project_dir) + if project: + _print_project_git_add_hint([_project_scope_root(skill_dst, project_dir), project_dir / "GEMINI.md", project_dir / ".gemini"]) + print() + print("Gemini CLI will now check the knowledge graph before answering") + print("codebase questions and rebuild it after code changes.") +def _install_gemini_hook(project_dir: Path) -> None: + settings_path = project_dir / ".gemini" / "settings.json" + settings_path.parent.mkdir(parents=True, exist_ok=True) + try: + settings = ( + json.loads(settings_path.read_text(encoding="utf-8")) + if settings_path.exists() + else {} + ) + except json.JSONDecodeError: + settings = {} + before_tool = settings.setdefault("hooks", {}).setdefault("BeforeTool", []) + settings["hooks"]["BeforeTool"] = [ + h for h in before_tool if "graphify" not in str(h) + ] + settings["hooks"]["BeforeTool"].append(_gemini_hook()) + settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") + print(" .gemini/settings.json -> BeforeTool hook registered") +def _uninstall_gemini_hook(project_dir: Path) -> None: + settings_path = project_dir / ".gemini" / "settings.json" + if not settings_path.exists(): + return + try: + settings = json.loads(settings_path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + return + before_tool = settings.get("hooks", {}).get("BeforeTool", []) + filtered = [h for h in before_tool if "graphify" not in str(h)] + if len(filtered) == len(before_tool): + return + settings["hooks"]["BeforeTool"] = filtered + settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") + print(" .gemini/settings.json -> BeforeTool hook removed") +def gemini_uninstall(project_dir: Path | None = None, *, project: bool = False) -> None: + """Remove the graphify section from GEMINI.md, uninstall hook, and remove skill file.""" + project_dir = project_dir or Path(".") + _remove_skill_file("gemini", project=project, project_dir=project_dir) + + target = project_dir / "GEMINI.md" + if not target.exists(): + print("No GEMINI.md found in current directory - nothing to do") + return + content = target.read_text(encoding="utf-8") + if _GEMINI_MD_MARKER not in content: + print("graphify section not found in GEMINI.md - nothing to do") + return + cleaned = re.sub( + r"\n*## graphify\n.*?(?=\n## |\Z)", "", content, flags=re.DOTALL + ).rstrip() + if cleaned: + target.write_text(cleaned + "\n", encoding="utf-8") + print(f"graphify section removed from {target.resolve()}") + else: + target.unlink() + print(f"GEMINI.md was empty after removal - deleted {target.resolve()}") + _uninstall_gemini_hook(project_dir) +_VSCODE_INSTRUCTIONS_MARKER = "## graphify" +def vscode_install(project_dir: Path | None = None) -> None: + """Install graphify skill for VS Code Copilot Chat + write .github/copilot-instructions.md.""" + skill_src = Path(__file__).parent / "skill-vscode.md" + refs_bundle = "vscode" + if not skill_src.exists(): + skill_src = Path(__file__).parent / "skill-copilot.md" + refs_bundle = "copilot" + skill_dst = Path.home() / ".copilot" / "skills" / "graphify" / "SKILL.md" + skill_dst.parent.mkdir(parents=True, exist_ok=True) + tmp_dst = skill_dst.with_suffix(skill_dst.suffix + ".tmp") + try: + shutil.copy(skill_src, tmp_dst) + os.replace(tmp_dst, skill_dst) + except Exception: + try: + tmp_dst.unlink(missing_ok=True) + except OSError: + pass + raise + # Progressive-capable: install the packaged references/ sidecar when present. + refs_src = Path(__file__).parent / "skills" / refs_bundle / "references" + if refs_src.exists(): + _install_skill_references(skill_dst, refs_src) + print(f" references -> {skill_dst.parent / 'references'}") + else: + orphan_refs = skill_dst.parent / "references" + if orphan_refs.exists(): + shutil.rmtree(orphan_refs) + (skill_dst.parent / ".graphify_version").write_text(__version__, encoding="utf-8") + print(f" skill installed -> {skill_dst}") + + instructions = (project_dir or Path(".")) / ".github" / "copilot-instructions.md" + instructions.parent.mkdir(parents=True, exist_ok=True) + if instructions.exists(): + content = instructions.read_text(encoding="utf-8") + new_content = _replace_or_append_section( + content, _VSCODE_INSTRUCTIONS_MARKER, _always_on("vscode-instructions") + ) + if new_content == content: + print(f" {instructions} -> already configured (no change)") + else: + instructions.write_text(new_content, encoding="utf-8") + print(f" {instructions} -> graphify section {'updated' if _VSCODE_INSTRUCTIONS_MARKER in content else 'added'}") + else: + instructions.write_text(_always_on("vscode-instructions"), encoding="utf-8") + print(f" {instructions} -> created") + + print() + print( + "VS Code Copilot Chat configured. Type /graphify in the chat panel to build the graph." + ) + print("Note: for GitHub Copilot CLI (terminal), use: graphify copilot install") +def vscode_uninstall(project_dir: Path | None = None) -> None: + """Remove graphify VS Code Copilot Chat skill and .github/copilot-instructions.md section.""" + skill_dst = Path.home() / ".copilot" / "skills" / "graphify" / "SKILL.md" + if skill_dst.exists(): + skill_dst.unlink() + print(f" skill removed -> {skill_dst}") + version_file = skill_dst.parent / ".graphify_version" + if version_file.exists(): + version_file.unlink() + refs_dir = skill_dst.parent / "references" + if refs_dir.exists(): + shutil.rmtree(refs_dir) + for d in ( + skill_dst.parent, + skill_dst.parent.parent, + skill_dst.parent.parent.parent, + ): + try: + d.rmdir() + except OSError: + break + + instructions = (project_dir or Path(".")) / ".github" / "copilot-instructions.md" + if not instructions.exists(): + return + content = instructions.read_text(encoding="utf-8") + if _VSCODE_INSTRUCTIONS_MARKER not in content: + return + cleaned = re.sub( + r"\n*## graphify\n.*?(?=\n## |\Z)", "", content, flags=re.DOTALL + ).rstrip() + if cleaned: + instructions.write_text(cleaned + "\n", encoding="utf-8") + print(f" graphify section removed from {instructions}") + else: + instructions.unlink() + print(f" {instructions} -> deleted (was empty after removal)") +_ANTIGRAVITY_RULES_PATH = Path(".agents") / "rules" / "graphify.md" +_ANTIGRAVITY_WORKFLOW_PATH = Path(".agents") / "workflows" / "graphify.md" +_ANTIGRAVITY_WORKFLOW = """\ +--- +name: graphify +description: Turn any folder of files into a navigable knowledge graph +--- + +# Workflow: graphify + +Follow the graphify skill installed at ~/.gemini/config/skills/graphify/SKILL.md to run the full pipeline. + +If no path argument is given, use `.` (current directory). +""" +def _kiro_install(project_dir: Path) -> None: + """Write graphify skill + steering file for Kiro IDE/CLI.""" + project_dir = project_dir or Path(".") + + # Skill file + references/ sidecar + .graphify_version stamp via the shared + # progressive-disclosure helper. Previously this used a bare write_text that + # bypassed _copy_skill_file, so the references/ dir and version stamp were + # never written even though kiro declares skill_refs: "kiro" (#1142). + _copy_skill_file("kiro", project=True, project_dir=project_dir) + + # Steering file → .kiro/steering/graphify.md (always-on) + steering_dir = project_dir / ".kiro" / "steering" + steering_dir.mkdir(parents=True, exist_ok=True) + steering_dst = steering_dir / "graphify.md" + if steering_dst.exists() and steering_dst.read_text(encoding="utf-8") == _always_on("kiro-steering"): + print(f" .kiro/steering/graphify.md -> already configured (no change)") + else: + # File is wholly graphify-owned. Overwrite on upgrade so older + # report-first wording does not silently linger (issue #580). + action = "updated" if steering_dst.exists() else "written" + steering_dst.write_text(_always_on("kiro-steering"), encoding="utf-8") + print(f" .kiro/steering/graphify.md -> always-on steering {action}") + + print() + print("Kiro will now read the knowledge graph before every conversation.") + print("Use /graphify to build or update the graph.") +def _kiro_uninstall(project_dir: Path) -> None: + """Remove graphify skill + steering file for Kiro.""" + project_dir = project_dir or Path(".") + removed = [] + + # Skill + .graphify_version + references/ sidecar + empty-dir walk. + skill_dst = _platform_skill_destination("kiro", project=True, project_dir=project_dir) + if _remove_skill_file("kiro", project=True, project_dir=project_dir): + removed.append(str(skill_dst.relative_to(project_dir))) + + steering_dst = project_dir / ".kiro" / "steering" / "graphify.md" + if steering_dst.exists(): + steering_dst.unlink() + removed.append(str(steering_dst.relative_to(project_dir))) + + print("Removed: " + (", ".join(removed) if removed else "nothing to remove")) +def _antigravity_finalize(skill_dst: Path, project_dir: Path) -> None: + """Write Antigravity's always-on layer next to an installed skill. + + Injects the native tool-discovery YAML frontmatter into *skill_dst*, then + writes ``.agents/rules/graphify.md`` and ``.agents/workflows/graphify.md`` + under *project_dir*. Shared by the global ``antigravity install`` and the + project-scoped ``install --project --platform antigravity`` paths, so both lay + down the rules/workflows that the uninstall path already expects to remove. + """ + # Inject YAML frontmatter for native Antigravity tool discovery. + if skill_dst.exists(): + content = skill_dst.read_text(encoding="utf-8") + if not content.startswith("---\n"): + frontmatter = "---\nname: graphify-manager\ndescription: Rebuild the code graph or perform manual CLI queries when MCP server is offline.\n---\n\n" + skill_dst.write_text(frontmatter + content, encoding="utf-8") + + # .agents/rules/graphify.md + rules_path = project_dir / _ANTIGRAVITY_RULES_PATH + rules_path.parent.mkdir(parents=True, exist_ok=True) + if rules_path.exists(): + existing = rules_path.read_text(encoding="utf-8") + if _always_on("antigravity-rules").strip() != existing.strip(): + rules_path.write_text(_always_on("antigravity-rules"), encoding="utf-8") + print(f"graphify rule updated at {rules_path.resolve()}") + else: + print(f"graphify rule already configured at {rules_path.resolve()} (no change)") + else: + rules_path.write_text(_always_on("antigravity-rules"), encoding="utf-8") + print(f"graphify rule written to {rules_path.resolve()}") + + # .agents/workflows/graphify.md + wf_path = project_dir / _ANTIGRAVITY_WORKFLOW_PATH + wf_path.parent.mkdir(parents=True, exist_ok=True) + if wf_path.exists(): + existing = wf_path.read_text(encoding="utf-8") + if _ANTIGRAVITY_WORKFLOW.strip() != existing.strip(): + wf_path.write_text(_ANTIGRAVITY_WORKFLOW, encoding="utf-8") + print(f"graphify workflow updated at {wf_path.resolve()}") + else: + print(f"graphify workflow already configured at {wf_path.resolve()} (no change)") + else: + wf_path.write_text(_ANTIGRAVITY_WORKFLOW, encoding="utf-8") + print(f"graphify workflow written to {wf_path.resolve()}") +def _antigravity_install(project_dir: Path) -> None: + """Install graphify for Google Antigravity (global skill + .agents/rules + .agents/workflows).""" + # Copy the skill to ~/.gemini/config/skills/graphify/SKILL.md (global), then + # lay down the always-on rules/workflows under the project dir. + install(platform="antigravity") + _antigravity_finalize(_platform_skill_destination("antigravity"), project_dir) + + print() + print("Antigravity will now check the knowledge graph before answering") + print("codebase questions. Run /graphify first to build the graph.") + print() + print( + "To enable full MCP architecture navigation, add this to ~/.gemini/antigravity/mcp_config.json:" + ) + print(' "graphify": {') + print(' "command": "uv",') + print( + ' "args": ["run", "--with", "graphifyy", "--with", "mcp", "-m", "graphify.serve", "${workspace.path}/graphify-out/graph.json"]' + ) + print(" }") +def _antigravity_uninstall(project_dir: Path, *, project: bool = False) -> None: + """Remove graphify Antigravity rules, workflow, and skill files.""" + # Remove rules file + rules_path = project_dir / _ANTIGRAVITY_RULES_PATH + if rules_path.exists(): + rules_path.unlink() + print(f"graphify rule removed from {rules_path.resolve()}") + else: + print("No graphify Antigravity rule found - nothing to do") + + # Remove workflow file + wf_path = project_dir / _ANTIGRAVITY_WORKFLOW_PATH + if wf_path.exists(): + wf_path.unlink() + print(f"graphify workflow removed from {wf_path.resolve()}") + + # Remove skill file + skill_dst = _platform_skill_destination("antigravity", project=project, project_dir=project_dir) + if skill_dst.exists(): + skill_dst.unlink() + print(f"graphify skill removed from {skill_dst}") + version_file = skill_dst.parent / ".graphify_version" + if version_file.exists(): + version_file.unlink() + refs_dir = skill_dst.parent / "references" + if refs_dir.exists(): + shutil.rmtree(refs_dir) + for d in ( + skill_dst.parent, + skill_dst.parent.parent, + skill_dst.parent.parent.parent, + ): + try: + d.rmdir() + except OSError: + break +_CURSOR_RULE_PATH = Path(".cursor") / "rules" / "graphify.mdc" +_CURSOR_RULE = """\ +--- +description: graphify knowledge graph context +alwaysApply: true +--- + +This project has a graphify knowledge graph at graphify-out/. + +**MANDATORY: Before using Read, Grep, Glob, or Bash to explore the codebase, you MUST run graphify first:** +- `graphify query ""` — scoped subgraph for any codebase or architecture question +- `graphify path "" ""` — dependency path between two symbols +- `graphify explain ""` — all nodes related to a concept + +This applies to YOU and to every subagent you spawn. Include this rule explicitly in every subagent prompt that involves code exploration. Do not skip graphify because files are "already known" or because you are executing a plan — the graph surfaces cross-file dependencies and INFERRED edges that grep and Read cannot find. + +Only use Read/Grep/Glob directly when: +1. graphify has already oriented you and you need to modify or debug specific lines +2. `graphify-out/graph.json` does not exist yet + +- If `graphify-out/wiki/index.md` exists, navigate it instead of reading raw files +- Read `graphify-out/GRAPH_REPORT.md` only for broad architecture review when query/path/explain do not surface enough context +- After modifying code files, run `graphify update .` to keep the graph current (AST-only, no API cost) +""" +def _cursor_install(project_dir: Path) -> None: + """Write .cursor/rules/graphify.mdc with alwaysApply: true.""" + rule_path = (project_dir or Path(".")) / _CURSOR_RULE_PATH + rule_path.parent.mkdir(parents=True, exist_ok=True) + if rule_path.exists() and rule_path.read_text(encoding="utf-8") == _CURSOR_RULE: + print(f"graphify rule at {rule_path} already configured (no change)") + return + # File is wholly graphify-owned. Overwrite on upgrade so older + # report-first wording does not silently linger (issue #580). + action = "updated" if rule_path.exists() else "written" + rule_path.write_text(_CURSOR_RULE, encoding="utf-8") + print(f"graphify rule {action} at {rule_path.resolve()}") + print() + print("Cursor will now always include the knowledge graph context.") + print("Run /graphify . first to build the graph if you haven't already.") +def _cursor_uninstall(project_dir: Path) -> None: + """Remove .cursor/rules/graphify.mdc.""" + rule_path = (project_dir or Path(".")) / _CURSOR_RULE_PATH + if not rule_path.exists(): + print("No graphify Cursor rule found - nothing to do") + return + rule_path.unlink() + print(f"graphify Cursor rule removed from {rule_path.resolve()}") +# Devin CLI — .windsurf/rules/graphify.md (always-on context) +# Devin reads .windsurf/rules/*.md files the same way Windsurf IDE does. +_DEVIN_RULES_PATH = Path(".windsurf") / "rules" / "graphify.md" +_DEVIN_RULES = """\ +## graphify + +This project has a graphify knowledge graph at graphify-out/. + +Rules: +- For codebase or architecture questions, when `graphify-out/graph.json` exists, first run `graphify query ""` (or `graphify path "" ""` / `graphify explain ""`). These return a scoped subgraph, usually much smaller than `GRAPH_REPORT.md` or raw grep output. +- If graphify-out/wiki/index.md exists, navigate it instead of reading raw files +- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain do not surface enough context +- After modifying code files in this session, run `graphify update .` to keep the graph current (AST-only, no API cost) +""" +def _devin_rules_install(project_dir: Path) -> None: + """Write .windsurf/rules/graphify.md for always-on Devin context.""" + rules_path = (project_dir or Path(".")) / _DEVIN_RULES_PATH + rules_path.parent.mkdir(parents=True, exist_ok=True) + if rules_path.exists() and rules_path.read_text(encoding="utf-8") == _DEVIN_RULES: + print(f" {rules_path} -> already configured (no change)") + return + action = "updated" if rules_path.exists() else "written" + rules_path.write_text(_DEVIN_RULES, encoding="utf-8") + print(f" rules {action} -> {rules_path}") +def _devin_rules_uninstall(project_dir: Path) -> None: + """Remove .windsurf/rules/graphify.md.""" + rules_path = (project_dir or Path(".")) / _DEVIN_RULES_PATH + if not rules_path.exists(): + return + rules_path.unlink() + print(f" rules removed -> {rules_path}") +_KILO_PLUGIN_JS = """\ +// graphify Kilo plugin +// Injects a knowledge graph reminder before bash tool calls when the graph exists. +import { existsSync } from "fs"; +import { join } from "path"; + +export const GraphifyPlugin = async ({ directory }) => { + let reminded = false; + + return { + "tool.execute.before": async (input, output) => { + if (reminded) return; + if (!existsSync(join(directory, "graphify-out", "graph.json"))) return; + + if (input.tool === "bash") { + // Separate with ';' not '&&' — Windows PowerShell 5.1 rejects '&&' as a + // statement separator ("not a valid statement separator"), which broke + // the first bash command in every OpenCode session on Windows (#1646). + // ';' works in PowerShell 5.1, Bash, and POSIX shells alike. + output.args.command = + 'echo "[graphify] Knowledge graph available. Read graphify-out/GRAPH_REPORT.md for god nodes and architecture context before searching files." ; ' + + output.args.command; + reminded = true; + } + }, + }; +}; +""" +_KILO_PLUGIN_PATH = Path(".kilo") / "plugins" / "graphify.js" +_KILO_CONFIG_JSON_PATH = Path(".kilo") / "kilo.json" +_KILO_CONFIG_JSONC_PATH = Path(".kilo") / "kilo.jsonc" +def _strip_json_comments(raw: str) -> str: + """Remove JSONC-style comments while leaving string content intact.""" + result: list[str] = [] + in_string = False + escaped = False + line_comment = False + block_comment = False + i = 0 + + while i < len(raw): + ch = raw[i] + nxt = raw[i + 1] if i + 1 < len(raw) else "" + + if line_comment: + if ch == "\n": + line_comment = False + result.append(ch) + i += 1 + continue + + if block_comment: + if ch == "*" and nxt == "/": + block_comment = False + i += 2 + else: + i += 1 + continue + + if in_string: + result.append(ch) + if escaped: + escaped = False + elif ch == "\\": + escaped = True + elif ch == '"': + in_string = False + i += 1 + continue + + if ch == "/" and nxt == "/": + line_comment = True + i += 2 + continue + if ch == "/" and nxt == "*": + block_comment = True + i += 2 + continue + + result.append(ch) + if ch == '"': + in_string = True + i += 1 + + return re.sub(r",(\s*[}\]])", r"\1", "".join(result)) +def _load_json_like(config_file: Path) -> dict: + if not config_file.exists(): + return {} + try: + raw = config_file.read_text(encoding="utf-8") + if config_file.suffix == ".jsonc": + raw = _strip_json_comments(raw) + loaded = json.loads(raw) + except (OSError, json.JSONDecodeError): + return {} + return loaded if isinstance(loaded, dict) else {} +def _kilo_config_path(project_dir: Path) -> Path: + kilo_dir = (project_dir or Path(".")) / ".kilo" + json_path = kilo_dir / _KILO_CONFIG_JSON_PATH.name + if json_path.exists(): + return json_path + jsonc_path = kilo_dir / _KILO_CONFIG_JSONC_PATH.name + if jsonc_path.exists(): + return jsonc_path + return json_path +def _kilo_config_write_path(project_dir: Path) -> Path: + """Write automated Kilo edits to kilo.json so existing JSONC stays untouched.""" + kilo_dir = (project_dir or Path(".")) / ".kilo" + return kilo_dir / _KILO_CONFIG_JSON_PATH.name +def _install_kilo_plugin(project_dir: Path) -> None: + """Write graphify.js plugin and register it without rewriting user JSONC.""" + plugin_file = project_dir / _KILO_PLUGIN_PATH + plugin_file.parent.mkdir(parents=True, exist_ok=True) + plugin_file.write_text(_KILO_PLUGIN_JS, encoding="utf-8") + print(f" {_KILO_PLUGIN_PATH} -> tool.execute.before hook written") + + config_file = _kilo_config_path(project_dir) + write_config_file = _kilo_config_write_path(project_dir) + write_config_file.parent.mkdir(parents=True, exist_ok=True) + config = _load_json_like(config_file) + plugins = config.get("plugin") + if not isinstance(plugins, list): + plugins = [] + config["plugin"] = plugins + entry = plugin_file.resolve().as_uri() + if entry not in plugins: + plugins.append(entry) + write_config_file.write_text(json.dumps(config, indent=2), encoding="utf-8") + print(f" {write_config_file.relative_to(project_dir)} -> plugin registered") + else: + print( + f" {config_file.relative_to(project_dir)} -> plugin already registered (no change)" + ) +def _uninstall_kilo_plugin(project_dir: Path) -> None: + """Remove graphify.js plugin and deregister it without rewriting user JSONC.""" + plugin_file = project_dir / _KILO_PLUGIN_PATH + if plugin_file.exists(): + plugin_file.unlink() + print(f" {_KILO_PLUGIN_PATH} -> removed") + + config_file = _kilo_config_path(project_dir) + if not config_file.exists(): + return + write_config_file = _kilo_config_write_path(project_dir) + config = _load_json_like(config_file) + plugins = config.get("plugin", []) + if not isinstance(plugins, list): + plugins = [] + entry = plugin_file.resolve().as_uri() + if entry in plugins: + config["plugin"] = [plugin for plugin in plugins if plugin != entry] + if not config["plugin"]: + config.pop("plugin") + write_config_file.parent.mkdir(parents=True, exist_ok=True) + write_config_file.write_text(json.dumps(config, indent=2), encoding="utf-8") + print( + f" {write_config_file.relative_to(project_dir)} -> plugin deregistered" + ) +# OpenCode tool.execute.before plugin — fires before every tool call. +# Injects a graph reminder into bash command output when graph.json exists. +_OPENCODE_PLUGIN_JS = """\ +// graphify OpenCode plugin +// Injects a knowledge graph reminder before bash tool calls when the graph exists. +// +// IMPORTANT: keep the reminder string free of backticks and $(...) constructs. +// The hook prepends `echo "" && ` to the user's bash command; +// backticks inside the double-quoted echo trigger bash command substitution, +// which both corrupts tool output and silently executes the very graphify +// command we are only suggesting. Plain words render fine in opencode's TUI. +import { existsSync } from "fs"; +import { join } from "path"; + +export const GraphifyPlugin = async ({ directory }) => { + let reminded = false; + + return { + "tool.execute.before": async (input, output) => { + if (reminded) return; + if (!existsSync(join(directory, "graphify-out", "graph.json"))) return; + + if (input.tool === "bash") { + // ';' not '&&' — Windows PowerShell 5.1 rejects '&&' as a statement + // separator, breaking the first bash command of the session (#1646). + output.args.command = + 'echo "[graphify] knowledge graph at graphify-out/. For focused questions, run graphify query with your question (scoped subgraph, usually much smaller than GRAPH_REPORT.md) instead of grepping raw files. Read GRAPH_REPORT.md only for broad architecture context." ; ' + + output.args.command; + reminded = true; + } + }, + }; +}; +""" +_OPENCODE_PLUGIN_PATH = Path(".opencode") / "plugins" / "graphify.js" +_OPENCODE_CONFIG_PATH = Path(".opencode") / "opencode.json" +def _install_opencode_plugin(project_dir: Path) -> None: + """Write graphify.js plugin and register it in opencode.json.""" + plugin_file = project_dir / _OPENCODE_PLUGIN_PATH + plugin_file.parent.mkdir(parents=True, exist_ok=True) + plugin_file.write_text(_OPENCODE_PLUGIN_JS, encoding="utf-8") + print(f" {_OPENCODE_PLUGIN_PATH} -> tool.execute.before hook written") + + config_file = project_dir / _OPENCODE_CONFIG_PATH + if config_file.exists(): + try: + config = json.loads(config_file.read_text(encoding="utf-8")) + except json.JSONDecodeError: + config = {} + else: + config = {} + + plugins = config.setdefault("plugin", []) + entry = _OPENCODE_PLUGIN_PATH.as_posix() + if entry not in plugins: + plugins.append(entry) + config_file.write_text(json.dumps(config, indent=2), encoding="utf-8") + print(f" {_OPENCODE_CONFIG_PATH} -> plugin registered") + else: + print(f" {_OPENCODE_CONFIG_PATH} -> plugin already registered (no change)") +def _uninstall_opencode_plugin(project_dir: Path) -> None: + """Remove graphify.js plugin and deregister from opencode.json.""" + plugin_file = project_dir / _OPENCODE_PLUGIN_PATH + if plugin_file.exists(): + plugin_file.unlink() + print(f" {_OPENCODE_PLUGIN_PATH} -> removed") + + config_file = project_dir / _OPENCODE_CONFIG_PATH + if not config_file.exists(): + return + try: + config = json.loads(config_file.read_text(encoding="utf-8")) + except json.JSONDecodeError: + return + plugins = config.get("plugin", []) + entry = _OPENCODE_PLUGIN_PATH.as_posix() + if entry in plugins: + plugins.remove(entry) + if not plugins: + config.pop("plugin") + config_file.write_text(json.dumps(config, indent=2), encoding="utf-8") + print(f" {_OPENCODE_CONFIG_PATH} -> plugin deregistered") +def _resolve_graphify_exe() -> str: + """Return the absolute path to the graphify executable. + + Falls back to bare 'graphify' if resolution fails. Using an absolute path + ensures the hook works in environments where the venv Scripts/ directory is + not on PATH (e.g. VS Code Codex extension on Windows). + """ + import shutil + found = shutil.which("graphify") + if found: + return found + # Derive from sys.executable: same Scripts/ (Windows) or bin/ (Unix) dir + scripts_dir = Path(sys.executable).parent + for name in ("graphify.exe", "graphify"): + candidate = scripts_dir / name + if candidate.exists(): + return str(candidate) + return "graphify" +def _install_codex_hook(project_dir: Path) -> None: + """Add graphify PreToolUse hook to .codex/hooks.json.""" + hooks_path = project_dir / ".codex" / "hooks.json" + hooks_path.parent.mkdir(parents=True, exist_ok=True) + + if hooks_path.exists(): + try: + existing = json.loads(hooks_path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + existing = {} + else: + existing = {} + + graphify_exe = _resolve_graphify_exe() + hook_entry = { + "hooks": { + "PreToolUse": [ + { + "matcher": "Bash", + "hooks": [{"type": "command", "command": f"{graphify_exe} hook-check"}], + } + ] + } + } + + pre_tool = existing.setdefault("hooks", {}).setdefault("PreToolUse", []) + existing["hooks"]["PreToolUse"] = [h for h in pre_tool if "graphify" not in str(h)] + existing["hooks"]["PreToolUse"].extend(hook_entry["hooks"]["PreToolUse"]) + hooks_path.write_text(json.dumps(existing, indent=2), encoding="utf-8") + print(f" .codex/hooks.json -> PreToolUse hook registered ({graphify_exe} hook-check)") +def _uninstall_codex_hook(project_dir: Path) -> None: + """Remove graphify PreToolUse hook from .codex/hooks.json.""" + hooks_path = project_dir / ".codex" / "hooks.json" + if not hooks_path.exists(): + return + try: + existing = json.loads(hooks_path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + return + pre_tool = existing.get("hooks", {}).get("PreToolUse", []) + filtered = [h for h in pre_tool if "graphify" not in str(h)] + existing["hooks"]["PreToolUse"] = filtered + hooks_path.write_text(json.dumps(existing, indent=2), encoding="utf-8") + print(f" .codex/hooks.json -> PreToolUse hook removed") +def _agents_install(project_dir: Path, platform: str) -> None: + """Write the graphify section to the local AGENTS.md for always-on platforms.""" + target = (project_dir or Path(".")) / "AGENTS.md" + + if target.exists(): + content = target.read_text(encoding="utf-8") + new_content = _replace_or_append_section( + content, _AGENTS_MD_MARKER, _always_on("agents-md") + ) + else: + new_content = _always_on("agents-md") + + if target.exists() and new_content == target.read_text(encoding="utf-8"): + print(f"graphify already configured in {target.resolve()} (no change)") + else: + target.write_text(new_content, encoding="utf-8") + print(f"graphify section written to {target.resolve()}") + + if platform == "codex": + _install_codex_hook(project_dir or Path(".")) + elif platform == "opencode": + _install_opencode_plugin(project_dir or Path(".")) + elif platform == "kilo": + _install_kilo_plugin(project_dir or Path(".")) + + print() + print( + f"{platform.capitalize()} will now check the knowledge graph before answering" + ) + print("codebase questions and rebuild it after code changes.") + if platform not in ("codex", "opencode", "kilo"): + print() + print("Note: unlike Claude Code, there is no PreToolUse hook equivalent for") + print( + f"{platform.capitalize()} — the AGENTS.md rules are the always-on mechanism." + ) +def _amp_legacy_cleanup() -> None: + """Best-effort removal of the pre-fix ~/.amp/skills/graphify install dir. + + Older graphify versions wrote the Amp skill to ~/.amp/skills, which Amp does + not search. Clean it up on install so a stale, never-loaded copy does not + linger. Failures are ignored (the new path is what matters). + """ + legacy = Path.home() / ".amp" / "skills" / "graphify" + if legacy.exists(): + shutil.rmtree(legacy, ignore_errors=True) + if not legacy.exists(): + print(f" legacy removed -> {legacy}") +def _amp_install(project_dir: Path | None = None) -> None: + """User-scope Amp install: skill into ~/.config/agents/skills + AGENTS.md.""" + _amp_legacy_cleanup() + _copy_skill_file("amp") + _agents_install(project_dir or Path("."), "amp") +def _amp_uninstall(project_dir: Path | None = None) -> None: + """User-scope Amp uninstall: remove the skill and the AGENTS.md section.""" + removed = _remove_skill_file("amp") + if removed: + print("skill removed") + _agents_uninstall(project_dir or Path("."), platform="amp") +def _agents_platform_install(project_dir: Path | None = None) -> None: + """`graphify agents install`: skill into ~/.agents/skills + AGENTS.md. + + The amp-twin of the generic Agent-Skills target. Mirrors _amp_install but + lands the skill at the spec's user-global ~/.agents/skills (set in + _platform_skill_destination). Wiring AGENTS.md keeps it honest with the + rendered hooks reference, which points at `graphify agents install`. The bare + `graphify install --platform agents` path stays skill-only (via install()), + exactly as amp's `--platform amp` does. + """ + _copy_skill_file("agents") + _agents_install(project_dir or Path("."), "agents") +def _agents_platform_uninstall(project_dir: Path | None = None) -> None: + """`graphify agents uninstall`: remove the skill and the AGENTS.md section.""" + removed = _remove_skill_file("agents") + if removed: + print("skill removed") + _agents_uninstall(project_dir or Path("."), platform="agents") +def _project_install(platform_name: str, project_dir: Path | None = None) -> None: + """Install platform skill/config files in the current project.""" + project_dir = project_dir or Path(".") + platform_name = _canonical_platform(platform_name) + if platform_name in ("claude", "windows"): + install(platform=platform_name, project=True, project_dir=project_dir) + claude_install(project_dir) + _print_project_git_add_hint([project_dir / ".claude", project_dir / "CLAUDE.md"]) + elif platform_name == "gemini": + gemini_install(project_dir, project=True) + elif platform_name == "cursor": + _cursor_install(project_dir) + _print_project_git_add_hint([project_dir / ".cursor"]) + elif platform_name == "kiro": + _kiro_install(project_dir) + _print_project_git_add_hint([project_dir / ".kiro"]) + elif platform_name in ("aider", "amp", "codex", "opencode", "claw", "droid", "trae", "trae-cn", "hermes"): + skill_dst = _copy_skill_file(platform_name, project=True, project_dir=project_dir) + _agents_install(project_dir, platform_name) + hint_paths = [_project_scope_root(skill_dst, project_dir), project_dir / "AGENTS.md"] + if platform_name == "opencode": + hint_paths.append(project_dir / ".opencode") + elif platform_name == "codex": + hint_paths.append(project_dir / ".codex") + _print_project_git_add_hint(hint_paths) + elif platform_name == "devin": + skill_dst = _copy_skill_file("devin", project=True, project_dir=project_dir) + _devin_rules_install(project_dir) + _print_project_git_add_hint([_project_scope_root(skill_dst, project_dir), project_dir / ".windsurf"]) + elif platform_name == "antigravity": + # Project-scoped: skill in .agents/skills/ PLUS the .agents/rules + + # .agents/workflows always-on layer (previously this path wrote only the + # skill, leaving the rules/workflows the uninstall path removes unset). + skill_dst = _copy_skill_file("antigravity", project=True, project_dir=project_dir) + _antigravity_finalize(skill_dst, project_dir) + _print_project_git_add_hint([_project_scope_root(skill_dst, project_dir), project_dir / ".agents"]) + elif platform_name in ("copilot", "pi", "kimi", "agents"): + # Skill-only project install: drop SKILL.md (+ references) at the scope + # root. `agents` -> ./.agents/skills/graphify/SKILL.md. + skill_dst = _copy_skill_file(platform_name, project=True, project_dir=project_dir) + _print_project_git_add_hint([_project_scope_root(skill_dst, project_dir)]) + else: + install(platform=platform_name, project=True, project_dir=project_dir) +def _project_uninstall(platform_name: str, project_dir: Path | None = None) -> None: + """Remove project-scoped platform skill/config files only.""" + project_dir = project_dir or Path(".") + platform_name = _canonical_platform(platform_name) + if platform_name in ("claude", "windows"): + _remove_skill_file(platform_name, project=True, project_dir=project_dir) + _remove_claude_skill_registration(project_dir) + claude_uninstall(project_dir, project=True) + elif platform_name == "gemini": + gemini_uninstall(project_dir, project=True) + elif platform_name == "cursor": + _cursor_uninstall(project_dir) + elif platform_name == "kiro": + _kiro_uninstall(project_dir) + elif platform_name in ("aider", "amp", "codex", "opencode", "claw", "droid", "trae", "trae-cn", "hermes"): + _remove_skill_file(platform_name, project=True, project_dir=project_dir) + _agents_uninstall(project_dir, platform=platform_name) + if platform_name == "codex": + _uninstall_codex_hook(project_dir) + elif platform_name == "antigravity": + _antigravity_uninstall(project_dir, project=True) + elif platform_name == "devin": + removed = _remove_skill_file("devin", project=True, project_dir=project_dir) + _devin_rules_uninstall(project_dir) + if not removed: + print("nothing to remove") + elif platform_name in ("copilot", "pi", "kimi", "agents"): + removed = _remove_skill_file(platform_name, project=True, project_dir=project_dir) + if not removed: + print("nothing to remove") + elif platform_name == "codebuddy": + codebuddy_uninstall(project_dir) + else: + _remove_skill_file(platform_name, project=True, project_dir=project_dir) +def _project_uninstall_all(project_dir: Path | None = None) -> None: + """Remove project-scoped install files without touching user-scope installs.""" + project_dir = project_dir or Path(".") + print("Uninstalling project-scoped graphify files...\n") + for platform_name in _PLATFORM_CONFIG: + _project_uninstall(platform_name, project_dir) + for platform_name in ("gemini", "cursor"): + _project_uninstall(platform_name, project_dir) + print("\nDone.") +def _agents_uninstall(project_dir: Path, platform: str = "") -> None: + """Remove the graphify section from the local AGENTS.md.""" + target = (project_dir or Path(".")) / "AGENTS.md" + + if not target.exists(): + print("No AGENTS.md found in current directory - nothing to do") + if platform == "opencode": + _uninstall_opencode_plugin(project_dir or Path(".")) + elif platform == "kilo": + _uninstall_kilo_plugin(project_dir or Path(".")) + return + + content = target.read_text(encoding="utf-8") + if _AGENTS_MD_MARKER not in content: + print("graphify section not found in AGENTS.md - nothing to do") + if platform == "opencode": + _uninstall_opencode_plugin(project_dir or Path(".")) + elif platform == "kilo": + _uninstall_kilo_plugin(project_dir or Path(".")) + return + + cleaned = re.sub( + r"\n*## graphify\n.*?(?=\n## |\Z)", + "", + content, + flags=re.DOTALL, + ).rstrip() + if cleaned: + target.write_text(cleaned + "\n", encoding="utf-8") + print(f"graphify section removed from {target.resolve()}") + else: + target.unlink() + print(f"AGENTS.md was empty after removal - deleted {target.resolve()}") + + if platform == "opencode": + _uninstall_opencode_plugin(project_dir or Path(".")) + elif platform == "kilo": + _uninstall_kilo_plugin(project_dir or Path(".")) +def _kilo_uninstall_global() -> list[str]: + removed = [] + command_dst = Path.home() / ".config" / "kilo" / "command" / "graphify.md" + if command_dst.exists(): + command_dst.unlink() + removed.append(f"command removed: {command_dst}") + try: + command_dst.parent.rmdir() + except OSError: + pass + + skill_dst = Path.home() / _PLATFORM_CONFIG["kilo"]["skill_dst"] + if skill_dst.exists(): + skill_dst.unlink() + removed.append(f"skill removed: {skill_dst}") + version_file = skill_dst.parent / ".graphify_version" + if version_file.exists(): + version_file.unlink() + for d in ( + skill_dst.parent, + skill_dst.parent.parent, + skill_dst.parent.parent.parent, + ): + try: + d.rmdir() + except OSError: + break + + return removed +def _kilo_install(project_dir: Path) -> None: + """Install native Kilo skill + command globally and always-on project wiring locally.""" + install(platform="kilo") + _agents_install(project_dir or Path("."), "kilo") +def _kilo_uninstall(project_dir: Path) -> None: + """Remove Kilo always-on project wiring and global skill/command files.""" + _agents_uninstall(project_dir or Path("."), platform="kilo") + removed = _kilo_uninstall_global() + print("; ".join(removed) if removed else "nothing to remove") +def claude_install(project_dir: Path | None = None) -> None: + """Write the graphify section to the local CLAUDE.md.""" + target = (project_dir or Path(".")) / "CLAUDE.md" + + if target.exists(): + content = target.read_text(encoding="utf-8") + new_content = _replace_or_append_section( + content, _CLAUDE_MD_MARKER, _always_on("claude-md") + ) + else: + new_content = _always_on("claude-md") + + if target.exists() and new_content == target.read_text(encoding="utf-8"): + print(f"graphify already configured in {target.resolve()} (no change)") + else: + target.write_text(new_content, encoding="utf-8") + print(f"graphify section written to {target.resolve()}") + + # Always re-install the Claude Code PreToolUse hook so an old hook + # payload (e.g. pre-issue-#580 wording) is replaced on upgrade. + _install_claude_hook(project_dir or Path(".")) + + print() + print("Claude Code will now check the knowledge graph before answering") + print("codebase questions and rebuild it after code changes.") +def _install_claude_hook(project_dir: Path) -> None: + """Add graphify PreToolUse hook to .claude/settings.json.""" + settings_path = project_dir / ".claude" / "settings.json" + settings_path.parent.mkdir(parents=True, exist_ok=True) + + if settings_path.exists(): + try: + settings = json.loads(settings_path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + settings = {} + else: + settings = {} + + hooks = settings.setdefault("hooks", {}) + pre_tool = hooks.setdefault("PreToolUse", []) + + hooks["PreToolUse"] = [h for h in pre_tool if not (h.get("matcher") in ("Glob|Grep", "Bash", "Read|Glob") and "graphify" in str(h))] + hooks["PreToolUse"].extend(_claude_pretooluse_hooks()) + settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") + print(f" .claude/settings.json -> PreToolUse hooks registered (Bash search + Read/Glob)") +def _uninstall_claude_hook(project_dir: Path) -> None: + """Remove the graphify PreToolUse hook from .claude/settings.json and its + local-only sibling .claude/settings.local.json. + + A user may relocate the hook into settings.local.json so it is not committed + to a shared repo, so uninstall has to clean whichever file holds it (#1731). + """ + claude_dir = project_dir / ".claude" + for name in ("settings.json", "settings.local.json"): + _strip_graphify_hook(claude_dir / name) +def _strip_graphify_hook(settings_path: Path) -> None: + """Drop graphify PreToolUse hooks from a single Claude settings file, if present.""" + if not settings_path.exists(): + return + try: + settings = json.loads(settings_path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + return + pre_tool = settings.get("hooks", {}).get("PreToolUse", []) + filtered = [h for h in pre_tool if not (h.get("matcher") in ("Glob|Grep", "Bash", "Read|Glob") and "graphify" in str(h))] + if len(filtered) == len(pre_tool): + return + settings["hooks"]["PreToolUse"] = filtered + settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") + print(f" .claude/{settings_path.name} -> PreToolUse hook removed") +def uninstall_all(project_dir: Path | None = None, purge: bool = False) -> None: + """Remove graphify from every platform detected in the current project.""" + pd = project_dir or Path(".") + print("Uninstalling graphify from all detected platforms...\n") + + # Skill-file / config-section uninstallers + claude_uninstall(pd) + codebuddy_uninstall(pd) + gemini_uninstall(pd) + vscode_uninstall(pd) + _cursor_uninstall(pd) + _kiro_uninstall(pd) + _antigravity_uninstall(pd) + # AGENTS.md covers: codex, aider, opencode, claw, droid, trae, trae-cn, hermes, copilot + _agents_uninstall(pd) + # Amp also drops a user-scope skill at ~/.config/agents/skills, which the + # AGENTS.md cleanup above does not touch. + _remove_skill_file("amp") + # The generic agents platform's user-scope skill lives at ~/.agents/skills, + # which neither the AGENTS.md cleanup nor amp's removal reaches. + _remove_skill_file("agents") + _uninstall_opencode_plugin(pd) + _uninstall_codex_hook(pd) + + # Git hook + try: + from graphify.hooks import uninstall as hook_uninstall + result = hook_uninstall(pd) + if result: + print(result) + except Exception: + pass + + if purge: + import shutil as _shutil + out = pd / _GRAPHIFY_OUT + if out.exists(): + _shutil.rmtree(out) + print(f"\n {_GRAPHIFY_OUT}/ -> deleted (--purge)") + else: + print(f"\n {_GRAPHIFY_OUT}/ -> not found (nothing to purge)") + + print("\nDone. Run 'pip uninstall graphifyy' to remove the package itself.") +def claude_uninstall(project_dir: Path | None = None, *, project: bool = False) -> None: + """Remove the graphify skill tree (SKILL.md + references/) and the graphify + section from CLAUDE.md and its local-only variants, plus the PreToolUse hook. + + Mirrors gemini_uninstall: the bare `graphify uninstall` and `graphify claude + uninstall` must remove the installed skill, not just strip CLAUDE.md, or the + progressive-disclosure tree (SKILL.md + references/) is orphaned (#1121). + + A user may relocate the section/hook into the local-only files Claude Code + supports so they are not committed to a shared repo, so uninstall also cleans + CLAUDE.local.md, .claude/CLAUDE.local.md and .claude/settings.local.json (#1731). + """ + project_dir = project_dir or Path(".") + _remove_skill_file("claude", project=project, project_dir=project_dir) + + md_targets = [ + project_dir / "CLAUDE.md", + project_dir / "CLAUDE.local.md", + project_dir / ".claude" / "CLAUDE.local.md", + ] + existing = [t for t in md_targets if t.exists()] + removed_any = False + for target in existing: + # Not short-circuited: every present file must be cleaned, not just the first. + if _strip_graphify_md_section(target): + removed_any = True + + if not existing: + print("No CLAUDE.md found in current directory - nothing to do") + elif not removed_any: + print("graphify section not found in CLAUDE.md - nothing to do") + + _uninstall_claude_hook(project_dir) +def _strip_graphify_md_section(target: Path) -> bool: + """Strip the ## graphify section from one CLAUDE.md-style file. + + Returns True if a section was removed. Deletes the file if nothing else + remains after removal. + """ + try: + content = target.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError): + # An unreadable/undecodable CLAUDE.md-style file (e.g. non-UTF-8, or a + # directory of that name) must not abort uninstall - nothing to strip. + return False + if _CLAUDE_MD_MARKER not in content: + return False + # Remove the ## graphify section: from the marker to the next ## heading or EOF + cleaned = re.sub( + r"\n*## graphify\n.*?(?=\n## |\Z)", + "", + content, + flags=re.DOTALL, + ).rstrip() + if cleaned: + target.write_text(cleaned + "\n", encoding="utf-8") + print(f"graphify section removed from {target.resolve()}") + else: + target.unlink() + print(f"{target.name} was empty after removal - deleted {target.resolve()}") + return True +def codebuddy_install(project_dir: Path | None = None) -> None: + """Install the graphify skill and CODEBUDDY.md section for CodeBuddy.""" + _copy_skill_file("codebuddy", project=bool(project_dir), project_dir=project_dir) + target = (project_dir or Path(".")) / "CODEBUDDY.md" + + if target.exists(): + content = target.read_text(encoding="utf-8") + new_content = _replace_or_append_section( + content, _CODEBUDDY_MD_MARKER, _always_on("claude-md") + ) + else: + new_content = _always_on("claude-md") + + if target.exists() and new_content == target.read_text(encoding="utf-8"): + print(f"graphify already configured in {target.resolve()} (no change)") + else: + target.write_text(new_content, encoding="utf-8") + print(f"graphify section written to {target.resolve()}") + + # Also write CodeBuddy PreToolUse hook to .codebuddy/settings.json + _install_codebuddy_hook(project_dir or Path(".")) + + print() + print("CodeBuddy will now check the knowledge graph before answering") + print("codebase questions and rebuild it after code changes.") +def _install_codebuddy_hook(project_dir: Path) -> None: + """Add graphify PreToolUse hook to .codebuddy/settings.json.""" + settings_path = project_dir / ".codebuddy" / "settings.json" + settings_path.parent.mkdir(parents=True, exist_ok=True) + + if settings_path.exists(): + try: + settings = json.loads(settings_path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + settings = {} + else: + settings = {} + + hooks = settings.setdefault("hooks", {}) + pre_tool = hooks.setdefault("PreToolUse", []) + + hooks["PreToolUse"] = [h for h in pre_tool if not (h.get("matcher") in ("Glob|Grep", "Bash", "Read|Glob") and "graphify" in str(h))] + hooks["PreToolUse"].extend(_claude_pretooluse_hooks()) + settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") + print(f" .codebuddy/settings.json -> PreToolUse hooks registered") +def _uninstall_codebuddy_hook(project_dir: Path) -> None: + """Remove graphify PreToolUse hook from .codebuddy/settings.json.""" + settings_path = project_dir / ".codebuddy" / "settings.json" + if not settings_path.exists(): + return + try: + settings = json.loads(settings_path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + return + pre_tool = settings.get("hooks", {}).get("PreToolUse", []) + filtered = [h for h in pre_tool if not (h.get("matcher") in ("Glob|Grep", "Bash", "Read|Glob") and "graphify" in str(h))] + if len(filtered) == len(pre_tool): + return + settings["hooks"]["PreToolUse"] = filtered + settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") + print(f" .codebuddy/settings.json -> PreToolUse hook removed") +def codebuddy_uninstall(project_dir: Path | None = None, *, project: bool = False) -> None: + """Remove the graphify skill tree (SKILL.md + references/) and the CODEBUDDY.md section.""" + project_dir = project_dir or Path(".") + _remove_skill_file("codebuddy", project=project, project_dir=project_dir) + target = project_dir / "CODEBUDDY.md" + + if not target.exists(): + print("No CODEBUDDY.md found in current directory - nothing to do") + return + + content = target.read_text(encoding="utf-8") + if _CODEBUDDY_MD_MARKER not in content: + print("graphify section not found in CODEBUDDY.md - nothing to do") + return + + # Remove the ## graphify section: from the marker to the next ## heading or EOF + cleaned = re.sub( + r"\n*## graphify\n.*?(?=\n## |\Z)", + "", + content, + flags=re.DOTALL, + ).rstrip() + if cleaned: + target.write_text(cleaned + "\n", encoding="utf-8") + print(f"graphify section removed from {target.resolve()}") + else: + target.unlink() + print(f"CODEBUDDY.md was empty after removal - deleted {target.resolve()}") + + _uninstall_codebuddy_hook(project_dir or Path(".")) From 9fc30e680d8ab98a2cc4418ae5a62dbbe192c2ca Mon Sep 17 00:00:00 2001 From: tpateeq Date: Thu, 9 Jul 2026 01:11:37 +0530 Subject: [PATCH 040/292] refactor(extract): migrate 18 bespoke language extractors to extractors/ package Continues the graphify/extract.py -> graphify/extractors/ split (MIGRATION.md, upstream #1212), which already moved blade/zig/elixir/razor. Moves the independent bespoke extractors whose closures are fully private (no shared _extract_generic core, no shared mutable caches), verbatim, following the documented invariants: dart, rust, go, powershell (+psd1 manifest), fortran, sql, dm (dm/dmm/dmi/dmf), bash, apex, terraform, sln, pascal_forms (delphi .dfm + lazarus .lfm), json_config Each language's private helper funcs and constants move with it; only the `extract_` entry points (plus fortran's _cpp_preprocess, which has a direct unit test) are re-exported from extract.py's facade block, so every existing importer (__main__.py, watch.py, tests) is unchanged and object identity is preserved. Registry (extractors/__init__.py) grows 4 -> 22 langs. Verified: AST closure-privacy analysis (no symbol referenced from outside its moved set except via the facade); byte-identity of every moved span; extract._DISPATCH still resolves every extension; ruff clean; skillgen --check OK. extract.py drops 17,054 -> 13,121 LOC. Full suite unchanged: 3036 passed, 29 skipped (excluding env-only openai tests). Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/extract.py | 4167 +-------------------------- graphify/extractors/MIGRATION.md | 16 +- graphify/extractors/__init__.py | 31 + graphify/extractors/apex.py | 215 ++ graphify/extractors/bash.py | 230 ++ graphify/extractors/dart.py | 528 ++++ graphify/extractors/dm.py | 494 ++++ graphify/extractors/fortran.py | 309 ++ graphify/extractors/go.py | 396 +++ graphify/extractors/json_config.py | 206 ++ graphify/extractors/pascal_forms.py | 196 ++ graphify/extractors/powershell.py | 496 ++++ graphify/extractors/rust.py | 410 +++ graphify/extractors/sln.py | 81 + graphify/extractors/sql.py | 276 ++ graphify/extractors/terraform.py | 181 ++ 16 files changed, 4181 insertions(+), 4051 deletions(-) create mode 100644 graphify/extractors/apex.py create mode 100644 graphify/extractors/bash.py create mode 100644 graphify/extractors/dart.py create mode 100644 graphify/extractors/dm.py create mode 100644 graphify/extractors/fortran.py create mode 100644 graphify/extractors/go.py create mode 100644 graphify/extractors/json_config.py create mode 100644 graphify/extractors/pascal_forms.py create mode 100644 graphify/extractors/powershell.py create mode 100644 graphify/extractors/rust.py create mode 100644 graphify/extractors/sln.py create mode 100644 graphify/extractors/sql.py create mode 100644 graphify/extractors/terraform.py diff --git a/graphify/extract.py b/graphify/extract.py index 16a3f753a..332da7e1f 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -28,13 +28,26 @@ _make_id, _read_text, ) +from graphify.extractors.apex import extract_apex # noqa: F401 +from graphify.extractors.bash import extract_bash # noqa: F401 from graphify.extractors.blade import extract_blade # noqa: F401 from graphify.extractors.csharp import ( _resolve_cross_file_csharp_imports, _resolve_csharp_type_references, ) +from graphify.extractors.dart import extract_dart # noqa: F401 +from graphify.extractors.dm import extract_dm, extract_dmf, extract_dmi, extract_dmm # noqa: F401 from graphify.extractors.elixir import extract_elixir # noqa: F401 +from graphify.extractors.fortran import _cpp_preprocess, extract_fortran # noqa: F401 +from graphify.extractors.go import extract_go # noqa: F401 +from graphify.extractors.json_config import extract_json # noqa: F401 +from graphify.extractors.pascal_forms import extract_delphi_form, extract_lazarus_form # noqa: F401 +from graphify.extractors.powershell import extract_powershell, extract_powershell_manifest # noqa: F401 from graphify.extractors.razor import extract_razor # noqa: F401 +from graphify.extractors.rust import extract_rust # noqa: F401 +from graphify.extractors.sln import extract_sln # noqa: F401 +from graphify.extractors.sql import extract_sql # noqa: F401 +from graphify.extractors.terraform import extract_terraform # noqa: F401 from graphify.extractors.zig import extract_zig # noqa: F401 from graphify.security import sanitize_metadata from graphify.paths import disambiguate_ambiguous_candidates @@ -67,10 +80,6 @@ def _safe_extract(extractor: Callable, path: Path) -> dict: return {"nodes": [], "edges": [], "error": f"{type(e).__name__}: {e}"} - - - - def _file_node_id(rel_path: Path) -> str: """File-level node ID matching the skill.md spec: ``{parent_dir}_{stem}`` — one parent directory level, no extension. ``rel_path`` MUST be relative to @@ -624,7 +633,6 @@ class LanguageConfig: # ── Generic helpers ─────────────────────────────────────────────────────────── - _PYTHON_TYPE_CONTAINERS = frozenset({ "list", "dict", "set", "tuple", "frozenset", "type", "List", "Dict", "Set", "Tuple", "FrozenSet", "Type", @@ -1010,97 +1018,6 @@ def _java_annotation_names(declaration_node, source: bytes) -> list[str]: return names -_GO_PREDECLARED_TYPES = frozenset({ - "bool", "byte", "complex64", "complex128", "error", "float32", "float64", - "int", "int8", "int16", "int32", "int64", "rune", "string", - "uint", "uint8", "uint16", "uint32", "uint64", "uintptr", "any", "comparable", -}) - - -def _go_collect_type_refs(node, source: bytes, generic: bool, out: list[tuple[str, str]]) -> None: - """Walk a Go type expression; append (name, role) tuples.""" - if node is None: - return - t = node.type - if t == "type_identifier": - text = _read_text(node, source) - if text and text not in _GO_PREDECLARED_TYPES: - out.append((text, "generic_arg" if generic else "type")) - return - if t == "qualified_type": - text = _read_text(node, source).rsplit(".", 1)[-1] - if text and text not in _GO_PREDECLARED_TYPES: - out.append((text, "generic_arg" if generic else "type")) - return - if t == "generic_type": - type_field = node.child_by_field_name("type") - if type_field is not None: - sub: list[tuple[str, str]] = [] - _go_collect_type_refs(type_field, source, generic, sub) - out.extend(sub) - for c in node.children: - if c.type == "type_arguments": - for arg in c.children: - if arg.is_named: - _go_collect_type_refs(arg, source, True, out) - return - if t in ("pointer_type", "slice_type", "array_type", "map_type", - "channel_type", "parenthesized_type"): - for c in node.children: - if c.is_named: - _go_collect_type_refs(c, source, generic, out) - return - if node.is_named: - for c in node.children: - if c.is_named: - _go_collect_type_refs(c, source, generic, out) - - -def _rust_collect_type_refs(node, source: bytes, generic: bool, out: list[tuple[str, str]]) -> None: - """Walk a Rust type expression; append (name, role) tuples.""" - if node is None: - return - t = node.type - if t == "primitive_type": - return - if t == "type_identifier": - text = _read_text(node, source) - if text: - out.append((text, "generic_arg" if generic else "type")) - return - if t == "scoped_type_identifier": - text = _read_text(node, source).rsplit("::", 1)[-1] - if text: - out.append((text, "generic_arg" if generic else "type")) - return - if t == "generic_type": - name_node = node.child_by_field_name("type") - if name_node is None: - for c in node.children: - if c.type in ("type_identifier", "scoped_type_identifier"): - name_node = c - break - if name_node is not None: - text = _read_text(name_node, source).rsplit("::", 1)[-1] - if text: - out.append((text, "generic_arg" if generic else "type")) - for c in node.children: - if c.type == "type_arguments": - for arg in c.children: - if arg.is_named: - _rust_collect_type_refs(arg, source, True, out) - return - if t in ("reference_type", "pointer_type", "array_type", "tuple_type", "slice_type"): - for c in node.children: - if c.is_named: - _rust_collect_type_refs(c, source, generic, out) - return - if node.is_named: - for c in node.children: - if c.is_named: - _rust_collect_type_refs(c, source, generic, out) - - def _php_name_text(node, source: bytes) -> str | None: """Return the unqualified name text from a PHP `name`/`qualified_name` node.""" if node is None: @@ -6474,215 +6391,6 @@ def extract_csharp(path: Path) -> dict: return _extract_generic(path, _CSHARP_CONFIG) -def extract_apex(path: Path) -> dict: - """Extract classes, interfaces, enums, methods, and Salesforce constructs from - Apex .cls and .trigger files using regex (no tree-sitter grammar on PyPI).""" - import re as _re - try: - source = path.read_text(encoding="utf-8", errors="replace") - except OSError: - return {"nodes": [], "edges": []} - - str_path = str(path) - stem = _file_stem(path) - file_nid = _make_id(str_path) - - nodes: list[dict] = [] - edges: list[dict] = [] - seen_ids: set[str] = set() - - def add_node(nid: str, label: str, line: int) -> None: - if nid not in seen_ids: - seen_ids.add(nid) - nodes.append({ - "id": nid, - "label": label, - "file_type": "code", - "source_file": str_path, - "source_location": f"L{line}", - }) - - def add_edge(src: str, tgt: str, relation: str, line: int, - confidence: str = "EXTRACTED") -> None: - edges.append({ - "source": src, - "target": tgt, - "relation": relation, - "confidence": confidence, - "source_file": str_path, - "source_location": f"L{line}", - "weight": 1.0, - }) - - add_node(file_nid, path.name, 1) - - lines = source.splitlines() - - _ACCESS = r"(?:public|private|protected|global|webService)?" - _SHARING = r"(?:\s+(?:with|without|inherited)\s+sharing)?" - _MOD = r"(?:\s+(?:abstract|virtual|override|static|final|transient|testMethod))?" - _ANNOTATION = r"(?:\s*@\w+(?:\s*\([^)]*\))?\s*)*" - - cls_re = _re.compile( - rf"^{_ANNOTATION}\s*{_ACCESS}{_SHARING}{_MOD}\s*class\s+(\w+)" - rf"(?:\s+extends\s+(\w+))?(?:\s+implements\s+([\w,\s]+))?\s*\{{?", - _re.IGNORECASE, - ) - iface_re = _re.compile( - rf"^{_ANNOTATION}\s*{_ACCESS}{_SHARING}{_MOD}\s*interface\s+(\w+)" - rf"(?:\s+extends\s+([\w,\s]+))?\s*\{{?", - _re.IGNORECASE, - ) - enum_re = _re.compile( - rf"^{_ANNOTATION}\s*{_ACCESS}{_SHARING}{_MOD}\s*enum\s+(\w+)\s*\{{?", - _re.IGNORECASE, - ) - trigger_re = _re.compile( - r"^\s*trigger\s+(\w+)\s+on\s+(\w+)\s*\(", - _re.IGNORECASE, - ) - method_re = _re.compile( - rf"^{_ANNOTATION}\s*{_ACCESS}{_MOD}\s*(?:static\s+)?[\w<>\[\]]+\s+(\w+)\s*\([^)]*\)\s*(?:throws\s+\w+\s*)?\{{?", - _re.IGNORECASE, - ) - annotation_re = _re.compile(r"@(\w+)", _re.IGNORECASE) - soql_re = _re.compile(r"\[\s*SELECT\b[^\]]+FROM\s+(\w+)", _re.IGNORECASE) - dml_re = _re.compile(r"\b(insert|update|delete|upsert|merge|undelete)\s+\w", _re.IGNORECASE) - - _CONTROL_FLOW = frozenset({ - "if", "else", "for", "while", "do", "switch", "try", "catch", - "finally", "return", "throw", "new", "void", "null", - "true", "false", "this", "super", "class", "interface", "enum", - "trigger", "on", - }) - - current_class_nid: str | None = None - pending_annotations: list[str] = [] - - for lineno, line_text in enumerate(lines, start=1): - stripped = line_text.strip() - - if stripped.startswith("@"): - for m in annotation_re.finditer(stripped): - pending_annotations.append(m.group(1).lower()) - continue - - tm = trigger_re.match(stripped) - if tm: - trig_name, sobject = tm.group(1), tm.group(2) - trig_nid = _make_id(stem, trig_name) - add_node(trig_nid, trig_name, lineno) - add_edge(file_nid, trig_nid, "contains", lineno) - sob_nid = _make_id(sobject) - if sob_nid not in seen_ids: - add_node(sob_nid, sobject, lineno) - add_edge(trig_nid, sob_nid, "uses", lineno, confidence="INFERRED") - current_class_nid = trig_nid - pending_annotations = [] - continue - - cm = cls_re.match(stripped) - if cm: - class_name = cm.group(1) - if class_name.lower() in _CONTROL_FLOW: - pending_annotations = [] - continue - class_nid = _make_id(stem, class_name) - add_node(class_nid, class_name, lineno) - add_edge(file_nid, class_nid, "contains", lineno) - if cm.group(2): - base = cm.group(2).strip() - base_nid = _make_id(stem, base) - if base_nid not in seen_ids: - base_nid = _make_id(base) - if base_nid not in seen_ids: - add_node(base_nid, base, lineno) - add_edge(class_nid, base_nid, "extends", lineno, confidence="INFERRED") - if cm.group(3): - for iface in cm.group(3).split(","): - iface = iface.strip() - if iface: - iface_nid = _make_id(stem, iface) - if iface_nid not in seen_ids: - iface_nid = _make_id(iface) - if iface_nid not in seen_ids: - add_node(iface_nid, iface, lineno) - add_edge(class_nid, iface_nid, "implements", lineno, confidence="INFERRED") - current_class_nid = class_nid - pending_annotations = [] - continue - - im = iface_re.match(stripped) - if im: - iface_name = im.group(1) - if iface_name.lower() in _CONTROL_FLOW: - pending_annotations = [] - continue - iface_nid = _make_id(stem, iface_name) - add_node(iface_nid, iface_name, lineno) - add_edge(file_nid if current_class_nid is None else current_class_nid, - iface_nid, "contains", lineno) - if im.group(2): - for parent in im.group(2).split(","): - parent = parent.strip() - if parent: - parent_nid = _make_id(stem, parent) - if parent_nid not in seen_ids: - parent_nid = _make_id(parent) - if parent_nid not in seen_ids: - add_node(parent_nid, parent, lineno) - add_edge(iface_nid, parent_nid, "extends", lineno, confidence="INFERRED") - pending_annotations = [] - continue - - em = enum_re.match(stripped) - if em: - enum_name = em.group(1) - if enum_name.lower() in _CONTROL_FLOW: - pending_annotations = [] - continue - enum_nid = _make_id(stem, enum_name) - add_node(enum_nid, enum_name, lineno) - add_edge(file_nid if current_class_nid is None else current_class_nid, - enum_nid, "contains", lineno) - pending_annotations = [] - continue - - if current_class_nid is not None: - mm = method_re.match(stripped) - if mm: - method_name = mm.group(1) - if method_name.lower() not in _CONTROL_FLOW: - method_nid = _make_id(current_class_nid, method_name) - method_label = f".{method_name}()" - add_node(method_nid, method_label, lineno) - add_edge(current_class_nid, method_nid, "method", lineno) - if "auraenabled" in pending_annotations or "invocablemethod" in pending_annotations: - add_edge(file_nid, method_nid, "contains", lineno, confidence="INFERRED") - pending_annotations = [] - continue - - pending_annotations = [] - - for sm in soql_re.finditer(line_text): - sobject = sm.group(1) - sob_nid = _make_id(sobject) - if sob_nid not in seen_ids: - add_node(sob_nid, sobject, lineno) - src = current_class_nid or file_nid - add_edge(src, sob_nid, "uses", lineno, confidence="INFERRED") - - for dm in dml_re.finditer(line_text): - dml_op = dm.group(1).lower() - dml_nid = _make_id(f"dml_{dml_op}") - if dml_nid not in seen_ids: - add_node(dml_nid, dml_op, lineno) - src = current_class_nid or file_nid - add_edge(src, dml_nid, "uses", lineno, confidence="INFERRED") - - return {"nodes": nodes, "edges": edges} - - def extract_kotlin(path: Path) -> dict: """Extract classes, objects, functions, and imports from a .kt/.kts file.""" return _extract_generic(path, _KOTLIN_CONFIG) @@ -6698,529 +6406,6 @@ def extract_php(path: Path) -> dict: return _extract_generic(path, _PHP_CONFIG) - - -def extract_dart(path: Path) -> dict: - """Extract classes, mixins, functions, imports, generic calls, and annotations from a .dart file using regex.""" - try: - src = path.read_text(encoding="utf-8", errors="replace") - except OSError: - return {"error": f"cannot read {path}"} - - # Remove inline and multi-line comments while leaving string literals untouched to prevent stripping URLs/paths inside strings - comment_string_pattern = re.compile( - r'"""(?:\\.|[\s\S])*?"""' - r"|'''(?:\\.|[\s\S])*?'''" - r'|"(?:\\.|[^"\\])*"' - r"|'(?:\\.|[^'\\])*'" - r"|/\*[\s\S]*?\*/" - r"|//[^\n]*" - ) - def _comment_replace(match: re.Match) -> str: - token = match.group(0) - if token.startswith("/"): - return "" - return token - src_clean = comment_string_pattern.sub(_comment_replace, src) - - stem = _file_stem(path) - file_nid = _make_id(str(path)) - - # Check if this is a part-of file and redirect to parent - part_of_match = re.search(r"^\s*part\s+of\s+['\"]([^'\"]+)['\"]", src_clean, re.MULTILINE) - is_part = False - if part_of_match: - parent_ref = part_of_match.group(1) - if parent_ref.endswith(".dart"): - try: - parent_path = (path.parent / parent_ref).resolve() - if parent_path.exists(): - stem = _file_stem(parent_path) - file_nid = _make_id(str(parent_path)) - is_part = True - except Exception: - pass - - nodes = [] - if not is_part: - nodes.append({"id": file_nid, "label": path.name, "file_type": "code", - "source_file": str(path), "source_location": None}) - edges = [] - defined: set[str] = set() - - def add_node(nid: str, label: str, ftype: str = "code", source_file: str | None = str(path)) -> None: - if nid not in defined: - nodes.append({"id": nid, "label": label, "file_type": ftype, - "source_file": source_file, "source_location": None}) - defined.add(nid) - - def add_edge(src_id: str, tgt_id: str, relation: str, weight: float = 1.0, context: str | None = None) -> None: - edge = {"source": src_id, "target": tgt_id, "relation": relation, - "confidence": "EXTRACTED", "confidence_score": 1.0, - "source_file": str(path), "source_location": None, "weight": weight} - if context: - edge["context"] = context - edges.append(edge) - - def _split_types(text: str) -> list[str]: - parts = [] - current = [] - depth = 0 - for char in text: - if char == "<": - depth += 1 - current.append(char) - elif char == ">": - depth -= 1 - current.append(char) - elif char == "," and depth == 0: - parts.append("".join(current).strip()) - current = [] - else: - current.append(char) - if current: - parts.append("".join(current).strip()) - return [p for p in parts if p] - - def _find_matching_brace(text: str, start_pos: int) -> int: - brace_count = 0 - in_double_quote = False - in_single_quote = False - escape = False - - first_brace = text.find("{", start_pos) - if first_brace == -1: - return len(text) - - brace_count = 1 - i = first_brace + 1 - n = len(text) - while i < n: - char = text[i] - if escape: - escape = False - i += 1 - continue - if char == "\\": - escape = True - i += 1 - continue - if text[i:i+3] == '"""' and not in_single_quote: - i += 3 - end = text.find('"""', i) - i = end + 3 if end != -1 else n - continue - if text[i:i+3] == "'''" and not in_double_quote: - i += 3 - end = text.find("'''", i) - i = end + 3 if end != -1 else n - continue - if char == '"' and not in_single_quote: - in_double_quote = not in_double_quote - elif char == "'" and not in_double_quote: - in_single_quote = not in_single_quote - elif not in_double_quote and not in_single_quote: - if char == "{": - brace_count += 1 - elif char == "}": - brace_count -= 1 - if brace_count == 0: - return i + 1 - i += 1 - return len(text) - - # 1. Classes, mixins, and enums declarations (with inheritance, mixins, interfaces, and generics) - # Supports multiple combined modifiers (e.g., abstract base class, mixin class) without capturing "class" as a name - class_pattern = r"^\s*(?:(?:abstract|sealed|base|interface|final|mixin)\s+)*(?:class|mixin|enum|extension\s+type)\s+(\w+)" - for m in re.finditer(class_pattern, src_clean, re.MULTILINE): - class_name = m.group(1) - class_nid = _make_id(stem, class_name) - add_node(class_nid, class_name) - add_edge(file_nid, class_nid, "defines") - - # Manually parse extends/on, with, and implements in header to handle nested generics brackets balanced - start_idx = m.end() - rest = src_clean[start_idx : start_idx + 500] - - # Skip class generic parameters - if rest.lstrip().startswith("<"): - offset = rest.find("<") - depth = 1 - i = offset + 1 - while i < len(rest) and depth > 0: - if rest[i] == "<": depth += 1 - elif rest[i] == ">": depth -= 1 - i += 1 - rest = rest[i:] - - # Skip primary constructor (e.g. extension type MyExt(int id)) - if rest.lstrip().startswith("("): - offset = rest.find("(") - depth = 1 - i = offset + 1 - while i < len(rest) and depth > 0: - if rest[i] == "(": depth += 1 - elif rest[i] == ")": depth -= 1 - i += 1 - rest = rest[i:] - - header_end = rest.find("{") - if header_end == -1: - header_end = rest.find(";") - if header_end == -1: - header_end = len(rest) - header = rest[:header_end] - - base_class = None - generics = None - mixins_list = [] - interfaces_list = [] - - # Parse extends or on - extends_m = re.search(r"^\s*(?:extends|on)\s+([a-zA-Z0-9_.]+)", header) - if extends_m: - base_class = extends_m.group(1) - rest_header = header[extends_m.end():] - if rest_header.strip().startswith("<"): - start_idx = rest_header.find("<") - depth = 1 - i = start_idx + 1 - while i < len(rest_header) and depth > 0: - if rest_header[i] == "<": - depth += 1 - elif rest_header[i] == ">": - depth -= 1 - if depth == 0: - generics = rest_header[start_idx + 1 : i] - break - i += 1 - if generics is not None: - header = rest_header[i + 1:] - else: - header = rest_header - else: - header = rest_header - - # Parse with - with_m = re.search(r"^\s*with\s+", header) - if with_m: - rest_header = header[with_m.end():] - impl_idx = rest_header.find("implements") - if impl_idx != -1: - mixins_str = rest_header[:impl_idx] - header = rest_header[impl_idx:] - else: - mixins_str = rest_header - header = "" - mixins_list = _split_types(mixins_str) - - # Parse implements - impl_m = re.search(r"^\s*implements\s+", header) - if impl_m: - interfaces_list = _split_types(header[impl_m.end():]) - - # Map extends inheritance relation - if base_class: - base_nid = _make_id(base_class) - add_node(base_nid, base_class, source_file=None) - add_edge(class_nid, base_nid, "inherits") - - # Map generic type arguments (e.g. MyBloc extends Bloc) - if generics: - for gen in _split_types(generics): - gen_clean = gen.split("<")[0].strip() - if gen_clean not in {"String", "int", "double", "bool", "num", "dynamic", "Object", "void"}: - gen_nid = _make_id(gen_clean) - add_node(gen_nid, gen_clean, source_file=None) - add_edge(class_nid, gen_nid, "references") - - # Map mixins - for mixin in mixins_list: - mixin_clean = mixin.split("<")[0].strip() - mixin_nid = _make_id(mixin_clean) - add_node(mixin_nid, mixin_clean, source_file=None) - add_edge(class_nid, mixin_nid, "mixes_in") - - # Map interfaces - for interface in interfaces_list: - interface_clean = interface.split("<")[0].strip() - interface_nid = _make_id(interface_clean) - add_node(interface_nid, interface_clean, source_file=None) - add_edge(class_nid, interface_nid, "implements") - - # Extract class body for precise framework dependencies and event handling - start_idx = m.start() - brace_pos = src_clean.find("{", start_idx) - semi_pos = src_clean.find(";", start_idx) - - has_body = brace_pos != -1 - if has_body and semi_pos != -1 and semi_pos < brace_pos: - has_body = False - - if has_body: - end_pos = _find_matching_brace(src_clean, start_idx) - class_body = src_clean[brace_pos:end_pos] - - # Bloc event registration: on() - for em in re.finditer(r"\bon<(\w+)>\s*\(", class_body): - event_name = em.group(1) - event_nid = _make_id(event_name) - add_node(event_nid, event_name, source_file=None) - add_edge(class_nid, event_nid, "calls", context="bloc_event") - - # Bloc state emissions: emit(MyState) or yield MyState - for sm in re.finditer(r"\b(?:emit|yield)\s*\(?\s*(?:const\s+)?([A-Z]\w*)\b", class_body): - state_name = sm.group(1) - if state_name not in {"String", "List", "Map", "Set", "Future", "Stream", "Object"}: - state_nid = _make_id(state_name) - add_node(state_nid, state_name, source_file=None) - add_edge(class_nid, state_nid, "calls", context="emit_state") - - # Bloc event additions: widget.add(MyEvent()) or bloc.add(MyEvent()) - for am in re.finditer(r"\b(?:\w*[Bb]loc\w*|context\.read<\w+>\(\))\.add\(\s*(?:const\s+)?([A-Z]\w*)\b", class_body): - event_name = am.group(1) - if event_name not in {"String", "List", "Map", "Set", "Future", "Stream", "Object"}: - event_nid = _make_id(event_name) - add_node(event_nid, event_name, source_file=None) - add_edge(class_nid, event_nid, "calls", context="bloc_add_event") - - # Riverpod provider references: ref.watch(provider) - for rm in re.finditer(r"\bref\.(?:watch|read|listen)\s*\(\s*(\w+)\b", class_body): - provider_name = rm.group(1) - provider_nid = _make_id(provider_name) - add_node(provider_nid, provider_name, source_file=None) - add_edge(class_nid, provider_nid, "references", context="riverpod_reference") - - # Widget to Bloc references: BlocBuilder - for bm in re.finditer(r"\bBloc(?:Builder|Listener|Consumer|Provider|Selector)\s*<\s*([a-zA-Z0-9_]+)\b", class_body): - bloc_name = bm.group(1) - if bloc_name not in {"String", "int", "double", "bool", "num", "dynamic", "Object", "void"}: - bloc_nid = _make_id(bloc_name) - add_node(bloc_nid, bloc_name, source_file=None) - add_edge(class_nid, bloc_nid, "references", context="bloc_widget_binding") - - # context.read() or BlocProvider.of(context) - for lm in re.finditer(r"\b(?:read|watch|select|of)\s*<([a-zA-Z0-9_]+)>", class_body): - bloc_name = lm.group(1) - if bloc_name not in {"String", "int", "double", "bool", "num", "dynamic", "Object", "void"}: - bloc_nid = _make_id(bloc_name) - add_node(bloc_nid, bloc_name, source_file=None) - add_edge(class_nid, bloc_nid, "references", context="bloc_lookup") - - # 2. Annotations mapping (class, mixin, enum, or function level annotations) - # Support: @riverpod, @Riverpod(...), @injectable, @singleton, @RoutePage(), @HiveType(typeId: 0), @RestApi() - # Matches `@annotation` and links it to the next class/mixin/enum/function declaration in the file - annotation_pattern = r"@(\w+)(?:\([^)]*\))?" - for am in re.finditer(annotation_pattern, src_clean): - annotation_name = am.group(1) - if annotation_name in {"override", "deprecated", "required", "protected", "mustCallSuper"}: - continue - annotation_pos = am.end() - intervening_text = src_clean[annotation_pos : annotation_pos + 300] - - class_m = re.search(r"^\s*(?:(?:abstract|sealed|base|interface|final|mixin)\s+)*(?:class|mixin|enum|extension\s+type)\s+(\w+)", intervening_text, re.MULTILINE) - func_m = re.search(r"^\s*(?:factory\s+|static\s+|async\s+|external\s+|abstract\s+)?(?:\([^)]+\)|[a-zA-Z0-9_<>,.?]+)(?:\s+[a-zA-Z0-9_<>,.?]+){0,3}\s+(\w+)\s*\(", intervening_text, re.MULTILINE) - - target_nid = None - target_name = None - target_type = None - - if class_m and func_m: - if class_m.start() < func_m.start(): - target_name = class_m.group(1) - target_type = "class" - target_nid = _make_id(stem, target_name) - else: - target_name = func_m.group(1) - target_type = "function" - target_nid = _make_id(stem, target_name) - elif class_m: - target_name = class_m.group(1) - target_type = "class" - target_nid = _make_id(stem, target_name) - elif func_m: - target_name = func_m.group(1) - target_type = "function" - target_nid = _make_id(stem, target_name) - - if target_nid and target_name: - actual_intervening = intervening_text[:min(class_m.start() if class_m else 300, func_m.start() if func_m else 300)] - if ";" not in actual_intervening and "}" not in actual_intervening and "{" not in actual_intervening: - annotation_nid = _make_id("annotation", annotation_name.lower()) - add_node(annotation_nid, f"@{annotation_name}", ftype="concept", source_file=None) - add_edge(target_nid, annotation_nid, "configures") - - # Riverpod specific provider generation mapping (supports camelCase class and functional providers) - if annotation_name.lower() == "riverpod": - if target_type == "class": - provider_name = target_name[0].lower() + target_name[1:] + "Provider" if len(target_name) > 1 else target_name.lower() + "Provider" - else: - provider_name = target_name + "Provider" - provider_nid = _make_id(provider_name) - add_node(provider_nid, provider_name, ftype="concept", source_file=str(path)) - add_edge(target_nid, provider_nid, "defines", context="riverpod_provider") - - # 2.5 Typedefs (Type Aliases) - typedef_pattern = r"^\s*typedef\s+(\w+)\s*(?:<[^>]+>)?\s*=\s*([a-zA-Z0-9_<>,.?\s]+);" - for m in re.finditer(typedef_pattern, src_clean, re.MULTILINE): - typedef_name = m.group(1) - target_type = m.group(2).split("<")[0].split(".")[-1].strip() - if target_type not in {"String", "int", "double", "bool", "num", "dynamic", "Object", "List", "Map", "Set", "void", "Function"}: - typedef_nid = _make_id(stem, typedef_name) - add_node(typedef_nid, typedef_name) - add_edge(file_nid, typedef_nid, "defines") - target_nid = _make_id(target_type) - add_node(target_nid, target_type, source_file=None) - add_edge(typedef_nid, target_nid, "references", context="typedef") - - # 3. Extensions (extension MyExt on MyClass) - ext_pattern = r"^\s{0,4}extension\s+(\w+)?(?:<[^>]+>)?\s+on\s+(\w+)" - for m in re.finditer(ext_pattern, src_clean, re.MULTILINE): - ext_name = m.group(1) or f"{stem}_anonymous_extension" - target_class = m.group(2) - - ext_nid = _make_id(stem, ext_name) - label = m.group(1) or f"Extension on {target_class}" - add_node(ext_nid, label) - add_edge(file_nid, ext_nid, "defines") - - target_nid = _make_id(target_class) - add_node(target_nid, target_class, source_file=None) - add_edge(ext_nid, target_nid, "extends") - - # 4. Top-level and class-level variable declarations (generic variables, records, late, and destructuring) - # Restrict indentation to 0-2 spaces to avoid matching local variables inside functions or switch expressions - var_pattern = r"^\s{0,2}(?:late\s+)?(?:(?:final|const|var)\s+)?(?:\([^)]+\)\s+|([a-zA-Z0-9_<>,.?]+(?:\s+[a-zA-Z0-9_<>,.?]+){0,3})\s+)?(?:(\w+)|(?:\w+\s*)?\(([^)]+)\))\s*(?:=|$|;)" - for m in re.finditer(var_pattern, src_clean, re.MULTILINE): - var_type = m.group(1) - single_name = m.group(2) - destructured_names = m.group(3) - - if not re.match(r"^\s*(?:late|final|const|var)\b", m.group(0)) and not var_type: - continue - - if single_name: - if single_name not in {"if", "for", "while", "switch", "catch", "return"}: - var_nid = _make_id(stem, single_name) - add_node(var_nid, single_name) - add_edge(file_nid, var_nid, "defines") - - if var_type and var_type not in {"String", "int", "double", "bool", "num", "dynamic", "Object", "List", "Map", "Set", "void"}: - clean_type = var_type.split("<")[0].split(".")[-1].strip() - type_nid = _make_id(clean_type) - add_node(type_nid, clean_type, source_file=None) - add_edge(file_nid, type_nid, "references", context="variable_type") - elif destructured_names: - for name in [n.strip() for n in destructured_names.split(",") if n.strip()]: - if ":" in name: - name = name.split(":")[-1].strip() - if re.match(r"^[a-zA-Z_]\w*$", name) and not re.match(r"^[A-Z]", name): - if name not in {"if", "for", "while", "switch", "catch", "return"}: - var_nid = _make_id(stem, name) - add_node(var_nid, name) - add_edge(file_nid, var_nid, "defines") - - # 5. Top-level and member functions/methods (supports typed/generic/record return types and Riverpod/Bloc references) - # Restrict indentation to 0-2 spaces to avoid matching nested local functions or methods inside multiline switch statements - method_pattern = r"^\s{0,2}(?:factory\s+|static\s+|async\s+|external\s+|abstract\s+)?(?:\([^)]+\)|[a-zA-Z0-9_<>,.?]+)(?:\s+[a-zA-Z0-9_<>,.?]+){0,3}\s+(\w+(?:\.\w+)?)\s*\(" - for m in re.finditer(method_pattern, src_clean, re.MULTILINE): - raw_name = m.group(1) - name = raw_name.split(".")[-1] - if name in {"if", "for", "while", "switch", "catch", "return", "void", "dynamic", "final", "const", "get", "set"}: - continue - if re.match(r"^[A-Z]", name): - continue - nid = _make_id(stem, name) - add_node(nid, name) - add_edge(file_nid, nid, "defines") - - # Get function body using matching brace to extract Riverpod reference patterns - start_idx = m.start() - brace_pos = src_clean.find("{", start_idx) - semi_pos = src_clean.find(";", start_idx) - arrow_pos = src_clean.find("=>", start_idx) - - has_body = brace_pos != -1 - if has_body and semi_pos != -1 and semi_pos < brace_pos: - has_body = False - if has_body and arrow_pos != -1 and arrow_pos < brace_pos: - has_body = False - - if has_body: - end_pos = _find_matching_brace(src_clean, start_idx) - func_body = src_clean[brace_pos:end_pos] - - # Extract Riverpod provider references: ref.watch(provider) - for rm in re.finditer(r"\bref\.(?:watch|read|listen)\s*\(\s*(\w+)\b", func_body): - provider_name = rm.group(1) - provider_nid = _make_id(provider_name) - add_node(provider_nid, provider_name, source_file=None) - add_edge(nid, provider_nid, "references", context="riverpod_reference") - - # Extract Bloc event additions: widget.add(MyEvent()) or bloc.add(MyEvent()) - for am in re.finditer(r"\b(?:\w*[Bb]loc\w*|context\.read<\w+>\(\))\.add\(\s*(?:const\s+)?([A-Z]\w*)\b", func_body): - event_name = am.group(1) - if event_name not in {"String", "List", "Map", "Set", "Future", "Stream", "Object"}: - event_nid = _make_id(event_name) - add_node(event_nid, event_name, source_file=None) - add_edge(nid, event_nid, "calls", context="bloc_add_event") - - # context.read() or BlocProvider.of(context) - for lm in re.finditer(r"\b(?:read|watch|select|of)\s*<([a-zA-Z0-9_]+)>", func_body): - bloc_name = lm.group(1) - if bloc_name not in {"String", "int", "double", "bool", "num", "dynamic", "Object", "void"}: - bloc_nid = _make_id(bloc_name) - add_node(bloc_nid, bloc_name, source_file=None) - add_edge(nid, bloc_nid, "references", context="bloc_lookup") - - # Universal Navigation Patters (GoRouter, AutoRoute, Navigator) - for nm in re.finditer(r"\b(?:go|push|goNamed|pushNamed|replace|replaceNamed)\s*\(\s*(?:context\s*,\s*)?['\"]([a-zA-Z0-9_/?=&%-]+)['\"]", func_body): - route_path = nm.group(1) - route_nid = _make_id("route", route_path.replace("/", "_").replace("?", "_").replace("=", "_").replace("&", "_")) - add_node(route_nid, f"Route {route_path}", ftype="concept", source_file=None) - add_edge(nid, route_nid, "navigates", context="route_path") - - for cm in re.finditer(r"\b(?:go|push|goNamed|pushNamed|replace|replaceNamed)\s*\(\s*(?:context\s*,\s*)?([A-Z][a-zA-Z0-9_]*\.[a-zA-Z0-9_]+)", func_body): - route_const = cm.group(1) - route_nid = _make_id("route", route_const.replace(".", "_")) - add_node(route_nid, route_const, ftype="concept", source_file=None) - add_edge(nid, route_nid, "navigates", context="route_const") - - for om in re.finditer(r"\b(?:push|replace)\s*\(\s*(?:context\s*,\s*)?.*?\b([A-Z]\w*(?:Route|Screen|Page))\b", func_body): - route_class = om.group(1) - route_nid = _make_id(route_class) - add_node(route_nid, route_class, source_file=None) - add_edge(nid, route_nid, "navigates", context="route_object") - - # 6. Imports and Exports - for m in re.finditer(r"""^\s*import\s+['"]([^'"]+)['"]""", src_clean, re.MULTILINE): - pkg = m.group(1) - tgt_nid = _make_id(pkg) - add_node(tgt_nid, pkg, source_file=None) - add_edge(file_nid, tgt_nid, "imports") - - for m in re.finditer(r"""^\s*export\s+['"]([^'"]+)['"]""", src_clean, re.MULTILINE): - pkg = m.group(1) - tgt_nid = _make_id(pkg) - add_node(tgt_nid, pkg, source_file=None) - add_edge(file_nid, tgt_nid, "exports") - - # 7. Generic Invocations / Type Lookups (Universal Dependency Lookup) - # Matches any method call with type parameters: methodName() or object.methodName() - # Automatically extracts GetIt, Injectable, Riverpod, Provider, BlocProvider, and InheritedWidget type lookups! - generic_call_pattern = r"\b\w+<([a-zA-Z0-9_.]+(?:<[a-zA-Z0-9_.,\s<>]+>)?)\s*>\s*\(" - type_blacklist = {"String", "int", "double", "bool", "num", "dynamic", "Object", "List", "Map", "Set", "Future", "Stream", "void"} - for m in re.finditer(generic_call_pattern, src_clean): - type_name = m.group(1).split(".")[-1].strip() - clean_name = type_name.split("<")[0].strip() - if clean_name not in type_blacklist: - target_nid = _make_id(clean_name) - add_node(target_nid, clean_name, source_file=None) - add_edge(file_nid, target_nid, "references", context="type_lookup") - - return {"nodes": nodes, "edges": edges} - - def _sv_first_identifier(node, source: bytes) -> str | None: """First `simple_identifier` under node in pre-order, or None. @@ -7551,275 +6736,6 @@ def walk(node, module_nid: str | None = None) -> None: return {"nodes": nodes, "edges": edges} -def extract_sql(path: Path, content: str | bytes | None = None) -> dict: - """Extract tables, views, functions, and relationships from .sql files via tree-sitter.""" - try: - import tree_sitter_sql as tssql - from tree_sitter import Language, Parser - except ImportError: - return {"nodes": [], "edges": [], "error": "tree_sitter_sql not installed. Run: pip install tree-sitter-sql"} - - try: - language = Language(tssql.language()) - parser = Parser(language) - source = ( - content.encode("utf-8") if isinstance(content, str) - else content if content is not None - else path.read_bytes() - ) - tree = parser.parse(source) - root = tree.root_node - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - - stem = _file_stem(path) - str_path = str(path) - file_nid = _make_id(str_path) - nodes: list[dict] = [{"id": file_nid, "label": path.name, "file_type": "code", - "source_file": str_path, "source_location": None}] - edges: list[dict] = [] - seen_ids: set[str] = {file_nid} - table_nids: dict[str, str] = {} # name → nid for reference resolution - - def _read(n) -> str: - return source[n.start_byte:n.end_byte].decode("utf-8", errors="replace") - - def _obj_name(n) -> str | None: - for c in n.children: - if c.type == "object_reference": - return _read(c) - return None - - def _add_node(nid: str, label: str, line: int) -> None: - if nid not in seen_ids: - seen_ids.add(nid) - nodes.append({"id": nid, "label": label, "file_type": "code", - "source_file": str_path, "source_location": f"L{line}"}) - edges.append({"source": file_nid, "target": nid, "relation": "contains", - "confidence": "EXTRACTED", "source_file": str_path, - "source_location": f"L{line}", "weight": 1.0}) - - def _add_edge(src: str, tgt: str, relation: str, line: int) -> None: - edges.append({"source": src, "target": tgt, "relation": relation, - "confidence": "EXTRACTED", "source_file": str_path, - "source_location": f"L{line}", "weight": 1.0}) - - def walk(node) -> None: - t = node.type - line = node.start_point[0] + 1 - - if t == "create_table": - name = _obj_name(node) - if name: - nid = _make_id(stem, name) - _add_node(nid, name, line) - table_nids[name.lower()] = nid - # Foreign key REFERENCES - for col in node.children: - if col.type == "column_definitions": - has_error = any(cd.type == "ERROR" for cd in col.children) - seen_refs: set[str] = set() - for cd in col.children: - if cd.type == "column_definition": - # Inline column-level REFERENCES - ref_name: str | None = None - found_ref = False - for cc in cd.children: - if cc.type == "keyword_references": - found_ref = True - elif found_ref and cc.type == "object_reference": - ref_name = _read(cc) - break - if ref_name: - ref_nid = table_nids.get(ref_name.lower()) or _make_id(stem, ref_name) - _add_edge(nid, ref_nid, "references", line) - seen_refs.add(ref_name.lower()) - elif cd.type == "constraints": - # Table-level FOREIGN KEY ... REFERENCES ... constraints - for constraint in cd.children: - if constraint.type != "constraint": - continue - ref_name = None - found_ref = False - for cc in constraint.children: - if cc.type == "keyword_references": - found_ref = True - elif found_ref and cc.type == "object_reference": - ref_name = _read(cc) - break - if ref_name: - ref_nid = table_nids.get(ref_name.lower()) or _make_id(stem, ref_name) - _add_edge(nid, ref_nid, "references", line) - seen_refs.add(ref_name.lower()) - if has_error: - # Dialect-specific syntax (e.g. Firebird COMPUTED BY) causes ERROR - # nodes that make the parser drop the trailing constraints block. - # Regex-scan the raw column_definitions text as fallback. - col_text = _read(col) - for rm in re.finditer(r"\bREFERENCES\s+([\w$]+)", col_text, re.IGNORECASE): - ref_name = rm.group(1) - if ref_name.lower() not in seen_refs: - ref_nid = table_nids.get(ref_name.lower()) or _make_id(stem, ref_name) - _add_edge(nid, ref_nid, "references", line) - seen_refs.add(ref_name.lower()) - - elif t == "create_view": - name = _obj_name(node) - if name: - nid = _make_id(stem, name) - _add_node(nid, name, line) - table_nids[name.lower()] = nid - # FROM/JOIN table references inside view body - _walk_from_refs(node, nid, line) - - elif t == "create_function": - name = _obj_name(node) - if name: - nid = _make_id(stem, name) - _add_node(nid, f"{name}()", line) - _walk_from_refs(node, nid, line) - - elif t == "create_procedure": - name = _obj_name(node) - if name: - nid = _make_id(stem, name) - _add_node(nid, f"{name}()", line) - _walk_from_refs(node, nid, line) - - elif t == "alter_table": - name = _obj_name(node) - if name: - src_nid = table_nids.get(name.lower()) - if not src_nid: - src_nid = _make_id(stem, name) - _add_node(src_nid, name, line) - table_nids[name.lower()] = src_nid - for child in node.children: - if child.type == "add_constraint": - for cc in child.children: - if cc.type != "constraint": - continue - found_ref = False - ref_name: str | None = None - for ccc in cc.children: - if ccc.type == "keyword_references": - found_ref = True - elif found_ref and ccc.type == "object_reference": - ref_name = _read(ccc) - break - if ref_name: - ref_nid = table_nids.get(ref_name.lower()) - if not ref_nid: - ref_nid = _make_id(stem, ref_name) - _add_edge(src_nid, ref_nid, "references", line) - - elif t == "create_trigger": - trig_name: str | None = None - tbl_name: str | None = None - after_trigger = False - after_for = False - for c in node.children: - if c.type == "keyword_trigger": - after_trigger = True - elif after_trigger and not trig_name and c.type == "object_reference": - trig_name = _read(c) - elif c.type == "keyword_for": - after_for = True - elif after_for and not tbl_name and c.type == "object_reference": - tbl_name = _read(c) - if trig_name: - trig_nid = _make_id(stem, trig_name) - _add_node(trig_nid, trig_name, line) - if tbl_name: - tbl_nid = table_nids.get(tbl_name.lower()) or _make_id(stem, tbl_name) - _add_edge(trig_nid, tbl_nid, "triggers", line) - - elif t == "fb_proc_or_trigger": - text = _read(node) - m = re.match( - r"CREATE\s+(?:OR\s+(?:REPLACE|ALTER)\s+)?" - r"(PROCEDURE|TRIGGER|FUNCTION)\s+([\w$]+)", - text, re.IGNORECASE, - ) - if m: - obj_type = m.group(1).upper() - obj_name = m.group(2) - obj_nid = _make_id(stem, obj_name) - label = obj_name if obj_type == "TRIGGER" else f"{obj_name}()" - _add_node(obj_nid, label, line) - if obj_type == "TRIGGER": - fm = re.search(r"\bFOR\s+([\w$]+)", text, re.IGNORECASE) - if fm: - tbl = fm.group(1) - tbl_nid = table_nids.get(tbl.lower()) or _make_id(stem, tbl) - _add_edge(obj_nid, tbl_nid, "triggers", line) - _NON_TABLES = { - "select", "where", "set", "dual", "null", "true", "false", - "first", "skip", "rows", "next", "only", "lateral", - } - seen_tbls: set[str] = set() - for rm in re.finditer(r"\b(?:FROM|JOIN|INTO)\s+([\w$]+)", text, re.IGNORECASE): - tbl = rm.group(1) - if tbl.lower() not in _NON_TABLES and tbl.lower() not in seen_tbls: - seen_tbls.add(tbl.lower()) - tbl_nid = table_nids.get(tbl.lower()) or _make_id(stem, tbl) - _add_edge(obj_nid, tbl_nid, "reads_from", line) - for rm in re.finditer(r"\bUPDATE\s+([\w$]+)", text, re.IGNORECASE): - tbl = rm.group(1) - if tbl.lower() not in _NON_TABLES and tbl.lower() not in seen_tbls: - seen_tbls.add(tbl.lower()) - tbl_nid = table_nids.get(tbl.lower()) or _make_id(stem, tbl) - _add_edge(obj_nid, tbl_nid, "reads_from", line) - - for child in node.children: - walk(child) - - def _walk_from_refs(node, caller_nid: str, line: int) -> None: - """Recursively find FROM/JOIN table references inside a node.""" - if node.type in ("from", "join"): - for c in node.children: - if c.type == "relation": - for cc in c.children: - if cc.type == "object_reference": - tbl = _read(cc) - tbl_nid = _make_id(stem, tbl) - _add_edge(caller_nid, tbl_nid, "reads_from", - c.start_point[0] + 1) - for child in node.children: - _walk_from_refs(child, caller_nid, line) - - for stmt in root.children: - if stmt.type == "statement": - for child in stmt.children: - walk(child) - elif stmt.type in ("fb_proc_or_trigger", "set_term", "declare_external_function"): - walk(stmt) - - # Global regex fallback: catch any REFERENCES missed due to ERROR nodes in the parse tree - # (e.g. Firebird COMPUTED BY columns push constraints out of the tree entirely). - # Snapshot after tree walk so we don't re-emit edges already captured above. - emitted = {(e["source"], e["target"]) for e in edges if e["relation"] == "references"} - src_text = source.decode("utf-8", errors="replace") - for m in re.finditer(r"CREATE\s+TABLE\s+([\w$]+)\s*\(", src_text, re.IGNORECASE): - tbl_name = m.group(1) - tbl_nid = table_nids.get(tbl_name.lower()) - if tbl_nid is None: - continue - tbl_line = src_text[: m.start()].count("\n") + 1 - tail = src_text[m.start():] - end = re.search(r"(?:^|\n)(?:CREATE|SET\s+TERM|ALTER)\s", tail[1:], re.IGNORECASE) - block = tail[: end.start() + 1] if end else tail - for rm in re.finditer(r"\bREFERENCES\s+([\w$]+)", block, re.IGNORECASE): - ref_name = rm.group(1) - ref_nid = table_nids.get(ref_name.lower()) or _make_id(stem, ref_name) - if (tbl_nid, ref_nid) not in emitted: - _add_edge(tbl_nid, ref_nid, "references", tbl_line) - emitted.add((tbl_nid, ref_nid)) - - return {"nodes": nodes, "edges": edges} - - def extract_lua(path: Path) -> dict: """Extract functions, methods, require() imports, and calls from a .lua file.""" return _extract_generic(path, _LUA_CONFIG) @@ -8101,1523 +7017,25 @@ def _emit_import(name): return {"nodes": nodes, "edges": edges} -_FORTRAN_CPP_EXTS = {".F", ".F90", ".F95", ".F03", ".F08"} +# ── Go extractor (custom walk) ──────────────────────────────────────────────── -def _cpp_preprocess(path: Path) -> bytes: - """Run cpp -w -P on a capital-F Fortran file and return preprocessed bytes. +# ── Rust extractor (custom walk) ────────────────────────────────────────────── - Falls back to raw file bytes if cpp is not available. Capital-F extensions - conventionally require C preprocessor expansion (#ifdef MPI, #define REAL8, etc.) - before parsing. +# Common Rust trait/stdlib method names that appear in virtually every codebase. +# Resolving these cross-file produces spurious INFERRED edges across crate +# boundaries (issue #908) — skip them from the unresolved-call queue entirely. - Security (F-007): we pass `-nostdinc` and `-I /dev/null` so a malicious - source file containing `#include "/home/victim/.ssh/id_rsa"` (or any other - include directive) cannot inline arbitrary host files into the output that - we then ship to an LLM. Without these flags `cpp` happily resolves any - relative or absolute include path it can read, which is a corpus-side - file-exfiltration vector. - """ - import shutil - import subprocess - if not shutil.which("cpp"): - return path.read_bytes() - try: - # Pass an absolute path so a corpus file named like "-I/etc/x.F90" cannot - # be parsed by cpp as an option (cpp does not accept a "--" end-of-options - # terminator). An absolute path always begins with "/". - result = subprocess.run( - ["cpp", "-w", "-P", "-nostdinc", "-I", "/dev/null", str(path.resolve())], - capture_output=True, - timeout=30, - ) - if result.returncode == 0 and result.stdout: - return result.stdout - except Exception: - pass - return path.read_bytes() - - -def extract_fortran(path: Path) -> dict: - """Extract programs, modules, subroutines, functions, use statements, and calls from Fortran files. - - Capital-F extensions (.F, .F90, etc.) are run through the C preprocessor before - parsing so #ifdef/#define macros are resolved. - """ - try: - import tree_sitter_fortran as tsfortran - from tree_sitter import Language, Parser - except ImportError: - return {"nodes": [], "edges": [], "error": "tree-sitter-fortran not installed"} - - try: - language = Language(tsfortran.language()) - parser = Parser(language) - source = _cpp_preprocess(path) if path.suffix in _FORTRAN_CPP_EXTS else path.read_bytes() - tree = parser.parse(source) - root = tree.root_node - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - stem = _file_stem(path) - str_path = str(path) - nodes: list[dict] = [] - edges: list[dict] = [] - seen_ids: set[str] = set() - scope_bodies: list[tuple[str, object]] = [] - - def add_node(nid: str, label: str, line: int) -> None: - if nid not in seen_ids: - seen_ids.add(nid) - nodes.append({ - "id": nid, - "label": label, - "file_type": "code", - "source_file": str_path, - "source_location": f"L{line}", - }) - - def add_edge(src: str, tgt: str, relation: str, line: int, - confidence: str = "EXTRACTED", weight: float = 1.0, - context: str | None = None) -> None: - edge = { - "source": src, - "target": tgt, - "relation": relation, - "confidence": confidence, - "source_file": str_path, - "source_location": f"L{line}", - "weight": weight, - } - if context: - edge["context"] = context - edges.append(edge) - - file_nid = _make_id(str(path)) - add_node(file_nid, path.name, 1) - - def _fortran_name(stmt_node) -> str | None: - """Extract name from a *_statement node. Fortran is case-insensitive; lowercase.""" - for child in stmt_node.children: - if child.type in ("name", "identifier"): - return _read_text(child, source).lower() - return None - - def ensure_named_node(name: str, line: int) -> str: - nid = _make_id(stem, name) - if nid in seen_ids: - return nid - nid = _make_id(name) - if nid not in seen_ids: - # The name isn't defined in this file, so this is a cross-file reference - # (e.g. a `Thing` type annotation imported from another module). Emit a - # SOURCELESS stub — like the inheritance-base path below — so the - # corpus-level rewire can collapse it onto the real definition. A sourced - # stub here makes _disambiguate_colliding_node_ids bake the referencing - # file's path (with extension) into the id and blocks the rewire, which is - # the phantom-duplicate-node bug (#1402). - seen_ids.add(nid) - nodes.append({ - "id": nid, - "label": name, - "file_type": "code", - "source_file": "", - "source_location": "", - "origin_file": str_path, - }) - return nid - - def emit_signature_refs(scope_node, fn_nid: str, is_function: bool) -> None: - """Emit references[parameter_type] / references[return_type] edges for - a subroutine/function based on its variable_declaration siblings.""" - stmt_type = "function_statement" if is_function else "subroutine_statement" - stmt = next((c for c in scope_node.children if c.type == stmt_type), None) - if stmt is None: - return - param_names: set[str] = set() - params_node = next((c for c in stmt.children if c.type == "parameters"), None) - if params_node is not None: - for c in params_node.children: - if c.type == "identifier": - param_names.add(_read_text(c, source).lower()) - result_name: str | None = None - if is_function: - result_node = next((c for c in stmt.children if c.type == "function_result"), None) - if result_node is not None: - res_id = next((c for c in result_node.children if c.type == "identifier"), None) - if res_id is not None: - result_name = _read_text(res_id, source).lower() - else: - # implicit result variable: same name as the function - result_name = _fortran_name(stmt) - for child in scope_node.children: - if child.type != "variable_declaration": - continue - derived = next((c for c in child.children if c.type == "derived_type"), None) - if derived is None: - continue - type_name_node = next((c for c in derived.children if c.type == "type_name"), None) - if type_name_node is None: - continue - type_name = _read_text(type_name_node, source).lower() - for var in child.children: - if var.type != "identifier": - continue - var_name = _read_text(var, source).lower() - var_line = var.start_point[0] + 1 - if var_name in param_names: - tgt = ensure_named_node(type_name, var_line) - if tgt != fn_nid: - add_edge(fn_nid, tgt, "references", var_line, context="parameter_type") - elif is_function and var_name == result_name: - tgt = ensure_named_node(type_name, var_line) - if tgt != fn_nid: - add_edge(fn_nid, tgt, "references", var_line, context="return_type") - - def walk_calls(node, scope_nid: str) -> None: - if node is None: - return - t = node.type - if t in ("subroutine", "function", "module", "program", "internal_procedures"): - return - # call FOO(args) — tree-sitter-fortran uses subroutine_call - if t == "subroutine_call": - name_node = next((c for c in node.children if c.type == "identifier"), None) - if name_node: - callee = _read_text(name_node, source).lower() - target_nid = _make_id(stem, callee) - add_edge(scope_nid, target_nid, "calls", node.start_point[0] + 1, - confidence="EXTRACTED", context="call") - # x = compute(args) — function invocations are `call_expression`, which - # shares Fortran's `name(...)` syntax with array indexing. Only emit a - # call edge when the callee resolves to a procedure defined in this file - # (an array variable produces no matching node), so array accesses can't - # fabricate spurious `calls` edges. - elif t == "call_expression": - name_node = next((c for c in node.children if c.type == "identifier"), None) - if name_node: - callee = _read_text(name_node, source).lower() - target_nid = _make_id(stem, callee) - if target_nid in seen_ids and target_nid != scope_nid: - add_edge(scope_nid, target_nid, "calls", node.start_point[0] + 1, - confidence="EXTRACTED", context="call") - for child in node.children: - walk_calls(child, scope_nid) - - def walk(node, scope_nid: str) -> None: - t = node.type - - if t == "program": - stmt = next((c for c in node.children if c.type == "program_statement"), None) - name = _fortran_name(stmt) if stmt else None - if name: - nid = _make_id(stem, name) - line = node.start_point[0] + 1 - add_node(nid, name, line) - add_edge(file_nid, nid, "defines", line) - scope_bodies.append((nid, node)) - for child in node.children: - walk(child, nid) - return - - if t == "module": - stmt = next((c for c in node.children if c.type == "module_statement"), None) - name = _fortran_name(stmt) if stmt else None - if name: - nid = _make_id(stem, name) - line = node.start_point[0] + 1 - add_node(nid, name, line) - add_edge(file_nid, nid, "defines", line) - for child in node.children: - walk(child, nid) - return - - # subroutines/functions inside a module live under internal_procedures - if t == "internal_procedures": - for child in node.children: - walk(child, scope_nid) - return - - if t == "derived_type_definition": - stmt = next((c for c in node.children if c.type == "derived_type_statement"), None) - if stmt is not None: - name_node = next((c for c in stmt.children if c.type == "type_name"), None) - if name_node is not None: - type_name = _read_text(name_node, source).lower() - type_nid = _make_id(stem, type_name) - line = node.start_point[0] + 1 - add_node(type_nid, type_name, line) - add_edge(scope_nid, type_nid, "defines", line) - return - - if t == "subroutine": - stmt = next((c for c in node.children if c.type == "subroutine_statement"), None) - name = _fortran_name(stmt) if stmt else None - if name: - nid = _make_id(stem, name) - line = node.start_point[0] + 1 - add_node(nid, f"{name}()", line) - add_edge(scope_nid, nid, "defines", line) - scope_bodies.append((nid, node)) - emit_signature_refs(node, nid, is_function=False) - for child in node.children: - walk(child, nid) - return - - if t == "function": - stmt = next((c for c in node.children if c.type == "function_statement"), None) - name = _fortran_name(stmt) if stmt else None - if name: - nid = _make_id(stem, name) - line = node.start_point[0] + 1 - add_node(nid, f"{name}()", line) - add_edge(scope_nid, nid, "defines", line) - scope_bodies.append((nid, node)) - emit_signature_refs(node, nid, is_function=True) - for child in node.children: - walk(child, nid) - return - - if t == "use_statement": - line = node.start_point[0] + 1 - # tree-sitter-fortran uses module_name node for the used module - name_node = next((c for c in node.children if c.type in ("module_name", "name", "identifier")), None) - if name_node: - mod_name = _read_text(name_node, source).lower() - imp_nid = _make_id(mod_name) - add_node(imp_nid, mod_name, line) - add_edge(scope_nid, imp_nid, "imports", line, context="use") - return - - for child in node.children: - walk(child, scope_nid) - - walk(root, file_nid) - - _stmt_headers = { - "subroutine_statement", "function_statement", - "program_statement", "module_statement", - } - for scope_nid, body_node in scope_bodies: - for child in body_node.children: - if child.type not in _stmt_headers: - walk_calls(child, scope_nid) - - return {"nodes": nodes, "edges": edges} - - -# ── Go extractor (custom walk) ──────────────────────────────────────────────── - -def extract_go(path: Path) -> dict: - """Extract functions, methods, type declarations, and imports from a .go file.""" - try: - import tree_sitter_go as tsgo - from tree_sitter import Language, Parser - except ImportError: - return {"nodes": [], "edges": [], "error": "tree-sitter-go not installed"} - - try: - language = Language(tsgo.language()) - parser = Parser(language) - source = path.read_bytes() - tree = parser.parse(source) - root = tree.root_node - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - stem = _file_stem(path) - # Use directory name as package scope so methods on the same type across - # multiple files in a package share one canonical type node. - pkg_scope = path.parent.name or stem - str_path = str(path) - nodes: list[dict] = [] - edges: list[dict] = [] - seen_ids: set[str] = set() - function_bodies: list[tuple[str, object]] = [] - go_imported_pkgs: set[str] = set() # local names of imported packages - - def add_node(nid: str, label: str, line: int) -> None: - if nid not in seen_ids: - seen_ids.add(nid) - nodes.append({ - "id": nid, - "label": label, - "file_type": "code", - "source_file": str_path, - "source_location": f"L{line}", - }) - - def add_edge(src: str, tgt: str, relation: str, line: int, - confidence: str = "EXTRACTED", weight: float = 1.0, - context: str | None = None) -> None: - edge = { - "source": src, - "target": tgt, - "relation": relation, - "confidence": confidence, - "source_file": str_path, - "source_location": f"L{line}", - "weight": weight, - } - if context: - edge["context"] = context - edges.append(edge) - - file_nid = _make_id(str(path)) - add_node(file_nid, path.name, 1) - - def ensure_named_node(name: str, line: int) -> str: - nid = _make_id(pkg_scope, name) - if nid in seen_ids: - return nid - nid = _make_id(name) - if nid not in seen_ids: - # The name isn't declared in this file, so this is a cross-file reference - # (e.g. a type defined in another file of the package). Emit a SOURCELESS - # stub — like the inheritance-base path in the other extractors — so the - # corpus-level rewire can collapse it onto the real definition. A sourced - # stub here makes _disambiguate_colliding_node_ids bake the referencing - # file's path (with extension) into the id and blocks the rewire, which is - # the phantom-duplicate-node bug (#1402). - seen_ids.add(nid) - nodes.append({ - "id": nid, - "label": name, - "file_type": "code", - "source_file": "", - "source_location": "", - "origin_file": str_path, - }) - return nid - - def emit_go_method_refs(func_node, func_nid: str, line: int) -> None: - params = func_node.child_by_field_name("parameters") - if params is not None: - for p in params.children: - if p.type != "parameter_declaration": - continue - type_node = p.child_by_field_name("type") - refs: list[tuple[str, str]] = [] - _go_collect_type_refs(type_node, source, False, refs) - for ref_name, role in refs: - ctx = "generic_arg" if role == "generic_arg" else "parameter_type" - tgt = ensure_named_node(ref_name, line) - if tgt != func_nid: - add_edge(func_nid, tgt, "references", line, context=ctx) - result = func_node.child_by_field_name("result") - if result is not None: - if result.type == "parameter_list": - for p in result.children: - if p.type != "parameter_declaration": - continue - type_node = p.child_by_field_name("type") - if type_node is None: - for c in p.children: - if c.is_named: - type_node = c - break - refs = [] - _go_collect_type_refs(type_node, source, False, refs) - for ref_name, role in refs: - ctx = "generic_arg" if role == "generic_arg" else "return_type" - tgt = ensure_named_node(ref_name, line) - if tgt != func_nid: - add_edge(func_nid, tgt, "references", line, context=ctx) - else: - refs = [] - _go_collect_type_refs(result, source, False, refs) - for ref_name, role in refs: - ctx = "generic_arg" if role == "generic_arg" else "return_type" - tgt = ensure_named_node(ref_name, line) - if tgt != func_nid: - add_edge(func_nid, tgt, "references", line, context=ctx) - - def walk(node) -> None: - t = node.type - - if t == "function_declaration": - name_node = node.child_by_field_name("name") - if name_node: - func_name = _read_text(name_node, source) - line = node.start_point[0] + 1 - func_nid = _make_id(stem, func_name) - add_node(func_nid, f"{func_name}()", line) - add_edge(file_nid, func_nid, "contains", line) - emit_go_method_refs(node, func_nid, line) - body = node.child_by_field_name("body") - if body: - function_bodies.append((func_nid, body)) - return - - if t == "method_declaration": - receiver = node.child_by_field_name("receiver") - receiver_type: str | None = None - if receiver: - for param in receiver.children: - if param.type == "parameter_declaration": - type_node = param.child_by_field_name("type") - if type_node: - receiver_type = _read_text(type_node, source).lstrip("*").strip() - break - name_node = node.child_by_field_name("name") - if not name_node: - return - method_name = _read_text(name_node, source) - line = node.start_point[0] + 1 - - if receiver_type: - parent_nid = _make_id(pkg_scope, receiver_type) - add_node(parent_nid, receiver_type, line) - method_nid = _make_id(parent_nid, method_name) - add_node(method_nid, f".{method_name}()", line) - add_edge(parent_nid, method_nid, "method", line) - else: - method_nid = _make_id(stem, method_name) - add_node(method_nid, f"{method_name}()", line) - add_edge(file_nid, method_nid, "contains", line) - - emit_go_method_refs(node, method_nid, line) - body = node.child_by_field_name("body") - if body: - function_bodies.append((method_nid, body)) - return - - if t == "type_declaration": - for child in node.children: - if child.type != "type_spec": - continue - name_node = child.child_by_field_name("name") - if not name_node: - continue - type_name = _read_text(name_node, source) - line = child.start_point[0] + 1 - type_nid = _make_id(pkg_scope, type_name) - add_node(type_nid, type_name, line) - add_edge(file_nid, type_nid, "contains", line) - # Type body: struct fields (with embeds) or interface embedding. - type_body = None - for tc in child.children: - if tc.type in ("struct_type", "interface_type"): - type_body = tc - break - if type_body is None: - continue - if type_body.type == "struct_type": - for fdl in type_body.children: - if fdl.type != "field_declaration_list": - continue - for field in fdl.children: - if field.type != "field_declaration": - continue - has_name = any( - fc.type == "field_identifier" for fc in field.children - ) - type_node = field.child_by_field_name("type") - if type_node is None: - for fc in field.children: - if fc.is_named and fc.type != "field_identifier": - type_node = fc - break - refs: list[tuple[str, str]] = [] - _go_collect_type_refs(type_node, source, False, refs) - for ref_name, role in refs: - tgt = ensure_named_node(ref_name, field.start_point[0] + 1) - if tgt == type_nid: - continue - if not has_name and role == "type": - add_edge(type_nid, tgt, "embeds", - field.start_point[0] + 1) - else: - ctx = "generic_arg" if role == "generic_arg" else "field" - add_edge(type_nid, tgt, "references", - field.start_point[0] + 1, context=ctx) - elif type_body.type == "interface_type": - for elem in type_body.children: - if elem.type != "type_elem": - continue - refs = [] - for sub in elem.children: - if sub.is_named: - _go_collect_type_refs(sub, source, False, refs) - for ref_name, role in refs: - tgt = ensure_named_node(ref_name, elem.start_point[0] + 1) - if tgt == type_nid: - continue - if role == "type": - add_edge(type_nid, tgt, "embeds", - elem.start_point[0] + 1) - else: - add_edge(type_nid, tgt, "references", - elem.start_point[0] + 1, context="generic_arg") - return - - if t == "import_declaration": - for child in node.children: - if child.type == "import_spec_list": - for spec in child.children: - if spec.type == "import_spec": - path_node = spec.child_by_field_name("path") - if path_node: - raw = _read_text(path_node, source).strip('"') - # Prefix with go_pkg_ so stdlib names (e.g. "context") - # don't collide with local files of the same basename. - tgt_nid = _make_id("go", "pkg", raw) - add_edge(file_nid, tgt_nid, "imports_from", spec.start_point[0] + 1, context="import") - # Track local name (alias or last path segment) - alias = spec.child_by_field_name("name") - local_name = _read_text(alias, source) if alias else raw.split("/")[-1] - if local_name and local_name != "_" and local_name != ".": - go_imported_pkgs.add(local_name) - elif child.type == "import_spec": - path_node = child.child_by_field_name("path") - if path_node: - raw = _read_text(path_node, source).strip('"') - tgt_nid = _make_id("go", "pkg", raw) - add_edge(file_nid, tgt_nid, "imports_from", child.start_point[0] + 1, context="import") - alias = child.child_by_field_name("name") - local_name = _read_text(alias, source) if alias else raw.split("/")[-1] - if local_name and local_name != "_" and local_name != ".": - go_imported_pkgs.add(local_name) - return - - for child in node.children: - walk(child) - - walk(root) - - label_to_nid: dict[str, str] = {} - for n in nodes: - raw = n["label"] - normalised = raw.strip("()").lstrip(".") - label_to_nid[normalised] = n["id"] - - seen_call_pairs: set[tuple[str, str]] = set() - raw_calls: list[dict] = [] - - def walk_calls(node, caller_nid: str) -> None: - if node.type in ("function_declaration", "method_declaration"): - return - if node.type == "call_expression": - func_node = node.child_by_field_name("function") - callee_name: str | None = None - is_member_call: bool = False - if func_node: - if func_node.type == "identifier": - callee_name = _read_text(func_node, source) - elif func_node.type == "selector_expression": - field = func_node.child_by_field_name("field") - operand = func_node.child_by_field_name("operand") - receiver_name = _read_text(operand, source) if operand else "" - # Package-qualified call (e.g. fmt.Println) → allow cross-file resolution. - # Receiver method call (e.g. s.logger.Log) → skip, no import evidence. - is_member_call = receiver_name not in go_imported_pkgs - if field: - callee_name = _read_text(field, source) - if callee_name and callee_name not in _LANGUAGE_BUILTIN_GLOBALS: - tgt_nid = label_to_nid.get(callee_name) - if tgt_nid and tgt_nid != caller_nid: - pair = (caller_nid, tgt_nid) - if pair not in seen_call_pairs: - seen_call_pairs.add(pair) - line = node.start_point[0] + 1 - edges.append({ - "source": caller_nid, - "target": tgt_nid, - "relation": "calls", - "context": "call", - "confidence": "EXTRACTED", - "source_file": str_path, - "source_location": f"L{line}", - "weight": 1.0, - }) - elif callee_name: - raw_calls.append({ - "caller_nid": caller_nid, - "callee": callee_name, - "is_member_call": is_member_call, - "source_file": str_path, - "source_location": f"L{node.start_point[0] + 1}", - }) - for child in node.children: - walk_calls(child, caller_nid) - - for caller_nid, body_node in function_bodies: - walk_calls(body_node, caller_nid) - - valid_ids = seen_ids - clean_edges = [] - for edge in edges: - src, tgt = edge["source"], edge["target"] - if src in valid_ids and (tgt in valid_ids or edge["relation"] in ("imports", "imports_from")): - clean_edges.append(edge) - - return {"nodes": nodes, "edges": clean_edges, "raw_calls": raw_calls} - - -# ── Rust extractor (custom walk) ────────────────────────────────────────────── - -# Common Rust trait/stdlib method names that appear in virtually every codebase. -# Resolving these cross-file produces spurious INFERRED edges across crate -# boundaries (issue #908) — skip them from the unresolved-call queue entirely. -_RUST_TRAIT_METHOD_BLOCKLIST: frozenset[str] = frozenset({ - "new", "default", "parse", "from_str", "now", "clone", "into", "from", - "to_string", "to_owned", "len", "is_empty", "iter", "next", "build", - "start", "run", "init", "app", "get", "set", "push", "pop", "insert", - "remove", "contains", "collect", "map", "filter", "unwrap", "expect", - "ok", "err", "some", "none", "send", "recv", "lock", "read", "write", -}) - -def extract_rust(path: Path) -> dict: - """Extract functions, structs, enums, traits, impl methods, and use declarations from a .rs file.""" - try: - import tree_sitter_rust as tsrust - from tree_sitter import Language, Parser - except ImportError: - return {"nodes": [], "edges": [], "error": "tree-sitter-rust not installed"} - - try: - language = Language(tsrust.language()) - parser = Parser(language) - source = path.read_bytes() - tree = parser.parse(source) - root = tree.root_node - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - stem = _file_stem(path) - str_path = str(path) - nodes: list[dict] = [] - edges: list[dict] = [] - seen_ids: set[str] = set() - function_bodies: list[tuple[str, object]] = [] - - def add_node(nid: str, label: str, line: int) -> None: - if nid not in seen_ids: - seen_ids.add(nid) - nodes.append({ - "id": nid, - "label": label, - "file_type": "code", - "source_file": str_path, - "source_location": f"L{line}", - }) - - def add_edge(src: str, tgt: str, relation: str, line: int, - confidence: str = "EXTRACTED", weight: float = 1.0, - context: str | None = None) -> None: - edge = { - "source": src, - "target": tgt, - "relation": relation, - "confidence": confidence, - "source_file": str_path, - "source_location": f"L{line}", - "weight": weight, - } - if context: - edge["context"] = context - edges.append(edge) - - file_nid = _make_id(str(path)) - add_node(file_nid, path.name, 1) - - def ensure_named_node(name: str, line: int) -> str: - nid = _make_id(stem, name) - if nid in seen_ids: - return nid - nid = _make_id(name) - if nid not in seen_ids: - # The name isn't defined in this file, so this is a cross-file reference - # (e.g. a `Thing` type annotation imported from another module). Emit a - # SOURCELESS stub — like the inheritance-base path below — so the - # corpus-level rewire can collapse it onto the real definition. A sourced - # stub here makes _disambiguate_colliding_node_ids bake the referencing - # file's path (with extension) into the id and blocks the rewire, which is - # the phantom-duplicate-node bug (#1402). - seen_ids.add(nid) - nodes.append({ - "id": nid, - "label": name, - "file_type": "code", - "source_file": "", - "source_location": "", - "origin_file": str_path, - }) - return nid - - def emit_param_return_refs(func_node, func_nid: str, line: int) -> None: - params = func_node.child_by_field_name("parameters") - if params is not None: - for p in params.children: - if p.type != "parameter": - continue - type_node = p.child_by_field_name("type") - refs: list[tuple[str, str]] = [] - _rust_collect_type_refs(type_node, source, False, refs) - for ref_name, role in refs: - ctx = "generic_arg" if role == "generic_arg" else "parameter_type" - tgt = ensure_named_node(ref_name, line) - if tgt != func_nid: - add_edge(func_nid, tgt, "references", line, context=ctx) - return_type = func_node.child_by_field_name("return_type") - if return_type is not None: - refs = [] - _rust_collect_type_refs(return_type, source, False, refs) - for ref_name, role in refs: - ctx = "generic_arg" if role == "generic_arg" else "return_type" - tgt = ensure_named_node(ref_name, line) - if tgt != func_nid: - add_edge(func_nid, tgt, "references", line, context=ctx) - - def walk(node, parent_impl_nid: str | None = None) -> None: - t = node.type - - if t == "function_item": - name_node = node.child_by_field_name("name") - if name_node: - func_name = _read_text(name_node, source) - line = node.start_point[0] + 1 - if parent_impl_nid: - func_nid = _make_id(parent_impl_nid, func_name) - add_node(func_nid, f".{func_name}()", line) - add_edge(parent_impl_nid, func_nid, "method", line) - else: - func_nid = _make_id(stem, func_name) - add_node(func_nid, f"{func_name}()", line) - add_edge(file_nid, func_nid, "contains", line) - emit_param_return_refs(node, func_nid, line) - body = node.child_by_field_name("body") - if body: - function_bodies.append((func_nid, body)) - return - - if t in ("struct_item", "enum_item", "trait_item"): - name_node = node.child_by_field_name("name") - if name_node: - item_name = _read_text(name_node, source) - line = node.start_point[0] + 1 - item_nid = _make_id(stem, item_name) - add_node(item_nid, item_name, line) - add_edge(file_nid, item_nid, "contains", line) - if t == "trait_item": - for c in node.children: - if c.type != "trait_bounds": - continue - for sub in c.children: - if not sub.is_named: - continue - refs: list[tuple[str, str]] = [] - _rust_collect_type_refs(sub, source, False, refs) - for idx, (ref_name, _role) in enumerate(refs): - tgt = ensure_named_node(ref_name, line) - if tgt == item_nid: - continue - rel = "inherits" if idx == 0 else "references" - if rel == "inherits": - add_edge(item_nid, tgt, "inherits", line) - else: - add_edge(item_nid, tgt, "references", line, - context="generic_arg") - if t == "struct_item": - for c in node.children: - if c.type != "field_declaration_list": - continue - for field in c.children: - if field.type != "field_declaration": - continue - type_node = field.child_by_field_name("type") - if type_node is None: - for fc in field.children: - if fc.type in ("type_identifier", "generic_type", - "scoped_type_identifier", - "reference_type", "primitive_type"): - type_node = fc - break - refs = [] - _rust_collect_type_refs(type_node, source, False, refs) - for ref_name, role in refs: - ctx = "generic_arg" if role == "generic_arg" else "field" - tgt = ensure_named_node(ref_name, field.start_point[0] + 1) - if tgt != item_nid: - add_edge(item_nid, tgt, "references", - field.start_point[0] + 1, context=ctx) - # Tuple structs (`struct Wrapper(pub Logger, Config);`) nest their - # positional field types directly under ordered_field_declaration_list - # with no field_declaration wrapper -- the same shape handled for tuple - # enum variants below. Without this branch these field type references - # are silently dropped. - for c in node.children: - if c.type != "ordered_field_declaration_list": - continue - fline = c.start_point[0] + 1 - for tc in c.children: - if tc.type not in ("type_identifier", "generic_type", - "scoped_type_identifier", "reference_type", - "primitive_type", "tuple_type", "array_type"): - continue - refs = [] - _rust_collect_type_refs(tc, source, False, refs) - for ref_name, role in refs: - ctx = "generic_arg" if role == "generic_arg" else "field" - tgt = ensure_named_node(ref_name, fline) - if tgt != item_nid: - add_edge(item_nid, tgt, "references", fline, context=ctx) - if t == "enum_item": - # Variant payload types nest under enum_variant_list -> - # enum_variant -> ordered_field_declaration_list (tuple variant, - # `Click(Logger)`) | field_declaration_list (struct variant, - # `Resize { size: Dim }`). Neither was traversed, so every - # enum-variant type reference was silently dropped. - _TYPE_NODES = ("type_identifier", "generic_type", - "scoped_type_identifier", "reference_type", - "primitive_type", "tuple_type", "array_type") - - def _emit_enum_type(type_node, at_line): - if type_node is None: - return - refs2: list[tuple[str, str]] = [] - _rust_collect_type_refs(type_node, source, False, refs2) - for ref_name, role in refs2: - ctx = "generic_arg" if role == "generic_arg" else "field" - tgt = ensure_named_node(ref_name, at_line) - if tgt != item_nid: - add_edge(item_nid, tgt, "references", at_line, context=ctx) - - for c in node.children: - if c.type != "enum_variant_list": - continue - for variant in c.children: - if variant.type != "enum_variant": - continue - vline = variant.start_point[0] + 1 - for vc in variant.children: - if vc.type == "ordered_field_declaration_list": - for tc in vc.children: - if tc.type in _TYPE_NODES: - _emit_enum_type(tc, vline) - elif vc.type == "field_declaration_list": - for field in vc.children: - if field.type != "field_declaration": - continue - type_node = field.child_by_field_name("type") - _emit_enum_type(type_node, field.start_point[0] + 1) - return - - if t == "impl_item": - type_node = node.child_by_field_name("type") - trait_node = node.child_by_field_name("trait") - impl_nid: str | None = None - if type_node: - type_name = _read_text(type_node, source).strip() - impl_nid = _make_id(stem, type_name) - add_node(impl_nid, type_name, node.start_point[0] + 1) - if trait_node is not None and impl_nid is not None: - refs: list[tuple[str, str]] = [] - _rust_collect_type_refs(trait_node, source, False, refs) - for idx, (ref_name, _role) in enumerate(refs): - tgt = ensure_named_node(ref_name, node.start_point[0] + 1) - if tgt == impl_nid: - continue - if idx == 0: - add_edge(impl_nid, tgt, "implements", node.start_point[0] + 1) - else: - add_edge(impl_nid, tgt, "references", node.start_point[0] + 1, - context="generic_arg") - body = node.child_by_field_name("body") - if body: - for child in body.children: - walk(child, parent_impl_nid=impl_nid) - return - - if t == "use_declaration": - arg = node.child_by_field_name("argument") - if arg: - raw = _read_text(arg, source) - clean = raw.split("{")[0].rstrip(":").rstrip("*").rstrip(":") - module_name = clean.split("::")[-1].strip() - if module_name: - tgt_nid = _make_id(module_name) - add_edge(file_nid, tgt_nid, "imports_from", node.start_point[0] + 1, context="import") - return - - for child in node.children: - walk(child, parent_impl_nid=None) - - walk(root) - - label_to_nid: dict[str, str] = {} - for n in nodes: - raw = n["label"] - normalised = raw.strip("()").lstrip(".") - label_to_nid[normalised] = n["id"] - - seen_call_pairs: set[tuple[str, str]] = set() - raw_calls: list[dict] = [] - - def walk_calls(node, caller_nid: str) -> None: - if node.type == "function_item": - return - if node.type == "call_expression": - func_node = node.child_by_field_name("function") - callee_name: str | None = None - is_member_call: bool = False - is_scoped_call: bool = False - if func_node: - if func_node.type == "identifier": - callee_name = _read_text(func_node, source) - elif func_node.type == "field_expression": - is_member_call = True - field = func_node.child_by_field_name("field") - if field: - callee_name = _read_text(field, source) - elif func_node.type == "scoped_identifier": - # Type::method() — still allow in-file EXTRACTED match, but - # skip cross-file resolution: bare last-segment lookup ignores - # crate boundaries and produces spurious INFERRED edges (#908). - is_scoped_call = True - name = func_node.child_by_field_name("name") - if name: - callee_name = _read_text(name, source) - if callee_name and callee_name not in _LANGUAGE_BUILTIN_GLOBALS: - tgt_nid = label_to_nid.get(callee_name) - if tgt_nid and tgt_nid != caller_nid: - pair = (caller_nid, tgt_nid) - if pair not in seen_call_pairs: - seen_call_pairs.add(pair) - line = node.start_point[0] + 1 - edges.append({ - "source": caller_nid, - "target": tgt_nid, - "relation": "calls", - "context": "call", - "confidence": "EXTRACTED", - "source_file": str_path, - "source_location": f"L{line}", - "weight": 1.0, - }) - elif not is_scoped_call and callee_name.lower() not in _RUST_TRAIT_METHOD_BLOCKLIST: - raw_calls.append({ - "caller_nid": caller_nid, - "callee": callee_name, - "is_member_call": is_member_call, - "source_file": str_path, - "source_location": f"L{node.start_point[0] + 1}", - }) - for child in node.children: - walk_calls(child, caller_nid) - - for caller_nid, body_node in function_bodies: - walk_calls(body_node, caller_nid) - - valid_ids = seen_ids - clean_edges = [] - for edge in edges: - src, tgt = edge["source"], edge["target"] - if src in valid_ids and (tgt in valid_ids or edge["relation"] in ("imports", "imports_from")): - clean_edges.append(edge) - - return {"nodes": nodes, "edges": clean_edges, "raw_calls": raw_calls} - - -# ── Zig ─────────────────────────────────────────────────────────────────────── - - - -# ── PowerShell ──────────────────────────────────────────────────────────────── - -def extract_powershell(path: Path) -> dict: - """Extract functions, classes, methods, and using statements from a .ps1 file.""" - try: - import tree_sitter_powershell as tsps - from tree_sitter import Language, Parser - except ImportError: - return {"nodes": [], "edges": [], "error": "tree_sitter_powershell not installed"} - - try: - language = Language(tsps.language()) - parser = Parser(language) - source = path.read_bytes() - tree = parser.parse(source) - root = tree.root_node - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - stem = _file_stem(path) - str_path = str(path) - nodes: list[dict] = [] - edges: list[dict] = [] - seen_ids: set[str] = set() - function_bodies: list[tuple[str, Any]] = [] - - def add_node(nid: str, label: str, line: int) -> None: - if nid not in seen_ids: - seen_ids.add(nid) - nodes.append({"id": nid, "label": label, "file_type": "code", - "source_file": str_path, "source_location": f"L{line}"}) - - def add_edge(src: str, tgt: str, relation: str, line: int, - confidence: str = "EXTRACTED", weight: float = 1.0, - context: str | None = None) -> None: - edge = {"source": src, "target": tgt, "relation": relation, - "confidence": confidence, "source_file": str_path, - "source_location": f"L{line}", "weight": weight} - if context: - edge["context"] = context - edges.append(edge) - - file_nid = _make_id(str(path)) - add_node(file_nid, path.name, 1) - - _PS_SKIP = frozenset({ - "using", "return", "if", "else", "elseif", "foreach", "for", - "while", "do", "switch", "try", "catch", "finally", "throw", - "break", "continue", "exit", "param", "begin", "process", "end", - # Import commands — handled as import edges, not function calls - "import-module", - }) - - def _find_script_block_body(node): - for child in node.children: - if child.type == "script_block": - for sc in child.children: - if sc.type == "script_block_body": - return sc - return child - return None - - def ensure_named_node(name: str, line: int) -> str: - nid = _make_id(stem, name) - if nid in seen_ids: - return nid - nid = _make_id(name) - if nid not in seen_ids: - # The name isn't defined in this file, so this is a cross-file reference - # (e.g. a `Thing` type annotation imported from another module). Emit a - # SOURCELESS stub — like the inheritance-base path below — so the - # corpus-level rewire can collapse it onto the real definition. A sourced - # stub here makes _disambiguate_colliding_node_ids bake the referencing - # file's path (with extension) into the id and blocks the rewire, which is - # the phantom-duplicate-node bug (#1402). - seen_ids.add(nid) - nodes.append({ - "id": nid, - "label": name, - "file_type": "code", - "source_file": "", - "source_location": "", - "origin_file": str_path, - }) - return nid - - def _ps_type_name(type_literal_node) -> str | None: - """Drill into a type_literal node and return the inner type_identifier text.""" - if type_literal_node is None: - return None - for spec in type_literal_node.children: - if spec.type != "type_spec": - continue - for tname in spec.children: - if tname.type != "type_name": - continue - for tid in tname.children: - if tid.type == "type_identifier": - return _read_text(tid, source) - return None - - def walk(node, parent_class_nid: str | None = None) -> None: - t = node.type - - if t == "function_statement": - name_node = next((c for c in node.children if c.type == "function_name"), None) - if name_node: - func_name = _read_text(name_node, source) - line = node.start_point[0] + 1 - func_nid = _make_id(stem, func_name) - add_node(func_nid, f"{func_name}()", line) - add_edge(file_nid, func_nid, "contains", line) - body = _find_script_block_body(node) - if body: - function_bodies.append((func_nid, body)) - # Also walk the body during the main pass so that - # Import-Module / dot-source inside functions emit - # file-level imports_from edges (#1331). - walk(body, parent_class_nid) - return - - if t == "class_statement": - name_node = next((c for c in node.children if c.type == "simple_name"), None) - if name_node: - class_name = _read_text(name_node, source) - line = node.start_point[0] + 1 - class_nid = _make_id(stem, class_name) - add_node(class_nid, class_name, line) - add_edge(file_nid, class_nid, "contains", line) - # Base type(s) after ':'. PowerShell has no syntactic base vs - # interface split, so (matching the C# convention) treat the - # first base as the superclass (inherits) and the rest as - # interfaces (implements). Bases are the simple_name children - # after the ':' token. - colon_seen = False - base_index = 0 - for child in node.children: - if child.type == ":": - colon_seen = True - elif colon_seen and child.type == "simple_name": - base_nid = ensure_named_node(_read_text(child, source), line) - if base_nid != class_nid: - rel = "inherits" if base_index == 0 else "implements" - add_edge(class_nid, base_nid, rel, line) - base_index += 1 - for child in node.children: - walk(child, parent_class_nid=class_nid) - return - - if t == "class_property_definition" and parent_class_nid: - type_literal = next((c for c in node.children if c.type == "type_literal"), None) - type_name = _ps_type_name(type_literal) - if type_name: - line = node.start_point[0] + 1 - target_nid = ensure_named_node(type_name, line) - if target_nid != parent_class_nid: - add_edge(parent_class_nid, target_nid, "references", - line, context="field") - return - - if t == "class_method_definition": - name_node = next((c for c in node.children if c.type == "simple_name"), None) - if name_node: - method_name = _read_text(name_node, source) - line = node.start_point[0] + 1 - if parent_class_nid: - method_nid = _make_id(parent_class_nid, method_name) - add_node(method_nid, f".{method_name}()", line) - add_edge(parent_class_nid, method_nid, "method", line) - else: - method_nid = _make_id(stem, method_name) - add_node(method_nid, f"{method_name}()", line) - add_edge(file_nid, method_nid, "contains", line) - # Return type: type_literal sibling of simple_name - return_type_literal = next( - (c for c in node.children if c.type == "type_literal"), None) - return_type_name = _ps_type_name(return_type_literal) - if return_type_name: - target_nid = ensure_named_node(return_type_name, line) - if target_nid != method_nid: - add_edge(method_nid, target_nid, "references", - line, context="return_type") - # Parameter types: class_method_parameter_list - param_list = next( - (c for c in node.children if c.type == "class_method_parameter_list"), None) - if param_list is not None: - for p in param_list.children: - if p.type != "class_method_parameter": - continue - ptype_literal = next( - (c for c in p.children if c.type == "type_literal"), None) - ptype_name = _ps_type_name(ptype_literal) - if not ptype_name: - continue - p_line = p.start_point[0] + 1 - target_nid = ensure_named_node(ptype_name, p_line) - if target_nid != method_nid: - add_edge(method_nid, target_nid, "references", - p_line, context="parameter_type") - body = _find_script_block_body(node) - if body: - function_bodies.append((method_nid, body)) - return - - if t == "command": - # Dot-sourcing: `. ./Shared.psm1` - # Uses command_invokation_operator '.' + command_name_expr (not command_name) - invoke_op = next( - (c for c in node.children if c.type == "command_invokation_operator"), None - ) - if invoke_op is not None and _read_text(invoke_op, source).strip() == ".": - name_expr = next( - (c for c in node.children if c.type == "command_name_expr"), None - ) - if name_expr is not None: - name_node = next( - (c for c in name_expr.children if c.type == "command_name"), None - ) - if name_node: - raw_path = _read_text(name_node, source) - # Strip relative path prefix (./ or .\ or just the dot) - module_stem = re.sub(r'^[./\\]+', '', raw_path) - # Drop extension to get bare module name - module_stem = re.sub(r'\.[^.]+$', '', module_stem).replace('\\', '/') - module_name = module_stem.split('/')[-1] - if module_name: - add_edge(file_nid, _make_id(module_name), "imports_from", - node.start_point[0] + 1) - return - - cmd_name_node = next((c for c in node.children if c.type == "command_name"), None) - if cmd_name_node: - cmd_text = _read_text(cmd_name_node, source).lower() - if cmd_text == "using": - tokens = [] - for child in node.children: - if child.type == "command_elements": - for el in child.children: - if el.type == "generic_token": - tokens.append(_read_text(el, source)) - module_tokens = [t for t in tokens - if t.lower() not in ("namespace", "module", "assembly")] - if module_tokens: - module_name = module_tokens[-1].split(".")[-1] - add_edge(file_nid, _make_id(module_name), "imports_from", - node.start_point[0] + 1) - elif cmd_text == "import-module": - # Collect generic_token args; skip command_parameter flags like -Name - # The module name is the first generic_token (or the one after -Name) - module_name: str | None = None - expect_name = False - for child in node.children: - if child.type != "command_elements": - continue - for el in child.children: - if el.type == "command_parameter": - param_text = _read_text(el, source).lstrip("-").lower() - expect_name = param_text in ("name", "n") - elif el.type == "generic_token": - token = _read_text(el, source) - if module_name is None or expect_name: - module_name = token - expect_name = False - if module_name: - # Strip extension; keep only the stem for the node ID - bare = re.sub(r'\.[^.]+$', '', module_name).split('/')[-1].split('\\')[-1] - if bare: - add_edge(file_nid, _make_id(bare), "imports_from", - node.start_point[0] + 1) - return - - for child in node.children: - walk(child, parent_class_nid) - - walk(root) - - label_to_nid = {n["label"].strip("()").lstrip(".").lower(): n["id"] for n in nodes} - seen_call_pairs: set[tuple[str, str]] = set() - raw_calls: list[dict] = [] - - def walk_calls(node, caller_nid: str) -> None: - if node.type in ("function_statement", "class_statement"): - return - if node.type == "command": - cmd_name_node = next((c for c in node.children if c.type == "command_name"), None) - if cmd_name_node: - cmd_text = _read_text(cmd_name_node, source) - if cmd_text.lower() not in _PS_SKIP: - tgt_nid = label_to_nid.get(cmd_text.lower()) - if tgt_nid and tgt_nid != caller_nid: - pair = (caller_nid, tgt_nid) - if pair not in seen_call_pairs: - seen_call_pairs.add(pair) - add_edge(caller_nid, tgt_nid, "calls", - node.start_point[0] + 1, - confidence="EXTRACTED", weight=1.0) - elif cmd_text: - raw_calls.append({ - "caller_nid": caller_nid, - "callee": cmd_text, - "is_member_call": False, - "source_file": str_path, - "source_location": f"L{node.start_point[0] + 1}", - }) - for child in node.children: - walk_calls(child, caller_nid) - - for caller_nid, body_node in function_bodies: - walk_calls(body_node, caller_nid) - - clean_edges = [e for e in edges if e["source"] in seen_ids and - (e["target"] in seen_ids or e["relation"] in ("imports_from", "imports"))] - return {"nodes": nodes, "edges": clean_edges, "raw_calls": raw_calls} - - -# ── PowerShell manifest (.psd1) ────────────────────────────────────────────── - -# Keys in a .psd1 whose values are module names/paths we treat as imports. -_PSD1_IMPORT_KEYS = frozenset({"RootModule", "NestedModules", "RequiredModules"}) - - -def _psd1_collect_string_literals(node, source: bytes) -> list[str]: - """Recursively collect all string_literal text values under *node*.""" - results: list[str] = [] - - def _walk(n) -> None: - if n.type == "string_literal": - raw = source[n.start_byte:n.end_byte].decode(errors="replace") - # Strip surrounding quote chars (' or ") - results.append(raw.strip("'\"")) - return - for child in n.children: - _walk(child) - - _walk(node) - return results - - -def _psd1_module_name(raw: str) -> str: - """Derive a bare module name from a raw string value. - - e.g. 'MyModule.psm1' → 'MyModule', './sub/Util.psm1' → 'Util', 'PSReadLine' → 'PSReadLine' - """ - # Strip path prefix and extension - name = raw.replace("\\", "/").split("/")[-1] - name = re.sub(r"\.[^.]+$", "", name) # remove last extension - return name.strip() - - -def extract_powershell_manifest(path: Path) -> dict: - """Extract module dependency edges from a PowerShell .psd1 manifest file. - - .psd1 files are PowerShell data hashtables, not scripts. tree-sitter-powershell - parses them correctly (they are syntactically valid PS). We walk the AST looking - for RootModule, NestedModules, and RequiredModules keys and emit imports_from - edges for every referenced module. - - RequiredModules supports two forms: - - Simple string: 'PSReadLine' - - Module specification: @{ ModuleName = 'Pester'; ModuleVersion = '5.0' } - For the hashtable form we only follow the ModuleName key. - """ - try: - import tree_sitter_powershell as tsps - from tree_sitter import Language, Parser - except ImportError: - return {"nodes": [], "edges": [], "error": "tree_sitter_powershell not installed"} - - try: - language = Language(tsps.language()) - parser = Parser(language) - source = path.read_bytes() - tree = parser.parse(source) - root = tree.root_node - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - str_path = str(path) - nodes: list[dict] = [] - edges: list[dict] = [] - seen_ids: set[str] = set() - - def add_node(nid: str, label: str, line: int) -> None: - if nid not in seen_ids: - seen_ids.add(nid) - nodes.append({"id": nid, "label": label, "file_type": "code", - "source_file": str_path, "source_location": f"L{line}"}) - - def add_import_edge(src: str, module_raw: str, line: int) -> None: - name = _psd1_module_name(module_raw) - if not name: - return - tgt_nid = _make_id(name) - edges.append({ - "source": src, - "target": tgt_nid, - "relation": "imports_from", - "confidence": "EXTRACTED", - "source_file": str_path, - "source_location": f"L{line}", - "weight": 1.0, - "context": "import", - }) - - file_nid = _make_id(str(path)) - add_node(file_nid, path.name, 1) - - def walk_manifest(node) -> None: - """Walk the AST and emit edges for import-relevant hash_entry nodes.""" - if node.type != "hash_entry": - for child in node.children: - walk_manifest(child) - return - - # Identify the key - key_node = next((c for c in node.children if c.type == "key_expression"), None) - if key_node is None: - return - key_text = source[key_node.start_byte:key_node.end_byte].decode(errors="replace").strip() - - if key_text not in _PSD1_IMPORT_KEYS: - # Still recurse in case there are nested hashes (e.g. ModuleVersion entries - # contain sub-hashes, but we only care about top-level keys for imports) - return - - line = node.start_point[0] + 1 - value_node = next((c for c in node.children if c.type == "pipeline"), None) - if value_node is None: - return - - if key_text == "RootModule": - # Value is a single string - strings = _psd1_collect_string_literals(value_node, source) - for s in strings: - add_import_edge(file_nid, s, line) - - elif key_text == "NestedModules": - # Value is a string or @('a', 'b', ...) array — collect all string literals - strings = _psd1_collect_string_literals(value_node, source) - for s in strings: - add_import_edge(file_nid, s, line) - - elif key_text == "RequiredModules": - # Two forms: - # 1) 'SimpleModule' — direct string literals in the array - # 2) @{ ModuleName = 'Foo'; ModuleVersion = '2.0' } — use ModuleName only - # - # Strategy: walk the value for hash_entry nodes whose key is 'ModuleName'; - # collect their string values. For the remaining string_literal nodes that - # are NOT inside a hash_entry subtree, treat them as simple module names. - module_name_strings: list[str] = [] - inside_hash_entries: set[int] = set() # byte offsets of handled strings - - def find_modulename_entries(n) -> None: - if n.type == "hash_entry": - sub_key = next((c for c in n.children if c.type == "key_expression"), None) - if sub_key is not None: - sk_text = source[sub_key.start_byte:sub_key.end_byte].decode(errors="replace").strip() - # Collect strings inside *all* sub-keys so we can exclude them - for c in n.children: - if c.type == "pipeline": - for s_node in _collect_string_nodes(c): - inside_hash_entries.add(s_node.start_byte) - if sk_text == "ModuleName": - for c in n.children: - if c.type == "pipeline": - for s in _psd1_collect_string_literals(c, source): - module_name_strings.append(s) - return # don't recurse further into this hash_entry - for child in n.children: - find_modulename_entries(child) - def _collect_string_nodes(n): - """Return all string_literal nodes in subtree.""" - if n.type == "string_literal": - yield n - return - for child in n.children: - yield from _collect_string_nodes(child) +# ── Zig ─────────────────────────────────────────────────────────────────────── - find_modulename_entries(value_node) - # Now gather direct string literals not inside hash entries - direct_strings: list[str] = [] - for s_node in _collect_string_nodes(value_node): - if s_node.start_byte not in inside_hash_entries: - raw = source[s_node.start_byte:s_node.end_byte].decode(errors="replace") - direct_strings.append(raw.strip("'\"")) +# ── PowerShell ──────────────────────────────────────────────────────────────── - for s in direct_strings + module_name_strings: - add_import_edge(file_nid, s, line) - walk_manifest(root) +# ── PowerShell manifest (.psd1) ────────────────────────────────────────────── - return {"nodes": nodes, "edges": edges, "raw_calls": []} +# Keys in a .psd1 whose values are module names/paths we treat as imports. # ── Cross-file import resolution ────────────────────────────────────────────── @@ -12840,8 +10258,6 @@ def walk_calls(n) -> None: return result - - # Inline markdown link: [text](target "optional title"). The negative lookbehind # excludes images (![alt](src)). The target stops at whitespace/closing paren so # an optional "title" after the URL is dropped; an optional <...> wrapper is too. @@ -13623,196 +11039,6 @@ def walk_calls(node, caller_nid: str) -> None: # type: ignore[no-untyped-def] return {"nodes": nodes, "edges": edges, "input_tokens": 0, "output_tokens": 0} -def extract_lazarus_form(path: Path) -> dict: - """Extract component hierarchy from Lazarus .lfm form files. - - .lfm is a text-based declarative format for UI component trees, structured as: - object ComponentName: TClassName - PropertyName = Value - OnEvent = HandlerName - object ChildName: TChildClass - ... - end - end - - Produces nodes for: - - The form file itself - - Each component class encountered (TForm1, TButton, TPanel, ...) - - Event handler names referenced by OnXxx properties - - Produces edges for: - - file --contains--> root form class - - parent component --contains--> child component class - - component --references--> event handler (context: "event") - """ - try: - text = path.read_text(encoding="utf-8", errors="replace") - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - import re - str_path = str(path) - stem = _file_stem(path) - nodes: list[dict] = [] - edges: list[dict] = [] - seen_ids: set[str] = set() - seen_edge_pairs: set[tuple[str, str, str]] = set() - - def add_node(nid: str, label: str, line: int) -> None: - if nid not in seen_ids: - seen_ids.add(nid) - nodes.append({ - "id": nid, "label": label, "file_type": "code", - "source_file": str_path, "source_location": f"L{line}", - }) - - def add_edge( - src: str, tgt: str, relation: str, line: int, - context: str | None = None, - ) -> None: - key = (src, tgt, relation) - if key in seen_edge_pairs: - return - seen_edge_pairs.add(key) - edge: dict[str, Any] = { - "source": src, "target": tgt, "relation": relation, - "confidence": "EXTRACTED", "source_file": str_path, - "source_location": f"L{line}", "weight": 1.0, - } - if context: - edge["context"] = context - edges.append(edge) - - file_nid = _make_id(str(path)) - add_node(file_nid, path.name, 1) - - obj_re = re.compile(r"^\s*object\s+\w+\s*:\s*(\w+)", re.IGNORECASE) - event_re = re.compile(r"^\s*On\w+\s*=\s*(\w+)", re.IGNORECASE) - end_re = re.compile(r"^\s*end\s*$", re.IGNORECASE) - - # Stack of node IDs representing the nesting of object...end blocks - stack: list[str] = [file_nid] - - for lineno, line in enumerate(text.splitlines(), 1): - m = obj_re.match(line) - if m: - class_name = m.group(1) - nid = _make_id(stem, class_name) - add_node(nid, class_name, lineno) - add_edge(stack[-1], nid, "contains", lineno) - stack.append(nid) - continue - - m = event_re.match(line) - if m and len(stack) > 1: - handler = m.group(1) - handler_nid = _make_id(stem, handler) - add_node(handler_nid, f"{handler}()", lineno) - add_edge(stack[-1], handler_nid, "references", lineno, context="event") - continue - - if end_re.match(line) and len(stack) > 1: - stack.pop() - - return {"nodes": nodes, "edges": edges, "input_tokens": 0, "output_tokens": 0} - - -def extract_delphi_form(path: Path) -> dict: - """Extract component hierarchy from Delphi .dfm form files. - - .dfm files come in two formats: - - Text (same `object Name: TClassName ... end` syntax as .lfm) - - Binary (starts with a TPF0/FF0A magic header — unreadable as text) - - Binary .dfm files are skipped gracefully: an empty result is returned - so the rest of the pipeline is unaffected. Convert binary forms to - text in the Delphi IDE via File → Save As (Text DFM) if you want them - indexed. - - Text .dfm files are parsed identically to .lfm: component containment - (`contains`) and event handler references (`references`, context "event"). - """ - try: - raw = path.read_bytes() - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - # Detect binary DFM: Delphi binary resource streams start with FF 0A - if raw[:2] == b"\xff\x0a": - return { - "nodes": [], "edges": [], - "error": f"binary DFM (convert to text in Delphi IDE to index): {path.name}", - } - - # Text DFM — delegate to the shared form parser (same syntax as .lfm) - try: - text = raw.decode("utf-8", errors="replace") - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - import re - str_path = str(path) - stem = _file_stem(path) - nodes: list[dict] = [] - edges: list[dict] = [] - seen_ids: set[str] = set() - seen_edge_pairs: set[tuple[str, str, str]] = set() - - def add_node(nid: str, label: str, line: int) -> None: - if nid not in seen_ids: - seen_ids.add(nid) - nodes.append({ - "id": nid, "label": label, "file_type": "code", - "source_file": str_path, "source_location": f"L{line}", - }) - - def add_edge( - src: str, tgt: str, relation: str, line: int, - context: str | None = None, - ) -> None: - key = (src, tgt, relation) - if key in seen_edge_pairs: - return - seen_edge_pairs.add(key) - edge: dict[str, Any] = { - "source": src, "target": tgt, "relation": relation, - "confidence": "EXTRACTED", "source_file": str_path, - "source_location": f"L{line}", "weight": 1.0, - } - if context: - edge["context"] = context - edges.append(edge) - - file_nid = _make_id(str(path)) - add_node(file_nid, path.name, 1) - - obj_re = re.compile(r"^\s*object\s+\w+\s*:\s*(\w+)", re.IGNORECASE) - event_re = re.compile(r"^\s*On\w+\s*=\s*(\w+)", re.IGNORECASE) - end_re = re.compile(r"^\s*end\s*$", re.IGNORECASE) - stack: list[str] = [file_nid] - - for lineno, line in enumerate(text.splitlines(), 1): - m = obj_re.match(line) - if m: - class_name = m.group(1) - nid = _make_id(stem, class_name) - add_node(nid, class_name, lineno) - add_edge(stack[-1], nid, "contains", lineno) - stack.append(nid) - continue - m = event_re.match(line) - if m and len(stack) > 1: - handler = m.group(1) - handler_nid = _make_id(stem, handler) - add_node(handler_nid, f"{handler}()", lineno) - add_edge(stack[-1], handler_nid, "references", lineno, context="event") - continue - if end_re.match(line) and len(stack) > 1: - stack.pop() - - return {"nodes": nodes, "edges": edges, "input_tokens": 0, "output_tokens": 0} - - # Size cap for project XML files we parse with stdlib ElementTree. # Real .csproj/.fsproj/.vbproj/.lpk files are well under 2 MiB; anything # larger is either malformed or hostile. @@ -13938,313 +11164,17 @@ def _check_tree_sitter_version() -> None: raise ImportError( "tree-sitter is not installed. Run: pip install 'tree-sitter>=0.23.0'" ) - # Language API v2 starts at LANGUAGE_VERSION 14 - if LANGUAGE_VERSION < 14: - import tree_sitter as _ts - raise RuntimeError( - f"tree-sitter {getattr(_ts, '__version__', 'unknown')} is too old. " - f"graphify requires tree-sitter >= 0.23.0 (Language API v2). " - f"Run: pip install --upgrade tree-sitter" - ) - - -def extract_bash(path: Path) -> dict: - """Extract functions, source imports, and cross-function calls from a .sh file.""" - try: - import tree_sitter_bash as tsbash - from tree_sitter import Language, Parser - except ImportError: - return {"nodes": [], "edges": [], "error": "tree-sitter-bash not installed"} - - try: - language = Language(tsbash.language()) - parser = Parser(language) - source = path.read_bytes() - tree = parser.parse(source) - root = tree.root_node - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - stem = _file_stem(path) - str_path = str(path) - nodes: list[dict] = [] - edges: list[dict] = [] - seen_ids: set[str] = set() - function_bodies: list[tuple[str, Any]] = [] - defined_functions: set[str] = set() - - from graphify.security import sanitize_metadata # module-level cached import - - def add_node(nid: str, label: str, line: int, kind: str = "code") -> None: - if nid and nid not in seen_ids: - seen_ids.add(nid) - nodes.append({"id": nid, "label": label, "file_type": "code", - "source_file": str_path, "source_location": f"L{line}", - "metadata": sanitize_metadata({"language": "bash", "kind": kind})}) # noqa: E501 - - def add_edge(src: str, tgt: str, relation: str, line: int, - confidence: str = "EXTRACTED", weight: float = 1.0, - context: str | None = None) -> None: - if not src or not tgt or src == tgt: - return - edge = {"source": src, "target": tgt, "relation": relation, - "confidence": confidence, "source_file": str_path, - "source_location": f"L{line}", "weight": weight} - if context: - edge["context"] = context - edges.append(edge) - - file_nid = _make_id(str(path)) - # file_nid is fully path-derived and never produced by _make_id(stem, func_name), - # so appending "__entry" guarantees a distinct ID from any function node. - entry_nid = file_nid + "__entry" - add_node(file_nid, path.name, 1, kind="file") - add_node(entry_nid, f"{path.name} script", 1, kind="bash_entrypoint") - add_edge(file_nid, entry_nid, "contains", 1) - - _BASH_SOURCE_COMMANDS = frozenset({"source", "."}) - # Parent node types that mean a contained command is part of a substitution - # or expansion, not a real function call. Token-level filtering misses - # these because `$(build)` exposes `build` as a child command whose name - # token has no metacharacters — only the parent does. - _BASH_EXPANSION_PARENTS = frozenset({ - "command_substitution", - "process_substitution", - }) - - def text(node) -> str: - return source[node.start_byte:node.end_byte].decode("utf-8", errors="replace") - - def is_inside_expansion(node) -> bool: - parent = node.parent - while parent is not None: - if parent.type in _BASH_EXPANSION_PARENTS: - return True - parent = parent.parent - return False - - def literal(node) -> str | None: - # Token-level filter: rejects names containing shell metacharacters. - # Combined with `is_inside_expansion` for parent-context rejection. - raw = text(node).strip() - if not raw: - return None - if raw[0:1] in {"'", '"'} and raw[-1:] == raw[0]: - raw = raw[1:-1] - if any(token in raw for token in ("$", "`", "$(", "<(", ">", "|", ";", "&")): - return None - return raw - - def _bash_func_name(node) -> str | None: - """Get the name from a function_definition node.""" - # bash grammar: function_definition has a word child (the name) - for child in node.children: - if child.type == "word": - return literal(child) - return None - - def walk_calls(body_node, func_nid: str, seen_calls: set) -> None: - if body_node is None: - return - for child in body_node.children: - if child.type == "function_definition": - # Skip nested function definitions — their bodies are walked - # separately, so we don't attribute their calls to the - # enclosing scope. - continue - if child.type == "command" and not is_inside_expansion(child): - cmd_name_node = child.child_by_field_name("name") - if cmd_name_node is None and child.children: - cmd_name_node = child.children[0] - if cmd_name_node: - name = literal(cmd_name_node) - # Defined-functions wins. Skip-lists for external commands - # would create false negatives when a user defines a - # function shadowing an external (`install`, `find`, etc.). - if name and name in defined_functions: - tgt = _make_id(stem, name) - key = (func_nid, tgt) - if tgt and key not in seen_calls: - seen_calls.add(key) - add_edge(func_nid, tgt, "calls", - child.start_point[0] + 1, - confidence="EXTRACTED", context="call") - walk_calls(child, func_nid, seen_calls) - - def walk(node, parent_nid: str) -> None: - t = node.type - if t == "function_definition": - name = _bash_func_name(node) - if name: - fn_nid = _make_id(stem, name) - line = node.start_point[0] + 1 - add_node(fn_nid, f"{name}()", line, kind="bash_function") - add_edge(parent_nid, fn_nid, "defines", line) - defined_functions.add(name) - # find the compound_statement body - body = None - for child in node.children: - if child.type == "compound_statement": - body = child - break - function_bodies.append((fn_nid, body)) - # Recurse into the body so nested function definitions are discovered - # and added to function_bodies for the second-pass walk_calls. - if body is not None: - walk(body, fn_nid) - return - - if t == "command": - if is_inside_expansion(node): - return - cmd_name_node = node.child_by_field_name("name") - if cmd_name_node is None and node.children: - cmd_name_node = node.children[0] - if cmd_name_node: - cmd = literal(cmd_name_node) - if cmd in _BASH_SOURCE_COMMANDS and cmd not in defined_functions: - # find the path argument (first word after command name) - args = [c for c in node.children - if c.type in ("word", "string", "concatenation") - and c != cmd_name_node] - if args: - raw = _read_text(args[0], source).strip().strip("'\"") - line = node.start_point[0] + 1 - if raw.startswith((".", "/")): - resolved = (path.parent / raw).resolve() - # Only emit the edge if the target actually exists on - # disk — prevents graph pollution from crafted paths - # like `source ../../etc/passwd` that traverse outside - # the project tree (B-1). - if resolved.exists(): - tgt_nid = _make_id(str(resolved)) - add_edge(file_nid, tgt_nid, "imports_from", line, - context="import") - else: - tgt_nid = _make_id(raw) - if tgt_nid: - add_edge(file_nid, tgt_nid, "imports", line, - context="import") - return - - if t == "declaration_command": - # export/declare/readonly VAR=value at program level - if node.parent and node.parent.type == "program": - for child in node.children: - if child.type == "variable_assignment": - var_node = child.child_by_field_name("name") - if var_node: - var = _read_text(var_node, source).strip() - if var: - var_nid = _make_id(stem, var) - line = child.start_point[0] + 1 - add_node(var_nid, var, line) - add_edge(file_nid, var_nid, "defines", line) - return - - for child in node.children: - walk(child, parent_nid) - - # Pre-pass: collect all defined function names so the source-command handler - # in walk() can detect user-defined functions that shadow 'source' / '.' - # regardless of definition order in the file. - def _prescan_functions(node) -> None: - if node.type == "function_definition": - name = _bash_func_name(node) - if name: - defined_functions.add(name) - for child in node.children: - _prescan_functions(child) - else: - for child in node.children: - _prescan_functions(child) - - _prescan_functions(root) - walk(root, file_nid) - - # Second pass: cross-function calls - top_seen: set = set() - walk_calls(root, entry_nid, top_seen) # top-level calls attributed to the entrypoint - for fn_nid, body in function_bodies: - walk_calls(body, fn_nid, set()) - - return {"nodes": nodes, "edges": edges} - - -# ── .NET project files (.sln, .slnx, .csproj, .razor) ─────────────────────── - -def extract_sln(path: Path) -> dict: - """Extract projects and inter-project dependencies from a .sln file.""" - try: - src = path.read_text(encoding="utf-8", errors="replace") - except OSError: - return {"nodes": [], "edges": [], "error": f"cannot read {path}"} - - file_nid = _make_id(str(path)) - str_path = str(path) - nodes: list[dict] = [{"id": file_nid, "label": path.name, "file_type": "code", - "source_file": str_path, "source_location": None}] - edges: list[dict] = [] - seen_ids: set[str] = set() - seen_ids.add(file_nid) - - _PROJECT_RE = re.compile( - r'Project\("[^"]*"\)\s*=\s*"([^"]+)"\s*,\s*"([^"]+)"\s*,\s*"([^"]*)"' - ) - _DEP_RE = re.compile(r'\{([0-9a-fA-F-]+)\}\s*=\s*\{([0-9a-fA-F-]+)\}') - - guid_to_nid: dict[str, str] = {} - - for m in _PROJECT_RE.finditer(src): - proj_name = m.group(1) - proj_path = m.group(2).replace("\\", "/") - proj_guid = m.group(3).strip("{}") + # Language API v2 starts at LANGUAGE_VERSION 14 + if LANGUAGE_VERSION < 14: + import tree_sitter as _ts + raise RuntimeError( + f"tree-sitter {getattr(_ts, '__version__', 'unknown')} is too old. " + f"graphify requires tree-sitter >= 0.23.0 (Language API v2). " + f"Run: pip install --upgrade tree-sitter" + ) - try: - abs_proj = str((path.parent / proj_path).resolve()) - except Exception: - abs_proj = proj_path - proj_nid = _make_id(abs_proj) - if proj_nid and proj_nid not in seen_ids: - seen_ids.add(proj_nid) - nodes.append({"id": proj_nid, "label": proj_name, - "file_type": "code", "source_file": abs_proj, - "source_location": None}) - edges.append({"source": file_nid, "target": proj_nid, - "relation": "contains", "confidence": "EXTRACTED", - "source_file": str_path, "weight": 1.0}) - if proj_guid: - guid_to_nid[proj_guid.lower()] = proj_nid - - in_dep_section = False - current_proj_guid: str | None = None - _PROJECT_LINE_RE = re.compile(r'Project\("[^"]*"\)\s*=\s*"[^"]+"\s*,\s*"[^"]+"\s*,\s*"\{([^}]+)\}"') - for line in src.splitlines(): - proj_line_m = _PROJECT_LINE_RE.search(line) - if proj_line_m: - current_proj_guid = proj_line_m.group(1).lower() - continue - if line.strip() == "EndProject": - current_proj_guid = None - continue - if "ProjectSection(ProjectDependencies)" in line: - in_dep_section = True - continue - if in_dep_section and "EndProjectSection" in line: - in_dep_section = False - continue - if in_dep_section and current_proj_guid: - dep_m = _DEP_RE.search(line) - if dep_m: - to_guid = dep_m.group(1).lower() - from_nid = guid_to_nid.get(current_proj_guid) - to_nid = guid_to_nid.get(to_guid) - if from_nid and to_nid and from_nid != to_nid: - edges.append({"source": from_nid, "target": to_nid, - "relation": "imports", "confidence": "EXTRACTED", - "source_file": str_path, "weight": 1.0}) - return {"nodes": nodes, "edges": edges} +# ── .NET project files (.sln, .slnx, .csproj, .razor) ─────────────────────── def extract_slnx(path: Path) -> dict: @@ -14968,541 +11898,100 @@ def add_existing_edge(edge: dict) -> None: confidence=vm_confidence, ) generated_members, generated_member_edges = _xaml_communitytoolkit_members(vm_node) - for member in generated_members.values(): - add_existing_node(member) - for member_edge in generated_member_edges: - add_existing_edge(member_edge) - - for elem in tree.iter(): - elem_type = _xml_local_name(elem.tag) - elem_name = None - for key, value in elem.attrib.items(): - if _xml_local_name(key) == "Name" and value: - elem_name = value.strip() - break - owner_nid = root_nid - if elem_name: - owner_nid = _make_id(stem, elem_name) - add_node(owner_nid, elem_name, line_for(elem_name)) - add_edge(root_nid, owner_nid, "contains", line_for(elem_name)) - type_nid = _make_id("xaml", elem_type) - add_node(type_nid, elem_type, line_for(elem_name), file_type="concept") - add_edge(owner_nid, type_nid, "references", line_for(elem_name), context="type") - - for key, value in elem.attrib.items(): - value = value or "" - # Event wiring: an attribute references a handler only when its local - # name isn't a known free-form/identity property, its value is a bare - # identifier (a method name, not markup or a sentence), and the matched - # code-behind method actually has an event-handler signature (the gate - # in _xaml_codebehind_symbols). This stops Content="Save" / Tag="..." - # from fabricating event edges against same-named ordinary methods. - attr_local = _xml_local_name(key) - if attr_local not in _XAML_NON_EVENT_ATTRS and _XAML_IDENT_RE.fullmatch(value): - method = codebehind_methods.get(value) - if method: - add_existing_node(method) - add_edge(owner_nid, method["id"], "references", line_for(value), context="event") - for method_edge in class_method_edges: - if method_edge.get("target") == method["id"]: - add_existing_node(class_node) - add_existing_edge(method_edge) - break - binding_path, binding_converter = _xaml_binding_refs(value) - if binding_path: - bind_nid = _make_id("binding", binding_path) - add_node(bind_nid, binding_path, line_for(value), file_type="concept") - binding_context = ( - "binding_command" - if attr_local == "Command" or attr_local.endswith(".Command") - else "binding_path" - ) - add_edge(owner_nid, bind_nid, "references", line_for(value), context=binding_context) - generated_member = generated_members.get(binding_path) - if generated_member: - add_existing_node(generated_member) - add_edge( - owner_nid, - generated_member["id"], - "references", - line_for(value), - context=binding_context, - confidence="INFERRED", - ) - if binding_converter: - converter_nid = _make_id("binding_converter", binding_converter) - add_node(converter_nid, binding_converter, line_for(value), file_type="concept") - add_edge(owner_nid, converter_nid, "references", line_for(value), context="binding_converter") - if elem_type == "Binding" and attr_local == "Path": - direct_path = value.strip() - if direct_path and "{" not in direct_path and "}" not in direct_path: - bind_nid = _make_id("binding", direct_path) - add_node(bind_nid, direct_path, line_for(value), file_type="concept") - add_edge(owner_nid, bind_nid, "references", line_for(value), context="binding_path") - if elem_type == "Binding" and attr_local == "Converter": - direct_converter = _xaml_static_resource_key(value) - if direct_converter: - converter_nid = _make_id("binding_converter", direct_converter) - add_node(converter_nid, direct_converter, line_for(value), file_type="concept") - add_edge(owner_nid, converter_nid, "references", line_for(value), context="binding_converter") - - return {"nodes": nodes, "edges": edges} - - - - -# Config/manifest JSON filenames the structural extractor understands. Anything -# else (eval fixtures, datasets, GeoJSON, API dumps) is *data* and must NOT be -# AST-walked into per-key nodes — that floods the graph with orphan key-nodes -# and near-duplicate communities (#1224). Data JSON is left to the LLM semantic -# pass instead. Matched case-insensitively against the bare filename. -_CONFIG_JSON_NAMES = frozenset({ - "package.json", "tsconfig.json", "jsconfig.json", "composer.json", - "deno.json", "deno.jsonc", "bower.json", "manifest.json", - "app.json", "now.json", "vercel.json", "angular.json", "nest-cli.json", - "biome.json", "biome.jsonc", "renovate.json", ".babelrc", ".babelrc.json", - ".eslintrc.json", ".prettierrc.json", ".prettierrc", "babel.config.json", -}) - -# Top-level keys that prove a JSON object is a config/manifest the extractor can -# draw *cross-file* edges from (deps, extends chains, schema refs). -_CONFIG_JSON_KEYS = frozenset({ - "dependencies", "devDependencies", "peerDependencies", - "optionalDependencies", "bundleDependencies", "bundledDependencies", - "extends", "$ref", "$schema", "compilerOptions", -}) - - -def _is_config_json(path: Path, obj_node, source: bytes) -> bool: - """True if a .json file is a recognized config/manifest worth AST-extracting. - - Matches by filename first (cheap), then falls back to a top-level key probe - so arbitrarily-named config files (e.g. ``api.tsconfig.json``, - ``foo.eslintrc.json``) are still picked up. Returns False for data JSON so it - is skipped by the structural pass (#1224).""" - name = path.name.casefold() - if name in _CONFIG_JSON_NAMES: - return True - # Common compound config names: *.eslintrc.json, *.prettierrc.json, etc. - if name.endswith((".eslintrc.json", ".prettierrc.json", ".babelrc.json", - "tsconfig.json", "jsconfig.json")): - return True - # Top-level key probe: scan the root object's immediate keys (no deep walk). - for top_key in obj_node.children: - if top_key.type != "pair": - continue - key_node = top_key.child_by_field_name("key") - if key_node is None: - continue - kc = key_node.child_by_field_name("string_content") - text = _read_text(kc, source) if kc else _read_text(key_node, source).strip('"\'') - if text in _CONFIG_JSON_KEYS: - return True - return False - - -def extract_json(path: Path) -> dict: - """Extract structure and dependency edges from a *config/manifest* .json file. - - Data-shaped JSON (eval fixtures, datasets, GeoJSON, API response dumps) is - deliberately skipped — AST-walking it produced hundreds of orphan key-nodes - and duplicate communities that swamped real structure (#1224). Recognition - is by filename (package.json, tsconfig.json, …) or a top-level key probe - (dependencies / extends / $ref / $schema / compilerOptions).""" - _JSON_MAX_BYTES = 1_048_576 # 1 MiB — skip large fixture dumps / GeoJSON blobs - - try: - import tree_sitter_json as tsjson - from tree_sitter import Language, Parser - except ImportError: - return {"nodes": [], "edges": [], "error": "tree-sitter-json not installed"} - - try: - # Bounded read instead of stat()+read() to eliminate TOCTOU (J-1): - # read one byte beyond the limit so we can detect oversized files even - # if the file grows between stat and read. - with path.open("rb") as _f: - source = _f.read(_JSON_MAX_BYTES + 1) - if len(source) > _JSON_MAX_BYTES: - return {"nodes": [], "edges": [], "error": "json file too large to index"} - language = Language(tsjson.language()) - parser = Parser(language) - tree = parser.parse(source) - root = tree.root_node - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - stem = _file_stem(path) - str_path = str(path) - nodes: list[dict] = [] - edges: list[dict] = [] - seen_ids: set[str] = set() - - # Keys whose string values become imports (package.json dep blocks) - _DEP_KEYS = frozenset({ - "dependencies", "devDependencies", "peerDependencies", - "optionalDependencies", "bundleDependencies", "bundledDependencies", - }) - - def add_node(nid: str, label: str, line: int) -> None: - if nid and nid not in seen_ids: - seen_ids.add(nid) - nodes.append({"id": nid, "label": label, "file_type": "code", - "source_file": str_path, "source_location": f"L{line}"}) - - def add_edge(src: str, tgt: str, relation: str, line: int, - context: str | None = None) -> None: - if not src or not tgt or src == tgt: - return - edge = {"source": src, "target": tgt, "relation": relation, - "confidence": "EXTRACTED", "source_file": str_path, - "source_location": f"L{line}", "weight": 1.0} - if context: - edge["context"] = context - edges.append(edge) - - file_nid = _make_id(str(path)) - add_node(file_nid, path.name, 1) - - def _key_text(pair_node) -> str | None: - """Extract the string content of a pair's key.""" - key_node = pair_node.child_by_field_name("key") - if key_node is None: - return None - if key_node.type == "string": - content = key_node.child_by_field_name("string_content") - if content: - return _read_text(content, source) - # fallback: strip surrounding quotes - raw = _read_text(key_node, source) - return raw.strip('"\'') - return _read_text(key_node, source) - - def _val_node(pair_node): - return pair_node.child_by_field_name("value") - - def walk_object(obj_node, parent_nid: str, parent_key: str | None, - depth: int, pair_count: list) -> None: - if depth > 6: - return - for child in obj_node.children: - if child.type != "pair": - continue - if pair_count[0] >= 500: # check per-pair so the cap is honoured exactly (J-3) - return - pair_count[0] += 1 - key = _key_text(child) - if not key: - continue - key_nid = _make_id(stem, *(([parent_key] if parent_key else []) + [key])) - if not key_nid: - continue - line = child.start_point[0] + 1 - add_node(key_nid, key, line) - add_edge(parent_nid, key_nid, "contains", line) - - val = _val_node(child) - if val is None: - continue - - if val.type == "object": - walk_object(val, key_nid, key, depth + 1, pair_count) - - elif val.type == "array": - # For "extends" arrays (tsconfig, eslint): each string element. - # Prefix with "ref_" so external refs don't collide with real - # code/file node IDs that share the same collapsed _make_id (J-4). - for item in val.children: - if item.type == "string": - content = item.child_by_field_name("string_content") - ref = _read_text(content, source) if content else _read_text(item, source).strip('"\'') - if ref: - ref_nid = _make_id("ref", ref) - if ref_nid: - add_edge(key_nid, ref_nid, "extends", line, context="import") - - elif val.type == "string": - content = val.child_by_field_name("string_content") - val_text = _read_text(content, source) if content else _read_text(val, source).strip('"\'') - - if key == "extends" and val_text: - # Namespace external refs to avoid ID collision with file nodes (J-4) - ref_nid = _make_id("ref", val_text) - if ref_nid: - add_edge(file_nid, ref_nid, "extends", line, context="import") - - elif key == "$ref" and val_text: - # Namespace $ref values to prevent edge hijacking into code nodes (J-4) - ref_nid = _make_id("ref", val_text) - if ref_nid: - add_edge(parent_nid, ref_nid, "references", line) - - elif parent_key in _DEP_KEYS and val_text: - dep_nid = _make_id(key) - if dep_nid: - add_edge(key_nid, dep_nid, "imports", line, context="import") - - # Entry: find root document → object - doc = root - if doc.type == "document" and doc.child_count > 0: - doc = doc.children[0] - if doc.type == "object": - # Only AST-extract recognized config/manifest JSON. Data JSON (fixtures, - # datasets, GeoJSON, API dumps) is skipped so it doesn't explode into - # orphan key-nodes (#1224); it's left to the LLM semantic pass. - if not _is_config_json(path, doc, source): - return {"nodes": [], "edges": [], "skipped": "data json (not a config/manifest)"} - walk_object(doc, file_nid, None, 0, [0]) - else: - # Top-level array or scalar => data JSON, never a config/manifest. - return {"nodes": [], "edges": [], "skipped": "data json (non-object root)"} - - return {"nodes": nodes, "edges": edges} - - -# ── DM (BYOND DreamMaker) extractor ────────────────────────────────────────── -# DM identity is path-based (`/datum/object/proc/New()`), not block-based, so -# the generic class-body walker doesn't fit well. - -def extract_dm(path: Path) -> dict: - """Extract types, procs, includes, and calls from a .dm/.dme file.""" - try: - import tree_sitter_dm as tsdm - from tree_sitter import Language, Parser - except ImportError: - return {"nodes": [], "edges": [], "error": "tree-sitter-dm not installed"} - try: - language = Language(tsdm.language()) - parser = Parser(language) - source = path.read_bytes() - tree = parser.parse(source) - root = tree.root_node - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - stem = _file_stem(path) - str_path = str(path) - nodes: list[dict] = [] - edges: list[dict] = [] - seen_ids: set[str] = set() - function_bodies: list[tuple[str, Any, "str | None"]] = [] - - def add_node(nid: str, label: str, line: int) -> None: - if nid and nid not in seen_ids: - seen_ids.add(nid) - nodes.append({"id": nid, "label": label, "file_type": "code", - "source_file": str_path, "source_location": f"L{line}"}) - - def add_edge(src: str, tgt: str, relation: str, line: int, - confidence: str = "EXTRACTED", weight: float = 1.0, - context: str | None = None) -> None: - if not src or not tgt or src == tgt: - return - edge: dict = {"source": src, "target": tgt, "relation": relation, - "confidence": confidence, "source_file": str_path, - "source_location": f"L{line}", "weight": weight} - if context: - edge["context"] = context - edges.append(edge) - - file_nid = _make_id(str(path)) - add_node(file_nid, path.name, 1) - - def _type_path_text(node) -> str: - return _read_text(node, source).strip() - - def _ensure_type(path_text: str, line: int) -> str: - nid = _make_id(stem, path_text) - add_node(nid, path_text, line) - return nid - - def _find_child(node, type_name: str): - for c in node.children: - if c.type == type_name: - return c - return None - - def _read_include_path(file_node) -> str: - if file_node is None: - return "" - if file_node.type == "string_literal": - parts = [] - for c in file_node.children: - if c.type == "string_content": - parts.append(_read_text(c, source)) - return "".join(parts) - return _read_text(file_node, source).strip("'\"") - - def walk(node, parent_type_path: "str | None" = None, - parent_type_nid: "str | None" = None) -> None: - t = node.type - line = node.start_point[0] + 1 - - if t == "preproc_include": - file_node = node.child_by_field_name("file") - raw = _read_include_path(file_node) - if raw: - norm = raw.replace("\\", "/").lstrip("./") - resolved = (path.parent / norm).resolve() - edge: dict = { - "source": file_nid, - "target": _make_id(str(resolved)) if resolved.exists() else _make_id(norm), - "relation": "imports_from" if resolved.exists() else "imports", - "context": "import", - "confidence": "EXTRACTED", - "source_file": str_path, - "source_location": f"L{line}", - "weight": 1.0, - } - if not resolved.exists(): - edge["external"] = True - edges.append(edge) - return - - if t == "type_definition": - tp_node = _find_child(node, "type_path") - if tp_node is None: - return - type_path_str = _type_path_text(tp_node) - type_nid = _ensure_type(type_path_str, line) - add_edge(file_nid, type_nid, "contains", line) - body = _find_child(node, "type_body") - if body is not None: - for c in body.children: - walk(c, parent_type_path=type_path_str, parent_type_nid=type_nid) - return - - if t in ("type_body_intended", "type_body_braced"): - for c in node.children: - walk(c, parent_type_path, parent_type_nid) - return - - if t in ("type_proc_definition", "type_proc_override"): - if parent_type_nid is None or parent_type_path is None: - return - name_node = node.child_by_field_name("name") - if name_node is None: - return - proc_name = _read_text(name_node, source) - proc_nid = _make_id(stem, parent_type_path, proc_name) - add_node(proc_nid, f"{parent_type_path}/{proc_name}()", line) - add_edge(parent_type_nid, proc_nid, "method", line) - block = _find_child(node, "block") - if block is not None: - function_bodies.append((proc_nid, block, parent_type_path)) - return - - if t in ("proc_definition", "proc_override"): - tp_node = _find_child(node, "type_path") - owner_path: "str | None" = None - owner_nid: "str | None" = None - if tp_node is not None: - owner_path = _type_path_text(tp_node) - owner_nid = _ensure_type(owner_path, line) - add_edge(file_nid, owner_nid, "contains", line) - name_node = node.child_by_field_name("name") - if name_node is None: - return - proc_name = _read_text(name_node, source) - if owner_path and owner_nid: - proc_nid = _make_id(stem, owner_path, proc_name) - add_node(proc_nid, f"{owner_path}/{proc_name}()", line) - add_edge(owner_nid, proc_nid, "method", line) - else: - proc_nid = _make_id(stem, proc_name) - add_node(proc_nid, f"{proc_name}()", line) - add_edge(file_nid, proc_nid, "contains", line) - block = _find_child(node, "block") - if block is not None: - function_bodies.append((proc_nid, block, owner_path)) - return - - if t in ("operator_override", "type_operator_override"): - return + for member in generated_members.values(): + add_existing_node(member) + for member_edge in generated_member_edges: + add_existing_edge(member_edge) - for child in node.children: - walk(child, parent_type_path, parent_type_nid) + for elem in tree.iter(): + elem_type = _xml_local_name(elem.tag) + elem_name = None + for key, value in elem.attrib.items(): + if _xml_local_name(key) == "Name" and value: + elem_name = value.strip() + break + owner_nid = root_nid + if elem_name: + owner_nid = _make_id(stem, elem_name) + add_node(owner_nid, elem_name, line_for(elem_name)) + add_edge(root_nid, owner_nid, "contains", line_for(elem_name)) + type_nid = _make_id("xaml", elem_type) + add_node(type_nid, elem_type, line_for(elem_name), file_type="concept") + add_edge(owner_nid, type_nid, "references", line_for(elem_name), context="type") - walk(root) + for key, value in elem.attrib.items(): + value = value or "" + # Event wiring: an attribute references a handler only when its local + # name isn't a known free-form/identity property, its value is a bare + # identifier (a method name, not markup or a sentence), and the matched + # code-behind method actually has an event-handler signature (the gate + # in _xaml_codebehind_symbols). This stops Content="Save" / Tag="..." + # from fabricating event edges against same-named ordinary methods. + attr_local = _xml_local_name(key) + if attr_local not in _XAML_NON_EVENT_ATTRS and _XAML_IDENT_RE.fullmatch(value): + method = codebehind_methods.get(value) + if method: + add_existing_node(method) + add_edge(owner_nid, method["id"], "references", line_for(value), context="event") + for method_edge in class_method_edges: + if method_edge.get("target") == method["id"]: + add_existing_node(class_node) + add_existing_edge(method_edge) + break + binding_path, binding_converter = _xaml_binding_refs(value) + if binding_path: + bind_nid = _make_id("binding", binding_path) + add_node(bind_nid, binding_path, line_for(value), file_type="concept") + binding_context = ( + "binding_command" + if attr_local == "Command" or attr_local.endswith(".Command") + else "binding_path" + ) + add_edge(owner_nid, bind_nid, "references", line_for(value), context=binding_context) + generated_member = generated_members.get(binding_path) + if generated_member: + add_existing_node(generated_member) + add_edge( + owner_nid, + generated_member["id"], + "references", + line_for(value), + context=binding_context, + confidence="INFERRED", + ) + if binding_converter: + converter_nid = _make_id("binding_converter", binding_converter) + add_node(converter_nid, binding_converter, line_for(value), file_type="concept") + add_edge(owner_nid, converter_nid, "references", line_for(value), context="binding_converter") + if elem_type == "Binding" and attr_local == "Path": + direct_path = value.strip() + if direct_path and "{" not in direct_path and "}" not in direct_path: + bind_nid = _make_id("binding", direct_path) + add_node(bind_nid, direct_path, line_for(value), file_type="concept") + add_edge(owner_nid, bind_nid, "references", line_for(value), context="binding_path") + if elem_type == "Binding" and attr_local == "Converter": + direct_converter = _xaml_static_resource_key(value) + if direct_converter: + converter_nid = _make_id("binding_converter", direct_converter) + add_node(converter_nid, direct_converter, line_for(value), file_type="concept") + add_edge(owner_nid, converter_nid, "references", line_for(value), context="binding_converter") - label_to_nids: dict[str, list[str]] = {} - path_to_nids: dict[str, list[str]] = {} - for n in nodes: - label = n["label"].strip("()") - last = label.rsplit("/", 1)[-1] if "/" in label else label - if last: - label_to_nids.setdefault(last.lower(), []).append(n["id"]) - if label.startswith("/"): - path_to_nids.setdefault(label.lower(), []).append(n["id"]) + return {"nodes": nodes, "edges": edges} - seen_call_pairs: set[tuple[str, str]] = set() - raw_calls: list[dict] = [] - def _emit_call(caller_nid: str, callee: str, line: int, is_member: bool) -> None: - candidates = label_to_nids.get(callee.lower(), []) - tgt_nid = candidates[0] if len(candidates) == 1 else None - if tgt_nid and tgt_nid != caller_nid: - pair = (caller_nid, tgt_nid) - if pair in seen_call_pairs: - return - seen_call_pairs.add(pair) - edges.append({ - "source": caller_nid, "target": tgt_nid, "relation": "calls", - "context": "call", "confidence": "EXTRACTED", - "source_file": str_path, "source_location": f"L{line}", "weight": 1.0, - }) - else: - raw_calls.append({ - "caller_nid": caller_nid, "callee": callee, - "is_member_call": is_member, "source_file": str_path, - "source_location": f"L{line}", - }) +# Config/manifest JSON filenames the structural extractor understands. Anything +# else (eval fixtures, datasets, GeoJSON, API dumps) is *data* and must NOT be +# AST-walked into per-key nodes — that floods the graph with orphan key-nodes +# and near-duplicate communities (#1224). Data JSON is left to the LLM semantic +# pass instead. Matched case-insensitively against the bare filename. - def walk_calls(body_node, caller_nid: str) -> None: - if body_node is None: - return - t = body_node.type - if t in ("proc_definition", "proc_override", "type_proc_definition", - "type_proc_override", "type_definition"): - return - if t == "call_expression": - name_node = body_node.child_by_field_name("name") - if name_node is not None: - callee = _read_text(name_node, source) - if callee and callee != "..": - _emit_call(caller_nid, callee, body_node.start_point[0] + 1, - is_member=False) - elif t == "field_proc_expression": - proc_field = body_node.child_by_field_name("proc") - if proc_field is not None: - callee = _read_text(proc_field, source) - if callee: - _emit_call(caller_nid, callee, body_node.start_point[0] + 1, - is_member=True) - elif t == "new_expression": - tp_node = _find_child(body_node, "type_path") - if tp_node is not None: - target_text = _type_path_text(tp_node) - candidates = path_to_nids.get(target_text.lower(), []) - tgt_nid = candidates[0] if len(candidates) == 1 else None - if tgt_nid and tgt_nid != caller_nid: - pair = (caller_nid, tgt_nid) - if pair not in seen_call_pairs: - seen_call_pairs.add(pair) - edges.append({ - "source": caller_nid, "target": tgt_nid, - "relation": "instantiates", "context": "call", - "confidence": "EXTRACTED", "source_file": str_path, - "source_location": f"L{body_node.start_point[0] + 1}", - "weight": 1.0, - }) - for child in body_node.children: - walk_calls(child, caller_nid) +# Top-level keys that prove a JSON object is a config/manifest the extractor can +# draw *cross-file* edges from (deps, extends chains, schema refs). - for proc_nid, block, _owner_path in function_bodies: - walk_calls(block, proc_nid) - return {"nodes": nodes, "edges": edges, "raw_calls": raw_calls} +# ── DM (BYOND DreamMaker) extractor ────────────────────────────────────────── +# DM identity is path-based (`/datum/object/proc/New()`), not block-based, so +# the generic class-body walker doesn't fit well. # ── DMI (BYOND icon files) ──────────────────────────────────────────────────── @@ -15510,440 +11999,18 @@ def walk_calls(body_node, caller_nid: str) -> None: # metadata. We want the icon state names (icon_state = "X" in DM code # references them). -def _read_dmi_description(data: bytes) -> str: - """Pull the BYOND metadata text out of a .dmi PNG, or empty string on failure.""" - import struct - import zlib as _zlib - if not data.startswith(b"\x89PNG\r\n\x1a\n"): - return "" - i = 8 - while i + 8 <= len(data): - length = struct.unpack(">I", data[i:i + 4])[0] - chunk_type = data[i + 4:i + 8] - payload = data[i + 8:i + 8 + length] - if chunk_type in (b"tEXt", b"zTXt"): - try: - null = payload.index(b"\x00") - except ValueError: - return "" - keyword = payload[:null] - if keyword == b"Description": - if chunk_type == b"zTXt": - return _zlib.decompressobj().decompress(payload[null + 2:], max_length=1024 * 1024).decode("utf-8", errors="replace") - return payload[null + 1:].decode("utf-8", errors="replace") - i += 8 + length + 4 - return "" - - -def extract_dmi(path: Path) -> dict: - """Extract icon state names from a .dmi (BYOND PNG icon sheet).""" - try: - data = path.read_bytes() - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - str_path = str(path) - stem = _file_stem(path) - file_nid = _make_id(str(path)) - nodes: list[dict] = [{"id": file_nid, "label": path.name, "file_type": "code", - "source_file": str_path, "source_location": "L1"}] - edges: list[dict] = [] - seen: set[str] = {file_nid} - - description = _read_dmi_description(data) - if not description: - return {"nodes": nodes, "edges": edges} - - line_no = 0 - for raw_line in description.splitlines(): - line_no += 1 - stripped = raw_line.strip() - if not stripped.startswith("state ="): - continue - value = stripped.split("=", 1)[1].strip() - if value.startswith('"') and value.endswith('"') and len(value) >= 2: - state_name = value[1:-1] - else: - state_name = value - if not state_name: - continue - nid = _make_id(stem, "state", state_name) - if nid in seen: - continue - seen.add(nid) - nodes.append({"id": nid, "label": f'"{state_name}"', "file_type": "code", - "source_file": str_path, "source_location": f"L{line_no}"}) - edges.append({"source": file_nid, "target": nid, "relation": "contains", - "confidence": "EXTRACTED", "source_file": str_path, - "source_location": f"L{line_no}", "weight": 1.0}) - - return {"nodes": nodes, "edges": edges} - # ── DMM (BYOND map files) ───────────────────────────────────────────────────── # A .dmm starts with a tile dictionary — each "key" = (type, type{var=val}, ...) # names one or more types that compose a tile — then a grid. We only need the # dictionary section: every type path referenced is a `uses` edge. -_DMM_GRID_RE = re.compile(r"^\(\s*\d+\s*,\s*\d+\s*,\s*\d+\s*\)\s*=", re.MULTILINE) - - -def _split_dmm_tile(body: str) -> list[str]: - out: list[str] = [] - buf: list[str] = [] - depth = 0 - in_string = False - escape = False - for ch in body: - if escape: - buf.append(ch) - escape = False - continue - if in_string: - buf.append(ch) - if ch == "\\": - escape = True - elif ch == '"': - in_string = False - continue - if ch == '"': - in_string = True - buf.append(ch) - elif ch in "({[": - depth += 1 - buf.append(ch) - elif ch in ")}]": - depth -= 1 - buf.append(ch) - elif ch == "," and depth == 0: - out.append("".join(buf).strip()) - buf = [] - else: - buf.append(ch) - tail = "".join(buf).strip() - if tail: - out.append(tail) - return out - - -def _dmm_type_path(entry: str) -> str: - brace = entry.find("{") - if brace != -1: - entry = entry[:brace] - return entry.strip() - - -def extract_dmm(path: Path) -> dict: - """Extract type-path references from a .dmm map file's tile dictionary.""" - try: - if path.stat().st_size > 50 * 1024 * 1024: - return {"nodes": [], "edges": [], "error": "file too large (>50 MB)"} - text = path.read_text(encoding="utf-8", errors="replace") - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - str_path = str(path) - file_nid = _make_id(str(path)) - nodes: list[dict] = [{"id": file_nid, "label": path.name, "file_type": "code", - "source_file": str_path, "source_location": "L1"}] - edges: list[dict] = [] - - grid_match = _DMM_GRID_RE.search(text) - dict_text = text[:grid_match.start()] if grid_match else text - - seen_targets: set[str] = set() - buf: list[str] = [] - open_line = 0 - depth = 0 - in_string = False - escape = False - for line_idx, line in enumerate(dict_text.splitlines(), start=1): - for ch in line: - if escape: - escape = False - elif in_string: - if ch == "\\": - escape = True - elif ch == '"': - in_string = False - elif ch == '"': - in_string = True - elif ch == "(": - if depth == 0: - open_line = line_idx - depth += 1 - elif ch == ")": - depth -= 1 - buf.append(ch) - buf.append("\n") - if depth == 0 and buf: - chunk = "".join(buf) - buf = [] - lp = chunk.find("(") - rp = chunk.rfind(")") - if lp == -1 or rp == -1 or rp <= lp: - continue - inner = chunk[lp + 1:rp] - for entry in _split_dmm_tile(inner): - tpath = _dmm_type_path(entry) - if not tpath.startswith("/"): - continue - tgt = _make_id(tpath) - if tgt in seen_targets: - continue - seen_targets.add(tgt) - edges.append({"source": file_nid, "target": tgt, "relation": "uses", - "context": "map", "confidence": "EXTRACTED", - "source_file": str_path, - "source_location": f"L{open_line}", "weight": 1.0}) - - return {"nodes": nodes, "edges": edges} - # ── DMF (BYOND interface forms) ─────────────────────────────────────────────── -_DMF_WINDOW_RE = re.compile(r'^\s*window\s+"([^"]+)"\s*$') -_DMF_ELEM_RE = re.compile(r'^\s*elem\s+"([^"]+)"\s*$') -_DMF_TYPE_RE = re.compile(r'^\s*type\s*=\s*(\S+)\s*$') - - -def extract_dmf(path: Path) -> dict: - """Extract windows and controls from a .dmf interface file.""" - try: - text = path.read_text(encoding="utf-8", errors="replace") - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - str_path = str(path) - stem = _file_stem(path) - file_nid = _make_id(str(path)) - nodes: list[dict] = [{"id": file_nid, "label": path.name, "file_type": "code", - "source_file": str_path, "source_location": "L1"}] - edges: list[dict] = [] - seen: set[str] = {file_nid} - - current_window_nid: str | None = None - current_elem_nid: str | None = None - current_elem_name: str | None = None - - for line_idx, line in enumerate(text.splitlines(), start=1): - m = _DMF_WINDOW_RE.match(line) - if m: - name = m.group(1) - nid = _make_id(stem, "window", name) - if nid not in seen: - seen.add(nid) - nodes.append({"id": nid, "label": f'window "{name}"', "file_type": "code", - "source_file": str_path, "source_location": f"L{line_idx}"}) - edges.append({"source": file_nid, "target": nid, "relation": "contains", - "confidence": "EXTRACTED", "source_file": str_path, - "source_location": f"L{line_idx}", "weight": 1.0}) - current_window_nid = nid - current_elem_nid = None - current_elem_name = None - continue - m = _DMF_ELEM_RE.match(line) - if m and current_window_nid is not None: - name = m.group(1) - nid = _make_id(stem, "elem", current_window_nid, name) - if nid not in seen: - seen.add(nid) - nodes.append({"id": nid, "label": f'elem "{name}"', "file_type": "code", - "source_file": str_path, "source_location": f"L{line_idx}"}) - edges.append({"source": current_window_nid, "target": nid, - "relation": "contains", "confidence": "EXTRACTED", - "source_file": str_path, "source_location": f"L{line_idx}", - "weight": 1.0}) - current_elem_nid = nid - current_elem_name = name - continue - m = _DMF_TYPE_RE.match(line) - if m and current_elem_nid is not None and current_elem_name is not None: - ctype = m.group(1) - for n in nodes: - if n["id"] == current_elem_nid and " [" not in n["label"]: - n["label"] = f'elem "{current_elem_name}" [{ctype}]' - break - - return {"nodes": nodes, "edges": edges} - # Head tokens in an HCL traversal that are meta/builtins, not references to a # block defined in the corpus (count.index, each.key, self.*, path.module, ...). -_TF_META_HEADS = frozenset({"count", "each", "self", "path", "terraform"}) - - -def extract_terraform(path: Path) -> dict: - """Extract Terraform/HCL blocks and the references between them via tree-sitter. - - Nodes: resources, data sources, modules, variables, outputs, providers, and - locals. Edges: `contains` (file -> block), `references` (block -> the blocks - it interpolates, e.g. `aws_instance.web` -> `var.region`), and `depends_on` - (explicit dependency edges). - - Node IDs are scoped by the parent directory, not the file stem, because - Terraform resources are module(directory)-scoped: a resource defined in - main.tf is referenced from other .tf files in the same directory. Directory - scoping lets those cross-file references resolve when per-file extractions - are merged (stem scoping would split a definition from its references). - """ - try: - import tree_sitter_hcl as tshcl - from tree_sitter import Language, Parser - except ImportError: - return {"nodes": [], "edges": [], "error": "tree_sitter_hcl not installed. Run: pip install tree-sitter-hcl"} - - try: - language = Language(tshcl.language()) - parser = Parser(language) - source = path.read_bytes() - tree = parser.parse(source) - root = tree.root_node - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - str_path = str(path) - file_nid = _make_id(str_path) - scope = path.parent.name or "tf" - - nodes: list[dict] = [{"id": file_nid, "label": path.name, "file_type": "code", - "source_file": str_path, "source_location": None}] - edges: list[dict] = [] - seen_ids: set[str] = {file_nid} - seen_edges: set[tuple[str, str, str]] = set() - - def _read(n) -> str: - return source[n.start_byte:n.end_byte].decode("utf-8", errors="replace") - - def _label_text(n) -> str: - return _read(n).strip().strip('"') - - def _add_node(address: str, label: str, line: int) -> str: - nid = _make_id(scope, address) - if nid not in seen_ids: - seen_ids.add(nid) - nodes.append({"id": nid, "label": label, "file_type": "code", - "source_file": str_path, "source_location": f"L{line}"}) - edges.append({"source": file_nid, "target": nid, "relation": "contains", - "confidence": "EXTRACTED", "source_file": str_path, - "source_location": f"L{line}", "weight": 1.0}) - return nid - - def _add_edge(src: str, address: str, relation: str, line: int) -> None: - tgt = _make_id(scope, address) - if src == tgt: - return - key = (src, tgt, relation) - if key in seen_edges: - return - seen_edges.add(key) - edges.append({"source": src, "target": tgt, "relation": relation, - "confidence": "EXTRACTED", "source_file": str_path, - "source_location": f"L{line}", "weight": 1.0}) - - def _block_parts(block) -> tuple: - btype = None - labels: list[str] = [] - for c in block.children: - if c.type in ("block_start", "body", "block_end"): - break - if c.type == "identifier" and btype is None: - btype = _read(c) - elif c.type in ("string_lit", "identifier"): - labels.append(_label_text(c)) - return btype, labels - - def _ref_address(expr): - head = _read(expr) - parent = expr.parent - attrs: list[str] = [] - if parent is not None: - seen_self = False - for c in parent.children: - if c.id == expr.id: - seen_self = True - continue - if seen_self and c.type == "get_attr": - name = None - for gc in c.children: - if gc.type == "identifier": - name = _read(gc) - break - if name is None: - break - attrs.append(name) - elif seen_self and c.type not in ("get_attr",): - break - if head in _TF_META_HEADS or not head: - return None - if head == "var": - return f"var.{attrs[0]}" if attrs else None - if head == "local": - return f"local.{attrs[0]}" if attrs else None - if head == "module": - return f"module.{attrs[0]}" if attrs else None - if head == "data": - return f"data.{attrs[0]}.{attrs[1]}" if len(attrs) >= 2 else None - return f"{head}.{attrs[0]}" if attrs else None - - def _collect_refs(node, owner_nid: str, relation: str) -> None: - rel = relation - if node.type == "attribute": - key_node = node.child_by_field_name("key") or ( - node.children[0] if node.children else None - ) - if key_node is not None and _read(key_node) == "depends_on": - rel = "depends_on" - if node.type == "variable_expr": - addr = _ref_address(node) - if addr: - _add_edge(owner_nid, addr, rel, node.start_point[0] + 1) - for c in node.children: - if c.is_named: - _collect_refs(c, owner_nid, rel) - - def _body_of(block): - for c in block.children: - if c.type == "body": - return c - return None - - body = next((c for c in root.children if c.type == "body"), root) - for block in body.children: - if block.type != "block": - continue - btype, labels = _block_parts(block) - line = block.start_point[0] + 1 - blk_body = _body_of(block) - if btype == "resource" and len(labels) >= 2: - owner = _add_node(f"{labels[0]}.{labels[1]}", f"{labels[0]}.{labels[1]}", line) - elif btype == "data" and len(labels) >= 2: - owner = _add_node(f"data.{labels[0]}.{labels[1]}", f"data.{labels[0]}.{labels[1]}", line) - elif btype == "module" and labels: - owner = _add_node(f"module.{labels[0]}", f"module.{labels[0]}", line) - elif btype == "variable" and labels: - owner = _add_node(f"var.{labels[0]}", f"var.{labels[0]}", line) - elif btype == "output" and labels: - owner = _add_node(f"output.{labels[0]}", f"output.{labels[0]}", line) - elif btype == "provider" and labels: - owner = _add_node(f"provider.{labels[0]}", f"provider.{labels[0]}", line) - elif btype == "locals" and blk_body is not None: - for attr in blk_body.children: - if attr.type != "attribute": - continue - key_node = attr.children[0] if attr.children else None - if key_node is None: - continue - key = _read(key_node) - lnid = _add_node(f"local.{key}", f"local.{key}", attr.start_point[0] + 1) - _collect_refs(attr, lnid, "references") - continue - else: - continue - if blk_body is not None: - _collect_refs(blk_body, owner, "references") - - return {"nodes": nodes, "edges": edges} _DISPATCH: dict[str, Any] = { diff --git a/graphify/extractors/MIGRATION.md b/graphify/extractors/MIGRATION.md index 1ddf6badf..75e1525e2 100644 --- a/graphify/extractors/MIGRATION.md +++ b/graphify/extractors/MIGRATION.md @@ -12,7 +12,21 @@ written so an AI agent can execute it in a single session. | zig | yes | | elixir | yes | | razor | yes | -| (40 more in extract.py) | no | +| dart | yes | +| rust | yes | +| go | yes | +| powershell (ps1 + psd1 manifest) | yes | +| fortran | yes | +| sql | yes | +| dm (dm/dmm/dmi/dmf) | yes | +| bash | yes | +| apex | yes | +| terraform | yes | +| sln | yes | +| pascal_forms (dfm + lfm) | yes | +| json_config | yes | +| (config-driven core: python, js, java, c, cpp, csharp, kotlin, scala, php, lua, swift, groovy, vue, svelte, astro, xaml, groovy) | no — shared _extract_generic core, move as one batch | +| (other bespoke: julia, verilog, markdown, objc, csproj, slnx, lazarus_package, pascal) | no | Note: config-driven extractors (python, js, java, c, cpp, ruby, csharp, kotlin, scala, php, lua, swift, groovy) depend on the shared diff --git a/graphify/extractors/__init__.py b/graphify/extractors/__init__.py index b2593b8c2..46b484b3d 100644 --- a/graphify/extractors/__init__.py +++ b/graphify/extractors/__init__.py @@ -10,14 +10,45 @@ from pathlib import Path from typing import Callable +from graphify.extractors.apex import extract_apex +from graphify.extractors.bash import extract_bash from graphify.extractors.blade import extract_blade +from graphify.extractors.dart import extract_dart +from graphify.extractors.dm import extract_dm, extract_dmf, extract_dmi, extract_dmm from graphify.extractors.elixir import extract_elixir +from graphify.extractors.fortran import extract_fortran +from graphify.extractors.go import extract_go +from graphify.extractors.json_config import extract_json +from graphify.extractors.pascal_forms import extract_delphi_form, extract_lazarus_form +from graphify.extractors.powershell import extract_powershell, extract_powershell_manifest from graphify.extractors.razor import extract_razor +from graphify.extractors.rust import extract_rust +from graphify.extractors.sln import extract_sln +from graphify.extractors.sql import extract_sql +from graphify.extractors.terraform import extract_terraform from graphify.extractors.zig import extract_zig LANGUAGE_EXTRACTORS: dict[str, Callable[[Path], dict]] = { + "apex": extract_apex, + "bash": extract_bash, "blade": extract_blade, + "dart": extract_dart, + "delphi_form": extract_delphi_form, + "dm": extract_dm, + "dmf": extract_dmf, + "dmi": extract_dmi, + "dmm": extract_dmm, "elixir": extract_elixir, + "fortran": extract_fortran, + "go": extract_go, + "json": extract_json, + "lazarus_form": extract_lazarus_form, + "powershell": extract_powershell, + "powershell_manifest": extract_powershell_manifest, "razor": extract_razor, + "rust": extract_rust, + "sln": extract_sln, + "sql": extract_sql, + "terraform": extract_terraform, "zig": extract_zig, } diff --git a/graphify/extractors/apex.py b/graphify/extractors/apex.py new file mode 100644 index 000000000..928923a64 --- /dev/null +++ b/graphify/extractors/apex.py @@ -0,0 +1,215 @@ +"""Apex extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + + +from pathlib import Path +from graphify.extractors.base import _file_stem, _make_id + + +def extract_apex(path: Path) -> dict: + """Extract classes, interfaces, enums, methods, and Salesforce constructs from + Apex .cls and .trigger files using regex (no tree-sitter grammar on PyPI).""" + import re as _re + try: + source = path.read_text(encoding="utf-8", errors="replace") + except OSError: + return {"nodes": [], "edges": []} + + str_path = str(path) + stem = _file_stem(path) + file_nid = _make_id(str_path) + + nodes: list[dict] = [] + edges: list[dict] = [] + seen_ids: set[str] = set() + + def add_node(nid: str, label: str, line: int) -> None: + if nid not in seen_ids: + seen_ids.add(nid) + nodes.append({ + "id": nid, + "label": label, + "file_type": "code", + "source_file": str_path, + "source_location": f"L{line}", + }) + + def add_edge(src: str, tgt: str, relation: str, line: int, + confidence: str = "EXTRACTED") -> None: + edges.append({ + "source": src, + "target": tgt, + "relation": relation, + "confidence": confidence, + "source_file": str_path, + "source_location": f"L{line}", + "weight": 1.0, + }) + + add_node(file_nid, path.name, 1) + + lines = source.splitlines() + + _ACCESS = r"(?:public|private|protected|global|webService)?" + _SHARING = r"(?:\s+(?:with|without|inherited)\s+sharing)?" + _MOD = r"(?:\s+(?:abstract|virtual|override|static|final|transient|testMethod))?" + _ANNOTATION = r"(?:\s*@\w+(?:\s*\([^)]*\))?\s*)*" + + cls_re = _re.compile( + rf"^{_ANNOTATION}\s*{_ACCESS}{_SHARING}{_MOD}\s*class\s+(\w+)" + rf"(?:\s+extends\s+(\w+))?(?:\s+implements\s+([\w,\s]+))?\s*\{{?", + _re.IGNORECASE, + ) + iface_re = _re.compile( + rf"^{_ANNOTATION}\s*{_ACCESS}{_SHARING}{_MOD}\s*interface\s+(\w+)" + rf"(?:\s+extends\s+([\w,\s]+))?\s*\{{?", + _re.IGNORECASE, + ) + enum_re = _re.compile( + rf"^{_ANNOTATION}\s*{_ACCESS}{_SHARING}{_MOD}\s*enum\s+(\w+)\s*\{{?", + _re.IGNORECASE, + ) + trigger_re = _re.compile( + r"^\s*trigger\s+(\w+)\s+on\s+(\w+)\s*\(", + _re.IGNORECASE, + ) + method_re = _re.compile( + rf"^{_ANNOTATION}\s*{_ACCESS}{_MOD}\s*(?:static\s+)?[\w<>\[\]]+\s+(\w+)\s*\([^)]*\)\s*(?:throws\s+\w+\s*)?\{{?", + _re.IGNORECASE, + ) + annotation_re = _re.compile(r"@(\w+)", _re.IGNORECASE) + soql_re = _re.compile(r"\[\s*SELECT\b[^\]]+FROM\s+(\w+)", _re.IGNORECASE) + dml_re = _re.compile(r"\b(insert|update|delete|upsert|merge|undelete)\s+\w", _re.IGNORECASE) + + _CONTROL_FLOW = frozenset({ + "if", "else", "for", "while", "do", "switch", "try", "catch", + "finally", "return", "throw", "new", "void", "null", + "true", "false", "this", "super", "class", "interface", "enum", + "trigger", "on", + }) + + current_class_nid: str | None = None + pending_annotations: list[str] = [] + + for lineno, line_text in enumerate(lines, start=1): + stripped = line_text.strip() + + if stripped.startswith("@"): + for m in annotation_re.finditer(stripped): + pending_annotations.append(m.group(1).lower()) + continue + + tm = trigger_re.match(stripped) + if tm: + trig_name, sobject = tm.group(1), tm.group(2) + trig_nid = _make_id(stem, trig_name) + add_node(trig_nid, trig_name, lineno) + add_edge(file_nid, trig_nid, "contains", lineno) + sob_nid = _make_id(sobject) + if sob_nid not in seen_ids: + add_node(sob_nid, sobject, lineno) + add_edge(trig_nid, sob_nid, "uses", lineno, confidence="INFERRED") + current_class_nid = trig_nid + pending_annotations = [] + continue + + cm = cls_re.match(stripped) + if cm: + class_name = cm.group(1) + if class_name.lower() in _CONTROL_FLOW: + pending_annotations = [] + continue + class_nid = _make_id(stem, class_name) + add_node(class_nid, class_name, lineno) + add_edge(file_nid, class_nid, "contains", lineno) + if cm.group(2): + base = cm.group(2).strip() + base_nid = _make_id(stem, base) + if base_nid not in seen_ids: + base_nid = _make_id(base) + if base_nid not in seen_ids: + add_node(base_nid, base, lineno) + add_edge(class_nid, base_nid, "extends", lineno, confidence="INFERRED") + if cm.group(3): + for iface in cm.group(3).split(","): + iface = iface.strip() + if iface: + iface_nid = _make_id(stem, iface) + if iface_nid not in seen_ids: + iface_nid = _make_id(iface) + if iface_nid not in seen_ids: + add_node(iface_nid, iface, lineno) + add_edge(class_nid, iface_nid, "implements", lineno, confidence="INFERRED") + current_class_nid = class_nid + pending_annotations = [] + continue + + im = iface_re.match(stripped) + if im: + iface_name = im.group(1) + if iface_name.lower() in _CONTROL_FLOW: + pending_annotations = [] + continue + iface_nid = _make_id(stem, iface_name) + add_node(iface_nid, iface_name, lineno) + add_edge(file_nid if current_class_nid is None else current_class_nid, + iface_nid, "contains", lineno) + if im.group(2): + for parent in im.group(2).split(","): + parent = parent.strip() + if parent: + parent_nid = _make_id(stem, parent) + if parent_nid not in seen_ids: + parent_nid = _make_id(parent) + if parent_nid not in seen_ids: + add_node(parent_nid, parent, lineno) + add_edge(iface_nid, parent_nid, "extends", lineno, confidence="INFERRED") + pending_annotations = [] + continue + + em = enum_re.match(stripped) + if em: + enum_name = em.group(1) + if enum_name.lower() in _CONTROL_FLOW: + pending_annotations = [] + continue + enum_nid = _make_id(stem, enum_name) + add_node(enum_nid, enum_name, lineno) + add_edge(file_nid if current_class_nid is None else current_class_nid, + enum_nid, "contains", lineno) + pending_annotations = [] + continue + + if current_class_nid is not None: + mm = method_re.match(stripped) + if mm: + method_name = mm.group(1) + if method_name.lower() not in _CONTROL_FLOW: + method_nid = _make_id(current_class_nid, method_name) + method_label = f".{method_name}()" + add_node(method_nid, method_label, lineno) + add_edge(current_class_nid, method_nid, "method", lineno) + if "auraenabled" in pending_annotations or "invocablemethod" in pending_annotations: + add_edge(file_nid, method_nid, "contains", lineno, confidence="INFERRED") + pending_annotations = [] + continue + + pending_annotations = [] + + for sm in soql_re.finditer(line_text): + sobject = sm.group(1) + sob_nid = _make_id(sobject) + if sob_nid not in seen_ids: + add_node(sob_nid, sobject, lineno) + src = current_class_nid or file_nid + add_edge(src, sob_nid, "uses", lineno, confidence="INFERRED") + + for dm in dml_re.finditer(line_text): + dml_op = dm.group(1).lower() + dml_nid = _make_id(f"dml_{dml_op}") + if dml_nid not in seen_ids: + add_node(dml_nid, dml_op, lineno) + src = current_class_nid or file_nid + add_edge(src, dml_nid, "uses", lineno, confidence="INFERRED") + + return {"nodes": nodes, "edges": edges} diff --git a/graphify/extractors/bash.py b/graphify/extractors/bash.py new file mode 100644 index 000000000..160ea032c --- /dev/null +++ b/graphify/extractors/bash.py @@ -0,0 +1,230 @@ +"""Bash extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + + +from pathlib import Path +from typing import Any +from graphify.extractors.base import _file_stem, _make_id, _read_text + + +def extract_bash(path: Path) -> dict: + """Extract functions, source imports, and cross-function calls from a .sh file.""" + try: + import tree_sitter_bash as tsbash + from tree_sitter import Language, Parser + except ImportError: + return {"nodes": [], "edges": [], "error": "tree-sitter-bash not installed"} + + try: + language = Language(tsbash.language()) + parser = Parser(language) + source = path.read_bytes() + tree = parser.parse(source) + root = tree.root_node + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + stem = _file_stem(path) + str_path = str(path) + nodes: list[dict] = [] + edges: list[dict] = [] + seen_ids: set[str] = set() + function_bodies: list[tuple[str, Any]] = [] + defined_functions: set[str] = set() + + from graphify.security import sanitize_metadata # module-level cached import + + def add_node(nid: str, label: str, line: int, kind: str = "code") -> None: + if nid and nid not in seen_ids: + seen_ids.add(nid) + nodes.append({"id": nid, "label": label, "file_type": "code", + "source_file": str_path, "source_location": f"L{line}", + "metadata": sanitize_metadata({"language": "bash", "kind": kind})}) # noqa: E501 + + def add_edge(src: str, tgt: str, relation: str, line: int, + confidence: str = "EXTRACTED", weight: float = 1.0, + context: str | None = None) -> None: + if not src or not tgt or src == tgt: + return + edge = {"source": src, "target": tgt, "relation": relation, + "confidence": confidence, "source_file": str_path, + "source_location": f"L{line}", "weight": weight} + if context: + edge["context"] = context + edges.append(edge) + + file_nid = _make_id(str(path)) + # file_nid is fully path-derived and never produced by _make_id(stem, func_name), + # so appending "__entry" guarantees a distinct ID from any function node. + entry_nid = file_nid + "__entry" + add_node(file_nid, path.name, 1, kind="file") + add_node(entry_nid, f"{path.name} script", 1, kind="bash_entrypoint") + add_edge(file_nid, entry_nid, "contains", 1) + + _BASH_SOURCE_COMMANDS = frozenset({"source", "."}) + # Parent node types that mean a contained command is part of a substitution + # or expansion, not a real function call. Token-level filtering misses + # these because `$(build)` exposes `build` as a child command whose name + # token has no metacharacters — only the parent does. + _BASH_EXPANSION_PARENTS = frozenset({ + "command_substitution", + "process_substitution", + }) + + def text(node) -> str: + return source[node.start_byte:node.end_byte].decode("utf-8", errors="replace") + + def is_inside_expansion(node) -> bool: + parent = node.parent + while parent is not None: + if parent.type in _BASH_EXPANSION_PARENTS: + return True + parent = parent.parent + return False + + def literal(node) -> str | None: + # Token-level filter: rejects names containing shell metacharacters. + # Combined with `is_inside_expansion` for parent-context rejection. + raw = text(node).strip() + if not raw: + return None + if raw[0:1] in {"'", '"'} and raw[-1:] == raw[0]: + raw = raw[1:-1] + if any(token in raw for token in ("$", "`", "$(", "<(", ">", "|", ";", "&")): + return None + return raw + + def _bash_func_name(node) -> str | None: + """Get the name from a function_definition node.""" + # bash grammar: function_definition has a word child (the name) + for child in node.children: + if child.type == "word": + return literal(child) + return None + + def walk_calls(body_node, func_nid: str, seen_calls: set) -> None: + if body_node is None: + return + for child in body_node.children: + if child.type == "function_definition": + # Skip nested function definitions — their bodies are walked + # separately, so we don't attribute their calls to the + # enclosing scope. + continue + if child.type == "command" and not is_inside_expansion(child): + cmd_name_node = child.child_by_field_name("name") + if cmd_name_node is None and child.children: + cmd_name_node = child.children[0] + if cmd_name_node: + name = literal(cmd_name_node) + # Defined-functions wins. Skip-lists for external commands + # would create false negatives when a user defines a + # function shadowing an external (`install`, `find`, etc.). + if name and name in defined_functions: + tgt = _make_id(stem, name) + key = (func_nid, tgt) + if tgt and key not in seen_calls: + seen_calls.add(key) + add_edge(func_nid, tgt, "calls", + child.start_point[0] + 1, + confidence="EXTRACTED", context="call") + walk_calls(child, func_nid, seen_calls) + + def walk(node, parent_nid: str) -> None: + t = node.type + if t == "function_definition": + name = _bash_func_name(node) + if name: + fn_nid = _make_id(stem, name) + line = node.start_point[0] + 1 + add_node(fn_nid, f"{name}()", line, kind="bash_function") + add_edge(parent_nid, fn_nid, "defines", line) + defined_functions.add(name) + # find the compound_statement body + body = None + for child in node.children: + if child.type == "compound_statement": + body = child + break + function_bodies.append((fn_nid, body)) + # Recurse into the body so nested function definitions are discovered + # and added to function_bodies for the second-pass walk_calls. + if body is not None: + walk(body, fn_nid) + return + + if t == "command": + if is_inside_expansion(node): + return + cmd_name_node = node.child_by_field_name("name") + if cmd_name_node is None and node.children: + cmd_name_node = node.children[0] + if cmd_name_node: + cmd = literal(cmd_name_node) + if cmd in _BASH_SOURCE_COMMANDS and cmd not in defined_functions: + # find the path argument (first word after command name) + args = [c for c in node.children + if c.type in ("word", "string", "concatenation") + and c != cmd_name_node] + if args: + raw = _read_text(args[0], source).strip().strip("'\"") + line = node.start_point[0] + 1 + if raw.startswith((".", "/")): + resolved = (path.parent / raw).resolve() + # Only emit the edge if the target actually exists on + # disk — prevents graph pollution from crafted paths + # like `source ../../etc/passwd` that traverse outside + # the project tree (B-1). + if resolved.exists(): + tgt_nid = _make_id(str(resolved)) + add_edge(file_nid, tgt_nid, "imports_from", line, + context="import") + else: + tgt_nid = _make_id(raw) + if tgt_nid: + add_edge(file_nid, tgt_nid, "imports", line, + context="import") + return + + if t == "declaration_command": + # export/declare/readonly VAR=value at program level + if node.parent and node.parent.type == "program": + for child in node.children: + if child.type == "variable_assignment": + var_node = child.child_by_field_name("name") + if var_node: + var = _read_text(var_node, source).strip() + if var: + var_nid = _make_id(stem, var) + line = child.start_point[0] + 1 + add_node(var_nid, var, line) + add_edge(file_nid, var_nid, "defines", line) + return + + for child in node.children: + walk(child, parent_nid) + + # Pre-pass: collect all defined function names so the source-command handler + # in walk() can detect user-defined functions that shadow 'source' / '.' + # regardless of definition order in the file. + def _prescan_functions(node) -> None: + if node.type == "function_definition": + name = _bash_func_name(node) + if name: + defined_functions.add(name) + for child in node.children: + _prescan_functions(child) + else: + for child in node.children: + _prescan_functions(child) + + _prescan_functions(root) + walk(root, file_nid) + + # Second pass: cross-function calls + top_seen: set = set() + walk_calls(root, entry_nid, top_seen) # top-level calls attributed to the entrypoint + for fn_nid, body in function_bodies: + walk_calls(body, fn_nid, set()) + + return {"nodes": nodes, "edges": edges} diff --git a/graphify/extractors/dart.py b/graphify/extractors/dart.py new file mode 100644 index 000000000..acbe19583 --- /dev/null +++ b/graphify/extractors/dart.py @@ -0,0 +1,528 @@ +"""Dart extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + +import re + +from pathlib import Path +from graphify.extractors.base import _file_stem, _make_id + + +def extract_dart(path: Path) -> dict: + """Extract classes, mixins, functions, imports, generic calls, and annotations from a .dart file using regex.""" + try: + src = path.read_text(encoding="utf-8", errors="replace") + except OSError: + return {"error": f"cannot read {path}"} + + # Remove inline and multi-line comments while leaving string literals untouched to prevent stripping URLs/paths inside strings + comment_string_pattern = re.compile( + r'"""(?:\\.|[\s\S])*?"""' + r"|'''(?:\\.|[\s\S])*?'''" + r'|"(?:\\.|[^"\\])*"' + r"|'(?:\\.|[^'\\])*'" + r"|/\*[\s\S]*?\*/" + r"|//[^\n]*" + ) + def _comment_replace(match: re.Match) -> str: + token = match.group(0) + if token.startswith("/"): + return "" + return token + src_clean = comment_string_pattern.sub(_comment_replace, src) + + stem = _file_stem(path) + file_nid = _make_id(str(path)) + + # Check if this is a part-of file and redirect to parent + part_of_match = re.search(r"^\s*part\s+of\s+['\"]([^'\"]+)['\"]", src_clean, re.MULTILINE) + is_part = False + if part_of_match: + parent_ref = part_of_match.group(1) + if parent_ref.endswith(".dart"): + try: + parent_path = (path.parent / parent_ref).resolve() + if parent_path.exists(): + stem = _file_stem(parent_path) + file_nid = _make_id(str(parent_path)) + is_part = True + except Exception: + pass + + nodes = [] + if not is_part: + nodes.append({"id": file_nid, "label": path.name, "file_type": "code", + "source_file": str(path), "source_location": None}) + edges = [] + defined: set[str] = set() + + def add_node(nid: str, label: str, ftype: str = "code", source_file: str | None = str(path)) -> None: + if nid not in defined: + nodes.append({"id": nid, "label": label, "file_type": ftype, + "source_file": source_file, "source_location": None}) + defined.add(nid) + + def add_edge(src_id: str, tgt_id: str, relation: str, weight: float = 1.0, context: str | None = None) -> None: + edge = {"source": src_id, "target": tgt_id, "relation": relation, + "confidence": "EXTRACTED", "confidence_score": 1.0, + "source_file": str(path), "source_location": None, "weight": weight} + if context: + edge["context"] = context + edges.append(edge) + + def _split_types(text: str) -> list[str]: + parts = [] + current = [] + depth = 0 + for char in text: + if char == "<": + depth += 1 + current.append(char) + elif char == ">": + depth -= 1 + current.append(char) + elif char == "," and depth == 0: + parts.append("".join(current).strip()) + current = [] + else: + current.append(char) + if current: + parts.append("".join(current).strip()) + return [p for p in parts if p] + + def _find_matching_brace(text: str, start_pos: int) -> int: + brace_count = 0 + in_double_quote = False + in_single_quote = False + escape = False + + first_brace = text.find("{", start_pos) + if first_brace == -1: + return len(text) + + brace_count = 1 + i = first_brace + 1 + n = len(text) + while i < n: + char = text[i] + if escape: + escape = False + i += 1 + continue + if char == "\\": + escape = True + i += 1 + continue + if text[i:i+3] == '"""' and not in_single_quote: + i += 3 + end = text.find('"""', i) + i = end + 3 if end != -1 else n + continue + if text[i:i+3] == "'''" and not in_double_quote: + i += 3 + end = text.find("'''", i) + i = end + 3 if end != -1 else n + continue + if char == '"' and not in_single_quote: + in_double_quote = not in_double_quote + elif char == "'" and not in_double_quote: + in_single_quote = not in_single_quote + elif not in_double_quote and not in_single_quote: + if char == "{": + brace_count += 1 + elif char == "}": + brace_count -= 1 + if brace_count == 0: + return i + 1 + i += 1 + return len(text) + + # 1. Classes, mixins, and enums declarations (with inheritance, mixins, interfaces, and generics) + # Supports multiple combined modifiers (e.g., abstract base class, mixin class) without capturing "class" as a name + class_pattern = r"^\s*(?:(?:abstract|sealed|base|interface|final|mixin)\s+)*(?:class|mixin|enum|extension\s+type)\s+(\w+)" + for m in re.finditer(class_pattern, src_clean, re.MULTILINE): + class_name = m.group(1) + class_nid = _make_id(stem, class_name) + add_node(class_nid, class_name) + add_edge(file_nid, class_nid, "defines") + + # Manually parse extends/on, with, and implements in header to handle nested generics brackets balanced + start_idx = m.end() + rest = src_clean[start_idx : start_idx + 500] + + # Skip class generic parameters + if rest.lstrip().startswith("<"): + offset = rest.find("<") + depth = 1 + i = offset + 1 + while i < len(rest) and depth > 0: + if rest[i] == "<": depth += 1 + elif rest[i] == ">": depth -= 1 + i += 1 + rest = rest[i:] + + # Skip primary constructor (e.g. extension type MyExt(int id)) + if rest.lstrip().startswith("("): + offset = rest.find("(") + depth = 1 + i = offset + 1 + while i < len(rest) and depth > 0: + if rest[i] == "(": depth += 1 + elif rest[i] == ")": depth -= 1 + i += 1 + rest = rest[i:] + + header_end = rest.find("{") + if header_end == -1: + header_end = rest.find(";") + if header_end == -1: + header_end = len(rest) + header = rest[:header_end] + + base_class = None + generics = None + mixins_list = [] + interfaces_list = [] + + # Parse extends or on + extends_m = re.search(r"^\s*(?:extends|on)\s+([a-zA-Z0-9_.]+)", header) + if extends_m: + base_class = extends_m.group(1) + rest_header = header[extends_m.end():] + if rest_header.strip().startswith("<"): + start_idx = rest_header.find("<") + depth = 1 + i = start_idx + 1 + while i < len(rest_header) and depth > 0: + if rest_header[i] == "<": + depth += 1 + elif rest_header[i] == ">": + depth -= 1 + if depth == 0: + generics = rest_header[start_idx + 1 : i] + break + i += 1 + if generics is not None: + header = rest_header[i + 1:] + else: + header = rest_header + else: + header = rest_header + + # Parse with + with_m = re.search(r"^\s*with\s+", header) + if with_m: + rest_header = header[with_m.end():] + impl_idx = rest_header.find("implements") + if impl_idx != -1: + mixins_str = rest_header[:impl_idx] + header = rest_header[impl_idx:] + else: + mixins_str = rest_header + header = "" + mixins_list = _split_types(mixins_str) + + # Parse implements + impl_m = re.search(r"^\s*implements\s+", header) + if impl_m: + interfaces_list = _split_types(header[impl_m.end():]) + + # Map extends inheritance relation + if base_class: + base_nid = _make_id(base_class) + add_node(base_nid, base_class, source_file=None) + add_edge(class_nid, base_nid, "inherits") + + # Map generic type arguments (e.g. MyBloc extends Bloc) + if generics: + for gen in _split_types(generics): + gen_clean = gen.split("<")[0].strip() + if gen_clean not in {"String", "int", "double", "bool", "num", "dynamic", "Object", "void"}: + gen_nid = _make_id(gen_clean) + add_node(gen_nid, gen_clean, source_file=None) + add_edge(class_nid, gen_nid, "references") + + # Map mixins + for mixin in mixins_list: + mixin_clean = mixin.split("<")[0].strip() + mixin_nid = _make_id(mixin_clean) + add_node(mixin_nid, mixin_clean, source_file=None) + add_edge(class_nid, mixin_nid, "mixes_in") + + # Map interfaces + for interface in interfaces_list: + interface_clean = interface.split("<")[0].strip() + interface_nid = _make_id(interface_clean) + add_node(interface_nid, interface_clean, source_file=None) + add_edge(class_nid, interface_nid, "implements") + + # Extract class body for precise framework dependencies and event handling + start_idx = m.start() + brace_pos = src_clean.find("{", start_idx) + semi_pos = src_clean.find(";", start_idx) + + has_body = brace_pos != -1 + if has_body and semi_pos != -1 and semi_pos < brace_pos: + has_body = False + + if has_body: + end_pos = _find_matching_brace(src_clean, start_idx) + class_body = src_clean[brace_pos:end_pos] + + # Bloc event registration: on() + for em in re.finditer(r"\bon<(\w+)>\s*\(", class_body): + event_name = em.group(1) + event_nid = _make_id(event_name) + add_node(event_nid, event_name, source_file=None) + add_edge(class_nid, event_nid, "calls", context="bloc_event") + + # Bloc state emissions: emit(MyState) or yield MyState + for sm in re.finditer(r"\b(?:emit|yield)\s*\(?\s*(?:const\s+)?([A-Z]\w*)\b", class_body): + state_name = sm.group(1) + if state_name not in {"String", "List", "Map", "Set", "Future", "Stream", "Object"}: + state_nid = _make_id(state_name) + add_node(state_nid, state_name, source_file=None) + add_edge(class_nid, state_nid, "calls", context="emit_state") + + # Bloc event additions: widget.add(MyEvent()) or bloc.add(MyEvent()) + for am in re.finditer(r"\b(?:\w*[Bb]loc\w*|context\.read<\w+>\(\))\.add\(\s*(?:const\s+)?([A-Z]\w*)\b", class_body): + event_name = am.group(1) + if event_name not in {"String", "List", "Map", "Set", "Future", "Stream", "Object"}: + event_nid = _make_id(event_name) + add_node(event_nid, event_name, source_file=None) + add_edge(class_nid, event_nid, "calls", context="bloc_add_event") + + # Riverpod provider references: ref.watch(provider) + for rm in re.finditer(r"\bref\.(?:watch|read|listen)\s*\(\s*(\w+)\b", class_body): + provider_name = rm.group(1) + provider_nid = _make_id(provider_name) + add_node(provider_nid, provider_name, source_file=None) + add_edge(class_nid, provider_nid, "references", context="riverpod_reference") + + # Widget to Bloc references: BlocBuilder + for bm in re.finditer(r"\bBloc(?:Builder|Listener|Consumer|Provider|Selector)\s*<\s*([a-zA-Z0-9_]+)\b", class_body): + bloc_name = bm.group(1) + if bloc_name not in {"String", "int", "double", "bool", "num", "dynamic", "Object", "void"}: + bloc_nid = _make_id(bloc_name) + add_node(bloc_nid, bloc_name, source_file=None) + add_edge(class_nid, bloc_nid, "references", context="bloc_widget_binding") + + # context.read() or BlocProvider.of(context) + for lm in re.finditer(r"\b(?:read|watch|select|of)\s*<([a-zA-Z0-9_]+)>", class_body): + bloc_name = lm.group(1) + if bloc_name not in {"String", "int", "double", "bool", "num", "dynamic", "Object", "void"}: + bloc_nid = _make_id(bloc_name) + add_node(bloc_nid, bloc_name, source_file=None) + add_edge(class_nid, bloc_nid, "references", context="bloc_lookup") + + # 2. Annotations mapping (class, mixin, enum, or function level annotations) + # Support: @riverpod, @Riverpod(...), @injectable, @singleton, @RoutePage(), @HiveType(typeId: 0), @RestApi() + # Matches `@annotation` and links it to the next class/mixin/enum/function declaration in the file + annotation_pattern = r"@(\w+)(?:\([^)]*\))?" + for am in re.finditer(annotation_pattern, src_clean): + annotation_name = am.group(1) + if annotation_name in {"override", "deprecated", "required", "protected", "mustCallSuper"}: + continue + annotation_pos = am.end() + intervening_text = src_clean[annotation_pos : annotation_pos + 300] + + class_m = re.search(r"^\s*(?:(?:abstract|sealed|base|interface|final|mixin)\s+)*(?:class|mixin|enum|extension\s+type)\s+(\w+)", intervening_text, re.MULTILINE) + func_m = re.search(r"^\s*(?:factory\s+|static\s+|async\s+|external\s+|abstract\s+)?(?:\([^)]+\)|[a-zA-Z0-9_<>,.?]+)(?:\s+[a-zA-Z0-9_<>,.?]+){0,3}\s+(\w+)\s*\(", intervening_text, re.MULTILINE) + + target_nid = None + target_name = None + target_type = None + + if class_m and func_m: + if class_m.start() < func_m.start(): + target_name = class_m.group(1) + target_type = "class" + target_nid = _make_id(stem, target_name) + else: + target_name = func_m.group(1) + target_type = "function" + target_nid = _make_id(stem, target_name) + elif class_m: + target_name = class_m.group(1) + target_type = "class" + target_nid = _make_id(stem, target_name) + elif func_m: + target_name = func_m.group(1) + target_type = "function" + target_nid = _make_id(stem, target_name) + + if target_nid and target_name: + actual_intervening = intervening_text[:min(class_m.start() if class_m else 300, func_m.start() if func_m else 300)] + if ";" not in actual_intervening and "}" not in actual_intervening and "{" not in actual_intervening: + annotation_nid = _make_id("annotation", annotation_name.lower()) + add_node(annotation_nid, f"@{annotation_name}", ftype="concept", source_file=None) + add_edge(target_nid, annotation_nid, "configures") + + # Riverpod specific provider generation mapping (supports camelCase class and functional providers) + if annotation_name.lower() == "riverpod": + if target_type == "class": + provider_name = target_name[0].lower() + target_name[1:] + "Provider" if len(target_name) > 1 else target_name.lower() + "Provider" + else: + provider_name = target_name + "Provider" + provider_nid = _make_id(provider_name) + add_node(provider_nid, provider_name, ftype="concept", source_file=str(path)) + add_edge(target_nid, provider_nid, "defines", context="riverpod_provider") + + # 2.5 Typedefs (Type Aliases) + typedef_pattern = r"^\s*typedef\s+(\w+)\s*(?:<[^>]+>)?\s*=\s*([a-zA-Z0-9_<>,.?\s]+);" + for m in re.finditer(typedef_pattern, src_clean, re.MULTILINE): + typedef_name = m.group(1) + target_type = m.group(2).split("<")[0].split(".")[-1].strip() + if target_type not in {"String", "int", "double", "bool", "num", "dynamic", "Object", "List", "Map", "Set", "void", "Function"}: + typedef_nid = _make_id(stem, typedef_name) + add_node(typedef_nid, typedef_name) + add_edge(file_nid, typedef_nid, "defines") + target_nid = _make_id(target_type) + add_node(target_nid, target_type, source_file=None) + add_edge(typedef_nid, target_nid, "references", context="typedef") + + # 3. Extensions (extension MyExt on MyClass) + ext_pattern = r"^\s{0,4}extension\s+(\w+)?(?:<[^>]+>)?\s+on\s+(\w+)" + for m in re.finditer(ext_pattern, src_clean, re.MULTILINE): + ext_name = m.group(1) or f"{stem}_anonymous_extension" + target_class = m.group(2) + + ext_nid = _make_id(stem, ext_name) + label = m.group(1) or f"Extension on {target_class}" + add_node(ext_nid, label) + add_edge(file_nid, ext_nid, "defines") + + target_nid = _make_id(target_class) + add_node(target_nid, target_class, source_file=None) + add_edge(ext_nid, target_nid, "extends") + + # 4. Top-level and class-level variable declarations (generic variables, records, late, and destructuring) + # Restrict indentation to 0-2 spaces to avoid matching local variables inside functions or switch expressions + var_pattern = r"^\s{0,2}(?:late\s+)?(?:(?:final|const|var)\s+)?(?:\([^)]+\)\s+|([a-zA-Z0-9_<>,.?]+(?:\s+[a-zA-Z0-9_<>,.?]+){0,3})\s+)?(?:(\w+)|(?:\w+\s*)?\(([^)]+)\))\s*(?:=|$|;)" + for m in re.finditer(var_pattern, src_clean, re.MULTILINE): + var_type = m.group(1) + single_name = m.group(2) + destructured_names = m.group(3) + + if not re.match(r"^\s*(?:late|final|const|var)\b", m.group(0)) and not var_type: + continue + + if single_name: + if single_name not in {"if", "for", "while", "switch", "catch", "return"}: + var_nid = _make_id(stem, single_name) + add_node(var_nid, single_name) + add_edge(file_nid, var_nid, "defines") + + if var_type and var_type not in {"String", "int", "double", "bool", "num", "dynamic", "Object", "List", "Map", "Set", "void"}: + clean_type = var_type.split("<")[0].split(".")[-1].strip() + type_nid = _make_id(clean_type) + add_node(type_nid, clean_type, source_file=None) + add_edge(file_nid, type_nid, "references", context="variable_type") + elif destructured_names: + for name in [n.strip() for n in destructured_names.split(",") if n.strip()]: + if ":" in name: + name = name.split(":")[-1].strip() + if re.match(r"^[a-zA-Z_]\w*$", name) and not re.match(r"^[A-Z]", name): + if name not in {"if", "for", "while", "switch", "catch", "return"}: + var_nid = _make_id(stem, name) + add_node(var_nid, name) + add_edge(file_nid, var_nid, "defines") + + # 5. Top-level and member functions/methods (supports typed/generic/record return types and Riverpod/Bloc references) + # Restrict indentation to 0-2 spaces to avoid matching nested local functions or methods inside multiline switch statements + method_pattern = r"^\s{0,2}(?:factory\s+|static\s+|async\s+|external\s+|abstract\s+)?(?:\([^)]+\)|[a-zA-Z0-9_<>,.?]+)(?:\s+[a-zA-Z0-9_<>,.?]+){0,3}\s+(\w+(?:\.\w+)?)\s*\(" + for m in re.finditer(method_pattern, src_clean, re.MULTILINE): + raw_name = m.group(1) + name = raw_name.split(".")[-1] + if name in {"if", "for", "while", "switch", "catch", "return", "void", "dynamic", "final", "const", "get", "set"}: + continue + if re.match(r"^[A-Z]", name): + continue + nid = _make_id(stem, name) + add_node(nid, name) + add_edge(file_nid, nid, "defines") + + # Get function body using matching brace to extract Riverpod reference patterns + start_idx = m.start() + brace_pos = src_clean.find("{", start_idx) + semi_pos = src_clean.find(";", start_idx) + arrow_pos = src_clean.find("=>", start_idx) + + has_body = brace_pos != -1 + if has_body and semi_pos != -1 and semi_pos < brace_pos: + has_body = False + if has_body and arrow_pos != -1 and arrow_pos < brace_pos: + has_body = False + + if has_body: + end_pos = _find_matching_brace(src_clean, start_idx) + func_body = src_clean[brace_pos:end_pos] + + # Extract Riverpod provider references: ref.watch(provider) + for rm in re.finditer(r"\bref\.(?:watch|read|listen)\s*\(\s*(\w+)\b", func_body): + provider_name = rm.group(1) + provider_nid = _make_id(provider_name) + add_node(provider_nid, provider_name, source_file=None) + add_edge(nid, provider_nid, "references", context="riverpod_reference") + + # Extract Bloc event additions: widget.add(MyEvent()) or bloc.add(MyEvent()) + for am in re.finditer(r"\b(?:\w*[Bb]loc\w*|context\.read<\w+>\(\))\.add\(\s*(?:const\s+)?([A-Z]\w*)\b", func_body): + event_name = am.group(1) + if event_name not in {"String", "List", "Map", "Set", "Future", "Stream", "Object"}: + event_nid = _make_id(event_name) + add_node(event_nid, event_name, source_file=None) + add_edge(nid, event_nid, "calls", context="bloc_add_event") + + # context.read() or BlocProvider.of(context) + for lm in re.finditer(r"\b(?:read|watch|select|of)\s*<([a-zA-Z0-9_]+)>", func_body): + bloc_name = lm.group(1) + if bloc_name not in {"String", "int", "double", "bool", "num", "dynamic", "Object", "void"}: + bloc_nid = _make_id(bloc_name) + add_node(bloc_nid, bloc_name, source_file=None) + add_edge(nid, bloc_nid, "references", context="bloc_lookup") + + # Universal Navigation Patters (GoRouter, AutoRoute, Navigator) + for nm in re.finditer(r"\b(?:go|push|goNamed|pushNamed|replace|replaceNamed)\s*\(\s*(?:context\s*,\s*)?['\"]([a-zA-Z0-9_/?=&%-]+)['\"]", func_body): + route_path = nm.group(1) + route_nid = _make_id("route", route_path.replace("/", "_").replace("?", "_").replace("=", "_").replace("&", "_")) + add_node(route_nid, f"Route {route_path}", ftype="concept", source_file=None) + add_edge(nid, route_nid, "navigates", context="route_path") + + for cm in re.finditer(r"\b(?:go|push|goNamed|pushNamed|replace|replaceNamed)\s*\(\s*(?:context\s*,\s*)?([A-Z][a-zA-Z0-9_]*\.[a-zA-Z0-9_]+)", func_body): + route_const = cm.group(1) + route_nid = _make_id("route", route_const.replace(".", "_")) + add_node(route_nid, route_const, ftype="concept", source_file=None) + add_edge(nid, route_nid, "navigates", context="route_const") + + for om in re.finditer(r"\b(?:push|replace)\s*\(\s*(?:context\s*,\s*)?.*?\b([A-Z]\w*(?:Route|Screen|Page))\b", func_body): + route_class = om.group(1) + route_nid = _make_id(route_class) + add_node(route_nid, route_class, source_file=None) + add_edge(nid, route_nid, "navigates", context="route_object") + + # 6. Imports and Exports + for m in re.finditer(r"""^\s*import\s+['"]([^'"]+)['"]""", src_clean, re.MULTILINE): + pkg = m.group(1) + tgt_nid = _make_id(pkg) + add_node(tgt_nid, pkg, source_file=None) + add_edge(file_nid, tgt_nid, "imports") + + for m in re.finditer(r"""^\s*export\s+['"]([^'"]+)['"]""", src_clean, re.MULTILINE): + pkg = m.group(1) + tgt_nid = _make_id(pkg) + add_node(tgt_nid, pkg, source_file=None) + add_edge(file_nid, tgt_nid, "exports") + + # 7. Generic Invocations / Type Lookups (Universal Dependency Lookup) + # Matches any method call with type parameters: methodName() or object.methodName() + # Automatically extracts GetIt, Injectable, Riverpod, Provider, BlocProvider, and InheritedWidget type lookups! + generic_call_pattern = r"\b\w+<([a-zA-Z0-9_.]+(?:<[a-zA-Z0-9_.,\s<>]+>)?)\s*>\s*\(" + type_blacklist = {"String", "int", "double", "bool", "num", "dynamic", "Object", "List", "Map", "Set", "Future", "Stream", "void"} + for m in re.finditer(generic_call_pattern, src_clean): + type_name = m.group(1).split(".")[-1].strip() + clean_name = type_name.split("<")[0].strip() + if clean_name not in type_blacklist: + target_nid = _make_id(clean_name) + add_node(target_nid, clean_name, source_file=None) + add_edge(file_nid, target_nid, "references", context="type_lookup") + + return {"nodes": nodes, "edges": edges} diff --git a/graphify/extractors/dm.py b/graphify/extractors/dm.py new file mode 100644 index 000000000..94d386271 --- /dev/null +++ b/graphify/extractors/dm.py @@ -0,0 +1,494 @@ +"""Dm extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + +import re + +from pathlib import Path +from typing import Any +from graphify.extractors.base import _file_stem, _make_id, _read_text + + +def extract_dm(path: Path) -> dict: + """Extract types, procs, includes, and calls from a .dm/.dme file.""" + try: + import tree_sitter_dm as tsdm + from tree_sitter import Language, Parser + except ImportError: + return {"nodes": [], "edges": [], "error": "tree-sitter-dm not installed"} + try: + language = Language(tsdm.language()) + parser = Parser(language) + source = path.read_bytes() + tree = parser.parse(source) + root = tree.root_node + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + stem = _file_stem(path) + str_path = str(path) + nodes: list[dict] = [] + edges: list[dict] = [] + seen_ids: set[str] = set() + function_bodies: list[tuple[str, Any, "str | None"]] = [] + + def add_node(nid: str, label: str, line: int) -> None: + if nid and nid not in seen_ids: + seen_ids.add(nid) + nodes.append({"id": nid, "label": label, "file_type": "code", + "source_file": str_path, "source_location": f"L{line}"}) + + def add_edge(src: str, tgt: str, relation: str, line: int, + confidence: str = "EXTRACTED", weight: float = 1.0, + context: str | None = None) -> None: + if not src or not tgt or src == tgt: + return + edge: dict = {"source": src, "target": tgt, "relation": relation, + "confidence": confidence, "source_file": str_path, + "source_location": f"L{line}", "weight": weight} + if context: + edge["context"] = context + edges.append(edge) + + file_nid = _make_id(str(path)) + add_node(file_nid, path.name, 1) + + def _type_path_text(node) -> str: + return _read_text(node, source).strip() + + def _ensure_type(path_text: str, line: int) -> str: + nid = _make_id(stem, path_text) + add_node(nid, path_text, line) + return nid + + def _find_child(node, type_name: str): + for c in node.children: + if c.type == type_name: + return c + return None + + def _read_include_path(file_node) -> str: + if file_node is None: + return "" + if file_node.type == "string_literal": + parts = [] + for c in file_node.children: + if c.type == "string_content": + parts.append(_read_text(c, source)) + return "".join(parts) + return _read_text(file_node, source).strip("'\"") + + def walk(node, parent_type_path: "str | None" = None, + parent_type_nid: "str | None" = None) -> None: + t = node.type + line = node.start_point[0] + 1 + + if t == "preproc_include": + file_node = node.child_by_field_name("file") + raw = _read_include_path(file_node) + if raw: + norm = raw.replace("\\", "/").lstrip("./") + resolved = (path.parent / norm).resolve() + edge: dict = { + "source": file_nid, + "target": _make_id(str(resolved)) if resolved.exists() else _make_id(norm), + "relation": "imports_from" if resolved.exists() else "imports", + "context": "import", + "confidence": "EXTRACTED", + "source_file": str_path, + "source_location": f"L{line}", + "weight": 1.0, + } + if not resolved.exists(): + edge["external"] = True + edges.append(edge) + return + + if t == "type_definition": + tp_node = _find_child(node, "type_path") + if tp_node is None: + return + type_path_str = _type_path_text(tp_node) + type_nid = _ensure_type(type_path_str, line) + add_edge(file_nid, type_nid, "contains", line) + body = _find_child(node, "type_body") + if body is not None: + for c in body.children: + walk(c, parent_type_path=type_path_str, parent_type_nid=type_nid) + return + + if t in ("type_body_intended", "type_body_braced"): + for c in node.children: + walk(c, parent_type_path, parent_type_nid) + return + + if t in ("type_proc_definition", "type_proc_override"): + if parent_type_nid is None or parent_type_path is None: + return + name_node = node.child_by_field_name("name") + if name_node is None: + return + proc_name = _read_text(name_node, source) + proc_nid = _make_id(stem, parent_type_path, proc_name) + add_node(proc_nid, f"{parent_type_path}/{proc_name}()", line) + add_edge(parent_type_nid, proc_nid, "method", line) + block = _find_child(node, "block") + if block is not None: + function_bodies.append((proc_nid, block, parent_type_path)) + return + + if t in ("proc_definition", "proc_override"): + tp_node = _find_child(node, "type_path") + owner_path: "str | None" = None + owner_nid: "str | None" = None + if tp_node is not None: + owner_path = _type_path_text(tp_node) + owner_nid = _ensure_type(owner_path, line) + add_edge(file_nid, owner_nid, "contains", line) + name_node = node.child_by_field_name("name") + if name_node is None: + return + proc_name = _read_text(name_node, source) + if owner_path and owner_nid: + proc_nid = _make_id(stem, owner_path, proc_name) + add_node(proc_nid, f"{owner_path}/{proc_name}()", line) + add_edge(owner_nid, proc_nid, "method", line) + else: + proc_nid = _make_id(stem, proc_name) + add_node(proc_nid, f"{proc_name}()", line) + add_edge(file_nid, proc_nid, "contains", line) + block = _find_child(node, "block") + if block is not None: + function_bodies.append((proc_nid, block, owner_path)) + return + + if t in ("operator_override", "type_operator_override"): + return + + for child in node.children: + walk(child, parent_type_path, parent_type_nid) + + walk(root) + + label_to_nids: dict[str, list[str]] = {} + path_to_nids: dict[str, list[str]] = {} + for n in nodes: + label = n["label"].strip("()") + last = label.rsplit("/", 1)[-1] if "/" in label else label + if last: + label_to_nids.setdefault(last.lower(), []).append(n["id"]) + if label.startswith("/"): + path_to_nids.setdefault(label.lower(), []).append(n["id"]) + + seen_call_pairs: set[tuple[str, str]] = set() + raw_calls: list[dict] = [] + + def _emit_call(caller_nid: str, callee: str, line: int, is_member: bool) -> None: + candidates = label_to_nids.get(callee.lower(), []) + tgt_nid = candidates[0] if len(candidates) == 1 else None + if tgt_nid and tgt_nid != caller_nid: + pair = (caller_nid, tgt_nid) + if pair in seen_call_pairs: + return + seen_call_pairs.add(pair) + edges.append({ + "source": caller_nid, "target": tgt_nid, "relation": "calls", + "context": "call", "confidence": "EXTRACTED", + "source_file": str_path, "source_location": f"L{line}", "weight": 1.0, + }) + else: + raw_calls.append({ + "caller_nid": caller_nid, "callee": callee, + "is_member_call": is_member, "source_file": str_path, + "source_location": f"L{line}", + }) + + def walk_calls(body_node, caller_nid: str) -> None: + if body_node is None: + return + t = body_node.type + if t in ("proc_definition", "proc_override", "type_proc_definition", + "type_proc_override", "type_definition"): + return + if t == "call_expression": + name_node = body_node.child_by_field_name("name") + if name_node is not None: + callee = _read_text(name_node, source) + if callee and callee != "..": + _emit_call(caller_nid, callee, body_node.start_point[0] + 1, + is_member=False) + elif t == "field_proc_expression": + proc_field = body_node.child_by_field_name("proc") + if proc_field is not None: + callee = _read_text(proc_field, source) + if callee: + _emit_call(caller_nid, callee, body_node.start_point[0] + 1, + is_member=True) + elif t == "new_expression": + tp_node = _find_child(body_node, "type_path") + if tp_node is not None: + target_text = _type_path_text(tp_node) + candidates = path_to_nids.get(target_text.lower(), []) + tgt_nid = candidates[0] if len(candidates) == 1 else None + if tgt_nid and tgt_nid != caller_nid: + pair = (caller_nid, tgt_nid) + if pair not in seen_call_pairs: + seen_call_pairs.add(pair) + edges.append({ + "source": caller_nid, "target": tgt_nid, + "relation": "instantiates", "context": "call", + "confidence": "EXTRACTED", "source_file": str_path, + "source_location": f"L{body_node.start_point[0] + 1}", + "weight": 1.0, + }) + for child in body_node.children: + walk_calls(child, caller_nid) + + for proc_nid, block, _owner_path in function_bodies: + walk_calls(block, proc_nid) + + return {"nodes": nodes, "edges": edges, "raw_calls": raw_calls} + +def _read_dmi_description(data: bytes) -> str: + """Pull the BYOND metadata text out of a .dmi PNG, or empty string on failure.""" + import struct + import zlib as _zlib + if not data.startswith(b"\x89PNG\r\n\x1a\n"): + return "" + i = 8 + while i + 8 <= len(data): + length = struct.unpack(">I", data[i:i + 4])[0] + chunk_type = data[i + 4:i + 8] + payload = data[i + 8:i + 8 + length] + if chunk_type in (b"tEXt", b"zTXt"): + try: + null = payload.index(b"\x00") + except ValueError: + return "" + keyword = payload[:null] + if keyword == b"Description": + if chunk_type == b"zTXt": + return _zlib.decompressobj().decompress(payload[null + 2:], max_length=1024 * 1024).decode("utf-8", errors="replace") + return payload[null + 1:].decode("utf-8", errors="replace") + i += 8 + length + 4 + return "" + +def extract_dmi(path: Path) -> dict: + """Extract icon state names from a .dmi (BYOND PNG icon sheet).""" + try: + data = path.read_bytes() + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + str_path = str(path) + stem = _file_stem(path) + file_nid = _make_id(str(path)) + nodes: list[dict] = [{"id": file_nid, "label": path.name, "file_type": "code", + "source_file": str_path, "source_location": "L1"}] + edges: list[dict] = [] + seen: set[str] = {file_nid} + + description = _read_dmi_description(data) + if not description: + return {"nodes": nodes, "edges": edges} + + line_no = 0 + for raw_line in description.splitlines(): + line_no += 1 + stripped = raw_line.strip() + if not stripped.startswith("state ="): + continue + value = stripped.split("=", 1)[1].strip() + if value.startswith('"') and value.endswith('"') and len(value) >= 2: + state_name = value[1:-1] + else: + state_name = value + if not state_name: + continue + nid = _make_id(stem, "state", state_name) + if nid in seen: + continue + seen.add(nid) + nodes.append({"id": nid, "label": f'"{state_name}"', "file_type": "code", + "source_file": str_path, "source_location": f"L{line_no}"}) + edges.append({"source": file_nid, "target": nid, "relation": "contains", + "confidence": "EXTRACTED", "source_file": str_path, + "source_location": f"L{line_no}", "weight": 1.0}) + + return {"nodes": nodes, "edges": edges} + +_DMM_GRID_RE = re.compile(r"^\(\s*\d+\s*,\s*\d+\s*,\s*\d+\s*\)\s*=", re.MULTILINE) + +def _split_dmm_tile(body: str) -> list[str]: + out: list[str] = [] + buf: list[str] = [] + depth = 0 + in_string = False + escape = False + for ch in body: + if escape: + buf.append(ch) + escape = False + continue + if in_string: + buf.append(ch) + if ch == "\\": + escape = True + elif ch == '"': + in_string = False + continue + if ch == '"': + in_string = True + buf.append(ch) + elif ch in "({[": + depth += 1 + buf.append(ch) + elif ch in ")}]": + depth -= 1 + buf.append(ch) + elif ch == "," and depth == 0: + out.append("".join(buf).strip()) + buf = [] + else: + buf.append(ch) + tail = "".join(buf).strip() + if tail: + out.append(tail) + return out + +def _dmm_type_path(entry: str) -> str: + brace = entry.find("{") + if brace != -1: + entry = entry[:brace] + return entry.strip() + +def extract_dmm(path: Path) -> dict: + """Extract type-path references from a .dmm map file's tile dictionary.""" + try: + if path.stat().st_size > 50 * 1024 * 1024: + return {"nodes": [], "edges": [], "error": "file too large (>50 MB)"} + text = path.read_text(encoding="utf-8", errors="replace") + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + str_path = str(path) + file_nid = _make_id(str(path)) + nodes: list[dict] = [{"id": file_nid, "label": path.name, "file_type": "code", + "source_file": str_path, "source_location": "L1"}] + edges: list[dict] = [] + + grid_match = _DMM_GRID_RE.search(text) + dict_text = text[:grid_match.start()] if grid_match else text + + seen_targets: set[str] = set() + buf: list[str] = [] + open_line = 0 + depth = 0 + in_string = False + escape = False + for line_idx, line in enumerate(dict_text.splitlines(), start=1): + for ch in line: + if escape: + escape = False + elif in_string: + if ch == "\\": + escape = True + elif ch == '"': + in_string = False + elif ch == '"': + in_string = True + elif ch == "(": + if depth == 0: + open_line = line_idx + depth += 1 + elif ch == ")": + depth -= 1 + buf.append(ch) + buf.append("\n") + if depth == 0 and buf: + chunk = "".join(buf) + buf = [] + lp = chunk.find("(") + rp = chunk.rfind(")") + if lp == -1 or rp == -1 or rp <= lp: + continue + inner = chunk[lp + 1:rp] + for entry in _split_dmm_tile(inner): + tpath = _dmm_type_path(entry) + if not tpath.startswith("/"): + continue + tgt = _make_id(tpath) + if tgt in seen_targets: + continue + seen_targets.add(tgt) + edges.append({"source": file_nid, "target": tgt, "relation": "uses", + "context": "map", "confidence": "EXTRACTED", + "source_file": str_path, + "source_location": f"L{open_line}", "weight": 1.0}) + + return {"nodes": nodes, "edges": edges} + +_DMF_WINDOW_RE = re.compile(r'^\s*window\s+"([^"]+)"\s*$') + +_DMF_ELEM_RE = re.compile(r'^\s*elem\s+"([^"]+)"\s*$') + +_DMF_TYPE_RE = re.compile(r'^\s*type\s*=\s*(\S+)\s*$') + +def extract_dmf(path: Path) -> dict: + """Extract windows and controls from a .dmf interface file.""" + try: + text = path.read_text(encoding="utf-8", errors="replace") + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + str_path = str(path) + stem = _file_stem(path) + file_nid = _make_id(str(path)) + nodes: list[dict] = [{"id": file_nid, "label": path.name, "file_type": "code", + "source_file": str_path, "source_location": "L1"}] + edges: list[dict] = [] + seen: set[str] = {file_nid} + + current_window_nid: str | None = None + current_elem_nid: str | None = None + current_elem_name: str | None = None + + for line_idx, line in enumerate(text.splitlines(), start=1): + m = _DMF_WINDOW_RE.match(line) + if m: + name = m.group(1) + nid = _make_id(stem, "window", name) + if nid not in seen: + seen.add(nid) + nodes.append({"id": nid, "label": f'window "{name}"', "file_type": "code", + "source_file": str_path, "source_location": f"L{line_idx}"}) + edges.append({"source": file_nid, "target": nid, "relation": "contains", + "confidence": "EXTRACTED", "source_file": str_path, + "source_location": f"L{line_idx}", "weight": 1.0}) + current_window_nid = nid + current_elem_nid = None + current_elem_name = None + continue + m = _DMF_ELEM_RE.match(line) + if m and current_window_nid is not None: + name = m.group(1) + nid = _make_id(stem, "elem", current_window_nid, name) + if nid not in seen: + seen.add(nid) + nodes.append({"id": nid, "label": f'elem "{name}"', "file_type": "code", + "source_file": str_path, "source_location": f"L{line_idx}"}) + edges.append({"source": current_window_nid, "target": nid, + "relation": "contains", "confidence": "EXTRACTED", + "source_file": str_path, "source_location": f"L{line_idx}", + "weight": 1.0}) + current_elem_nid = nid + current_elem_name = name + continue + m = _DMF_TYPE_RE.match(line) + if m and current_elem_nid is not None and current_elem_name is not None: + ctype = m.group(1) + for n in nodes: + if n["id"] == current_elem_nid and " [" not in n["label"]: + n["label"] = f'elem "{current_elem_name}" [{ctype}]' + break + + return {"nodes": nodes, "edges": edges} diff --git a/graphify/extractors/fortran.py b/graphify/extractors/fortran.py new file mode 100644 index 000000000..58ffedfc8 --- /dev/null +++ b/graphify/extractors/fortran.py @@ -0,0 +1,309 @@ +"""Fortran extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + + +from pathlib import Path +from graphify.extractors.base import _file_stem, _make_id, _read_text + + +_FORTRAN_CPP_EXTS = {".F", ".F90", ".F95", ".F03", ".F08"} + +def _cpp_preprocess(path: Path) -> bytes: + """Run cpp -w -P on a capital-F Fortran file and return preprocessed bytes. + + Falls back to raw file bytes if cpp is not available. Capital-F extensions + conventionally require C preprocessor expansion (#ifdef MPI, #define REAL8, etc.) + before parsing. + + Security (F-007): we pass `-nostdinc` and `-I /dev/null` so a malicious + source file containing `#include "/home/victim/.ssh/id_rsa"` (or any other + include directive) cannot inline arbitrary host files into the output that + we then ship to an LLM. Without these flags `cpp` happily resolves any + relative or absolute include path it can read, which is a corpus-side + file-exfiltration vector. + """ + import shutil + import subprocess + if not shutil.which("cpp"): + return path.read_bytes() + try: + # Pass an absolute path so a corpus file named like "-I/etc/x.F90" cannot + # be parsed by cpp as an option (cpp does not accept a "--" end-of-options + # terminator). An absolute path always begins with "/". + result = subprocess.run( + ["cpp", "-w", "-P", "-nostdinc", "-I", "/dev/null", str(path.resolve())], + capture_output=True, + timeout=30, + ) + if result.returncode == 0 and result.stdout: + return result.stdout + except Exception: + pass + return path.read_bytes() + +def extract_fortran(path: Path) -> dict: + """Extract programs, modules, subroutines, functions, use statements, and calls from Fortran files. + + Capital-F extensions (.F, .F90, etc.) are run through the C preprocessor before + parsing so #ifdef/#define macros are resolved. + """ + try: + import tree_sitter_fortran as tsfortran + from tree_sitter import Language, Parser + except ImportError: + return {"nodes": [], "edges": [], "error": "tree-sitter-fortran not installed"} + + try: + language = Language(tsfortran.language()) + parser = Parser(language) + source = _cpp_preprocess(path) if path.suffix in _FORTRAN_CPP_EXTS else path.read_bytes() + tree = parser.parse(source) + root = tree.root_node + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + stem = _file_stem(path) + str_path = str(path) + nodes: list[dict] = [] + edges: list[dict] = [] + seen_ids: set[str] = set() + scope_bodies: list[tuple[str, object]] = [] + + def add_node(nid: str, label: str, line: int) -> None: + if nid not in seen_ids: + seen_ids.add(nid) + nodes.append({ + "id": nid, + "label": label, + "file_type": "code", + "source_file": str_path, + "source_location": f"L{line}", + }) + + def add_edge(src: str, tgt: str, relation: str, line: int, + confidence: str = "EXTRACTED", weight: float = 1.0, + context: str | None = None) -> None: + edge = { + "source": src, + "target": tgt, + "relation": relation, + "confidence": confidence, + "source_file": str_path, + "source_location": f"L{line}", + "weight": weight, + } + if context: + edge["context"] = context + edges.append(edge) + + file_nid = _make_id(str(path)) + add_node(file_nid, path.name, 1) + + def _fortran_name(stmt_node) -> str | None: + """Extract name from a *_statement node. Fortran is case-insensitive; lowercase.""" + for child in stmt_node.children: + if child.type in ("name", "identifier"): + return _read_text(child, source).lower() + return None + + def ensure_named_node(name: str, line: int) -> str: + nid = _make_id(stem, name) + if nid in seen_ids: + return nid + nid = _make_id(name) + if nid not in seen_ids: + # The name isn't defined in this file, so this is a cross-file reference + # (e.g. a `Thing` type annotation imported from another module). Emit a + # SOURCELESS stub — like the inheritance-base path below — so the + # corpus-level rewire can collapse it onto the real definition. A sourced + # stub here makes _disambiguate_colliding_node_ids bake the referencing + # file's path (with extension) into the id and blocks the rewire, which is + # the phantom-duplicate-node bug (#1402). + seen_ids.add(nid) + nodes.append({ + "id": nid, + "label": name, + "file_type": "code", + "source_file": "", + "source_location": "", + "origin_file": str_path, + }) + return nid + + def emit_signature_refs(scope_node, fn_nid: str, is_function: bool) -> None: + """Emit references[parameter_type] / references[return_type] edges for + a subroutine/function based on its variable_declaration siblings.""" + stmt_type = "function_statement" if is_function else "subroutine_statement" + stmt = next((c for c in scope_node.children if c.type == stmt_type), None) + if stmt is None: + return + param_names: set[str] = set() + params_node = next((c for c in stmt.children if c.type == "parameters"), None) + if params_node is not None: + for c in params_node.children: + if c.type == "identifier": + param_names.add(_read_text(c, source).lower()) + result_name: str | None = None + if is_function: + result_node = next((c for c in stmt.children if c.type == "function_result"), None) + if result_node is not None: + res_id = next((c for c in result_node.children if c.type == "identifier"), None) + if res_id is not None: + result_name = _read_text(res_id, source).lower() + else: + # implicit result variable: same name as the function + result_name = _fortran_name(stmt) + for child in scope_node.children: + if child.type != "variable_declaration": + continue + derived = next((c for c in child.children if c.type == "derived_type"), None) + if derived is None: + continue + type_name_node = next((c for c in derived.children if c.type == "type_name"), None) + if type_name_node is None: + continue + type_name = _read_text(type_name_node, source).lower() + for var in child.children: + if var.type != "identifier": + continue + var_name = _read_text(var, source).lower() + var_line = var.start_point[0] + 1 + if var_name in param_names: + tgt = ensure_named_node(type_name, var_line) + if tgt != fn_nid: + add_edge(fn_nid, tgt, "references", var_line, context="parameter_type") + elif is_function and var_name == result_name: + tgt = ensure_named_node(type_name, var_line) + if tgt != fn_nid: + add_edge(fn_nid, tgt, "references", var_line, context="return_type") + + def walk_calls(node, scope_nid: str) -> None: + if node is None: + return + t = node.type + if t in ("subroutine", "function", "module", "program", "internal_procedures"): + return + # call FOO(args) — tree-sitter-fortran uses subroutine_call + if t == "subroutine_call": + name_node = next((c for c in node.children if c.type == "identifier"), None) + if name_node: + callee = _read_text(name_node, source).lower() + target_nid = _make_id(stem, callee) + add_edge(scope_nid, target_nid, "calls", node.start_point[0] + 1, + confidence="EXTRACTED", context="call") + # x = compute(args) — function invocations are `call_expression`, which + # shares Fortran's `name(...)` syntax with array indexing. Only emit a + # call edge when the callee resolves to a procedure defined in this file + # (an array variable produces no matching node), so array accesses can't + # fabricate spurious `calls` edges. + elif t == "call_expression": + name_node = next((c for c in node.children if c.type == "identifier"), None) + if name_node: + callee = _read_text(name_node, source).lower() + target_nid = _make_id(stem, callee) + if target_nid in seen_ids and target_nid != scope_nid: + add_edge(scope_nid, target_nid, "calls", node.start_point[0] + 1, + confidence="EXTRACTED", context="call") + for child in node.children: + walk_calls(child, scope_nid) + + def walk(node, scope_nid: str) -> None: + t = node.type + + if t == "program": + stmt = next((c for c in node.children if c.type == "program_statement"), None) + name = _fortran_name(stmt) if stmt else None + if name: + nid = _make_id(stem, name) + line = node.start_point[0] + 1 + add_node(nid, name, line) + add_edge(file_nid, nid, "defines", line) + scope_bodies.append((nid, node)) + for child in node.children: + walk(child, nid) + return + + if t == "module": + stmt = next((c for c in node.children if c.type == "module_statement"), None) + name = _fortran_name(stmt) if stmt else None + if name: + nid = _make_id(stem, name) + line = node.start_point[0] + 1 + add_node(nid, name, line) + add_edge(file_nid, nid, "defines", line) + for child in node.children: + walk(child, nid) + return + + # subroutines/functions inside a module live under internal_procedures + if t == "internal_procedures": + for child in node.children: + walk(child, scope_nid) + return + + if t == "derived_type_definition": + stmt = next((c for c in node.children if c.type == "derived_type_statement"), None) + if stmt is not None: + name_node = next((c for c in stmt.children if c.type == "type_name"), None) + if name_node is not None: + type_name = _read_text(name_node, source).lower() + type_nid = _make_id(stem, type_name) + line = node.start_point[0] + 1 + add_node(type_nid, type_name, line) + add_edge(scope_nid, type_nid, "defines", line) + return + + if t == "subroutine": + stmt = next((c for c in node.children if c.type == "subroutine_statement"), None) + name = _fortran_name(stmt) if stmt else None + if name: + nid = _make_id(stem, name) + line = node.start_point[0] + 1 + add_node(nid, f"{name}()", line) + add_edge(scope_nid, nid, "defines", line) + scope_bodies.append((nid, node)) + emit_signature_refs(node, nid, is_function=False) + for child in node.children: + walk(child, nid) + return + + if t == "function": + stmt = next((c for c in node.children if c.type == "function_statement"), None) + name = _fortran_name(stmt) if stmt else None + if name: + nid = _make_id(stem, name) + line = node.start_point[0] + 1 + add_node(nid, f"{name}()", line) + add_edge(scope_nid, nid, "defines", line) + scope_bodies.append((nid, node)) + emit_signature_refs(node, nid, is_function=True) + for child in node.children: + walk(child, nid) + return + + if t == "use_statement": + line = node.start_point[0] + 1 + # tree-sitter-fortran uses module_name node for the used module + name_node = next((c for c in node.children if c.type in ("module_name", "name", "identifier")), None) + if name_node: + mod_name = _read_text(name_node, source).lower() + imp_nid = _make_id(mod_name) + add_node(imp_nid, mod_name, line) + add_edge(scope_nid, imp_nid, "imports", line, context="use") + return + + for child in node.children: + walk(child, scope_nid) + + walk(root, file_nid) + + _stmt_headers = { + "subroutine_statement", "function_statement", + "program_statement", "module_statement", + } + for scope_nid, body_node in scope_bodies: + for child in body_node.children: + if child.type not in _stmt_headers: + walk_calls(child, scope_nid) + + return {"nodes": nodes, "edges": edges} diff --git a/graphify/extractors/go.py b/graphify/extractors/go.py new file mode 100644 index 000000000..a0db9a693 --- /dev/null +++ b/graphify/extractors/go.py @@ -0,0 +1,396 @@ +"""Go extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + + +from pathlib import Path +from graphify.extractors.base import _LANGUAGE_BUILTIN_GLOBALS, _file_stem, _make_id, _read_text + + +_GO_PREDECLARED_TYPES = frozenset({ + "bool", "byte", "complex64", "complex128", "error", "float32", "float64", + "int", "int8", "int16", "int32", "int64", "rune", "string", + "uint", "uint8", "uint16", "uint32", "uint64", "uintptr", "any", "comparable", +}) + +def _go_collect_type_refs(node, source: bytes, generic: bool, out: list[tuple[str, str]]) -> None: + """Walk a Go type expression; append (name, role) tuples.""" + if node is None: + return + t = node.type + if t == "type_identifier": + text = _read_text(node, source) + if text and text not in _GO_PREDECLARED_TYPES: + out.append((text, "generic_arg" if generic else "type")) + return + if t == "qualified_type": + text = _read_text(node, source).rsplit(".", 1)[-1] + if text and text not in _GO_PREDECLARED_TYPES: + out.append((text, "generic_arg" if generic else "type")) + return + if t == "generic_type": + type_field = node.child_by_field_name("type") + if type_field is not None: + sub: list[tuple[str, str]] = [] + _go_collect_type_refs(type_field, source, generic, sub) + out.extend(sub) + for c in node.children: + if c.type == "type_arguments": + for arg in c.children: + if arg.is_named: + _go_collect_type_refs(arg, source, True, out) + return + if t in ("pointer_type", "slice_type", "array_type", "map_type", + "channel_type", "parenthesized_type"): + for c in node.children: + if c.is_named: + _go_collect_type_refs(c, source, generic, out) + return + if node.is_named: + for c in node.children: + if c.is_named: + _go_collect_type_refs(c, source, generic, out) + +def extract_go(path: Path) -> dict: + """Extract functions, methods, type declarations, and imports from a .go file.""" + try: + import tree_sitter_go as tsgo + from tree_sitter import Language, Parser + except ImportError: + return {"nodes": [], "edges": [], "error": "tree-sitter-go not installed"} + + try: + language = Language(tsgo.language()) + parser = Parser(language) + source = path.read_bytes() + tree = parser.parse(source) + root = tree.root_node + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + stem = _file_stem(path) + # Use directory name as package scope so methods on the same type across + # multiple files in a package share one canonical type node. + pkg_scope = path.parent.name or stem + str_path = str(path) + nodes: list[dict] = [] + edges: list[dict] = [] + seen_ids: set[str] = set() + function_bodies: list[tuple[str, object]] = [] + go_imported_pkgs: set[str] = set() # local names of imported packages + + def add_node(nid: str, label: str, line: int) -> None: + if nid not in seen_ids: + seen_ids.add(nid) + nodes.append({ + "id": nid, + "label": label, + "file_type": "code", + "source_file": str_path, + "source_location": f"L{line}", + }) + + def add_edge(src: str, tgt: str, relation: str, line: int, + confidence: str = "EXTRACTED", weight: float = 1.0, + context: str | None = None) -> None: + edge = { + "source": src, + "target": tgt, + "relation": relation, + "confidence": confidence, + "source_file": str_path, + "source_location": f"L{line}", + "weight": weight, + } + if context: + edge["context"] = context + edges.append(edge) + + file_nid = _make_id(str(path)) + add_node(file_nid, path.name, 1) + + def ensure_named_node(name: str, line: int) -> str: + nid = _make_id(pkg_scope, name) + if nid in seen_ids: + return nid + nid = _make_id(name) + if nid not in seen_ids: + # The name isn't declared in this file, so this is a cross-file reference + # (e.g. a type defined in another file of the package). Emit a SOURCELESS + # stub — like the inheritance-base path in the other extractors — so the + # corpus-level rewire can collapse it onto the real definition. A sourced + # stub here makes _disambiguate_colliding_node_ids bake the referencing + # file's path (with extension) into the id and blocks the rewire, which is + # the phantom-duplicate-node bug (#1402). + seen_ids.add(nid) + nodes.append({ + "id": nid, + "label": name, + "file_type": "code", + "source_file": "", + "source_location": "", + "origin_file": str_path, + }) + return nid + + def emit_go_method_refs(func_node, func_nid: str, line: int) -> None: + params = func_node.child_by_field_name("parameters") + if params is not None: + for p in params.children: + if p.type != "parameter_declaration": + continue + type_node = p.child_by_field_name("type") + refs: list[tuple[str, str]] = [] + _go_collect_type_refs(type_node, source, False, refs) + for ref_name, role in refs: + ctx = "generic_arg" if role == "generic_arg" else "parameter_type" + tgt = ensure_named_node(ref_name, line) + if tgt != func_nid: + add_edge(func_nid, tgt, "references", line, context=ctx) + result = func_node.child_by_field_name("result") + if result is not None: + if result.type == "parameter_list": + for p in result.children: + if p.type != "parameter_declaration": + continue + type_node = p.child_by_field_name("type") + if type_node is None: + for c in p.children: + if c.is_named: + type_node = c + break + refs = [] + _go_collect_type_refs(type_node, source, False, refs) + for ref_name, role in refs: + ctx = "generic_arg" if role == "generic_arg" else "return_type" + tgt = ensure_named_node(ref_name, line) + if tgt != func_nid: + add_edge(func_nid, tgt, "references", line, context=ctx) + else: + refs = [] + _go_collect_type_refs(result, source, False, refs) + for ref_name, role in refs: + ctx = "generic_arg" if role == "generic_arg" else "return_type" + tgt = ensure_named_node(ref_name, line) + if tgt != func_nid: + add_edge(func_nid, tgt, "references", line, context=ctx) + + def walk(node) -> None: + t = node.type + + if t == "function_declaration": + name_node = node.child_by_field_name("name") + if name_node: + func_name = _read_text(name_node, source) + line = node.start_point[0] + 1 + func_nid = _make_id(stem, func_name) + add_node(func_nid, f"{func_name}()", line) + add_edge(file_nid, func_nid, "contains", line) + emit_go_method_refs(node, func_nid, line) + body = node.child_by_field_name("body") + if body: + function_bodies.append((func_nid, body)) + return + + if t == "method_declaration": + receiver = node.child_by_field_name("receiver") + receiver_type: str | None = None + if receiver: + for param in receiver.children: + if param.type == "parameter_declaration": + type_node = param.child_by_field_name("type") + if type_node: + receiver_type = _read_text(type_node, source).lstrip("*").strip() + break + name_node = node.child_by_field_name("name") + if not name_node: + return + method_name = _read_text(name_node, source) + line = node.start_point[0] + 1 + + if receiver_type: + parent_nid = _make_id(pkg_scope, receiver_type) + add_node(parent_nid, receiver_type, line) + method_nid = _make_id(parent_nid, method_name) + add_node(method_nid, f".{method_name}()", line) + add_edge(parent_nid, method_nid, "method", line) + else: + method_nid = _make_id(stem, method_name) + add_node(method_nid, f"{method_name}()", line) + add_edge(file_nid, method_nid, "contains", line) + + emit_go_method_refs(node, method_nid, line) + body = node.child_by_field_name("body") + if body: + function_bodies.append((method_nid, body)) + return + + if t == "type_declaration": + for child in node.children: + if child.type != "type_spec": + continue + name_node = child.child_by_field_name("name") + if not name_node: + continue + type_name = _read_text(name_node, source) + line = child.start_point[0] + 1 + type_nid = _make_id(pkg_scope, type_name) + add_node(type_nid, type_name, line) + add_edge(file_nid, type_nid, "contains", line) + # Type body: struct fields (with embeds) or interface embedding. + type_body = None + for tc in child.children: + if tc.type in ("struct_type", "interface_type"): + type_body = tc + break + if type_body is None: + continue + if type_body.type == "struct_type": + for fdl in type_body.children: + if fdl.type != "field_declaration_list": + continue + for field in fdl.children: + if field.type != "field_declaration": + continue + has_name = any( + fc.type == "field_identifier" for fc in field.children + ) + type_node = field.child_by_field_name("type") + if type_node is None: + for fc in field.children: + if fc.is_named and fc.type != "field_identifier": + type_node = fc + break + refs: list[tuple[str, str]] = [] + _go_collect_type_refs(type_node, source, False, refs) + for ref_name, role in refs: + tgt = ensure_named_node(ref_name, field.start_point[0] + 1) + if tgt == type_nid: + continue + if not has_name and role == "type": + add_edge(type_nid, tgt, "embeds", + field.start_point[0] + 1) + else: + ctx = "generic_arg" if role == "generic_arg" else "field" + add_edge(type_nid, tgt, "references", + field.start_point[0] + 1, context=ctx) + elif type_body.type == "interface_type": + for elem in type_body.children: + if elem.type != "type_elem": + continue + refs = [] + for sub in elem.children: + if sub.is_named: + _go_collect_type_refs(sub, source, False, refs) + for ref_name, role in refs: + tgt = ensure_named_node(ref_name, elem.start_point[0] + 1) + if tgt == type_nid: + continue + if role == "type": + add_edge(type_nid, tgt, "embeds", + elem.start_point[0] + 1) + else: + add_edge(type_nid, tgt, "references", + elem.start_point[0] + 1, context="generic_arg") + return + + if t == "import_declaration": + for child in node.children: + if child.type == "import_spec_list": + for spec in child.children: + if spec.type == "import_spec": + path_node = spec.child_by_field_name("path") + if path_node: + raw = _read_text(path_node, source).strip('"') + # Prefix with go_pkg_ so stdlib names (e.g. "context") + # don't collide with local files of the same basename. + tgt_nid = _make_id("go", "pkg", raw) + add_edge(file_nid, tgt_nid, "imports_from", spec.start_point[0] + 1, context="import") + # Track local name (alias or last path segment) + alias = spec.child_by_field_name("name") + local_name = _read_text(alias, source) if alias else raw.split("/")[-1] + if local_name and local_name != "_" and local_name != ".": + go_imported_pkgs.add(local_name) + elif child.type == "import_spec": + path_node = child.child_by_field_name("path") + if path_node: + raw = _read_text(path_node, source).strip('"') + tgt_nid = _make_id("go", "pkg", raw) + add_edge(file_nid, tgt_nid, "imports_from", child.start_point[0] + 1, context="import") + alias = child.child_by_field_name("name") + local_name = _read_text(alias, source) if alias else raw.split("/")[-1] + if local_name and local_name != "_" and local_name != ".": + go_imported_pkgs.add(local_name) + return + + for child in node.children: + walk(child) + + walk(root) + + label_to_nid: dict[str, str] = {} + for n in nodes: + raw = n["label"] + normalised = raw.strip("()").lstrip(".") + label_to_nid[normalised] = n["id"] + + seen_call_pairs: set[tuple[str, str]] = set() + raw_calls: list[dict] = [] + + def walk_calls(node, caller_nid: str) -> None: + if node.type in ("function_declaration", "method_declaration"): + return + if node.type == "call_expression": + func_node = node.child_by_field_name("function") + callee_name: str | None = None + is_member_call: bool = False + if func_node: + if func_node.type == "identifier": + callee_name = _read_text(func_node, source) + elif func_node.type == "selector_expression": + field = func_node.child_by_field_name("field") + operand = func_node.child_by_field_name("operand") + receiver_name = _read_text(operand, source) if operand else "" + # Package-qualified call (e.g. fmt.Println) → allow cross-file resolution. + # Receiver method call (e.g. s.logger.Log) → skip, no import evidence. + is_member_call = receiver_name not in go_imported_pkgs + if field: + callee_name = _read_text(field, source) + if callee_name and callee_name not in _LANGUAGE_BUILTIN_GLOBALS: + tgt_nid = label_to_nid.get(callee_name) + if tgt_nid and tgt_nid != caller_nid: + pair = (caller_nid, tgt_nid) + if pair not in seen_call_pairs: + seen_call_pairs.add(pair) + line = node.start_point[0] + 1 + edges.append({ + "source": caller_nid, + "target": tgt_nid, + "relation": "calls", + "context": "call", + "confidence": "EXTRACTED", + "source_file": str_path, + "source_location": f"L{line}", + "weight": 1.0, + }) + elif callee_name: + raw_calls.append({ + "caller_nid": caller_nid, + "callee": callee_name, + "is_member_call": is_member_call, + "source_file": str_path, + "source_location": f"L{node.start_point[0] + 1}", + }) + for child in node.children: + walk_calls(child, caller_nid) + + for caller_nid, body_node in function_bodies: + walk_calls(body_node, caller_nid) + + valid_ids = seen_ids + clean_edges = [] + for edge in edges: + src, tgt = edge["source"], edge["target"] + if src in valid_ids and (tgt in valid_ids or edge["relation"] in ("imports", "imports_from")): + clean_edges.append(edge) + + return {"nodes": nodes, "edges": clean_edges, "raw_calls": raw_calls} diff --git a/graphify/extractors/json_config.py b/graphify/extractors/json_config.py new file mode 100644 index 000000000..025ca20a8 --- /dev/null +++ b/graphify/extractors/json_config.py @@ -0,0 +1,206 @@ +"""Json_config extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + + +from pathlib import Path +from graphify.extractors.base import _file_stem, _make_id, _read_text + + +_CONFIG_JSON_NAMES = frozenset({ + "package.json", "tsconfig.json", "jsconfig.json", "composer.json", + "deno.json", "deno.jsonc", "bower.json", "manifest.json", + "app.json", "now.json", "vercel.json", "angular.json", "nest-cli.json", + "biome.json", "biome.jsonc", "renovate.json", ".babelrc", ".babelrc.json", + ".eslintrc.json", ".prettierrc.json", ".prettierrc", "babel.config.json", +}) + +_CONFIG_JSON_KEYS = frozenset({ + "dependencies", "devDependencies", "peerDependencies", + "optionalDependencies", "bundleDependencies", "bundledDependencies", + "extends", "$ref", "$schema", "compilerOptions", +}) + +def _is_config_json(path: Path, obj_node, source: bytes) -> bool: + """True if a .json file is a recognized config/manifest worth AST-extracting. + + Matches by filename first (cheap), then falls back to a top-level key probe + so arbitrarily-named config files (e.g. ``api.tsconfig.json``, + ``foo.eslintrc.json``) are still picked up. Returns False for data JSON so it + is skipped by the structural pass (#1224).""" + name = path.name.casefold() + if name in _CONFIG_JSON_NAMES: + return True + # Common compound config names: *.eslintrc.json, *.prettierrc.json, etc. + if name.endswith((".eslintrc.json", ".prettierrc.json", ".babelrc.json", + "tsconfig.json", "jsconfig.json")): + return True + # Top-level key probe: scan the root object's immediate keys (no deep walk). + for top_key in obj_node.children: + if top_key.type != "pair": + continue + key_node = top_key.child_by_field_name("key") + if key_node is None: + continue + kc = key_node.child_by_field_name("string_content") + text = _read_text(kc, source) if kc else _read_text(key_node, source).strip('"\'') + if text in _CONFIG_JSON_KEYS: + return True + return False + +def extract_json(path: Path) -> dict: + """Extract structure and dependency edges from a *config/manifest* .json file. + + Data-shaped JSON (eval fixtures, datasets, GeoJSON, API response dumps) is + deliberately skipped — AST-walking it produced hundreds of orphan key-nodes + and duplicate communities that swamped real structure (#1224). Recognition + is by filename (package.json, tsconfig.json, …) or a top-level key probe + (dependencies / extends / $ref / $schema / compilerOptions).""" + _JSON_MAX_BYTES = 1_048_576 # 1 MiB — skip large fixture dumps / GeoJSON blobs + + try: + import tree_sitter_json as tsjson + from tree_sitter import Language, Parser + except ImportError: + return {"nodes": [], "edges": [], "error": "tree-sitter-json not installed"} + + try: + # Bounded read instead of stat()+read() to eliminate TOCTOU (J-1): + # read one byte beyond the limit so we can detect oversized files even + # if the file grows between stat and read. + with path.open("rb") as _f: + source = _f.read(_JSON_MAX_BYTES + 1) + if len(source) > _JSON_MAX_BYTES: + return {"nodes": [], "edges": [], "error": "json file too large to index"} + language = Language(tsjson.language()) + parser = Parser(language) + tree = parser.parse(source) + root = tree.root_node + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + stem = _file_stem(path) + str_path = str(path) + nodes: list[dict] = [] + edges: list[dict] = [] + seen_ids: set[str] = set() + + # Keys whose string values become imports (package.json dep blocks) + _DEP_KEYS = frozenset({ + "dependencies", "devDependencies", "peerDependencies", + "optionalDependencies", "bundleDependencies", "bundledDependencies", + }) + + def add_node(nid: str, label: str, line: int) -> None: + if nid and nid not in seen_ids: + seen_ids.add(nid) + nodes.append({"id": nid, "label": label, "file_type": "code", + "source_file": str_path, "source_location": f"L{line}"}) + + def add_edge(src: str, tgt: str, relation: str, line: int, + context: str | None = None) -> None: + if not src or not tgt or src == tgt: + return + edge = {"source": src, "target": tgt, "relation": relation, + "confidence": "EXTRACTED", "source_file": str_path, + "source_location": f"L{line}", "weight": 1.0} + if context: + edge["context"] = context + edges.append(edge) + + file_nid = _make_id(str(path)) + add_node(file_nid, path.name, 1) + + def _key_text(pair_node) -> str | None: + """Extract the string content of a pair's key.""" + key_node = pair_node.child_by_field_name("key") + if key_node is None: + return None + if key_node.type == "string": + content = key_node.child_by_field_name("string_content") + if content: + return _read_text(content, source) + # fallback: strip surrounding quotes + raw = _read_text(key_node, source) + return raw.strip('"\'') + return _read_text(key_node, source) + + def _val_node(pair_node): + return pair_node.child_by_field_name("value") + + def walk_object(obj_node, parent_nid: str, parent_key: str | None, + depth: int, pair_count: list) -> None: + if depth > 6: + return + for child in obj_node.children: + if child.type != "pair": + continue + if pair_count[0] >= 500: # check per-pair so the cap is honoured exactly (J-3) + return + pair_count[0] += 1 + key = _key_text(child) + if not key: + continue + key_nid = _make_id(stem, *(([parent_key] if parent_key else []) + [key])) + if not key_nid: + continue + line = child.start_point[0] + 1 + add_node(key_nid, key, line) + add_edge(parent_nid, key_nid, "contains", line) + + val = _val_node(child) + if val is None: + continue + + if val.type == "object": + walk_object(val, key_nid, key, depth + 1, pair_count) + + elif val.type == "array": + # For "extends" arrays (tsconfig, eslint): each string element. + # Prefix with "ref_" so external refs don't collide with real + # code/file node IDs that share the same collapsed _make_id (J-4). + for item in val.children: + if item.type == "string": + content = item.child_by_field_name("string_content") + ref = _read_text(content, source) if content else _read_text(item, source).strip('"\'') + if ref: + ref_nid = _make_id("ref", ref) + if ref_nid: + add_edge(key_nid, ref_nid, "extends", line, context="import") + + elif val.type == "string": + content = val.child_by_field_name("string_content") + val_text = _read_text(content, source) if content else _read_text(val, source).strip('"\'') + + if key == "extends" and val_text: + # Namespace external refs to avoid ID collision with file nodes (J-4) + ref_nid = _make_id("ref", val_text) + if ref_nid: + add_edge(file_nid, ref_nid, "extends", line, context="import") + + elif key == "$ref" and val_text: + # Namespace $ref values to prevent edge hijacking into code nodes (J-4) + ref_nid = _make_id("ref", val_text) + if ref_nid: + add_edge(parent_nid, ref_nid, "references", line) + + elif parent_key in _DEP_KEYS and val_text: + dep_nid = _make_id(key) + if dep_nid: + add_edge(key_nid, dep_nid, "imports", line, context="import") + + # Entry: find root document → object + doc = root + if doc.type == "document" and doc.child_count > 0: + doc = doc.children[0] + if doc.type == "object": + # Only AST-extract recognized config/manifest JSON. Data JSON (fixtures, + # datasets, GeoJSON, API dumps) is skipped so it doesn't explode into + # orphan key-nodes (#1224); it's left to the LLM semantic pass. + if not _is_config_json(path, doc, source): + return {"nodes": [], "edges": [], "skipped": "data json (not a config/manifest)"} + walk_object(doc, file_nid, None, 0, [0]) + else: + # Top-level array or scalar => data JSON, never a config/manifest. + return {"nodes": [], "edges": [], "skipped": "data json (non-object root)"} + + return {"nodes": nodes, "edges": edges} diff --git a/graphify/extractors/pascal_forms.py b/graphify/extractors/pascal_forms.py new file mode 100644 index 000000000..1f81ad875 --- /dev/null +++ b/graphify/extractors/pascal_forms.py @@ -0,0 +1,196 @@ +"""Pascal_forms extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + + +from pathlib import Path +from typing import Any +from graphify.extractors.base import _file_stem, _make_id + + +def extract_lazarus_form(path: Path) -> dict: + """Extract component hierarchy from Lazarus .lfm form files. + + .lfm is a text-based declarative format for UI component trees, structured as: + object ComponentName: TClassName + PropertyName = Value + OnEvent = HandlerName + object ChildName: TChildClass + ... + end + end + + Produces nodes for: + - The form file itself + - Each component class encountered (TForm1, TButton, TPanel, ...) + - Event handler names referenced by OnXxx properties + + Produces edges for: + - file --contains--> root form class + - parent component --contains--> child component class + - component --references--> event handler (context: "event") + """ + try: + text = path.read_text(encoding="utf-8", errors="replace") + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + import re + str_path = str(path) + stem = _file_stem(path) + nodes: list[dict] = [] + edges: list[dict] = [] + seen_ids: set[str] = set() + seen_edge_pairs: set[tuple[str, str, str]] = set() + + def add_node(nid: str, label: str, line: int) -> None: + if nid not in seen_ids: + seen_ids.add(nid) + nodes.append({ + "id": nid, "label": label, "file_type": "code", + "source_file": str_path, "source_location": f"L{line}", + }) + + def add_edge( + src: str, tgt: str, relation: str, line: int, + context: str | None = None, + ) -> None: + key = (src, tgt, relation) + if key in seen_edge_pairs: + return + seen_edge_pairs.add(key) + edge: dict[str, Any] = { + "source": src, "target": tgt, "relation": relation, + "confidence": "EXTRACTED", "source_file": str_path, + "source_location": f"L{line}", "weight": 1.0, + } + if context: + edge["context"] = context + edges.append(edge) + + file_nid = _make_id(str(path)) + add_node(file_nid, path.name, 1) + + obj_re = re.compile(r"^\s*object\s+\w+\s*:\s*(\w+)", re.IGNORECASE) + event_re = re.compile(r"^\s*On\w+\s*=\s*(\w+)", re.IGNORECASE) + end_re = re.compile(r"^\s*end\s*$", re.IGNORECASE) + + # Stack of node IDs representing the nesting of object...end blocks + stack: list[str] = [file_nid] + + for lineno, line in enumerate(text.splitlines(), 1): + m = obj_re.match(line) + if m: + class_name = m.group(1) + nid = _make_id(stem, class_name) + add_node(nid, class_name, lineno) + add_edge(stack[-1], nid, "contains", lineno) + stack.append(nid) + continue + + m = event_re.match(line) + if m and len(stack) > 1: + handler = m.group(1) + handler_nid = _make_id(stem, handler) + add_node(handler_nid, f"{handler}()", lineno) + add_edge(stack[-1], handler_nid, "references", lineno, context="event") + continue + + if end_re.match(line) and len(stack) > 1: + stack.pop() + + return {"nodes": nodes, "edges": edges, "input_tokens": 0, "output_tokens": 0} + +def extract_delphi_form(path: Path) -> dict: + """Extract component hierarchy from Delphi .dfm form files. + + .dfm files come in two formats: + - Text (same `object Name: TClassName ... end` syntax as .lfm) + - Binary (starts with a TPF0/FF0A magic header — unreadable as text) + + Binary .dfm files are skipped gracefully: an empty result is returned + so the rest of the pipeline is unaffected. Convert binary forms to + text in the Delphi IDE via File → Save As (Text DFM) if you want them + indexed. + + Text .dfm files are parsed identically to .lfm: component containment + (`contains`) and event handler references (`references`, context "event"). + """ + try: + raw = path.read_bytes() + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + # Detect binary DFM: Delphi binary resource streams start with FF 0A + if raw[:2] == b"\xff\x0a": + return { + "nodes": [], "edges": [], + "error": f"binary DFM (convert to text in Delphi IDE to index): {path.name}", + } + + # Text DFM — delegate to the shared form parser (same syntax as .lfm) + try: + text = raw.decode("utf-8", errors="replace") + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + import re + str_path = str(path) + stem = _file_stem(path) + nodes: list[dict] = [] + edges: list[dict] = [] + seen_ids: set[str] = set() + seen_edge_pairs: set[tuple[str, str, str]] = set() + + def add_node(nid: str, label: str, line: int) -> None: + if nid not in seen_ids: + seen_ids.add(nid) + nodes.append({ + "id": nid, "label": label, "file_type": "code", + "source_file": str_path, "source_location": f"L{line}", + }) + + def add_edge( + src: str, tgt: str, relation: str, line: int, + context: str | None = None, + ) -> None: + key = (src, tgt, relation) + if key in seen_edge_pairs: + return + seen_edge_pairs.add(key) + edge: dict[str, Any] = { + "source": src, "target": tgt, "relation": relation, + "confidence": "EXTRACTED", "source_file": str_path, + "source_location": f"L{line}", "weight": 1.0, + } + if context: + edge["context"] = context + edges.append(edge) + + file_nid = _make_id(str(path)) + add_node(file_nid, path.name, 1) + + obj_re = re.compile(r"^\s*object\s+\w+\s*:\s*(\w+)", re.IGNORECASE) + event_re = re.compile(r"^\s*On\w+\s*=\s*(\w+)", re.IGNORECASE) + end_re = re.compile(r"^\s*end\s*$", re.IGNORECASE) + stack: list[str] = [file_nid] + + for lineno, line in enumerate(text.splitlines(), 1): + m = obj_re.match(line) + if m: + class_name = m.group(1) + nid = _make_id(stem, class_name) + add_node(nid, class_name, lineno) + add_edge(stack[-1], nid, "contains", lineno) + stack.append(nid) + continue + m = event_re.match(line) + if m and len(stack) > 1: + handler = m.group(1) + handler_nid = _make_id(stem, handler) + add_node(handler_nid, f"{handler}()", lineno) + add_edge(stack[-1], handler_nid, "references", lineno, context="event") + continue + if end_re.match(line) and len(stack) > 1: + stack.pop() + + return {"nodes": nodes, "edges": edges, "input_tokens": 0, "output_tokens": 0} diff --git a/graphify/extractors/powershell.py b/graphify/extractors/powershell.py new file mode 100644 index 000000000..9f8526c7b --- /dev/null +++ b/graphify/extractors/powershell.py @@ -0,0 +1,496 @@ +"""Powershell extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + +import re + +from pathlib import Path +from typing import Any +from graphify.extractors.base import _file_stem, _make_id, _read_text + + +def extract_powershell(path: Path) -> dict: + """Extract functions, classes, methods, and using statements from a .ps1 file.""" + try: + import tree_sitter_powershell as tsps + from tree_sitter import Language, Parser + except ImportError: + return {"nodes": [], "edges": [], "error": "tree_sitter_powershell not installed"} + + try: + language = Language(tsps.language()) + parser = Parser(language) + source = path.read_bytes() + tree = parser.parse(source) + root = tree.root_node + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + stem = _file_stem(path) + str_path = str(path) + nodes: list[dict] = [] + edges: list[dict] = [] + seen_ids: set[str] = set() + function_bodies: list[tuple[str, Any]] = [] + + def add_node(nid: str, label: str, line: int) -> None: + if nid not in seen_ids: + seen_ids.add(nid) + nodes.append({"id": nid, "label": label, "file_type": "code", + "source_file": str_path, "source_location": f"L{line}"}) + + def add_edge(src: str, tgt: str, relation: str, line: int, + confidence: str = "EXTRACTED", weight: float = 1.0, + context: str | None = None) -> None: + edge = {"source": src, "target": tgt, "relation": relation, + "confidence": confidence, "source_file": str_path, + "source_location": f"L{line}", "weight": weight} + if context: + edge["context"] = context + edges.append(edge) + + file_nid = _make_id(str(path)) + add_node(file_nid, path.name, 1) + + _PS_SKIP = frozenset({ + "using", "return", "if", "else", "elseif", "foreach", "for", + "while", "do", "switch", "try", "catch", "finally", "throw", + "break", "continue", "exit", "param", "begin", "process", "end", + # Import commands — handled as import edges, not function calls + "import-module", + }) + + def _find_script_block_body(node): + for child in node.children: + if child.type == "script_block": + for sc in child.children: + if sc.type == "script_block_body": + return sc + return child + return None + + def ensure_named_node(name: str, line: int) -> str: + nid = _make_id(stem, name) + if nid in seen_ids: + return nid + nid = _make_id(name) + if nid not in seen_ids: + # The name isn't defined in this file, so this is a cross-file reference + # (e.g. a `Thing` type annotation imported from another module). Emit a + # SOURCELESS stub — like the inheritance-base path below — so the + # corpus-level rewire can collapse it onto the real definition. A sourced + # stub here makes _disambiguate_colliding_node_ids bake the referencing + # file's path (with extension) into the id and blocks the rewire, which is + # the phantom-duplicate-node bug (#1402). + seen_ids.add(nid) + nodes.append({ + "id": nid, + "label": name, + "file_type": "code", + "source_file": "", + "source_location": "", + "origin_file": str_path, + }) + return nid + + def _ps_type_name(type_literal_node) -> str | None: + """Drill into a type_literal node and return the inner type_identifier text.""" + if type_literal_node is None: + return None + for spec in type_literal_node.children: + if spec.type != "type_spec": + continue + for tname in spec.children: + if tname.type != "type_name": + continue + for tid in tname.children: + if tid.type == "type_identifier": + return _read_text(tid, source) + return None + + def walk(node, parent_class_nid: str | None = None) -> None: + t = node.type + + if t == "function_statement": + name_node = next((c for c in node.children if c.type == "function_name"), None) + if name_node: + func_name = _read_text(name_node, source) + line = node.start_point[0] + 1 + func_nid = _make_id(stem, func_name) + add_node(func_nid, f"{func_name}()", line) + add_edge(file_nid, func_nid, "contains", line) + body = _find_script_block_body(node) + if body: + function_bodies.append((func_nid, body)) + # Also walk the body during the main pass so that + # Import-Module / dot-source inside functions emit + # file-level imports_from edges (#1331). + walk(body, parent_class_nid) + return + + if t == "class_statement": + name_node = next((c for c in node.children if c.type == "simple_name"), None) + if name_node: + class_name = _read_text(name_node, source) + line = node.start_point[0] + 1 + class_nid = _make_id(stem, class_name) + add_node(class_nid, class_name, line) + add_edge(file_nid, class_nid, "contains", line) + # Base type(s) after ':'. PowerShell has no syntactic base vs + # interface split, so (matching the C# convention) treat the + # first base as the superclass (inherits) and the rest as + # interfaces (implements). Bases are the simple_name children + # after the ':' token. + colon_seen = False + base_index = 0 + for child in node.children: + if child.type == ":": + colon_seen = True + elif colon_seen and child.type == "simple_name": + base_nid = ensure_named_node(_read_text(child, source), line) + if base_nid != class_nid: + rel = "inherits" if base_index == 0 else "implements" + add_edge(class_nid, base_nid, rel, line) + base_index += 1 + for child in node.children: + walk(child, parent_class_nid=class_nid) + return + + if t == "class_property_definition" and parent_class_nid: + type_literal = next((c for c in node.children if c.type == "type_literal"), None) + type_name = _ps_type_name(type_literal) + if type_name: + line = node.start_point[0] + 1 + target_nid = ensure_named_node(type_name, line) + if target_nid != parent_class_nid: + add_edge(parent_class_nid, target_nid, "references", + line, context="field") + return + + if t == "class_method_definition": + name_node = next((c for c in node.children if c.type == "simple_name"), None) + if name_node: + method_name = _read_text(name_node, source) + line = node.start_point[0] + 1 + if parent_class_nid: + method_nid = _make_id(parent_class_nid, method_name) + add_node(method_nid, f".{method_name}()", line) + add_edge(parent_class_nid, method_nid, "method", line) + else: + method_nid = _make_id(stem, method_name) + add_node(method_nid, f"{method_name}()", line) + add_edge(file_nid, method_nid, "contains", line) + # Return type: type_literal sibling of simple_name + return_type_literal = next( + (c for c in node.children if c.type == "type_literal"), None) + return_type_name = _ps_type_name(return_type_literal) + if return_type_name: + target_nid = ensure_named_node(return_type_name, line) + if target_nid != method_nid: + add_edge(method_nid, target_nid, "references", + line, context="return_type") + # Parameter types: class_method_parameter_list + param_list = next( + (c for c in node.children if c.type == "class_method_parameter_list"), None) + if param_list is not None: + for p in param_list.children: + if p.type != "class_method_parameter": + continue + ptype_literal = next( + (c for c in p.children if c.type == "type_literal"), None) + ptype_name = _ps_type_name(ptype_literal) + if not ptype_name: + continue + p_line = p.start_point[0] + 1 + target_nid = ensure_named_node(ptype_name, p_line) + if target_nid != method_nid: + add_edge(method_nid, target_nid, "references", + p_line, context="parameter_type") + body = _find_script_block_body(node) + if body: + function_bodies.append((method_nid, body)) + return + + if t == "command": + # Dot-sourcing: `. ./Shared.psm1` + # Uses command_invokation_operator '.' + command_name_expr (not command_name) + invoke_op = next( + (c for c in node.children if c.type == "command_invokation_operator"), None + ) + if invoke_op is not None and _read_text(invoke_op, source).strip() == ".": + name_expr = next( + (c for c in node.children if c.type == "command_name_expr"), None + ) + if name_expr is not None: + name_node = next( + (c for c in name_expr.children if c.type == "command_name"), None + ) + if name_node: + raw_path = _read_text(name_node, source) + # Strip relative path prefix (./ or .\ or just the dot) + module_stem = re.sub(r'^[./\\]+', '', raw_path) + # Drop extension to get bare module name + module_stem = re.sub(r'\.[^.]+$', '', module_stem).replace('\\', '/') + module_name = module_stem.split('/')[-1] + if module_name: + add_edge(file_nid, _make_id(module_name), "imports_from", + node.start_point[0] + 1) + return + + cmd_name_node = next((c for c in node.children if c.type == "command_name"), None) + if cmd_name_node: + cmd_text = _read_text(cmd_name_node, source).lower() + if cmd_text == "using": + tokens = [] + for child in node.children: + if child.type == "command_elements": + for el in child.children: + if el.type == "generic_token": + tokens.append(_read_text(el, source)) + module_tokens = [t for t in tokens + if t.lower() not in ("namespace", "module", "assembly")] + if module_tokens: + module_name = module_tokens[-1].split(".")[-1] + add_edge(file_nid, _make_id(module_name), "imports_from", + node.start_point[0] + 1) + elif cmd_text == "import-module": + # Collect generic_token args; skip command_parameter flags like -Name + # The module name is the first generic_token (or the one after -Name) + module_name: str | None = None + expect_name = False + for child in node.children: + if child.type != "command_elements": + continue + for el in child.children: + if el.type == "command_parameter": + param_text = _read_text(el, source).lstrip("-").lower() + expect_name = param_text in ("name", "n") + elif el.type == "generic_token": + token = _read_text(el, source) + if module_name is None or expect_name: + module_name = token + expect_name = False + if module_name: + # Strip extension; keep only the stem for the node ID + bare = re.sub(r'\.[^.]+$', '', module_name).split('/')[-1].split('\\')[-1] + if bare: + add_edge(file_nid, _make_id(bare), "imports_from", + node.start_point[0] + 1) + return + + for child in node.children: + walk(child, parent_class_nid) + + walk(root) + + label_to_nid = {n["label"].strip("()").lstrip(".").lower(): n["id"] for n in nodes} + seen_call_pairs: set[tuple[str, str]] = set() + raw_calls: list[dict] = [] + + def walk_calls(node, caller_nid: str) -> None: + if node.type in ("function_statement", "class_statement"): + return + if node.type == "command": + cmd_name_node = next((c for c in node.children if c.type == "command_name"), None) + if cmd_name_node: + cmd_text = _read_text(cmd_name_node, source) + if cmd_text.lower() not in _PS_SKIP: + tgt_nid = label_to_nid.get(cmd_text.lower()) + if tgt_nid and tgt_nid != caller_nid: + pair = (caller_nid, tgt_nid) + if pair not in seen_call_pairs: + seen_call_pairs.add(pair) + add_edge(caller_nid, tgt_nid, "calls", + node.start_point[0] + 1, + confidence="EXTRACTED", weight=1.0) + elif cmd_text: + raw_calls.append({ + "caller_nid": caller_nid, + "callee": cmd_text, + "is_member_call": False, + "source_file": str_path, + "source_location": f"L{node.start_point[0] + 1}", + }) + for child in node.children: + walk_calls(child, caller_nid) + + for caller_nid, body_node in function_bodies: + walk_calls(body_node, caller_nid) + + clean_edges = [e for e in edges if e["source"] in seen_ids and + (e["target"] in seen_ids or e["relation"] in ("imports_from", "imports"))] + return {"nodes": nodes, "edges": clean_edges, "raw_calls": raw_calls} + +_PSD1_IMPORT_KEYS = frozenset({"RootModule", "NestedModules", "RequiredModules"}) + +def _psd1_collect_string_literals(node, source: bytes) -> list[str]: + """Recursively collect all string_literal text values under *node*.""" + results: list[str] = [] + + def _walk(n) -> None: + if n.type == "string_literal": + raw = source[n.start_byte:n.end_byte].decode(errors="replace") + # Strip surrounding quote chars (' or ") + results.append(raw.strip("'\"")) + return + for child in n.children: + _walk(child) + + _walk(node) + return results + +def _psd1_module_name(raw: str) -> str: + """Derive a bare module name from a raw string value. + + e.g. 'MyModule.psm1' → 'MyModule', './sub/Util.psm1' → 'Util', 'PSReadLine' → 'PSReadLine' + """ + # Strip path prefix and extension + name = raw.replace("\\", "/").split("/")[-1] + name = re.sub(r"\.[^.]+$", "", name) # remove last extension + return name.strip() + +def extract_powershell_manifest(path: Path) -> dict: + """Extract module dependency edges from a PowerShell .psd1 manifest file. + + .psd1 files are PowerShell data hashtables, not scripts. tree-sitter-powershell + parses them correctly (they are syntactically valid PS). We walk the AST looking + for RootModule, NestedModules, and RequiredModules keys and emit imports_from + edges for every referenced module. + + RequiredModules supports two forms: + - Simple string: 'PSReadLine' + - Module specification: @{ ModuleName = 'Pester'; ModuleVersion = '5.0' } + For the hashtable form we only follow the ModuleName key. + """ + try: + import tree_sitter_powershell as tsps + from tree_sitter import Language, Parser + except ImportError: + return {"nodes": [], "edges": [], "error": "tree_sitter_powershell not installed"} + + try: + language = Language(tsps.language()) + parser = Parser(language) + source = path.read_bytes() + tree = parser.parse(source) + root = tree.root_node + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + str_path = str(path) + nodes: list[dict] = [] + edges: list[dict] = [] + seen_ids: set[str] = set() + + def add_node(nid: str, label: str, line: int) -> None: + if nid not in seen_ids: + seen_ids.add(nid) + nodes.append({"id": nid, "label": label, "file_type": "code", + "source_file": str_path, "source_location": f"L{line}"}) + + def add_import_edge(src: str, module_raw: str, line: int) -> None: + name = _psd1_module_name(module_raw) + if not name: + return + tgt_nid = _make_id(name) + edges.append({ + "source": src, + "target": tgt_nid, + "relation": "imports_from", + "confidence": "EXTRACTED", + "source_file": str_path, + "source_location": f"L{line}", + "weight": 1.0, + "context": "import", + }) + + file_nid = _make_id(str(path)) + add_node(file_nid, path.name, 1) + + def walk_manifest(node) -> None: + """Walk the AST and emit edges for import-relevant hash_entry nodes.""" + if node.type != "hash_entry": + for child in node.children: + walk_manifest(child) + return + + # Identify the key + key_node = next((c for c in node.children if c.type == "key_expression"), None) + if key_node is None: + return + key_text = source[key_node.start_byte:key_node.end_byte].decode(errors="replace").strip() + + if key_text not in _PSD1_IMPORT_KEYS: + # Still recurse in case there are nested hashes (e.g. ModuleVersion entries + # contain sub-hashes, but we only care about top-level keys for imports) + return + + line = node.start_point[0] + 1 + value_node = next((c for c in node.children if c.type == "pipeline"), None) + if value_node is None: + return + + if key_text == "RootModule": + # Value is a single string + strings = _psd1_collect_string_literals(value_node, source) + for s in strings: + add_import_edge(file_nid, s, line) + + elif key_text == "NestedModules": + # Value is a string or @('a', 'b', ...) array — collect all string literals + strings = _psd1_collect_string_literals(value_node, source) + for s in strings: + add_import_edge(file_nid, s, line) + + elif key_text == "RequiredModules": + # Two forms: + # 1) 'SimpleModule' — direct string literals in the array + # 2) @{ ModuleName = 'Foo'; ModuleVersion = '2.0' } — use ModuleName only + # + # Strategy: walk the value for hash_entry nodes whose key is 'ModuleName'; + # collect their string values. For the remaining string_literal nodes that + # are NOT inside a hash_entry subtree, treat them as simple module names. + module_name_strings: list[str] = [] + inside_hash_entries: set[int] = set() # byte offsets of handled strings + + def find_modulename_entries(n) -> None: + if n.type == "hash_entry": + sub_key = next((c for c in n.children if c.type == "key_expression"), None) + if sub_key is not None: + sk_text = source[sub_key.start_byte:sub_key.end_byte].decode(errors="replace").strip() + # Collect strings inside *all* sub-keys so we can exclude them + for c in n.children: + if c.type == "pipeline": + for s_node in _collect_string_nodes(c): + inside_hash_entries.add(s_node.start_byte) + if sk_text == "ModuleName": + for c in n.children: + if c.type == "pipeline": + for s in _psd1_collect_string_literals(c, source): + module_name_strings.append(s) + return # don't recurse further into this hash_entry + for child in n.children: + find_modulename_entries(child) + + def _collect_string_nodes(n): + """Return all string_literal nodes in subtree.""" + if n.type == "string_literal": + yield n + return + for child in n.children: + yield from _collect_string_nodes(child) + + find_modulename_entries(value_node) + + # Now gather direct string literals not inside hash entries + direct_strings: list[str] = [] + for s_node in _collect_string_nodes(value_node): + if s_node.start_byte not in inside_hash_entries: + raw = source[s_node.start_byte:s_node.end_byte].decode(errors="replace") + direct_strings.append(raw.strip("'\"")) + + for s in direct_strings + module_name_strings: + add_import_edge(file_nid, s, line) + + walk_manifest(root) + + return {"nodes": nodes, "edges": edges, "raw_calls": []} diff --git a/graphify/extractors/rust.py b/graphify/extractors/rust.py new file mode 100644 index 000000000..b663bd625 --- /dev/null +++ b/graphify/extractors/rust.py @@ -0,0 +1,410 @@ +"""Rust extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + + +from pathlib import Path +from graphify.extractors.base import _LANGUAGE_BUILTIN_GLOBALS, _file_stem, _make_id, _read_text + + +def _rust_collect_type_refs(node, source: bytes, generic: bool, out: list[tuple[str, str]]) -> None: + """Walk a Rust type expression; append (name, role) tuples.""" + if node is None: + return + t = node.type + if t == "primitive_type": + return + if t == "type_identifier": + text = _read_text(node, source) + if text: + out.append((text, "generic_arg" if generic else "type")) + return + if t == "scoped_type_identifier": + text = _read_text(node, source).rsplit("::", 1)[-1] + if text: + out.append((text, "generic_arg" if generic else "type")) + return + if t == "generic_type": + name_node = node.child_by_field_name("type") + if name_node is None: + for c in node.children: + if c.type in ("type_identifier", "scoped_type_identifier"): + name_node = c + break + if name_node is not None: + text = _read_text(name_node, source).rsplit("::", 1)[-1] + if text: + out.append((text, "generic_arg" if generic else "type")) + for c in node.children: + if c.type == "type_arguments": + for arg in c.children: + if arg.is_named: + _rust_collect_type_refs(arg, source, True, out) + return + if t in ("reference_type", "pointer_type", "array_type", "tuple_type", "slice_type"): + for c in node.children: + if c.is_named: + _rust_collect_type_refs(c, source, generic, out) + return + if node.is_named: + for c in node.children: + if c.is_named: + _rust_collect_type_refs(c, source, generic, out) + +_RUST_TRAIT_METHOD_BLOCKLIST: frozenset[str] = frozenset({ + "new", "default", "parse", "from_str", "now", "clone", "into", "from", + "to_string", "to_owned", "len", "is_empty", "iter", "next", "build", + "start", "run", "init", "app", "get", "set", "push", "pop", "insert", + "remove", "contains", "collect", "map", "filter", "unwrap", "expect", + "ok", "err", "some", "none", "send", "recv", "lock", "read", "write", +}) + +def extract_rust(path: Path) -> dict: + """Extract functions, structs, enums, traits, impl methods, and use declarations from a .rs file.""" + try: + import tree_sitter_rust as tsrust + from tree_sitter import Language, Parser + except ImportError: + return {"nodes": [], "edges": [], "error": "tree-sitter-rust not installed"} + + try: + language = Language(tsrust.language()) + parser = Parser(language) + source = path.read_bytes() + tree = parser.parse(source) + root = tree.root_node + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + stem = _file_stem(path) + str_path = str(path) + nodes: list[dict] = [] + edges: list[dict] = [] + seen_ids: set[str] = set() + function_bodies: list[tuple[str, object]] = [] + + def add_node(nid: str, label: str, line: int) -> None: + if nid not in seen_ids: + seen_ids.add(nid) + nodes.append({ + "id": nid, + "label": label, + "file_type": "code", + "source_file": str_path, + "source_location": f"L{line}", + }) + + def add_edge(src: str, tgt: str, relation: str, line: int, + confidence: str = "EXTRACTED", weight: float = 1.0, + context: str | None = None) -> None: + edge = { + "source": src, + "target": tgt, + "relation": relation, + "confidence": confidence, + "source_file": str_path, + "source_location": f"L{line}", + "weight": weight, + } + if context: + edge["context"] = context + edges.append(edge) + + file_nid = _make_id(str(path)) + add_node(file_nid, path.name, 1) + + def ensure_named_node(name: str, line: int) -> str: + nid = _make_id(stem, name) + if nid in seen_ids: + return nid + nid = _make_id(name) + if nid not in seen_ids: + # The name isn't defined in this file, so this is a cross-file reference + # (e.g. a `Thing` type annotation imported from another module). Emit a + # SOURCELESS stub — like the inheritance-base path below — so the + # corpus-level rewire can collapse it onto the real definition. A sourced + # stub here makes _disambiguate_colliding_node_ids bake the referencing + # file's path (with extension) into the id and blocks the rewire, which is + # the phantom-duplicate-node bug (#1402). + seen_ids.add(nid) + nodes.append({ + "id": nid, + "label": name, + "file_type": "code", + "source_file": "", + "source_location": "", + "origin_file": str_path, + }) + return nid + + def emit_param_return_refs(func_node, func_nid: str, line: int) -> None: + params = func_node.child_by_field_name("parameters") + if params is not None: + for p in params.children: + if p.type != "parameter": + continue + type_node = p.child_by_field_name("type") + refs: list[tuple[str, str]] = [] + _rust_collect_type_refs(type_node, source, False, refs) + for ref_name, role in refs: + ctx = "generic_arg" if role == "generic_arg" else "parameter_type" + tgt = ensure_named_node(ref_name, line) + if tgt != func_nid: + add_edge(func_nid, tgt, "references", line, context=ctx) + return_type = func_node.child_by_field_name("return_type") + if return_type is not None: + refs = [] + _rust_collect_type_refs(return_type, source, False, refs) + for ref_name, role in refs: + ctx = "generic_arg" if role == "generic_arg" else "return_type" + tgt = ensure_named_node(ref_name, line) + if tgt != func_nid: + add_edge(func_nid, tgt, "references", line, context=ctx) + + def walk(node, parent_impl_nid: str | None = None) -> None: + t = node.type + + if t == "function_item": + name_node = node.child_by_field_name("name") + if name_node: + func_name = _read_text(name_node, source) + line = node.start_point[0] + 1 + if parent_impl_nid: + func_nid = _make_id(parent_impl_nid, func_name) + add_node(func_nid, f".{func_name}()", line) + add_edge(parent_impl_nid, func_nid, "method", line) + else: + func_nid = _make_id(stem, func_name) + add_node(func_nid, f"{func_name}()", line) + add_edge(file_nid, func_nid, "contains", line) + emit_param_return_refs(node, func_nid, line) + body = node.child_by_field_name("body") + if body: + function_bodies.append((func_nid, body)) + return + + if t in ("struct_item", "enum_item", "trait_item"): + name_node = node.child_by_field_name("name") + if name_node: + item_name = _read_text(name_node, source) + line = node.start_point[0] + 1 + item_nid = _make_id(stem, item_name) + add_node(item_nid, item_name, line) + add_edge(file_nid, item_nid, "contains", line) + if t == "trait_item": + for c in node.children: + if c.type != "trait_bounds": + continue + for sub in c.children: + if not sub.is_named: + continue + refs: list[tuple[str, str]] = [] + _rust_collect_type_refs(sub, source, False, refs) + for idx, (ref_name, _role) in enumerate(refs): + tgt = ensure_named_node(ref_name, line) + if tgt == item_nid: + continue + rel = "inherits" if idx == 0 else "references" + if rel == "inherits": + add_edge(item_nid, tgt, "inherits", line) + else: + add_edge(item_nid, tgt, "references", line, + context="generic_arg") + if t == "struct_item": + for c in node.children: + if c.type != "field_declaration_list": + continue + for field in c.children: + if field.type != "field_declaration": + continue + type_node = field.child_by_field_name("type") + if type_node is None: + for fc in field.children: + if fc.type in ("type_identifier", "generic_type", + "scoped_type_identifier", + "reference_type", "primitive_type"): + type_node = fc + break + refs = [] + _rust_collect_type_refs(type_node, source, False, refs) + for ref_name, role in refs: + ctx = "generic_arg" if role == "generic_arg" else "field" + tgt = ensure_named_node(ref_name, field.start_point[0] + 1) + if tgt != item_nid: + add_edge(item_nid, tgt, "references", + field.start_point[0] + 1, context=ctx) + # Tuple structs (`struct Wrapper(pub Logger, Config);`) nest their + # positional field types directly under ordered_field_declaration_list + # with no field_declaration wrapper -- the same shape handled for tuple + # enum variants below. Without this branch these field type references + # are silently dropped. + for c in node.children: + if c.type != "ordered_field_declaration_list": + continue + fline = c.start_point[0] + 1 + for tc in c.children: + if tc.type not in ("type_identifier", "generic_type", + "scoped_type_identifier", "reference_type", + "primitive_type", "tuple_type", "array_type"): + continue + refs = [] + _rust_collect_type_refs(tc, source, False, refs) + for ref_name, role in refs: + ctx = "generic_arg" if role == "generic_arg" else "field" + tgt = ensure_named_node(ref_name, fline) + if tgt != item_nid: + add_edge(item_nid, tgt, "references", fline, context=ctx) + if t == "enum_item": + # Variant payload types nest under enum_variant_list -> + # enum_variant -> ordered_field_declaration_list (tuple variant, + # `Click(Logger)`) | field_declaration_list (struct variant, + # `Resize { size: Dim }`). Neither was traversed, so every + # enum-variant type reference was silently dropped. + _TYPE_NODES = ("type_identifier", "generic_type", + "scoped_type_identifier", "reference_type", + "primitive_type", "tuple_type", "array_type") + + def _emit_enum_type(type_node, at_line): + if type_node is None: + return + refs2: list[tuple[str, str]] = [] + _rust_collect_type_refs(type_node, source, False, refs2) + for ref_name, role in refs2: + ctx = "generic_arg" if role == "generic_arg" else "field" + tgt = ensure_named_node(ref_name, at_line) + if tgt != item_nid: + add_edge(item_nid, tgt, "references", at_line, context=ctx) + + for c in node.children: + if c.type != "enum_variant_list": + continue + for variant in c.children: + if variant.type != "enum_variant": + continue + vline = variant.start_point[0] + 1 + for vc in variant.children: + if vc.type == "ordered_field_declaration_list": + for tc in vc.children: + if tc.type in _TYPE_NODES: + _emit_enum_type(tc, vline) + elif vc.type == "field_declaration_list": + for field in vc.children: + if field.type != "field_declaration": + continue + type_node = field.child_by_field_name("type") + _emit_enum_type(type_node, field.start_point[0] + 1) + return + + if t == "impl_item": + type_node = node.child_by_field_name("type") + trait_node = node.child_by_field_name("trait") + impl_nid: str | None = None + if type_node: + type_name = _read_text(type_node, source).strip() + impl_nid = _make_id(stem, type_name) + add_node(impl_nid, type_name, node.start_point[0] + 1) + if trait_node is not None and impl_nid is not None: + refs: list[tuple[str, str]] = [] + _rust_collect_type_refs(trait_node, source, False, refs) + for idx, (ref_name, _role) in enumerate(refs): + tgt = ensure_named_node(ref_name, node.start_point[0] + 1) + if tgt == impl_nid: + continue + if idx == 0: + add_edge(impl_nid, tgt, "implements", node.start_point[0] + 1) + else: + add_edge(impl_nid, tgt, "references", node.start_point[0] + 1, + context="generic_arg") + body = node.child_by_field_name("body") + if body: + for child in body.children: + walk(child, parent_impl_nid=impl_nid) + return + + if t == "use_declaration": + arg = node.child_by_field_name("argument") + if arg: + raw = _read_text(arg, source) + clean = raw.split("{")[0].rstrip(":").rstrip("*").rstrip(":") + module_name = clean.split("::")[-1].strip() + if module_name: + tgt_nid = _make_id(module_name) + add_edge(file_nid, tgt_nid, "imports_from", node.start_point[0] + 1, context="import") + return + + for child in node.children: + walk(child, parent_impl_nid=None) + + walk(root) + + label_to_nid: dict[str, str] = {} + for n in nodes: + raw = n["label"] + normalised = raw.strip("()").lstrip(".") + label_to_nid[normalised] = n["id"] + + seen_call_pairs: set[tuple[str, str]] = set() + raw_calls: list[dict] = [] + + def walk_calls(node, caller_nid: str) -> None: + if node.type == "function_item": + return + if node.type == "call_expression": + func_node = node.child_by_field_name("function") + callee_name: str | None = None + is_member_call: bool = False + is_scoped_call: bool = False + if func_node: + if func_node.type == "identifier": + callee_name = _read_text(func_node, source) + elif func_node.type == "field_expression": + is_member_call = True + field = func_node.child_by_field_name("field") + if field: + callee_name = _read_text(field, source) + elif func_node.type == "scoped_identifier": + # Type::method() — still allow in-file EXTRACTED match, but + # skip cross-file resolution: bare last-segment lookup ignores + # crate boundaries and produces spurious INFERRED edges (#908). + is_scoped_call = True + name = func_node.child_by_field_name("name") + if name: + callee_name = _read_text(name, source) + if callee_name and callee_name not in _LANGUAGE_BUILTIN_GLOBALS: + tgt_nid = label_to_nid.get(callee_name) + if tgt_nid and tgt_nid != caller_nid: + pair = (caller_nid, tgt_nid) + if pair not in seen_call_pairs: + seen_call_pairs.add(pair) + line = node.start_point[0] + 1 + edges.append({ + "source": caller_nid, + "target": tgt_nid, + "relation": "calls", + "context": "call", + "confidence": "EXTRACTED", + "source_file": str_path, + "source_location": f"L{line}", + "weight": 1.0, + }) + elif not is_scoped_call and callee_name.lower() not in _RUST_TRAIT_METHOD_BLOCKLIST: + raw_calls.append({ + "caller_nid": caller_nid, + "callee": callee_name, + "is_member_call": is_member_call, + "source_file": str_path, + "source_location": f"L{node.start_point[0] + 1}", + }) + for child in node.children: + walk_calls(child, caller_nid) + + for caller_nid, body_node in function_bodies: + walk_calls(body_node, caller_nid) + + valid_ids = seen_ids + clean_edges = [] + for edge in edges: + src, tgt = edge["source"], edge["target"] + if src in valid_ids and (tgt in valid_ids or edge["relation"] in ("imports", "imports_from")): + clean_edges.append(edge) + + return {"nodes": nodes, "edges": clean_edges, "raw_calls": raw_calls} diff --git a/graphify/extractors/sln.py b/graphify/extractors/sln.py new file mode 100644 index 000000000..d77fa6e64 --- /dev/null +++ b/graphify/extractors/sln.py @@ -0,0 +1,81 @@ +"""Sln extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + +import re + +from pathlib import Path +from graphify.extractors.base import _make_id + + +def extract_sln(path: Path) -> dict: + """Extract projects and inter-project dependencies from a .sln file.""" + try: + src = path.read_text(encoding="utf-8", errors="replace") + except OSError: + return {"nodes": [], "edges": [], "error": f"cannot read {path}"} + + file_nid = _make_id(str(path)) + str_path = str(path) + nodes: list[dict] = [{"id": file_nid, "label": path.name, "file_type": "code", + "source_file": str_path, "source_location": None}] + edges: list[dict] = [] + seen_ids: set[str] = set() + seen_ids.add(file_nid) + + _PROJECT_RE = re.compile( + r'Project\("[^"]*"\)\s*=\s*"([^"]+)"\s*,\s*"([^"]+)"\s*,\s*"([^"]*)"' + ) + _DEP_RE = re.compile(r'\{([0-9a-fA-F-]+)\}\s*=\s*\{([0-9a-fA-F-]+)\}') + + guid_to_nid: dict[str, str] = {} + + for m in _PROJECT_RE.finditer(src): + proj_name = m.group(1) + proj_path = m.group(2).replace("\\", "/") + proj_guid = m.group(3).strip("{}") + + try: + abs_proj = str((path.parent / proj_path).resolve()) + except Exception: + abs_proj = proj_path + proj_nid = _make_id(abs_proj) + if proj_nid and proj_nid not in seen_ids: + seen_ids.add(proj_nid) + nodes.append({"id": proj_nid, "label": proj_name, + "file_type": "code", "source_file": abs_proj, + "source_location": None}) + edges.append({"source": file_nid, "target": proj_nid, + "relation": "contains", "confidence": "EXTRACTED", + "source_file": str_path, "weight": 1.0}) + if proj_guid: + guid_to_nid[proj_guid.lower()] = proj_nid + + in_dep_section = False + current_proj_guid: str | None = None + _PROJECT_LINE_RE = re.compile(r'Project\("[^"]*"\)\s*=\s*"[^"]+"\s*,\s*"[^"]+"\s*,\s*"\{([^}]+)\}"') + for line in src.splitlines(): + proj_line_m = _PROJECT_LINE_RE.search(line) + if proj_line_m: + current_proj_guid = proj_line_m.group(1).lower() + continue + if line.strip() == "EndProject": + current_proj_guid = None + continue + if "ProjectSection(ProjectDependencies)" in line: + in_dep_section = True + continue + if in_dep_section and "EndProjectSection" in line: + in_dep_section = False + continue + if in_dep_section and current_proj_guid: + dep_m = _DEP_RE.search(line) + if dep_m: + to_guid = dep_m.group(1).lower() + from_nid = guid_to_nid.get(current_proj_guid) + to_nid = guid_to_nid.get(to_guid) + if from_nid and to_nid and from_nid != to_nid: + edges.append({"source": from_nid, "target": to_nid, + "relation": "imports", "confidence": "EXTRACTED", + "source_file": str_path, "weight": 1.0}) + + return {"nodes": nodes, "edges": edges} diff --git a/graphify/extractors/sql.py b/graphify/extractors/sql.py new file mode 100644 index 000000000..fa87982f2 --- /dev/null +++ b/graphify/extractors/sql.py @@ -0,0 +1,276 @@ +"""Sql extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + +import re + +from pathlib import Path +from graphify.extractors.base import _file_stem, _make_id + + +def extract_sql(path: Path, content: str | bytes | None = None) -> dict: + """Extract tables, views, functions, and relationships from .sql files via tree-sitter.""" + try: + import tree_sitter_sql as tssql + from tree_sitter import Language, Parser + except ImportError: + return {"nodes": [], "edges": [], "error": "tree_sitter_sql not installed. Run: pip install tree-sitter-sql"} + + try: + language = Language(tssql.language()) + parser = Parser(language) + source = ( + content.encode("utf-8") if isinstance(content, str) + else content if content is not None + else path.read_bytes() + ) + tree = parser.parse(source) + root = tree.root_node + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + + stem = _file_stem(path) + str_path = str(path) + file_nid = _make_id(str_path) + nodes: list[dict] = [{"id": file_nid, "label": path.name, "file_type": "code", + "source_file": str_path, "source_location": None}] + edges: list[dict] = [] + seen_ids: set[str] = {file_nid} + table_nids: dict[str, str] = {} # name → nid for reference resolution + + def _read(n) -> str: + return source[n.start_byte:n.end_byte].decode("utf-8", errors="replace") + + def _obj_name(n) -> str | None: + for c in n.children: + if c.type == "object_reference": + return _read(c) + return None + + def _add_node(nid: str, label: str, line: int) -> None: + if nid not in seen_ids: + seen_ids.add(nid) + nodes.append({"id": nid, "label": label, "file_type": "code", + "source_file": str_path, "source_location": f"L{line}"}) + edges.append({"source": file_nid, "target": nid, "relation": "contains", + "confidence": "EXTRACTED", "source_file": str_path, + "source_location": f"L{line}", "weight": 1.0}) + + def _add_edge(src: str, tgt: str, relation: str, line: int) -> None: + edges.append({"source": src, "target": tgt, "relation": relation, + "confidence": "EXTRACTED", "source_file": str_path, + "source_location": f"L{line}", "weight": 1.0}) + + def walk(node) -> None: + t = node.type + line = node.start_point[0] + 1 + + if t == "create_table": + name = _obj_name(node) + if name: + nid = _make_id(stem, name) + _add_node(nid, name, line) + table_nids[name.lower()] = nid + # Foreign key REFERENCES + for col in node.children: + if col.type == "column_definitions": + has_error = any(cd.type == "ERROR" for cd in col.children) + seen_refs: set[str] = set() + for cd in col.children: + if cd.type == "column_definition": + # Inline column-level REFERENCES + ref_name: str | None = None + found_ref = False + for cc in cd.children: + if cc.type == "keyword_references": + found_ref = True + elif found_ref and cc.type == "object_reference": + ref_name = _read(cc) + break + if ref_name: + ref_nid = table_nids.get(ref_name.lower()) or _make_id(stem, ref_name) + _add_edge(nid, ref_nid, "references", line) + seen_refs.add(ref_name.lower()) + elif cd.type == "constraints": + # Table-level FOREIGN KEY ... REFERENCES ... constraints + for constraint in cd.children: + if constraint.type != "constraint": + continue + ref_name = None + found_ref = False + for cc in constraint.children: + if cc.type == "keyword_references": + found_ref = True + elif found_ref and cc.type == "object_reference": + ref_name = _read(cc) + break + if ref_name: + ref_nid = table_nids.get(ref_name.lower()) or _make_id(stem, ref_name) + _add_edge(nid, ref_nid, "references", line) + seen_refs.add(ref_name.lower()) + if has_error: + # Dialect-specific syntax (e.g. Firebird COMPUTED BY) causes ERROR + # nodes that make the parser drop the trailing constraints block. + # Regex-scan the raw column_definitions text as fallback. + col_text = _read(col) + for rm in re.finditer(r"\bREFERENCES\s+([\w$]+)", col_text, re.IGNORECASE): + ref_name = rm.group(1) + if ref_name.lower() not in seen_refs: + ref_nid = table_nids.get(ref_name.lower()) or _make_id(stem, ref_name) + _add_edge(nid, ref_nid, "references", line) + seen_refs.add(ref_name.lower()) + + elif t == "create_view": + name = _obj_name(node) + if name: + nid = _make_id(stem, name) + _add_node(nid, name, line) + table_nids[name.lower()] = nid + # FROM/JOIN table references inside view body + _walk_from_refs(node, nid, line) + + elif t == "create_function": + name = _obj_name(node) + if name: + nid = _make_id(stem, name) + _add_node(nid, f"{name}()", line) + _walk_from_refs(node, nid, line) + + elif t == "create_procedure": + name = _obj_name(node) + if name: + nid = _make_id(stem, name) + _add_node(nid, f"{name}()", line) + _walk_from_refs(node, nid, line) + + elif t == "alter_table": + name = _obj_name(node) + if name: + src_nid = table_nids.get(name.lower()) + if not src_nid: + src_nid = _make_id(stem, name) + _add_node(src_nid, name, line) + table_nids[name.lower()] = src_nid + for child in node.children: + if child.type == "add_constraint": + for cc in child.children: + if cc.type != "constraint": + continue + found_ref = False + ref_name: str | None = None + for ccc in cc.children: + if ccc.type == "keyword_references": + found_ref = True + elif found_ref and ccc.type == "object_reference": + ref_name = _read(ccc) + break + if ref_name: + ref_nid = table_nids.get(ref_name.lower()) + if not ref_nid: + ref_nid = _make_id(stem, ref_name) + _add_edge(src_nid, ref_nid, "references", line) + + elif t == "create_trigger": + trig_name: str | None = None + tbl_name: str | None = None + after_trigger = False + after_for = False + for c in node.children: + if c.type == "keyword_trigger": + after_trigger = True + elif after_trigger and not trig_name and c.type == "object_reference": + trig_name = _read(c) + elif c.type == "keyword_for": + after_for = True + elif after_for and not tbl_name and c.type == "object_reference": + tbl_name = _read(c) + if trig_name: + trig_nid = _make_id(stem, trig_name) + _add_node(trig_nid, trig_name, line) + if tbl_name: + tbl_nid = table_nids.get(tbl_name.lower()) or _make_id(stem, tbl_name) + _add_edge(trig_nid, tbl_nid, "triggers", line) + + elif t == "fb_proc_or_trigger": + text = _read(node) + m = re.match( + r"CREATE\s+(?:OR\s+(?:REPLACE|ALTER)\s+)?" + r"(PROCEDURE|TRIGGER|FUNCTION)\s+([\w$]+)", + text, re.IGNORECASE, + ) + if m: + obj_type = m.group(1).upper() + obj_name = m.group(2) + obj_nid = _make_id(stem, obj_name) + label = obj_name if obj_type == "TRIGGER" else f"{obj_name}()" + _add_node(obj_nid, label, line) + if obj_type == "TRIGGER": + fm = re.search(r"\bFOR\s+([\w$]+)", text, re.IGNORECASE) + if fm: + tbl = fm.group(1) + tbl_nid = table_nids.get(tbl.lower()) or _make_id(stem, tbl) + _add_edge(obj_nid, tbl_nid, "triggers", line) + _NON_TABLES = { + "select", "where", "set", "dual", "null", "true", "false", + "first", "skip", "rows", "next", "only", "lateral", + } + seen_tbls: set[str] = set() + for rm in re.finditer(r"\b(?:FROM|JOIN|INTO)\s+([\w$]+)", text, re.IGNORECASE): + tbl = rm.group(1) + if tbl.lower() not in _NON_TABLES and tbl.lower() not in seen_tbls: + seen_tbls.add(tbl.lower()) + tbl_nid = table_nids.get(tbl.lower()) or _make_id(stem, tbl) + _add_edge(obj_nid, tbl_nid, "reads_from", line) + for rm in re.finditer(r"\bUPDATE\s+([\w$]+)", text, re.IGNORECASE): + tbl = rm.group(1) + if tbl.lower() not in _NON_TABLES and tbl.lower() not in seen_tbls: + seen_tbls.add(tbl.lower()) + tbl_nid = table_nids.get(tbl.lower()) or _make_id(stem, tbl) + _add_edge(obj_nid, tbl_nid, "reads_from", line) + + for child in node.children: + walk(child) + + def _walk_from_refs(node, caller_nid: str, line: int) -> None: + """Recursively find FROM/JOIN table references inside a node.""" + if node.type in ("from", "join"): + for c in node.children: + if c.type == "relation": + for cc in c.children: + if cc.type == "object_reference": + tbl = _read(cc) + tbl_nid = _make_id(stem, tbl) + _add_edge(caller_nid, tbl_nid, "reads_from", + c.start_point[0] + 1) + for child in node.children: + _walk_from_refs(child, caller_nid, line) + + for stmt in root.children: + if stmt.type == "statement": + for child in stmt.children: + walk(child) + elif stmt.type in ("fb_proc_or_trigger", "set_term", "declare_external_function"): + walk(stmt) + + # Global regex fallback: catch any REFERENCES missed due to ERROR nodes in the parse tree + # (e.g. Firebird COMPUTED BY columns push constraints out of the tree entirely). + # Snapshot after tree walk so we don't re-emit edges already captured above. + emitted = {(e["source"], e["target"]) for e in edges if e["relation"] == "references"} + src_text = source.decode("utf-8", errors="replace") + for m in re.finditer(r"CREATE\s+TABLE\s+([\w$]+)\s*\(", src_text, re.IGNORECASE): + tbl_name = m.group(1) + tbl_nid = table_nids.get(tbl_name.lower()) + if tbl_nid is None: + continue + tbl_line = src_text[: m.start()].count("\n") + 1 + tail = src_text[m.start():] + end = re.search(r"(?:^|\n)(?:CREATE|SET\s+TERM|ALTER)\s", tail[1:], re.IGNORECASE) + block = tail[: end.start() + 1] if end else tail + for rm in re.finditer(r"\bREFERENCES\s+([\w$]+)", block, re.IGNORECASE): + ref_name = rm.group(1) + ref_nid = table_nids.get(ref_name.lower()) or _make_id(stem, ref_name) + if (tbl_nid, ref_nid) not in emitted: + _add_edge(tbl_nid, ref_nid, "references", tbl_line) + emitted.add((tbl_nid, ref_nid)) + + return {"nodes": nodes, "edges": edges} diff --git a/graphify/extractors/terraform.py b/graphify/extractors/terraform.py new file mode 100644 index 000000000..b5fb78f99 --- /dev/null +++ b/graphify/extractors/terraform.py @@ -0,0 +1,181 @@ +"""Terraform extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + + +from pathlib import Path +from graphify.extractors.base import _make_id + + +_TF_META_HEADS = frozenset({"count", "each", "self", "path", "terraform"}) + +def extract_terraform(path: Path) -> dict: + """Extract Terraform/HCL blocks and the references between them via tree-sitter. + + Nodes: resources, data sources, modules, variables, outputs, providers, and + locals. Edges: `contains` (file -> block), `references` (block -> the blocks + it interpolates, e.g. `aws_instance.web` -> `var.region`), and `depends_on` + (explicit dependency edges). + + Node IDs are scoped by the parent directory, not the file stem, because + Terraform resources are module(directory)-scoped: a resource defined in + main.tf is referenced from other .tf files in the same directory. Directory + scoping lets those cross-file references resolve when per-file extractions + are merged (stem scoping would split a definition from its references). + """ + try: + import tree_sitter_hcl as tshcl + from tree_sitter import Language, Parser + except ImportError: + return {"nodes": [], "edges": [], "error": "tree_sitter_hcl not installed. Run: pip install tree-sitter-hcl"} + + try: + language = Language(tshcl.language()) + parser = Parser(language) + source = path.read_bytes() + tree = parser.parse(source) + root = tree.root_node + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + str_path = str(path) + file_nid = _make_id(str_path) + scope = path.parent.name or "tf" + + nodes: list[dict] = [{"id": file_nid, "label": path.name, "file_type": "code", + "source_file": str_path, "source_location": None}] + edges: list[dict] = [] + seen_ids: set[str] = {file_nid} + seen_edges: set[tuple[str, str, str]] = set() + + def _read(n) -> str: + return source[n.start_byte:n.end_byte].decode("utf-8", errors="replace") + + def _label_text(n) -> str: + return _read(n).strip().strip('"') + + def _add_node(address: str, label: str, line: int) -> str: + nid = _make_id(scope, address) + if nid not in seen_ids: + seen_ids.add(nid) + nodes.append({"id": nid, "label": label, "file_type": "code", + "source_file": str_path, "source_location": f"L{line}"}) + edges.append({"source": file_nid, "target": nid, "relation": "contains", + "confidence": "EXTRACTED", "source_file": str_path, + "source_location": f"L{line}", "weight": 1.0}) + return nid + + def _add_edge(src: str, address: str, relation: str, line: int) -> None: + tgt = _make_id(scope, address) + if src == tgt: + return + key = (src, tgt, relation) + if key in seen_edges: + return + seen_edges.add(key) + edges.append({"source": src, "target": tgt, "relation": relation, + "confidence": "EXTRACTED", "source_file": str_path, + "source_location": f"L{line}", "weight": 1.0}) + + def _block_parts(block) -> tuple: + btype = None + labels: list[str] = [] + for c in block.children: + if c.type in ("block_start", "body", "block_end"): + break + if c.type == "identifier" and btype is None: + btype = _read(c) + elif c.type in ("string_lit", "identifier"): + labels.append(_label_text(c)) + return btype, labels + + def _ref_address(expr): + head = _read(expr) + parent = expr.parent + attrs: list[str] = [] + if parent is not None: + seen_self = False + for c in parent.children: + if c.id == expr.id: + seen_self = True + continue + if seen_self and c.type == "get_attr": + name = None + for gc in c.children: + if gc.type == "identifier": + name = _read(gc) + break + if name is None: + break + attrs.append(name) + elif seen_self and c.type not in ("get_attr",): + break + if head in _TF_META_HEADS or not head: + return None + if head == "var": + return f"var.{attrs[0]}" if attrs else None + if head == "local": + return f"local.{attrs[0]}" if attrs else None + if head == "module": + return f"module.{attrs[0]}" if attrs else None + if head == "data": + return f"data.{attrs[0]}.{attrs[1]}" if len(attrs) >= 2 else None + return f"{head}.{attrs[0]}" if attrs else None + + def _collect_refs(node, owner_nid: str, relation: str) -> None: + rel = relation + if node.type == "attribute": + key_node = node.child_by_field_name("key") or ( + node.children[0] if node.children else None + ) + if key_node is not None and _read(key_node) == "depends_on": + rel = "depends_on" + if node.type == "variable_expr": + addr = _ref_address(node) + if addr: + _add_edge(owner_nid, addr, rel, node.start_point[0] + 1) + for c in node.children: + if c.is_named: + _collect_refs(c, owner_nid, rel) + + def _body_of(block): + for c in block.children: + if c.type == "body": + return c + return None + + body = next((c for c in root.children if c.type == "body"), root) + for block in body.children: + if block.type != "block": + continue + btype, labels = _block_parts(block) + line = block.start_point[0] + 1 + blk_body = _body_of(block) + if btype == "resource" and len(labels) >= 2: + owner = _add_node(f"{labels[0]}.{labels[1]}", f"{labels[0]}.{labels[1]}", line) + elif btype == "data" and len(labels) >= 2: + owner = _add_node(f"data.{labels[0]}.{labels[1]}", f"data.{labels[0]}.{labels[1]}", line) + elif btype == "module" and labels: + owner = _add_node(f"module.{labels[0]}", f"module.{labels[0]}", line) + elif btype == "variable" and labels: + owner = _add_node(f"var.{labels[0]}", f"var.{labels[0]}", line) + elif btype == "output" and labels: + owner = _add_node(f"output.{labels[0]}", f"output.{labels[0]}", line) + elif btype == "provider" and labels: + owner = _add_node(f"provider.{labels[0]}", f"provider.{labels[0]}", line) + elif btype == "locals" and blk_body is not None: + for attr in blk_body.children: + if attr.type != "attribute": + continue + key_node = attr.children[0] if attr.children else None + if key_node is None: + continue + key = _read(key_node) + lnid = _add_node(f"local.{key}", f"local.{key}", attr.start_point[0] + 1) + _collect_refs(attr, lnid, "references") + continue + else: + continue + if blk_body is not None: + _collect_refs(blk_body, owner, "references") + + return {"nodes": nodes, "edges": edges} From 4859f9883c7e4b00b3372b757f614a1da4ab0389 Mon Sep 17 00:00:00 2001 From: tpateeq Date: Thu, 9 Jul 2026 01:20:49 +0530 Subject: [PATCH 041/292] refactor(export): split HTML and graph-DB exporters into graphify/exporters/ export.py mixed nine independent export targets in one 1,671-LOC module. Move the two largest self-contained ones into a new graphify/exporters/ package, verbatim, re-exported from export.py so every importer (from graphify.export import to_html / push_to_neo4j) is unchanged: - exporters/html.py: to_html + its private helpers (_html_script, _html_styles, _hyperedge_script, _viz_node_limit) and MAX_NODES_FOR_VIZ. - exporters/graphdb.py: push_to_neo4j, push_to_falkordb. - exporters/base.py: COMMUNITY_COLORS (shared by the HTML/SVG/Obsidian exporters), homed here so per-format modules and export.py both import it without a cycle. Verified: AST closure-privacy analysis, facade object identity, ruff clean. export.py drops 1,671 -> 962 LOC. Full suite unchanged: 3036 passed, 29 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/export.py | 716 +-------------------------------- graphify/exporters/__init__.py | 1 + graphify/exporters/base.py | 14 + graphify/exporters/graphdb.py | 173 ++++++++ graphify/exporters/html.py | 547 +++++++++++++++++++++++++ 5 files changed, 739 insertions(+), 712 deletions(-) create mode 100644 graphify/exporters/__init__.py create mode 100644 graphify/exporters/base.py create mode 100644 graphify/exporters/graphdb.py create mode 100644 graphify/exporters/html.py diff --git a/graphify/export.py b/graphify/export.py index 36e4f9949..bf27eedc4 100644 --- a/graphify/export.py +++ b/graphify/export.py @@ -17,6 +17,8 @@ from graphify.analyze import _node_community_map from graphify.build import edge_data +from graphify.exporters.graphdb import push_to_falkordb, push_to_neo4j # noqa: E402,F401 + # Artifacts worth preserving across rebuilds (non-regenerable without LLM or curation). _BACKUP_ARTIFACTS = [ @@ -149,313 +151,9 @@ def _yaml_str(s: str) -> str: return "".join(out) -COMMUNITY_COLORS = [ - "#4E79A7", "#F28E2B", "#E15759", "#76B7B2", "#59A14F", - "#EDC948", "#B07AA1", "#FF9DA7", "#9C755F", "#BAB0AC", -] - -MAX_NODES_FOR_VIZ = 5_000 +from graphify.exporters.base import COMMUNITY_COLORS # noqa: E402,F401 - -def _viz_node_limit() -> int: - """Return the effective viz node limit, honoring GRAPHIFY_VIZ_NODE_LIMIT env var. - - Falls back to MAX_NODES_FOR_VIZ when the env var is unset, empty, or non-integer. - Set to 0 to disable HTML viz unconditionally (useful for CI runners). - """ - import os - raw = os.environ.get("GRAPHIFY_VIZ_NODE_LIMIT") - if raw is None or not raw.strip(): - return MAX_NODES_FOR_VIZ - try: - return int(raw) - except ValueError: - return MAX_NODES_FOR_VIZ - - -def _html_styles() -> str: - return """""" - - -def _hyperedge_script(hyperedges_json: str) -> str: - return f"""""" - - -def _html_script(nodes_json: str, edges_json: str, legend_json: str) -> str: - return f"""""" +from graphify.exporters.html import to_html # noqa: E402,F401 _CONFIDENCE_SCORE_DEFAULTS = {"EXTRACTED": 1.0, "INFERRED": 0.5, "AMBIGUOUS": 0.2} @@ -629,244 +327,6 @@ def to_cypher(G: nx.Graph, output_path: str) -> None: f.write("\n".join(lines)) -def to_html( - G: nx.Graph, - communities: dict[int, list[str]], - output_path: str, - community_labels: dict[int, str] | None = None, - member_counts: dict[int, int] | None = None, - node_limit: int | None = None, - learning_overlay: dict | None = None, -) -> None: - """Generate an interactive vis.js HTML visualization of the graph. - - Features: node size by degree, click-to-inspect panel, search box, - community filter, physics clustering by community, confidence-styled edges. - Raises ValueError if graph exceeds MAX_NODES_FOR_VIZ. - - If member_counts is provided (aggregated community view), node sizes are - based on community member counts rather than graph degree. - - If node_limit is set and the graph exceeds it, automatically builds an - aggregated community-level meta-graph instead of raising ValueError. - """ - limit = node_limit if node_limit is not None else _viz_node_limit() - if G.number_of_nodes() > limit: - if node_limit is not None: - # Build aggregated community meta-graph - from collections import Counter as _Counter - import networkx as _nx - print(f"Graph has {G.number_of_nodes()} nodes (above {limit} limit). Building aggregated community view...") - node_to_community = {nid: cid for cid, members in communities.items() for nid in members} - meta = _nx.Graph() - for cid, members in communities.items(): - meta.add_node(str(cid), label=(community_labels or {}).get(cid, f"Community {cid}")) - edge_counts = _Counter() - for u, v in G.edges(): - cu, cv = node_to_community.get(u), node_to_community.get(v) - if cu is not None and cv is not None and cu != cv: - edge_counts[(min(cu, cv), max(cu, cv))] += 1 - for (cu, cv), w in edge_counts.items(): - meta.add_edge(str(cu), str(cv), weight=w, - relation=f"{w} cross-community edges", confidence="AGGREGATED") - if meta.number_of_nodes() <= 1: - print("Single community - aggregated view not useful. Skipping graph.html.") - return - meta_communities = {cid: [str(cid)] for cid in communities} - mc = {cid: len(members) for cid, members in communities.items()} - # Remap hyperedges from semantic node IDs to community IDs - raw_hyperedges = G.graph.get("hyperedges", []) - if raw_hyperedges: - remapped = [] - for he in raw_hyperedges: - he_members = he.get("nodes", []) - comm_ids, seen = [], set() - for nid in he_members: - c = node_to_community.get(nid) - if c is None: - continue - s = str(c) - if s in seen: - continue - seen.add(s) - comm_ids.append(s) - if len(comm_ids) < 2: - continue - remapped.append({ - "id": he.get("id", ""), - "label": he.get("label") or he.get("relation", "").replace("_", " "), - "nodes": comm_ids, - }) - meta.graph["hyperedges"] = remapped - to_html(meta, meta_communities, output_path, - community_labels=community_labels, member_counts=mc) - print(f"graph.html written (aggregated: {meta.number_of_nodes()} community nodes, {meta.number_of_edges()} cross-community edges)") - print("Tip: run with --obsidian for full node-level detail.") - return - raise ValueError( - f"Graph has {G.number_of_nodes()} nodes - too large for HTML viz " - f"(limit: {limit}). Use --no-viz, raise GRAPHIFY_VIZ_NODE_LIMIT, " - f"or reduce input size." - ) - - node_community = _node_community_map(communities) - degree = dict(G.degree()) - max_deg = max(degree.values(), default=1) or 1 - max_mc = (max(member_counts.values(), default=1) or 1) if member_counts else 1 - - # Work-memory overlay (derived sidecar). When not passed explicitly, load it - # best-effort from the sibling .graphify_learning.json next to the output - # graph.html (which lives beside graph.json). Empty/missing => no learning - # fields, so the un-annotated render is byte-identical to pre-feature. - if learning_overlay is None: - learning_overlay = {} - try: - from graphify.reflect import load_learning_overlay as _llo - learning_overlay = _llo(Path(output_path)) - except Exception: - learning_overlay = {} - # Status -> ring color. preferred=green, contested=amber. Tentative gets no - # ring (it's not yet trustworthy enough to highlight in the map). - _RING = {"preferred": "#22c55e", "contested": "#f59e0b"} - - # Build nodes list for vis.js - vis_nodes = [] - for node_id, data in G.nodes(data=True): - cid = node_community.get(node_id, 0) - color = COMMUNITY_COLORS[cid % len(COMMUNITY_COLORS)] - label = sanitize_label(data.get("label", node_id)) - deg = degree.get(node_id, 1) - if member_counts: - mc = member_counts.get(cid, 1) - size = 10 + 30 * (mc / max_mc) - font_size = 12 - else: - size = 10 + 30 * (deg / max_deg) - # Only show label for high-degree nodes by default; others show on hover - font_size = 12 if deg >= max_deg * 0.15 else 0 - node = { - "id": node_id, - "label": label, - "color": {"background": color, "border": color, "highlight": {"background": "#ffffff", "border": color}}, - "size": round(size, 1), - "font": {"size": font_size, "color": "#ffffff"}, - "title": _html.escape(label), - "community": cid, - "community_name": sanitize_label((community_labels or {}).get(cid, f"Community {cid}")), - "source_file": sanitize_label(str(data.get("source_file") or "")), - "file_type": data.get("file_type", ""), - "degree": deg, - } - # Conditional learning fields — only present for annotated nodes, so - # un-annotated output keeps the exact pre-feature node dict shape. - entry = learning_overlay.get(str(node_id)) if learning_overlay else None - if entry: - status = sanitize_label(str(entry.get("status", ""))) - stale = bool(entry.get("stale")) - node["learning_status"] = status - node["learning_stale"] = stale - ring = _RING.get(status) - if ring: - # Status-colored ring via the border; stale => desaturated + - # dashed (vis.js supports per-node `shapeProperties.borderDashes`). - if stale: - ring = "#9ca3af" - node["shapeProperties"] = {"borderDashes": [4, 4]} - node["borderWidth"] = 3 - node["color"] = { - "background": color, "border": ring, - "highlight": {"background": "#ffffff", "border": ring}, - } - # Lesson line appended to the hover title. - if status == "contested": - lesson = f"Lesson: contested (useful {entry.get('uses', 0)} / dead-end {entry.get('neg', 0)})" - elif status == "preferred": - lesson = f"Lesson: preferred source ({entry.get('uses', 0)} useful, score={entry.get('score', 0)})" - else: - lesson = f"Lesson: {status} ({entry.get('uses', 0)} useful)" - if stale: - lesson += " [code changed — re-verify]" - node["title"] = _html.escape(label) + "\n" + _html.escape(sanitize_label(lesson)) - vis_nodes.append(node) - - # Build edges list. Restore original edge direction from _src/_tgt - # (stashed by build.py for exactly this reason): undirected NetworkX - # canonicalizes endpoint order, which would otherwise flip the arrow - # for `calls` and `rationale_for` in the rendered graph (#563). - vis_edges = [] - for u, v, data in G.edges(data=True): - confidence = data.get("confidence", "EXTRACTED") - relation = data.get("relation", "") - true_src = data.get("_src", u) - true_tgt = data.get("_tgt", v) - vis_edges.append({ - "from": true_src, - "to": true_tgt, - "label": relation, - "title": _html.escape(f"{relation} [{confidence}]"), - "dashes": confidence != "EXTRACTED", - "width": 2 if confidence == "EXTRACTED" else 1, - "color": {"opacity": 0.7 if confidence == "EXTRACTED" else 0.35}, - "confidence": confidence, - }) - - # Build community legend data - legend_data = [] - for cid in sorted((community_labels or {}).keys()): - color = COMMUNITY_COLORS[cid % len(COMMUNITY_COLORS)] - lbl = _html.escape(sanitize_label((community_labels or {}).get(cid, f"Community {cid}"))) - n = member_counts.get(cid, len(communities.get(cid, []))) if member_counts else len(communities.get(cid, [])) - legend_data.append({"cid": cid, "color": color, "label": lbl, "count": n}) - - # Escape sequences so embedded JSON cannot break out of the script tag - def _js_safe(obj) -> str: - return json.dumps(obj).replace(" - - - -graphify - {title} - -{_html_styles()} - - -
- -{_html_script(nodes_json, edges_json, legend_json)} -{_hyperedge_script(hyperedges_json)} - -""" - - Path(output_path).write_text(html, encoding="utf-8") # nosec - - # Keep backward-compatible alias - skill.md calls generate_html generate_html = to_html @@ -1397,174 +857,6 @@ def safe_name(label: str) -> str: Path(output_path).write_text(json.dumps(canvas_data, indent=2), encoding="utf-8") # nosec -def push_to_neo4j( - G: nx.Graph, - uri: str, - user: str, - password: str, - communities: dict[int, list[str]] | None = None, -) -> dict[str, int]: - """Push graph directly to a running Neo4j instance via the Python driver. - - Requires: pip install neo4j - - Uses MERGE so re-running is safe - nodes and edges are upserted, not duplicated. - Returns a dict with counts of nodes and edges pushed. - """ - try: - from neo4j import GraphDatabase - except ImportError as e: - raise ImportError( - "neo4j driver not installed. Run: pip install neo4j" - ) from e - - node_community = _node_community_map(communities) if communities else {} - - def _safe_rel(relation: str) -> str: - return re.sub(r"[^A-Z0-9_]", "_", relation.upper().replace(" ", "_").replace("-", "_")) or "RELATED_TO" - - def _safe_label(label: str) -> str: - """Sanitize a Neo4j node label to prevent Cypher injection.""" - sanitized = re.sub(r"[^A-Za-z0-9_]", "", label) - return sanitized if sanitized else "Entity" - - driver = GraphDatabase.driver(uri, auth=(user, password)) - nodes_pushed = 0 - edges_pushed = 0 - - with driver.session() as session: - for node_id, data in G.nodes(data=True): - props = { - k: v for k, v in data.items() - if isinstance(v, (str, int, float, bool)) and not k.startswith("_") - } - props["id"] = node_id - cid = node_community.get(node_id) - if cid is not None: - props["community"] = cid - ftype = _safe_label(data.get("file_type", "Entity").capitalize()) - session.run( - f"MERGE (n:{ftype} {{id: $id}}) SET n += $props", - id=node_id, - props=props, - ) - nodes_pushed += 1 - - for u, v, data in G.edges(data=True): - rel = _safe_rel(data.get("relation", "RELATED_TO")) - props = { - k: v for k, v in data.items() - if isinstance(v, (str, int, float, bool)) and not k.startswith("_") - } - session.run( - f"MATCH (a {{id: $src}}), (b {{id: $tgt}}) " - f"MERGE (a)-[r:{rel}]->(b) SET r += $props", - src=u, - tgt=v, - props=props, - ) - edges_pushed += 1 - - driver.close() - return {"nodes": nodes_pushed, "edges": edges_pushed} - - -def push_to_falkordb( - G: nx.Graph, - uri: str, - user: str | None = None, - password: str | None = None, - communities: dict[int, list[str]] | None = None, - graph_name: str = "graphify", -) -> dict[str, int]: - """Push graph directly to a running FalkorDB instance via the Python SDK. - - Requires: pip install falkordb - - FalkorDB is OpenCypher-compatible, so the MERGE/SET upsert queries are - identical to push_to_neo4j. Differences from the Neo4j path: - - connects with FalkorDB(host, port, username, password) instead of a bolt - driver; only the host/port are read from the URI, so the scheme is - informational - "falkordb://localhost:6379", "redis://localhost:6379" - and a bare "localhost:6379" are all equivalent (default port 6379). - - a named graph is selected via db.select_graph(graph_name) (default - "graphify"); FalkorDB keys each graph by name in the same instance. - - queries run via graph.query(cypher, params) - there is no session object. - - auth is optional (FalkorDB runs without credentials by default), so user - and password may be None. - - no APOC: the Neo4j path does not use APOC either, so nothing to port. - - Uses MERGE so re-running is safe - nodes and edges are upserted, not - duplicated. Returns a dict with counts of nodes and edges pushed. - """ - try: - from falkordb import FalkorDB - except ImportError as e: - raise ImportError( - "falkordb SDK not installed. Run: pip install falkordb" - ) from e - - from urllib.parse import urlparse - - node_community = _node_community_map(communities) if communities else {} - - def _safe_rel(relation: str) -> str: - return re.sub(r"[^A-Z0-9_]", "_", relation.upper().replace(" ", "_").replace("-", "_")) or "RELATED_TO" - - def _safe_label(label: str) -> str: - """Sanitize a FalkorDB node label to prevent Cypher injection.""" - sanitized = re.sub(r"[^A-Za-z0-9_]", "", label) - return sanitized if sanitized else "Entity" - - parsed = urlparse(uri if "://" in uri else f"redis://{uri}") - # FalkorDB auth is optional. Only send credentials when a password is - # provided; otherwise connect anonymously and ignore any bolt-style default - # username (e.g. Neo4j's "neo4j"), which FalkorDB rejects as an unknown ACL - # user. Credentials embedded in the URI take precedence over the args. - connect_user = parsed.username or (user if password else None) - connect_password = parsed.password or (password or None) - db = FalkorDB( - host=parsed.hostname or "localhost", - port=parsed.port or 6379, - username=connect_user, - password=connect_password, - ) - graph = db.select_graph(graph_name) - nodes_pushed = 0 - edges_pushed = 0 - - for node_id, data in G.nodes(data=True): - props = { - k: v for k, v in data.items() - if isinstance(v, (str, int, float, bool)) and not k.startswith("_") - } - props["id"] = node_id - cid = node_community.get(node_id) - if cid is not None: - props["community"] = cid - ftype = _safe_label(data.get("file_type", "Entity").capitalize()) - graph.query( - f"MERGE (n:{ftype} {{id: $id}}) SET n += $props", - {"id": node_id, "props": props}, - ) - nodes_pushed += 1 - - for u, v, data in G.edges(data=True): - rel = _safe_rel(data.get("relation", "RELATED_TO")) - props = { - k: v for k, v in data.items() - if isinstance(v, (str, int, float, bool)) and not k.startswith("_") - } - graph.query( - f"MATCH (a {{id: $src}}), (b {{id: $tgt}}) " - f"MERGE (a)-[r:{rel}]->(b) SET r += $props", - {"src": u, "tgt": v, "props": props}, - ) - edges_pushed += 1 - - return {"nodes": nodes_pushed, "edges": edges_pushed} - - def to_graphml( G: nx.Graph, communities: dict[int, list[str]], diff --git a/graphify/exporters/__init__.py b/graphify/exporters/__init__.py new file mode 100644 index 000000000..41e1aeff9 --- /dev/null +++ b/graphify/exporters/__init__.py @@ -0,0 +1 @@ +"""exporters package.""" diff --git a/graphify/exporters/base.py b/graphify/exporters/base.py new file mode 100644 index 000000000..9c9419646 --- /dev/null +++ b/graphify/exporters/base.py @@ -0,0 +1,14 @@ +"""Shared constants/helpers for the graphify exporters package. + +Symbols used by more than one exporter live here so each exporter module can be +split out of graphify/export.py without a circular import (export.py and the +per-format modules both import from here, never from each other). +""" +from __future__ import annotations + +# Categorical palette for community coloring, shared by the HTML, SVG, and +# Obsidian exporters. Moved verbatim from graphify/export.py. +COMMUNITY_COLORS = [ + "#4E79A7", "#F28E2B", "#E15759", "#76B7B2", "#59A14F", + "#EDC948", "#B07AA1", "#FF9DA7", "#9C755F", "#BAB0AC", +] diff --git a/graphify/exporters/graphdb.py b/graphify/exporters/graphdb.py new file mode 100644 index 000000000..14c47f0d5 --- /dev/null +++ b/graphify/exporters/graphdb.py @@ -0,0 +1,173 @@ +"""graphdb — moved verbatim from graphify/export.py.""" +from __future__ import annotations + +from graphify.analyze import _node_community_map +import networkx as nx +import re + + +def push_to_neo4j( + G: nx.Graph, + uri: str, + user: str, + password: str, + communities: dict[int, list[str]] | None = None, +) -> dict[str, int]: + """Push graph directly to a running Neo4j instance via the Python driver. + + Requires: pip install neo4j + + Uses MERGE so re-running is safe - nodes and edges are upserted, not duplicated. + Returns a dict with counts of nodes and edges pushed. + """ + try: + from neo4j import GraphDatabase + except ImportError as e: + raise ImportError( + "neo4j driver not installed. Run: pip install neo4j" + ) from e + + node_community = _node_community_map(communities) if communities else {} + + def _safe_rel(relation: str) -> str: + return re.sub(r"[^A-Z0-9_]", "_", relation.upper().replace(" ", "_").replace("-", "_")) or "RELATED_TO" + + def _safe_label(label: str) -> str: + """Sanitize a Neo4j node label to prevent Cypher injection.""" + sanitized = re.sub(r"[^A-Za-z0-9_]", "", label) + return sanitized if sanitized else "Entity" + + driver = GraphDatabase.driver(uri, auth=(user, password)) + nodes_pushed = 0 + edges_pushed = 0 + + with driver.session() as session: + for node_id, data in G.nodes(data=True): + props = { + k: v for k, v in data.items() + if isinstance(v, (str, int, float, bool)) and not k.startswith("_") + } + props["id"] = node_id + cid = node_community.get(node_id) + if cid is not None: + props["community"] = cid + ftype = _safe_label(data.get("file_type", "Entity").capitalize()) + session.run( + f"MERGE (n:{ftype} {{id: $id}}) SET n += $props", + id=node_id, + props=props, + ) + nodes_pushed += 1 + + for u, v, data in G.edges(data=True): + rel = _safe_rel(data.get("relation", "RELATED_TO")) + props = { + k: v for k, v in data.items() + if isinstance(v, (str, int, float, bool)) and not k.startswith("_") + } + session.run( + f"MATCH (a {{id: $src}}), (b {{id: $tgt}}) " + f"MERGE (a)-[r:{rel}]->(b) SET r += $props", + src=u, + tgt=v, + props=props, + ) + edges_pushed += 1 + + driver.close() + return {"nodes": nodes_pushed, "edges": edges_pushed} + +def push_to_falkordb( + G: nx.Graph, + uri: str, + user: str | None = None, + password: str | None = None, + communities: dict[int, list[str]] | None = None, + graph_name: str = "graphify", +) -> dict[str, int]: + """Push graph directly to a running FalkorDB instance via the Python SDK. + + Requires: pip install falkordb + + FalkorDB is OpenCypher-compatible, so the MERGE/SET upsert queries are + identical to push_to_neo4j. Differences from the Neo4j path: + - connects with FalkorDB(host, port, username, password) instead of a bolt + driver; only the host/port are read from the URI, so the scheme is + informational - "falkordb://localhost:6379", "redis://localhost:6379" + and a bare "localhost:6379" are all equivalent (default port 6379). + - a named graph is selected via db.select_graph(graph_name) (default + "graphify"); FalkorDB keys each graph by name in the same instance. + - queries run via graph.query(cypher, params) - there is no session object. + - auth is optional (FalkorDB runs without credentials by default), so user + and password may be None. + - no APOC: the Neo4j path does not use APOC either, so nothing to port. + + Uses MERGE so re-running is safe - nodes and edges are upserted, not + duplicated. Returns a dict with counts of nodes and edges pushed. + """ + try: + from falkordb import FalkorDB + except ImportError as e: + raise ImportError( + "falkordb SDK not installed. Run: pip install falkordb" + ) from e + + from urllib.parse import urlparse + + node_community = _node_community_map(communities) if communities else {} + + def _safe_rel(relation: str) -> str: + return re.sub(r"[^A-Z0-9_]", "_", relation.upper().replace(" ", "_").replace("-", "_")) or "RELATED_TO" + + def _safe_label(label: str) -> str: + """Sanitize a FalkorDB node label to prevent Cypher injection.""" + sanitized = re.sub(r"[^A-Za-z0-9_]", "", label) + return sanitized if sanitized else "Entity" + + parsed = urlparse(uri if "://" in uri else f"redis://{uri}") + # FalkorDB auth is optional. Only send credentials when a password is + # provided; otherwise connect anonymously and ignore any bolt-style default + # username (e.g. Neo4j's "neo4j"), which FalkorDB rejects as an unknown ACL + # user. Credentials embedded in the URI take precedence over the args. + connect_user = parsed.username or (user if password else None) + connect_password = parsed.password or (password or None) + db = FalkorDB( + host=parsed.hostname or "localhost", + port=parsed.port or 6379, + username=connect_user, + password=connect_password, + ) + graph = db.select_graph(graph_name) + nodes_pushed = 0 + edges_pushed = 0 + + for node_id, data in G.nodes(data=True): + props = { + k: v for k, v in data.items() + if isinstance(v, (str, int, float, bool)) and not k.startswith("_") + } + props["id"] = node_id + cid = node_community.get(node_id) + if cid is not None: + props["community"] = cid + ftype = _safe_label(data.get("file_type", "Entity").capitalize()) + graph.query( + f"MERGE (n:{ftype} {{id: $id}}) SET n += $props", + {"id": node_id, "props": props}, + ) + nodes_pushed += 1 + + for u, v, data in G.edges(data=True): + rel = _safe_rel(data.get("relation", "RELATED_TO")) + props = { + k: v for k, v in data.items() + if isinstance(v, (str, int, float, bool)) and not k.startswith("_") + } + graph.query( + f"MATCH (a {{id: $src}}), (b {{id: $tgt}}) " + f"MERGE (a)-[r:{rel}]->(b) SET r += $props", + {"src": u, "tgt": v, "props": props}, + ) + edges_pushed += 1 + + return {"nodes": nodes_pushed, "edges": edges_pushed} diff --git a/graphify/exporters/html.py b/graphify/exporters/html.py new file mode 100644 index 000000000..5ba61ce24 --- /dev/null +++ b/graphify/exporters/html.py @@ -0,0 +1,547 @@ +"""html — moved verbatim from graphify/export.py.""" +from __future__ import annotations + +from graphify.exporters.base import COMMUNITY_COLORS # noqa: E402,F401 +from pathlib import Path +import html as _html +from graphify.analyze import _node_community_map +import json +import networkx as nx +from graphify.security import sanitize_label + + +MAX_NODES_FOR_VIZ = 5_000 + +def _viz_node_limit() -> int: + """Return the effective viz node limit, honoring GRAPHIFY_VIZ_NODE_LIMIT env var. + + Falls back to MAX_NODES_FOR_VIZ when the env var is unset, empty, or non-integer. + Set to 0 to disable HTML viz unconditionally (useful for CI runners). + """ + import os + raw = os.environ.get("GRAPHIFY_VIZ_NODE_LIMIT") + if raw is None or not raw.strip(): + return MAX_NODES_FOR_VIZ + try: + return int(raw) + except ValueError: + return MAX_NODES_FOR_VIZ + +def _html_styles() -> str: + return """""" + +def _hyperedge_script(hyperedges_json: str) -> str: + return f"""""" + +def _html_script(nodes_json: str, edges_json: str, legend_json: str) -> str: + return f"""""" + +def to_html( + G: nx.Graph, + communities: dict[int, list[str]], + output_path: str, + community_labels: dict[int, str] | None = None, + member_counts: dict[int, int] | None = None, + node_limit: int | None = None, + learning_overlay: dict | None = None, +) -> None: + """Generate an interactive vis.js HTML visualization of the graph. + + Features: node size by degree, click-to-inspect panel, search box, + community filter, physics clustering by community, confidence-styled edges. + Raises ValueError if graph exceeds MAX_NODES_FOR_VIZ. + + If member_counts is provided (aggregated community view), node sizes are + based on community member counts rather than graph degree. + + If node_limit is set and the graph exceeds it, automatically builds an + aggregated community-level meta-graph instead of raising ValueError. + """ + limit = node_limit if node_limit is not None else _viz_node_limit() + if G.number_of_nodes() > limit: + if node_limit is not None: + # Build aggregated community meta-graph + from collections import Counter as _Counter + import networkx as _nx + print(f"Graph has {G.number_of_nodes()} nodes (above {limit} limit). Building aggregated community view...") + node_to_community = {nid: cid for cid, members in communities.items() for nid in members} + meta = _nx.Graph() + for cid, members in communities.items(): + meta.add_node(str(cid), label=(community_labels or {}).get(cid, f"Community {cid}")) + edge_counts = _Counter() + for u, v in G.edges(): + cu, cv = node_to_community.get(u), node_to_community.get(v) + if cu is not None and cv is not None and cu != cv: + edge_counts[(min(cu, cv), max(cu, cv))] += 1 + for (cu, cv), w in edge_counts.items(): + meta.add_edge(str(cu), str(cv), weight=w, + relation=f"{w} cross-community edges", confidence="AGGREGATED") + if meta.number_of_nodes() <= 1: + print("Single community - aggregated view not useful. Skipping graph.html.") + return + meta_communities = {cid: [str(cid)] for cid in communities} + mc = {cid: len(members) for cid, members in communities.items()} + # Remap hyperedges from semantic node IDs to community IDs + raw_hyperedges = G.graph.get("hyperedges", []) + if raw_hyperedges: + remapped = [] + for he in raw_hyperedges: + he_members = he.get("nodes", []) + comm_ids, seen = [], set() + for nid in he_members: + c = node_to_community.get(nid) + if c is None: + continue + s = str(c) + if s in seen: + continue + seen.add(s) + comm_ids.append(s) + if len(comm_ids) < 2: + continue + remapped.append({ + "id": he.get("id", ""), + "label": he.get("label") or he.get("relation", "").replace("_", " "), + "nodes": comm_ids, + }) + meta.graph["hyperedges"] = remapped + to_html(meta, meta_communities, output_path, + community_labels=community_labels, member_counts=mc) + print(f"graph.html written (aggregated: {meta.number_of_nodes()} community nodes, {meta.number_of_edges()} cross-community edges)") + print("Tip: run with --obsidian for full node-level detail.") + return + raise ValueError( + f"Graph has {G.number_of_nodes()} nodes - too large for HTML viz " + f"(limit: {limit}). Use --no-viz, raise GRAPHIFY_VIZ_NODE_LIMIT, " + f"or reduce input size." + ) + + node_community = _node_community_map(communities) + degree = dict(G.degree()) + max_deg = max(degree.values(), default=1) or 1 + max_mc = (max(member_counts.values(), default=1) or 1) if member_counts else 1 + + # Work-memory overlay (derived sidecar). When not passed explicitly, load it + # best-effort from the sibling .graphify_learning.json next to the output + # graph.html (which lives beside graph.json). Empty/missing => no learning + # fields, so the un-annotated render is byte-identical to pre-feature. + if learning_overlay is None: + learning_overlay = {} + try: + from graphify.reflect import load_learning_overlay as _llo + learning_overlay = _llo(Path(output_path)) + except Exception: + learning_overlay = {} + # Status -> ring color. preferred=green, contested=amber. Tentative gets no + # ring (it's not yet trustworthy enough to highlight in the map). + _RING = {"preferred": "#22c55e", "contested": "#f59e0b"} + + # Build nodes list for vis.js + vis_nodes = [] + for node_id, data in G.nodes(data=True): + cid = node_community.get(node_id, 0) + color = COMMUNITY_COLORS[cid % len(COMMUNITY_COLORS)] + label = sanitize_label(data.get("label", node_id)) + deg = degree.get(node_id, 1) + if member_counts: + mc = member_counts.get(cid, 1) + size = 10 + 30 * (mc / max_mc) + font_size = 12 + else: + size = 10 + 30 * (deg / max_deg) + # Only show label for high-degree nodes by default; others show on hover + font_size = 12 if deg >= max_deg * 0.15 else 0 + node = { + "id": node_id, + "label": label, + "color": {"background": color, "border": color, "highlight": {"background": "#ffffff", "border": color}}, + "size": round(size, 1), + "font": {"size": font_size, "color": "#ffffff"}, + "title": _html.escape(label), + "community": cid, + "community_name": sanitize_label((community_labels or {}).get(cid, f"Community {cid}")), + "source_file": sanitize_label(str(data.get("source_file") or "")), + "file_type": data.get("file_type", ""), + "degree": deg, + } + # Conditional learning fields — only present for annotated nodes, so + # un-annotated output keeps the exact pre-feature node dict shape. + entry = learning_overlay.get(str(node_id)) if learning_overlay else None + if entry: + status = sanitize_label(str(entry.get("status", ""))) + stale = bool(entry.get("stale")) + node["learning_status"] = status + node["learning_stale"] = stale + ring = _RING.get(status) + if ring: + # Status-colored ring via the border; stale => desaturated + + # dashed (vis.js supports per-node `shapeProperties.borderDashes`). + if stale: + ring = "#9ca3af" + node["shapeProperties"] = {"borderDashes": [4, 4]} + node["borderWidth"] = 3 + node["color"] = { + "background": color, "border": ring, + "highlight": {"background": "#ffffff", "border": ring}, + } + # Lesson line appended to the hover title. + if status == "contested": + lesson = f"Lesson: contested (useful {entry.get('uses', 0)} / dead-end {entry.get('neg', 0)})" + elif status == "preferred": + lesson = f"Lesson: preferred source ({entry.get('uses', 0)} useful, score={entry.get('score', 0)})" + else: + lesson = f"Lesson: {status} ({entry.get('uses', 0)} useful)" + if stale: + lesson += " [code changed — re-verify]" + node["title"] = _html.escape(label) + "\n" + _html.escape(sanitize_label(lesson)) + vis_nodes.append(node) + + # Build edges list. Restore original edge direction from _src/_tgt + # (stashed by build.py for exactly this reason): undirected NetworkX + # canonicalizes endpoint order, which would otherwise flip the arrow + # for `calls` and `rationale_for` in the rendered graph (#563). + vis_edges = [] + for u, v, data in G.edges(data=True): + confidence = data.get("confidence", "EXTRACTED") + relation = data.get("relation", "") + true_src = data.get("_src", u) + true_tgt = data.get("_tgt", v) + vis_edges.append({ + "from": true_src, + "to": true_tgt, + "label": relation, + "title": _html.escape(f"{relation} [{confidence}]"), + "dashes": confidence != "EXTRACTED", + "width": 2 if confidence == "EXTRACTED" else 1, + "color": {"opacity": 0.7 if confidence == "EXTRACTED" else 0.35}, + "confidence": confidence, + }) + + # Build community legend data + legend_data = [] + for cid in sorted((community_labels or {}).keys()): + color = COMMUNITY_COLORS[cid % len(COMMUNITY_COLORS)] + lbl = _html.escape(sanitize_label((community_labels or {}).get(cid, f"Community {cid}"))) + n = member_counts.get(cid, len(communities.get(cid, []))) if member_counts else len(communities.get(cid, [])) + legend_data.append({"cid": cid, "color": color, "label": lbl, "count": n}) + + # Escape sequences so embedded JSON cannot break out of the script tag + def _js_safe(obj) -> str: + return json.dumps(obj).replace(" + + + +graphify - {title} + +{_html_styles()} + + +
+ +{_html_script(nodes_json, edges_json, legend_json)} +{_hyperedge_script(hyperedges_json)} + +""" + + Path(output_path).write_text(html, encoding="utf-8") # nosec From e8a22893c92afa2efd4cd46432c4103b69e9856b Mon Sep 17 00:00:00 2001 From: tpateeq Date: Thu, 9 Jul 2026 01:31:17 +0530 Subject: [PATCH 042/292] refactor(extract): migrate verilog and markdown extractors to extractors/ Continues the extract.py -> extractors/ split. Both are self-contained bespoke extractors (verified: closure-private, byte-identical verbatim move, facade identity + _DISPATCH resolution intact): - extractors/verilog.py: extract_verilog + SystemVerilog helpers (_sv_*, _augment_systemverilog_semantics) and _SV_* constants. - extractors/markdown.py: extract_markdown + _resolve_markdown_link and _MD_* regexes. extract.py 13,121 -> 12,632 LOC. Full suite unchanged: 3036 passed, 29 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/extract.py | 493 +------------------------------- graphify/extractors/__init__.py | 4 + graphify/extractors/markdown.py | 176 ++++++++++++ graphify/extractors/verilog.py | 329 +++++++++++++++++++++ 4 files changed, 511 insertions(+), 491 deletions(-) create mode 100644 graphify/extractors/markdown.py create mode 100644 graphify/extractors/verilog.py diff --git a/graphify/extract.py b/graphify/extract.py index 332da7e1f..8380e0dea 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -41,6 +41,7 @@ from graphify.extractors.fortran import _cpp_preprocess, extract_fortran # noqa: F401 from graphify.extractors.go import extract_go # noqa: F401 from graphify.extractors.json_config import extract_json # noqa: F401 +from graphify.extractors.markdown import extract_markdown # noqa: F401 from graphify.extractors.pascal_forms import extract_delphi_form, extract_lazarus_form # noqa: F401 from graphify.extractors.powershell import extract_powershell, extract_powershell_manifest # noqa: F401 from graphify.extractors.razor import extract_razor # noqa: F401 @@ -48,6 +49,7 @@ from graphify.extractors.sln import extract_sln # noqa: F401 from graphify.extractors.sql import extract_sql # noqa: F401 from graphify.extractors.terraform import extract_terraform # noqa: F401 +from graphify.extractors.verilog import extract_verilog # noqa: F401 from graphify.extractors.zig import extract_zig # noqa: F401 from graphify.security import sanitize_metadata from graphify.paths import disambiguate_ambiguous_candidates @@ -6406,334 +6408,8 @@ def extract_php(path: Path) -> dict: return _extract_generic(path, _PHP_CONFIG) -def _sv_first_identifier(node, source: bytes) -> str | None: - """First `simple_identifier` under node in pre-order, or None. - - tree-sitter-verilog 1.0.3 nests declaration names a few levels deep instead - of exposing a `name` field. Scope the search to the right child node (e.g. - `function_identifier`) or this returns the return-type instead of the name. - """ - if node is None: - return None - for child in node.children: - if child.type == "simple_identifier": - return _read_text(child, source) - found = _sv_first_identifier(child, source) - if found: - return found - return None - - -def _sv_child(node, type_name: str) -> object | None: - if node is None: - return None - for child in node.children: - if child.type == type_name: - return child - return None - - -_SV_BUILTIN_TYPES = frozenset({ - "bit", "logic", "reg", "wire", "int", "integer", "shortint", "longint", - "byte", "time", "real", "shortreal", "void", "string", "type", "event", - "mailbox", "semaphore", "process", "chandle", -}) - -_SV_NON_TYPE_WORDS = frozenset({ - "return", "if", "else", "for", "foreach", "while", "case", "begin", "end", - "function", "task", "class", "endclass", "endfunction", "endtask", -}) - # One level of balanced parens (e.g. `Foo #(Bar #(int))`) — bounded so malformed # input cannot trigger pathological backtracking. -_SV_PARENS_INNER = r"(?:[^()]|\([^()]*\))*" -_SV_PARENS = r"\(" + _SV_PARENS_INNER + r"\)" - -_SV_FUNC_RE = re.compile( - r"\bfunction\s+([A-Za-z_]\w*(?:\s*#\s*" + _SV_PARENS + r")?)\s+(\w+)\s*" - r"\((" + _SV_PARENS_INNER + r")\)\s*;", - re.MULTILINE, -) - -_SV_PARAM_RE = re.compile( - r"\s*(?:input|output|inout|ref|const\s+ref)?\s*" - r"([A-Za-z_]\w*(?:\s*#\s*" + _SV_PARENS + r")?)\s+\w+" -) - - -def _sv_strip_comments(text: str) -> str: - text = re.sub(r"/\*.*?\*/", "", text, flags=re.DOTALL) - return re.sub(r"//.*", "", text) - - -def _sv_split_type_list(text: str) -> list[str]: - parts: list[str] = [] - depth = 0 - start = 0 - for idx, ch in enumerate(text): - if ch == "(": - depth += 1 - elif ch == ")": - depth = max(0, depth - 1) - elif ch == "," and depth == 0: - item = text[start:idx].strip() - if item: - parts.append(item) - start = idx + 1 - item = text[start:].strip() - if item: - parts.append(item) - return parts - - -def _sv_collect_type_refs(type_text: str, generic: bool = False, - skip: frozenset[str] = frozenset()) -> list[tuple[str, str]]: - refs: list[tuple[str, str]] = [] - text = type_text.strip() - if not text: - return refs - head = re.match(r"([A-Za-z_]\w*)", text) - if head: - name = head.group(1) - # `skip` carries the enclosing class's `#(type T = ...)` parameters so - # they are not mistaken for referenced types. - if name not in _SV_BUILTIN_TYPES and name not in _SV_NON_TYPE_WORDS and name not in skip: - refs.append((name, "generic_arg" if generic else "type")) - params = re.search(r"#\s*\((" + _SV_PARENS_INNER + r")\)", text) - if params: - for arg in _sv_split_type_list(params.group(1)): - refs.extend(_sv_collect_type_refs(arg, generic=True, skip=skip)) - return refs - - -def _augment_systemverilog_semantics( - raw: str, - stem: str, - str_path: str, - file_nid: str, - nodes: list[dict], - edges: list[dict], - seen_ids: set[str], -) -> None: - label_to_nid = {node["label"]: node["id"] for node in nodes} - - def line_for(offset: int) -> int: - return raw.count("\n", 0, offset) + 1 - - def add_node(nid: str, label: str, line: int) -> None: - if nid not in seen_ids: - seen_ids.add(nid) - nodes.append({"id": nid, "label": label, "file_type": "code", - "source_file": str_path, "source_location": f"L{line}", - "confidence_score": 1.0}) - label_to_nid[label] = nid - - def ensure_type(label: str, line: int) -> str: - if label in label_to_nid: - return label_to_nid[label] - nid = _make_id(stem, label) - add_node(nid, label, line) - return nid - - def add_edge(src: str, target_label: str, relation: str, line: int, context: str | None = None) -> None: - tgt = ensure_type(target_label, line) - edge = {"source": src, "target": tgt, "relation": relation, - "confidence": "EXTRACTED", "confidence_score": 1.0, - "source_file": str_path, "source_location": f"L{line}", "weight": 1.0} - if context: - edge["context"] = context - edges.append(edge) - - text = _sv_strip_comments(raw) - # Consuming `endclass` (rather than a lookahead) makes each match own its - # terminator, so back-to-back or malformed classes cannot bleed bodies. - class_re = re.compile( - r"\b(?:(interface)\s+)?class\s+(\w+)([^;{]*)\s*;(.*?)\bendclass\b", - re.DOTALL, - ) - for match in class_re.finditer(text): - class_name = match.group(2) - header = match.group(3) or "" - body = match.group(4) or "" - line = line_for(match.start()) - # `#(type T = Payload)` declares `T` as a class type parameter, not a - # referenced type — collect these to skip below. - type_params = frozenset(re.findall(r"\btype\s+(\w+)", header)) - class_nid = _make_id(stem, class_name) - add_node(class_nid, class_name, line) - edges.append({"source": file_nid, "target": class_nid, "relation": "defines", - "confidence": "EXTRACTED", "confidence_score": 1.0, - "source_file": str_path, "source_location": f"L{line}", "weight": 1.0}) - - ext = re.search(r"\bextends\s+(\w+)", header) - if ext: - add_edge(class_nid, ext.group(1), "inherits", line) - impl = re.search(r"\bimplements\s+([^;{]+)", header) - if impl: - for iface_name in _sv_split_type_list(impl.group(1)): - add_edge(class_nid, iface_name.split("#", 1)[0].strip(), "implements", line) - - body_without_functions = re.sub( - r"\bfunction\b.*?\bendfunction\b", - lambda m: "\n" * m.group(0).count("\n"), - body, - flags=re.DOTALL, - ) - # Optional leading class-property qualifiers (rand/local/protected/etc.) - # must be consumed: otherwise a qualified field like `rand Config x;` - # (three tokens) fails the ` ;` shape and its type reference - # is silently dropped. - for field in re.finditer(r"^\s*(?:(?:rand|randc|local|protected|static|const|automatic|var)\s+)*([A-Za-z_]\w*(?:\s*#\s*\([^;]+?\))?)\s+\w+\s*;", body_without_functions, re.MULTILINE): - # Count to the start of the type token (group 1), not the match - # start: `^\s*` consumes the leading newline(s), so field.start() - # would resolve to the class's line instead of the field's. - field_line = line + body_without_functions.count("\n", 0, field.start(1)) - for ref_name, role in _sv_collect_type_refs(field.group(1), skip=type_params): - add_edge(class_nid, ref_name, "references", field_line, "generic_arg" if role == "generic_arg" else "field") - - for fm in _SV_FUNC_RE.finditer(body): - return_type, func_name, params = fm.group(1), fm.group(2), fm.group(3) - func_line = line + body.count("\n", 0, fm.start()) - func_nid = _make_id(class_nid, func_name) - add_node(func_nid, func_name, func_line) - edges.append({"source": class_nid, "target": func_nid, "relation": "method", - "confidence": "EXTRACTED", "confidence_score": 1.0, - "source_file": str_path, "source_location": f"L{func_line}", "weight": 1.0}) - for ref_name, role in _sv_collect_type_refs(return_type, skip=type_params): - add_edge(func_nid, ref_name, "references", func_line, "generic_arg" if role == "generic_arg" else "return_type") - for param in _sv_split_type_list(params): - pm = _SV_PARAM_RE.match(param) - if not pm: - continue - for ref_name, role in _sv_collect_type_refs(pm.group(1), skip=type_params): - add_edge(func_nid, ref_name, "references", func_line, "generic_arg" if role == "generic_arg" else "parameter_type") - - -def extract_verilog(path: Path) -> dict: - """Extract modules, functions, tasks, package imports, instantiations, and - SystemVerilog class semantics (inherits/implements edges, field/parameter/ - return-type references) from .v/.sv files.""" - try: - import tree_sitter_verilog as tsverilog - from tree_sitter import Language, Parser - except ImportError: - return {"nodes": [], "edges": [], "error": "tree_sitter_verilog not installed"} - - try: - language = Language(tsverilog.language()) - parser = Parser(language) - source = path.read_bytes() - tree = parser.parse(source) - root = tree.root_node - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - stem = _file_stem(path) - str_path = str(path) - nodes: list[dict] = [] - edges: list[dict] = [] - seen_ids: set[str] = set() - - def add_node(nid: str, label: str, line: int) -> None: - if nid not in seen_ids: - seen_ids.add(nid) - nodes.append({"id": nid, "label": label, "file_type": "code", - "source_file": str_path, "source_location": f"L{line}", - "confidence_score": 1.0}) - - def add_edge(src: str, tgt: str, relation: str, line: int, - confidence: str = "EXTRACTED", score: float = 1.0) -> None: - edges.append({"source": src, "target": tgt, "relation": relation, - "confidence": confidence, "confidence_score": score, - "source_file": str_path, "source_location": f"L{line}", "weight": 1.0}) - - file_nid = _make_id(str(path)) - add_node(file_nid, path.name, 1) - - def walk(node, module_nid: str | None = None) -> None: - t = node.type - - # SystemVerilog class bodies are handled by _augment_systemverilog_semantics - # (regex over source text). Skip their subtrees so in-class methods are not - # double-emitted here — and with the wrong, return-type-derived name. - if t in ("class_declaration", "interface_class_declaration"): - return - - if t == "module_declaration": - mod_name = _sv_first_identifier(_sv_child(node, "module_header"), source) - if mod_name: - line = node.start_point[0] + 1 - nid = _make_id(stem, mod_name) - add_node(nid, mod_name, line) - add_edge(file_nid, nid, "defines", line) - for child in node.children: - walk(child, nid) - return - - # `function_prototype` only appears inside class/interface-class bodies - # (skipped above) and nests its name differently; it is intentionally not - # handled here. - elif t == "function_declaration": - fn_body = _sv_child(node, "function_body_declaration") - func_name = _sv_first_identifier(_sv_child(fn_body, "function_identifier"), source) - if func_name: - line = node.start_point[0] + 1 - parent = module_nid or file_nid - nid = _make_id(parent, func_name) - add_node(nid, f"{func_name}()", line) - add_edge(parent, nid, "contains", line) - - elif t == "task_declaration": - tk_body = _sv_child(node, "task_body_declaration") - task_name = _sv_first_identifier(_sv_child(tk_body, "task_identifier"), source) - if task_name: - line = node.start_point[0] + 1 - parent = module_nid or file_nid - nid = _make_id(parent, task_name) - add_node(nid, task_name, line) - add_edge(parent, nid, "contains", line) - - elif t == "package_import_declaration": - for child in node.children: - if child.type == "package_import_item": - pkg_text = _read_text(child, source) - pkg_name = pkg_text.split("::")[0].strip() - if pkg_name: - line = node.start_point[0] + 1 - tgt_nid = _make_id(pkg_name) - add_node(tgt_nid, pkg_name, line) - src_nid = module_nid or file_nid - add_edge(src_nid, tgt_nid, "imports_from", line) - - elif t in ("module_instantiation", "checker_instantiation"): - # `leaf u_leaf();` parses as checker_instantiation in 1.0.3; - # module_instantiation (when it occurs) exposes a `module_type` field. - # Both reduce to the first identifier under the node — the instantiated - # type, not the instance name (which appears later). - if module_nid: - type_node = node.child_by_field_name("module_type") - inst_type = (_read_text(type_node, source).strip() if type_node - else _sv_first_identifier(node, source)) - if inst_type: - line = node.start_point[0] + 1 - tgt_nid = _make_id(inst_type) - add_node(tgt_nid, inst_type, line) - add_edge(module_nid, tgt_nid, "instantiates", line) - - for child in node.children: - walk(child, module_nid) - - walk(root) - _augment_systemverilog_semantics( - source.decode("utf-8", errors="replace"), - stem, - str_path, - file_nid, - nodes, - edges, - seen_ids, - ) - return {"nodes": nodes, "edges": edges} def extract_lua(path: Path) -> dict: @@ -10261,177 +9937,12 @@ def walk_calls(n) -> None: # Inline markdown link: [text](target "optional title"). The negative lookbehind # excludes images (![alt](src)). The target stops at whitespace/closing paren so # an optional "title" after the URL is dropped; an optional <...> wrapper is too. -_MD_INLINE_LINK_RE = re.compile(r'(?]+)>?(?:\s+[^)]*)?\)') # Reference-style link definition line: [label]: target "optional title" -_MD_REF_DEF_RE = re.compile(r'^\s{0,3}\[[^\]]+\]:\s*]+)>?') # Obsidian-style wikilink: [[target]] / [[target|alias]] / [[target#anchor]]. -_MD_WIKILINK_RE = re.compile(r'(? "Path | None": - """Resolve a markdown link target to the absolute path of a sibling document. - - Returns the resolved (normalized, not necessarily existing) path when the - target is a *local* relative/absolute file-path link to a document, or None - when it should be skipped: external URLs (http/https/mailto/protocol- - relative/data), pure in-page anchors (``#section``), and links to non-doc - file types (code/assets are handled by their own extractors). - - The anchor fragment (``#section``) and query (``?x=1``) are stripped before - resolution so ``./repo.md#setup`` resolves to the same node as ``./repo.md``. - Extension-less targets (typical of wikilinks) are treated as sibling ``.md``. - """ - target = raw.strip() - if not target: - return None - # Drop anchor / query so #section links still resolve to the target doc. - target = target.split("#", 1)[0].split("?", 1)[0].strip() - if not target: - return None - low = target.lower() - if "://" in target or low.startswith(("mailto:", "tel:", "//", "data:")): - return None - suffix = Path(target).suffix.lower() - if suffix == "": - target = target + ".md" - suffix = ".md" - if suffix not in _MD_LINKABLE_EXTS: - return None - candidate = Path(target) - if not candidate.is_absolute(): - candidate = source_dir / candidate - return Path(os.path.normpath(str(candidate))) - - -def extract_markdown(path: Path) -> dict: - """Extract structural nodes and edges from a Markdown file. - - Produces nodes for: - - The file itself - - Each heading (# / ## / ### etc.) - - Produces edges for: - - file --contains--> heading - - parent heading --contains--> child heading (nesting by level) - - heading --references--> other node (when backtick `Name` matches a known pattern) - - file --references--> linked document, for inline ``[text](./other.md)``, - reference-style ``[label]: ./other.md`` and ``[[wikilink]]`` links, so a - hub doc (``index.md`` / ``table-of-contents.md``) becomes a real hub node - instead of an under-connected orphan (#1376). The target node ID is built - from the resolved target path with the same recipe as the target file's - own node, so the edge merges into that node (no ghost node). External - URLs, in-page anchors, images and non-document targets are skipped. - - Fenced code blocks (``` ... ```) are skipped during parsing so their - contents don't get treated as headings, but no node is emitted for - them — they were always orphans (only a single contains edge to the - parent doc) and inflated the disconnected-component count (#1077). - - No tree-sitter dependency — pure line-by-line parsing. - """ - try: - source = path.read_text(encoding="utf-8", errors="replace") - except Exception as e: - return {"nodes": [], "edges": [], "error": str(e)} - - stem = _file_stem(path) - str_path = str(path) - nodes: list[dict] = [] - edges: list[dict] = [] - seen_ids: set[str] = set() - - def add_node(nid: str, label: str, line: int, file_type: str = "document") -> None: - if nid not in seen_ids: - seen_ids.add(nid) - nodes.append({"id": nid, "label": label, "file_type": file_type, - "source_file": str_path, "source_location": f"L{line}"}) - - def add_edge(src: str, tgt: str, relation: str, line: int, - confidence: str = "EXTRACTED", weight: float = 1.0) -> None: - edges.append({"source": src, "target": tgt, "relation": relation, - "confidence": confidence, "source_file": str_path, - "source_location": f"L{line}", "weight": weight}) - - file_nid = _make_id(str(path)) - add_node(file_nid, path.name, 1) - - source_dir = path.parent - # Dedup link edges by resolved target node so a hub doc that links to the - # same sibling many times yields one edge, not N (keeps weights meaningful). - linked_targets: set[str] = set() - - def add_link(raw: str, line: int) -> None: - resolved = _resolve_markdown_link(raw, source_dir) - if resolved is None: - return - # Build the target ID with the SAME recipe as the target file's own - # node (_make_id(str(path)) at extract time, canonicalized to - # _file_node_id(rel) by the extract() post-pass). Using the absolute - # resolved path means both endpoints get remapped identically, so the - # edge merges into the existing doc node instead of spawning a ghost. - tgt_nid = _make_id(str(resolved)) - if tgt_nid == file_nid or tgt_nid in linked_targets: - return - linked_targets.add(tgt_nid) - add_edge(file_nid, tgt_nid, "references", line) - - # Track heading stack for nesting: [(level, nid), ...] - heading_stack: list[tuple[int, str]] = [] - in_code_block = False - - lines = source.splitlines() - for line_num_0, line_text in enumerate(lines): - line_num = line_num_0 + 1 - - # Skip over fenced code blocks so their contents are not parsed as - # headings, but do not emit nodes/edges for them (#1077). - stripped = line_text.strip() - if stripped.startswith("```"): - in_code_block = not in_code_block - continue - - if in_code_block: - continue - - # Markdown links -> document references (#1376). Scanned on every - # non-fenced line (including heading lines, which the heading branch - # below `continue`s past) so links anywhere in the doc are captured. - for m in _MD_INLINE_LINK_RE.finditer(line_text): - add_link(m.group(1), line_num) - for m in _MD_WIKILINK_RE.finditer(line_text): - add_link(m.group(1), line_num) - ref_def = _MD_REF_DEF_RE.match(line_text) - if ref_def: - add_link(ref_def.group(1), line_num) - - # Detect headings: # Heading, ## Heading, etc. - heading_match = re.match(r'^(#{1,6})\s+(.+)', line_text) - if heading_match: - level = len(heading_match.group(1)) - title = heading_match.group(2).strip() - h_nid = _make_id(stem, title) - # Avoid duplicate heading IDs by appending line number - if h_nid in seen_ids: - h_nid = _make_id(stem, title, str(line_num)) - add_node(h_nid, title, line_num) - - # Pop headings at same or deeper level - while heading_stack and heading_stack[-1][0] >= level: - heading_stack.pop() - - # Connect to parent heading or file - parent = heading_stack[-1][1] if heading_stack else file_nid - add_edge(parent, h_nid, "contains", line_num) - - heading_stack.append((level, h_nid)) - continue - - return {"nodes": nodes, "edges": edges, "input_tokens": 0, "output_tokens": 0} # ── Pascal / Delphi extractor ───────────────────────────────────────────────── diff --git a/graphify/extractors/__init__.py b/graphify/extractors/__init__.py index 46b484b3d..883e1181e 100644 --- a/graphify/extractors/__init__.py +++ b/graphify/extractors/__init__.py @@ -19,6 +19,7 @@ from graphify.extractors.fortran import extract_fortran from graphify.extractors.go import extract_go from graphify.extractors.json_config import extract_json +from graphify.extractors.markdown import extract_markdown from graphify.extractors.pascal_forms import extract_delphi_form, extract_lazarus_form from graphify.extractors.powershell import extract_powershell, extract_powershell_manifest from graphify.extractors.razor import extract_razor @@ -26,6 +27,7 @@ from graphify.extractors.sln import extract_sln from graphify.extractors.sql import extract_sql from graphify.extractors.terraform import extract_terraform +from graphify.extractors.verilog import extract_verilog from graphify.extractors.zig import extract_zig LANGUAGE_EXTRACTORS: dict[str, Callable[[Path], dict]] = { @@ -43,6 +45,7 @@ "go": extract_go, "json": extract_json, "lazarus_form": extract_lazarus_form, + "markdown": extract_markdown, "powershell": extract_powershell, "powershell_manifest": extract_powershell_manifest, "razor": extract_razor, @@ -50,5 +53,6 @@ "sln": extract_sln, "sql": extract_sql, "terraform": extract_terraform, + "verilog": extract_verilog, "zig": extract_zig, } diff --git a/graphify/extractors/markdown.py b/graphify/extractors/markdown.py new file mode 100644 index 000000000..d6d45ac8f --- /dev/null +++ b/graphify/extractors/markdown.py @@ -0,0 +1,176 @@ +"""Markdown extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + +import re +import os + +from pathlib import Path +from graphify.extractors.base import _file_stem, _make_id + + +_MD_INLINE_LINK_RE = re.compile(r'(?]+)>?(?:\s+[^)]*)?\)') + +_MD_REF_DEF_RE = re.compile(r'^\s{0,3}\[[^\]]+\]:\s*]+)>?') + +_MD_WIKILINK_RE = re.compile(r'(? "Path | None": + """Resolve a markdown link target to the absolute path of a sibling document. + + Returns the resolved (normalized, not necessarily existing) path when the + target is a *local* relative/absolute file-path link to a document, or None + when it should be skipped: external URLs (http/https/mailto/protocol- + relative/data), pure in-page anchors (``#section``), and links to non-doc + file types (code/assets are handled by their own extractors). + + The anchor fragment (``#section``) and query (``?x=1``) are stripped before + resolution so ``./repo.md#setup`` resolves to the same node as ``./repo.md``. + Extension-less targets (typical of wikilinks) are treated as sibling ``.md``. + """ + target = raw.strip() + if not target: + return None + # Drop anchor / query so #section links still resolve to the target doc. + target = target.split("#", 1)[0].split("?", 1)[0].strip() + if not target: + return None + low = target.lower() + if "://" in target or low.startswith(("mailto:", "tel:", "//", "data:")): + return None + suffix = Path(target).suffix.lower() + if suffix == "": + target = target + ".md" + suffix = ".md" + if suffix not in _MD_LINKABLE_EXTS: + return None + candidate = Path(target) + if not candidate.is_absolute(): + candidate = source_dir / candidate + return Path(os.path.normpath(str(candidate))) + +def extract_markdown(path: Path) -> dict: + """Extract structural nodes and edges from a Markdown file. + + Produces nodes for: + - The file itself + - Each heading (# / ## / ### etc.) + + Produces edges for: + - file --contains--> heading + - parent heading --contains--> child heading (nesting by level) + - heading --references--> other node (when backtick `Name` matches a known pattern) + - file --references--> linked document, for inline ``[text](./other.md)``, + reference-style ``[label]: ./other.md`` and ``[[wikilink]]`` links, so a + hub doc (``index.md`` / ``table-of-contents.md``) becomes a real hub node + instead of an under-connected orphan (#1376). The target node ID is built + from the resolved target path with the same recipe as the target file's + own node, so the edge merges into that node (no ghost node). External + URLs, in-page anchors, images and non-document targets are skipped. + + Fenced code blocks (``` ... ```) are skipped during parsing so their + contents don't get treated as headings, but no node is emitted for + them — they were always orphans (only a single contains edge to the + parent doc) and inflated the disconnected-component count (#1077). + + No tree-sitter dependency — pure line-by-line parsing. + """ + try: + source = path.read_text(encoding="utf-8", errors="replace") + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + stem = _file_stem(path) + str_path = str(path) + nodes: list[dict] = [] + edges: list[dict] = [] + seen_ids: set[str] = set() + + def add_node(nid: str, label: str, line: int, file_type: str = "document") -> None: + if nid not in seen_ids: + seen_ids.add(nid) + nodes.append({"id": nid, "label": label, "file_type": file_type, + "source_file": str_path, "source_location": f"L{line}"}) + + def add_edge(src: str, tgt: str, relation: str, line: int, + confidence: str = "EXTRACTED", weight: float = 1.0) -> None: + edges.append({"source": src, "target": tgt, "relation": relation, + "confidence": confidence, "source_file": str_path, + "source_location": f"L{line}", "weight": weight}) + + file_nid = _make_id(str(path)) + add_node(file_nid, path.name, 1) + + source_dir = path.parent + # Dedup link edges by resolved target node so a hub doc that links to the + # same sibling many times yields one edge, not N (keeps weights meaningful). + linked_targets: set[str] = set() + + def add_link(raw: str, line: int) -> None: + resolved = _resolve_markdown_link(raw, source_dir) + if resolved is None: + return + # Build the target ID with the SAME recipe as the target file's own + # node (_make_id(str(path)) at extract time, canonicalized to + # _file_node_id(rel) by the extract() post-pass). Using the absolute + # resolved path means both endpoints get remapped identically, so the + # edge merges into the existing doc node instead of spawning a ghost. + tgt_nid = _make_id(str(resolved)) + if tgt_nid == file_nid or tgt_nid in linked_targets: + return + linked_targets.add(tgt_nid) + add_edge(file_nid, tgt_nid, "references", line) + + # Track heading stack for nesting: [(level, nid), ...] + heading_stack: list[tuple[int, str]] = [] + in_code_block = False + + lines = source.splitlines() + for line_num_0, line_text in enumerate(lines): + line_num = line_num_0 + 1 + + # Skip over fenced code blocks so their contents are not parsed as + # headings, but do not emit nodes/edges for them (#1077). + stripped = line_text.strip() + if stripped.startswith("```"): + in_code_block = not in_code_block + continue + + if in_code_block: + continue + + # Markdown links -> document references (#1376). Scanned on every + # non-fenced line (including heading lines, which the heading branch + # below `continue`s past) so links anywhere in the doc are captured. + for m in _MD_INLINE_LINK_RE.finditer(line_text): + add_link(m.group(1), line_num) + for m in _MD_WIKILINK_RE.finditer(line_text): + add_link(m.group(1), line_num) + ref_def = _MD_REF_DEF_RE.match(line_text) + if ref_def: + add_link(ref_def.group(1), line_num) + + # Detect headings: # Heading, ## Heading, etc. + heading_match = re.match(r'^(#{1,6})\s+(.+)', line_text) + if heading_match: + level = len(heading_match.group(1)) + title = heading_match.group(2).strip() + h_nid = _make_id(stem, title) + # Avoid duplicate heading IDs by appending line number + if h_nid in seen_ids: + h_nid = _make_id(stem, title, str(line_num)) + add_node(h_nid, title, line_num) + + # Pop headings at same or deeper level + while heading_stack and heading_stack[-1][0] >= level: + heading_stack.pop() + + # Connect to parent heading or file + parent = heading_stack[-1][1] if heading_stack else file_nid + add_edge(parent, h_nid, "contains", line_num) + + heading_stack.append((level, h_nid)) + continue + + return {"nodes": nodes, "edges": edges, "input_tokens": 0, "output_tokens": 0} diff --git a/graphify/extractors/verilog.py b/graphify/extractors/verilog.py new file mode 100644 index 000000000..2f5fea49b --- /dev/null +++ b/graphify/extractors/verilog.py @@ -0,0 +1,329 @@ +"""Verilog extractor. Moved verbatim from graphify/extract.py.""" +from __future__ import annotations + +import re + +from pathlib import Path +from graphify.extractors.base import _file_stem, _make_id, _read_text + + +def _sv_first_identifier(node, source: bytes) -> str | None: + """First `simple_identifier` under node in pre-order, or None. + + tree-sitter-verilog 1.0.3 nests declaration names a few levels deep instead + of exposing a `name` field. Scope the search to the right child node (e.g. + `function_identifier`) or this returns the return-type instead of the name. + """ + if node is None: + return None + for child in node.children: + if child.type == "simple_identifier": + return _read_text(child, source) + found = _sv_first_identifier(child, source) + if found: + return found + return None + +def _sv_child(node, type_name: str) -> object | None: + if node is None: + return None + for child in node.children: + if child.type == type_name: + return child + return None + +_SV_BUILTIN_TYPES = frozenset({ + "bit", "logic", "reg", "wire", "int", "integer", "shortint", "longint", + "byte", "time", "real", "shortreal", "void", "string", "type", "event", + "mailbox", "semaphore", "process", "chandle", +}) + +_SV_NON_TYPE_WORDS = frozenset({ + "return", "if", "else", "for", "foreach", "while", "case", "begin", "end", + "function", "task", "class", "endclass", "endfunction", "endtask", +}) + +_SV_PARENS_INNER = r"(?:[^()]|\([^()]*\))*" + +_SV_PARENS = r"\(" + _SV_PARENS_INNER + r"\)" + +_SV_FUNC_RE = re.compile( + r"\bfunction\s+([A-Za-z_]\w*(?:\s*#\s*" + _SV_PARENS + r")?)\s+(\w+)\s*" + r"\((" + _SV_PARENS_INNER + r")\)\s*;", + re.MULTILINE, +) + +_SV_PARAM_RE = re.compile( + r"\s*(?:input|output|inout|ref|const\s+ref)?\s*" + r"([A-Za-z_]\w*(?:\s*#\s*" + _SV_PARENS + r")?)\s+\w+" +) + +def _sv_strip_comments(text: str) -> str: + text = re.sub(r"/\*.*?\*/", "", text, flags=re.DOTALL) + return re.sub(r"//.*", "", text) + +def _sv_split_type_list(text: str) -> list[str]: + parts: list[str] = [] + depth = 0 + start = 0 + for idx, ch in enumerate(text): + if ch == "(": + depth += 1 + elif ch == ")": + depth = max(0, depth - 1) + elif ch == "," and depth == 0: + item = text[start:idx].strip() + if item: + parts.append(item) + start = idx + 1 + item = text[start:].strip() + if item: + parts.append(item) + return parts + +def _sv_collect_type_refs(type_text: str, generic: bool = False, + skip: frozenset[str] = frozenset()) -> list[tuple[str, str]]: + refs: list[tuple[str, str]] = [] + text = type_text.strip() + if not text: + return refs + head = re.match(r"([A-Za-z_]\w*)", text) + if head: + name = head.group(1) + # `skip` carries the enclosing class's `#(type T = ...)` parameters so + # they are not mistaken for referenced types. + if name not in _SV_BUILTIN_TYPES and name not in _SV_NON_TYPE_WORDS and name not in skip: + refs.append((name, "generic_arg" if generic else "type")) + params = re.search(r"#\s*\((" + _SV_PARENS_INNER + r")\)", text) + if params: + for arg in _sv_split_type_list(params.group(1)): + refs.extend(_sv_collect_type_refs(arg, generic=True, skip=skip)) + return refs + +def _augment_systemverilog_semantics( + raw: str, + stem: str, + str_path: str, + file_nid: str, + nodes: list[dict], + edges: list[dict], + seen_ids: set[str], +) -> None: + label_to_nid = {node["label"]: node["id"] for node in nodes} + + def line_for(offset: int) -> int: + return raw.count("\n", 0, offset) + 1 + + def add_node(nid: str, label: str, line: int) -> None: + if nid not in seen_ids: + seen_ids.add(nid) + nodes.append({"id": nid, "label": label, "file_type": "code", + "source_file": str_path, "source_location": f"L{line}", + "confidence_score": 1.0}) + label_to_nid[label] = nid + + def ensure_type(label: str, line: int) -> str: + if label in label_to_nid: + return label_to_nid[label] + nid = _make_id(stem, label) + add_node(nid, label, line) + return nid + + def add_edge(src: str, target_label: str, relation: str, line: int, context: str | None = None) -> None: + tgt = ensure_type(target_label, line) + edge = {"source": src, "target": tgt, "relation": relation, + "confidence": "EXTRACTED", "confidence_score": 1.0, + "source_file": str_path, "source_location": f"L{line}", "weight": 1.0} + if context: + edge["context"] = context + edges.append(edge) + + text = _sv_strip_comments(raw) + # Consuming `endclass` (rather than a lookahead) makes each match own its + # terminator, so back-to-back or malformed classes cannot bleed bodies. + class_re = re.compile( + r"\b(?:(interface)\s+)?class\s+(\w+)([^;{]*)\s*;(.*?)\bendclass\b", + re.DOTALL, + ) + for match in class_re.finditer(text): + class_name = match.group(2) + header = match.group(3) or "" + body = match.group(4) or "" + line = line_for(match.start()) + # `#(type T = Payload)` declares `T` as a class type parameter, not a + # referenced type — collect these to skip below. + type_params = frozenset(re.findall(r"\btype\s+(\w+)", header)) + class_nid = _make_id(stem, class_name) + add_node(class_nid, class_name, line) + edges.append({"source": file_nid, "target": class_nid, "relation": "defines", + "confidence": "EXTRACTED", "confidence_score": 1.0, + "source_file": str_path, "source_location": f"L{line}", "weight": 1.0}) + + ext = re.search(r"\bextends\s+(\w+)", header) + if ext: + add_edge(class_nid, ext.group(1), "inherits", line) + impl = re.search(r"\bimplements\s+([^;{]+)", header) + if impl: + for iface_name in _sv_split_type_list(impl.group(1)): + add_edge(class_nid, iface_name.split("#", 1)[0].strip(), "implements", line) + + body_without_functions = re.sub( + r"\bfunction\b.*?\bendfunction\b", + lambda m: "\n" * m.group(0).count("\n"), + body, + flags=re.DOTALL, + ) + # Optional leading class-property qualifiers (rand/local/protected/etc.) + # must be consumed: otherwise a qualified field like `rand Config x;` + # (three tokens) fails the ` ;` shape and its type reference + # is silently dropped. + for field in re.finditer(r"^\s*(?:(?:rand|randc|local|protected|static|const|automatic|var)\s+)*([A-Za-z_]\w*(?:\s*#\s*\([^;]+?\))?)\s+\w+\s*;", body_without_functions, re.MULTILINE): + # Count to the start of the type token (group 1), not the match + # start: `^\s*` consumes the leading newline(s), so field.start() + # would resolve to the class's line instead of the field's. + field_line = line + body_without_functions.count("\n", 0, field.start(1)) + for ref_name, role in _sv_collect_type_refs(field.group(1), skip=type_params): + add_edge(class_nid, ref_name, "references", field_line, "generic_arg" if role == "generic_arg" else "field") + + for fm in _SV_FUNC_RE.finditer(body): + return_type, func_name, params = fm.group(1), fm.group(2), fm.group(3) + func_line = line + body.count("\n", 0, fm.start()) + func_nid = _make_id(class_nid, func_name) + add_node(func_nid, func_name, func_line) + edges.append({"source": class_nid, "target": func_nid, "relation": "method", + "confidence": "EXTRACTED", "confidence_score": 1.0, + "source_file": str_path, "source_location": f"L{func_line}", "weight": 1.0}) + for ref_name, role in _sv_collect_type_refs(return_type, skip=type_params): + add_edge(func_nid, ref_name, "references", func_line, "generic_arg" if role == "generic_arg" else "return_type") + for param in _sv_split_type_list(params): + pm = _SV_PARAM_RE.match(param) + if not pm: + continue + for ref_name, role in _sv_collect_type_refs(pm.group(1), skip=type_params): + add_edge(func_nid, ref_name, "references", func_line, "generic_arg" if role == "generic_arg" else "parameter_type") + +def extract_verilog(path: Path) -> dict: + """Extract modules, functions, tasks, package imports, instantiations, and + SystemVerilog class semantics (inherits/implements edges, field/parameter/ + return-type references) from .v/.sv files.""" + try: + import tree_sitter_verilog as tsverilog + from tree_sitter import Language, Parser + except ImportError: + return {"nodes": [], "edges": [], "error": "tree_sitter_verilog not installed"} + + try: + language = Language(tsverilog.language()) + parser = Parser(language) + source = path.read_bytes() + tree = parser.parse(source) + root = tree.root_node + except Exception as e: + return {"nodes": [], "edges": [], "error": str(e)} + + stem = _file_stem(path) + str_path = str(path) + nodes: list[dict] = [] + edges: list[dict] = [] + seen_ids: set[str] = set() + + def add_node(nid: str, label: str, line: int) -> None: + if nid not in seen_ids: + seen_ids.add(nid) + nodes.append({"id": nid, "label": label, "file_type": "code", + "source_file": str_path, "source_location": f"L{line}", + "confidence_score": 1.0}) + + def add_edge(src: str, tgt: str, relation: str, line: int, + confidence: str = "EXTRACTED", score: float = 1.0) -> None: + edges.append({"source": src, "target": tgt, "relation": relation, + "confidence": confidence, "confidence_score": score, + "source_file": str_path, "source_location": f"L{line}", "weight": 1.0}) + + file_nid = _make_id(str(path)) + add_node(file_nid, path.name, 1) + + def walk(node, module_nid: str | None = None) -> None: + t = node.type + + # SystemVerilog class bodies are handled by _augment_systemverilog_semantics + # (regex over source text). Skip their subtrees so in-class methods are not + # double-emitted here — and with the wrong, return-type-derived name. + if t in ("class_declaration", "interface_class_declaration"): + return + + if t == "module_declaration": + mod_name = _sv_first_identifier(_sv_child(node, "module_header"), source) + if mod_name: + line = node.start_point[0] + 1 + nid = _make_id(stem, mod_name) + add_node(nid, mod_name, line) + add_edge(file_nid, nid, "defines", line) + for child in node.children: + walk(child, nid) + return + + # `function_prototype` only appears inside class/interface-class bodies + # (skipped above) and nests its name differently; it is intentionally not + # handled here. + elif t == "function_declaration": + fn_body = _sv_child(node, "function_body_declaration") + func_name = _sv_first_identifier(_sv_child(fn_body, "function_identifier"), source) + if func_name: + line = node.start_point[0] + 1 + parent = module_nid or file_nid + nid = _make_id(parent, func_name) + add_node(nid, f"{func_name}()", line) + add_edge(parent, nid, "contains", line) + + elif t == "task_declaration": + tk_body = _sv_child(node, "task_body_declaration") + task_name = _sv_first_identifier(_sv_child(tk_body, "task_identifier"), source) + if task_name: + line = node.start_point[0] + 1 + parent = module_nid or file_nid + nid = _make_id(parent, task_name) + add_node(nid, task_name, line) + add_edge(parent, nid, "contains", line) + + elif t == "package_import_declaration": + for child in node.children: + if child.type == "package_import_item": + pkg_text = _read_text(child, source) + pkg_name = pkg_text.split("::")[0].strip() + if pkg_name: + line = node.start_point[0] + 1 + tgt_nid = _make_id(pkg_name) + add_node(tgt_nid, pkg_name, line) + src_nid = module_nid or file_nid + add_edge(src_nid, tgt_nid, "imports_from", line) + + elif t in ("module_instantiation", "checker_instantiation"): + # `leaf u_leaf();` parses as checker_instantiation in 1.0.3; + # module_instantiation (when it occurs) exposes a `module_type` field. + # Both reduce to the first identifier under the node — the instantiated + # type, not the instance name (which appears later). + if module_nid: + type_node = node.child_by_field_name("module_type") + inst_type = (_read_text(type_node, source).strip() if type_node + else _sv_first_identifier(node, source)) + if inst_type: + line = node.start_point[0] + 1 + tgt_nid = _make_id(inst_type) + add_node(tgt_nid, inst_type, line) + add_edge(module_nid, tgt_nid, "instantiates", line) + + for child in node.children: + walk(child, module_nid) + + walk(root) + _augment_systemverilog_semantics( + source.decode("utf-8", errors="replace"), + stem, + str_path, + file_nid, + nodes, + edges, + seen_ids, + ) + return {"nodes": nodes, "edges": edges} From 4561da20888a1d8f278455e07ff4e173c3e4dae5 Mon Sep 17 00:00:00 2001 From: tpateeq Date: Thu, 9 Jul 2026 01:36:20 +0530 Subject: [PATCH 043/292] refactor(extract): extract symbol-resolution subsystem into extractors/ MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Moves the cross-file symbol-resolution / import-resolution / decl-def-merge passes — the largest remaining self-contained subsystem in extract.py — into two new modules, verbatim: - extractors/models.py: the shared data types (LanguageConfig, the _Symbol*Fact / _SymbolResolutionFacts dataclasses) and two shared caches (_WORKSPACE_PACKAGE_CACHE, _JS_CACHE_BYPASS_SUFFIXES). Homed here so both extract.py and resolution.py import them without a cycle; the mutable workspace cache keeps a single shared object identity (only ever .clear()'d in place), verified in the smoke test. - extractors/resolution.py: 60 resolution functions (_resolve_*, _collect_*, _apply_symbol_resolution_facts, _disambiguate_colliding_node_ids, _merge_decl_def_classes, the JS/TS/Python import walkers, tsconfig/workspace resolution) and their 12 private constants. extract.py re-exports every moved name, so importers (__main__, watch, tests that reach into _JS_RESOLVE_EXTS / _resolve_cross_file_imports / the fact dataclasses) are unchanged. Import direction is strictly extract.py -> resolution -> {models, base}; AST analysis confirmed no moved non-entry symbol is referenced from outside its new module. extract.py 12,632 -> 10,270 LOC (17,054 at the start of this branch, -40%). Full suite unchanged: 3036 passed, 29 skipped; skillgen --check OK. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/extract.py | 2598 ++--------------------------- graphify/extractors/models.py | 119 ++ graphify/extractors/resolution.py | 2287 +++++++++++++++++++++++++ 3 files changed, 2524 insertions(+), 2480 deletions(-) create mode 100644 graphify/extractors/models.py create mode 100644 graphify/extractors/resolution.py diff --git a/graphify/extract.py b/graphify/extract.py index 8380e0dea..0cac5060c 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -54,6 +54,82 @@ from graphify.security import sanitize_metadata from graphify.paths import disambiguate_ambiguous_candidates +from graphify.extractors.models import LanguageConfig, _JS_CACHE_BYPASS_SUFFIXES, _NamespaceExportFact, _StarExportFact, _SymbolAliasFact, _SymbolDeclarationFact, _SymbolExportFact, _SymbolImportFact, _SymbolResolutionFacts, _SymbolUseFact, _WORKSPACE_PACKAGE_CACHE # noqa: E402,F401 + +from graphify.extractors.resolution import ( # noqa: E402,F401 + _DECLDEF_HEADER_SUFFIXES, + _DECLDEF_IMPL_SUFFIXES, + _EXPORT_CONDITION_PRIORITY, + _JS_INDEX_FILES, + _JS_PRIMITIVE_TYPES, + _JS_RESOLVE_EXTS, + _TSCONFIG_ALIAS_CACHE, + _VUE_SCRIPT_LANG_RE, + _VUE_SCRIPT_RE, + _WORKSPACE_MANIFEST_NAMES, + _apply_symbol_resolution_facts, + _augment_symbol_resolution_edges, + _collect_js_symbol_resolution_facts, + _collect_python_symbol_resolution_facts, + _contained_in_package, + _decldef_class_stem, + _disambiguate_colliding_node_ids, + _find_workspace_root, + _is_type_like_definition, + _js_call_identifier, + _js_default_export_name, + _js_default_import_name, + _js_export_clause, + _js_export_statement_is_star, + _js_exported_declaration_names, + _js_lexical_aliases, + _js_module_specifier, + _js_named_specifiers, + _js_namespace_export_name, + _js_source_path, + _js_top_level_function_bodies, + _load_tsconfig_aliases, + _load_workspace_packages, + _match_tsconfig_alias, + _merge_decl_def_classes, + _node_disambiguation_source_key, + _package_entry_candidates, + _parse_js_tree, + _parse_python_tree, + _pascal_class_stem_cache, + _pascal_project_root, + _pascal_resolve_class, + _pascal_resolve_unit, + _pascal_unit_cache, + _pnpm_workspace_globs, + _python_call_identifier, + _python_import_from_module, + _python_imported_names, + _python_top_level_function_bodies, + _read_tsconfig_aliases, + _resolve_c_include_path, + _resolve_cross_file_imports, + _resolve_cross_file_java_imports, + _resolve_export_target, + _resolve_java_type_references, + _resolve_js_import_path, + _resolve_js_import_target, + _resolve_js_module_path, + _resolve_lua_import_target, + _resolve_python_module_path, + _resolve_tsconfig_alias, + _resolve_workspace_import, + _source_key, + _strip_jsonc, + _ts_collect_type_refs, + _ts_heritage_clause_entries, + _ts_walk_class_members, + _vue_mask_non_script, + _walk_js_tree, + _walk_python_tree, + _workspace_globs, +) + _RECURSION_LIMIT = 10_000 # Language built-in globals that AST may classify as call targets when used as @@ -97,14 +173,6 @@ def _csharp_namespace_id(dotted_name: str) -> str: return f"csharp_namespace:{digest}" -_TSCONFIG_ALIAS_CACHE: dict[str, dict[str, list[str]]] = {} -_WORKSPACE_PACKAGE_CACHE: dict[str, dict[str, Path]] = {} -_WORKSPACE_MANIFEST_NAMES = ("pnpm-workspace.yaml", "package.json") -_JS_CACHE_BYPASS_SUFFIXES = {".js", ".jsx", ".mjs", ".ts", ".tsx", ".mts", ".cts", ".vue", ".svelte"} -_JS_RESOLVE_EXTS = (".ts", ".tsx", ".mts", ".cts", ".svelte", ".js", ".jsx", ".mjs") -_JS_INDEX_FILES = ("index.ts", "index.tsx", "index.svelte", "index.js", "index.jsx", "index.mjs") - - SEMANTIC_RELATIONS = frozenset({ "inherits", "implements", "mixes_in", "embeds", "references", "calls", "imports", "imports_from", "re_exports", "contains", "method", @@ -144,493 +212,13 @@ def _semantic_reference_edge( } -def _resolve_js_import_path(candidate: Path) -> Path: - """Resolve a JS/TS/Svelte import target to a local file when it exists.""" - candidate = Path(os.path.normpath(candidate)) - if candidate.is_file(): - return candidate - - # TS ESM convention: imports often spell .js/.jsx while source is .ts/.tsx. - if candidate.suffix == ".js": - ts_candidate = candidate.with_suffix(".ts") - if ts_candidate.is_file(): - return ts_candidate - elif candidate.suffix == ".jsx": - tsx_candidate = candidate.with_suffix(".tsx") - if tsx_candidate.is_file(): - return tsx_candidate - - # Append extensions to the full filename, which covers extensionless imports, - # multi-dot helpers, and Svelte 5 rune files like Foo.svelte.ts. - for ext in _JS_RESOLVE_EXTS: - with_ext = candidate.parent / f"{candidate.name}{ext}" - if with_ext.is_file(): - return with_ext - - # Only fall back to directory indexes after file candidates lose. - if candidate.is_dir(): - for index_name in _JS_INDEX_FILES: - index_candidate = candidate / index_name - if index_candidate.is_file(): - return index_candidate - - return candidate - - -def _strip_jsonc(text: str) -> str: - """Strip // line comments, /* */ block comments, and trailing commas from JSONC. - - Preserves string contents (including // and /* inside strings) by skipping over - quoted spans first. Required for tsconfig.json files generated by SvelteKit, - NestJS, Vite, T3, Astro, etc., which use JSONC by default (#700). - """ - # Remove block and line comments while leaving string literals untouched. - pattern = re.compile( - r'"(?:\\.|[^"\\])*"' # double-quoted string (with escapes) - r"|/\*.*?\*/" # /* block comment */ - r"|//[^\n]*", # // line comment - re.DOTALL, - ) - - def _replace(match: re.Match) -> str: - token = match.group(0) - if token.startswith('"'): - return token - return "" - - stripped = pattern.sub(_replace, text) - # Remove trailing commas before } or ] (allowing whitespace between). - stripped = re.sub(r",(\s*[}\]])", r"\1", stripped) - return stripped - - -def _read_tsconfig_aliases(tsconfig: Path, base_dir: Path, seen: set) -> dict[str, list[str]]: - """Recursively read path aliases from a tsconfig, following extends chains. - - Child config paths override parent. Circular extends are detected via seen set. - npm package configs (e.g. @tsconfig/svelte) are skipped since they're not on disk. - Handles JSONC (comments + trailing commas) which is the default tsconfig format - for SvelteKit, NestJS, Vite, T3, Astro, etc. (#700). - """ - if str(tsconfig) in seen: - return {} - seen.add(str(tsconfig)) - try: - raw = tsconfig.read_text(encoding="utf-8") - except Exception as e: - print(f" warning: could not read {tsconfig} ({type(e).__name__}: {e})", file=sys.stderr, flush=True) - return {} - try: - data = json.loads(raw) - except json.JSONDecodeError: - try: - data = json.loads(_strip_jsonc(raw)) - except json.JSONDecodeError as e: - print(f" warning: failed to parse {tsconfig} as JSON/JSONC ({e.msg} at line {e.lineno} col {e.colno})", file=sys.stderr, flush=True) - return {} - except Exception as e: - print(f" warning: failed to parse {tsconfig} ({type(e).__name__}: {e})", file=sys.stderr, flush=True) - return {} - - aliases: dict[str, list[str]] = {} - # `extends` may be a string or, since TypeScript 5.0, an array of paths. - # For an array, parents are processed in order with later entries - # overriding earlier ones; the extending config (paths below) overrides - # all parents. Without the list branch, an array `extends` raised - # `AttributeError: 'list' object has no attribute 'startswith'`, which - # _safe_extract turned into a skip of the whole file. - extends = data.get("extends") - if isinstance(extends, str): - extends_list = [extends] - elif isinstance(extends, list): - extends_list = [e for e in extends if isinstance(e, str)] - else: - extends_list = [] - for ext in extends_list: - # Skip scoped npm package configs (e.g. @tsconfig/svelte) — not on disk. - if not ext or ext.startswith("@"): - continue - extended_path = (base_dir / ext).resolve() - if not extended_path.suffix: - extended_path = extended_path.with_suffix(".json") - if extended_path.exists(): - aliases.update(_read_tsconfig_aliases(extended_path, extended_path.parent, seen)) - - # tsconfig `paths` are resolved relative to `baseUrl` (itself relative to - # the tsconfig's directory), not the tsconfig directory directly. Honoring - # baseUrl is required for the common monorepo / NestJS layout where - # baseUrl points at a subdirectory, e.g. baseUrl "./src" with - # "@services/*": ["services/*"] must resolve to /src/services rather - # than /services. Defaults to "." so configs without baseUrl (paths - # relative to the tsconfig dir, the TS 4.1+ behavior) keep working. - compiler_options = data.get("compilerOptions", {}) - base_url = compiler_options.get("baseUrl") or "." - paths_base = base_dir / base_url - paths = compiler_options.get("paths", {}) - for alias, targets in paths.items(): - if not targets: - continue - # Keep ALL targets in declared order — tsc tries each until one resolves - # on disk. Discarding the fallbacks (#1531) misresolved/dropped imports - # whose file lived at a non-first target. Preserve wildcard tokens in - # both sides until the resolver substitutes the captured segment, then - # normalizes the concrete path (#927). Empty/non-string entries are skipped. - target_patterns = [ - str(paths_base / t) - for t in targets - if isinstance(t, str) and t - ] - if target_patterns: - aliases[alias] = target_patterns - - return aliases - - -def _load_tsconfig_aliases(start_dir: Path) -> dict[str, list[str]]: - """Walk up from start_dir to find tsconfig.json and return compilerOptions.paths aliases. - - Follows extends chains so SvelteKit/Nuxt/NestJS inherited aliases are included. - Returns a dict mapping alias patterns to ordered resolved target patterns; - wildcard tokens remain intact for substitution during resolution (#927). - Result is cached by tsconfig path string. - """ - current = start_dir.resolve() - for candidate in [current, *current.parents]: - tsconfig = candidate / "tsconfig.json" - if tsconfig.exists(): - key = str(tsconfig) - if key not in _TSCONFIG_ALIAS_CACHE: - _TSCONFIG_ALIAS_CACHE[key] = _read_tsconfig_aliases(tsconfig, candidate, seen=set()) - return _TSCONFIG_ALIAS_CACHE[key] - return {} - - -def _match_tsconfig_alias(raw: str, pattern: str) -> "tuple[tuple[int, int], str, bool] | None": - """Return (specificity, captured text, is_wildcard) when pattern matches raw. - - Exact aliases win first. Wildcard aliases follow TypeScript's longest-prefix - rule. The final branch preserves Graphify's existing support for treating a - non-wildcard alias as a directory prefix, but only after real wildcard matches. - """ - if "*" in pattern: - if pattern.count("*") != 1: - return None - prefix, suffix = pattern.split("*", 1) - if not raw.startswith(prefix) or not raw.endswith(suffix): - return None - end = len(raw) - len(suffix) if suffix else len(raw) - if end < len(prefix): - return None - return (1, -len(prefix)), raw[len(prefix):end], True - - if raw == pattern: - return (0, -len(pattern)), "", False - - prefix = pattern.rstrip("/") - if prefix and raw.startswith(prefix + "/"): - return (2, -len(prefix)), raw[len(prefix):].lstrip("/"), False - return None - - -def _resolve_tsconfig_alias(raw: str, aliases: dict[str, list[str]]) -> "Path | None": - """Resolve `raw` against the most specific matching tsconfig alias pattern. - - Within that pattern, try targets in declared order and return the first whose - candidate resolves to a real file. If none exist, return the first candidate - so existing phantom/external-edge behavior stays unchanged. - """ - best: "tuple[tuple[int, int], str, bool, list[str]] | None" = None - for pattern, targets in aliases.items(): - match = _match_tsconfig_alias(raw, pattern) - if match is None: - continue - specificity, captured, is_wildcard = match - if best is None or specificity < best[0]: - best = specificity, captured, is_wildcard, targets - - if best is None: - return None - - _, captured, is_wildcard, targets = best - first = None - for target in targets: - if is_wildcard: - # TypeScript substitutes only when the matched star is non-empty. - substituted = target.replace("*", captured, 1) if captured else target - cand = Path(os.path.normpath(substituted)) - else: - cand = Path(target) - if captured: - cand = Path(os.path.normpath(cand / captured)) - resolved = _resolve_js_import_path(cand) - if resolved.is_file(): - return resolved - if first is None: - first = cand - return first - - -def _find_workspace_root(start_dir: Path) -> Path | None: - current = start_dir.resolve() - for candidate in [current, *current.parents]: - if (candidate / "pnpm-workspace.yaml").exists(): - return candidate - package_json = candidate / "package.json" - if package_json.is_file(): - try: - data = json.loads(package_json.read_text(encoding="utf-8")) - except Exception: - continue - if "workspaces" in data: - return candidate - return None - - -def _pnpm_workspace_globs(workspace_file: Path) -> list[str]: - globs: list[str] = [] - in_packages = False - for raw_line in workspace_file.read_text(encoding="utf-8", errors="replace").splitlines(): - line = raw_line.strip() - if not line or line.startswith("#"): - continue - if line.startswith("packages:"): - in_packages = True - continue - if in_packages and line.startswith("-"): - value = line[1:].strip().strip("'\"") - if value and not value.startswith("!"): - globs.append(value) - continue - if in_packages and not raw_line.startswith((" ", "\t")): - break - return globs - - -def _workspace_globs(root: Path) -> list[str]: - pnpm_workspace = root / "pnpm-workspace.yaml" - if pnpm_workspace.exists(): - return _pnpm_workspace_globs(pnpm_workspace) - - package_json = root / "package.json" - try: - data = json.loads(package_json.read_text(encoding="utf-8")) - except Exception: - return [] - - workspaces = data.get("workspaces") - if isinstance(workspaces, list): - return [item for item in workspaces if isinstance(item, str) and not item.startswith("!")] - if isinstance(workspaces, dict): - packages = workspaces.get("packages") - if isinstance(packages, list): - return [item for item in packages if isinstance(item, str) and not item.startswith("!")] - return [] - - -def _load_workspace_packages(start_dir: Path) -> dict[str, Path]: - root = _find_workspace_root(start_dir) - if root is None: - return {} - manifest_mtimes = tuple( - (name, (root / name).stat().st_mtime_ns) - for name in _WORKSPACE_MANIFEST_NAMES - if (root / name).is_file() - ) - key = str((root, manifest_mtimes)) - if key in _WORKSPACE_PACKAGE_CACHE: - return _WORKSPACE_PACKAGE_CACHE[key] - - packages: dict[str, Path] = {} - for pattern in _workspace_globs(root): - package_dirs: list[Path] = [root] if pattern in (".", "./") else list(root.glob(pattern)) - for package_dir in package_dirs: - manifest = package_dir / "package.json" - if not manifest.is_file(): - continue - try: - data = json.loads(manifest.read_text(encoding="utf-8")) - except Exception: - continue - name = data.get("name") - if isinstance(name, str) and name: - packages[name] = package_dir - _WORKSPACE_PACKAGE_CACHE[key] = packages - return packages - - # Condition keys consulted when resolving an `exports` target, in priority # order. `default` is Node's catch-all and must be consulted LAST so a more # specific condition (source/import/module/etc.) wins when several match. -_EXPORT_CONDITION_PRIORITY = ( - "source", "import", "module", "svelte", "types", "require", "default", -) - - -def _resolve_export_target(value: Any) -> str | None: - """Resolve an `exports` map value (string or condition object) to a - relative target string, honouring _EXPORT_CONDITION_PRIORITY for objects - and recursing into nested condition objects.""" - if isinstance(value, str): - return value - if isinstance(value, dict): - for cond in _EXPORT_CONDITION_PRIORITY: - v = value.get(cond) - if isinstance(v, str): - return v - if isinstance(v, dict): - nested = _resolve_export_target(v) - if nested: - return nested - return None - - -def _contained_in_package(resolved: Path, package_dir: Path) -> bool: - """Guard against `exports` targets that escape the package directory - (e.g. "./evil": "../../../etc/passwd"). Only accept paths that stay - within package_dir after resolution.""" - try: - return resolved.resolve().is_relative_to(package_dir.resolve()) - except ValueError: - return False - - -def _package_entry_candidates(package_dir: Path, subpath: str) -> list[Path]: - manifest = package_dir / "package.json" - manifest_data: dict[str, Any] = {} - try: - manifest_data = json.loads(manifest.read_text(encoding="utf-8")) - except Exception: - pass - - if subpath: - # Consult the package's `exports` subpath map before the bare-path - # fallback (#1308): "./browser" -> conditions -> file, plus single - # wildcard "./*" patterns. Targets that escape the package dir are - # rejected; resolution then falls through to the bare path. - exports = manifest_data.get("exports") - if isinstance(exports, dict): - subpath_key = "./" + subpath - target = _resolve_export_target(exports.get(subpath_key)) - if target: - candidate = package_dir / target - if _contained_in_package(candidate, package_dir): - return [candidate] - else: - for pattern, pattern_value in exports.items(): - if "*" in pattern and pattern.count("*") == 1: - prefix, suffix = pattern.split("*", 1) - if (subpath_key.startswith(prefix) - and (not suffix or subpath_key.endswith(suffix))): - matched = subpath_key[len(prefix):len(subpath_key) - len(suffix) if suffix else None] - resolved = _resolve_export_target(pattern_value) - if resolved and "*" in resolved: - candidate = package_dir / resolved.replace("*", matched) - if _contained_in_package(candidate, package_dir): - return [candidate] - return [package_dir / subpath] - - exports = manifest_data.get("exports") - if isinstance(exports, str): - return [package_dir / exports] - if isinstance(exports, dict): - dot_target = _resolve_export_target(exports.get(".")) - if dot_target: - return [package_dir / dot_target] - - candidates: list[Path] = [] - for key in ("svelte", "module", "main", "types"): - value = manifest_data.get(key) - if isinstance(value, str): - candidates.append(package_dir / value) - candidates.append(package_dir / "src/index") - candidates.append(package_dir / "index") - return candidates - - -def _resolve_workspace_import(raw: str, start_dir: Path) -> Path | None: - packages = _load_workspace_packages(start_dir) - for package_name, package_dir in packages.items(): - if raw == package_name: - subpath = "" - elif raw.startswith(package_name + "/"): - subpath = raw[len(package_name) + 1:] - else: - continue - for candidate in _package_entry_candidates(package_dir, subpath): - resolved = _resolve_js_import_path(candidate) - if resolved.is_file(): - return resolved - return None - - -def _resolve_js_module_path(raw: str | Path, start_dir: Path | None = None) -> Path | None: - """Resolve a JS/TS module path or specifier to a local source file. - - With a Path argument this preserves the path-based helper API used by - import-extension tests. With a string plus start_dir it resolves JS/TS - module specifiers including relative paths, tsconfig aliases, and workspace - packages. - """ - if isinstance(raw, Path): - return _resolve_js_import_path(raw) - if start_dir is None: - return _resolve_js_import_path(Path(raw)) - if raw.startswith("."): - return _resolve_js_import_path(start_dir / raw) - - aliases = _load_tsconfig_aliases(start_dir) - hit = _resolve_tsconfig_alias(raw, aliases) - if hit is not None: - return _resolve_js_import_path(hit) - - return _resolve_workspace_import(raw, start_dir) # ── LanguageConfig dataclass ───────────────────────────────────────────────── -@dataclass -class LanguageConfig: - ts_module: str # e.g. "tree_sitter_python" - ts_language_fn: str = "language" # attr to call: e.g. tslang.language() - - class_types: frozenset = frozenset() - function_types: frozenset = frozenset() - import_types: frozenset = frozenset() - call_types: frozenset = frozenset() - static_prop_types: frozenset = frozenset() - helper_fn_names: frozenset = frozenset() - container_bind_methods: frozenset = frozenset() - event_listener_properties: frozenset = frozenset() - - # Name extraction - name_field: str = "name" - name_fallback_child_types: tuple = () - - # Body detection - body_field: str = "body" - body_fallback_child_types: tuple = () # e.g. ("declaration_list", "compound_statement") - - # Call name extraction - call_function_field: str = "function" # field on call node for callee - call_accessor_node_types: frozenset = frozenset() # member/attribute nodes - call_accessor_field: str = "attribute" # field on accessor for method name - call_accessor_object_field: str = "" # field on accessor for the receiver/object - - # Stop recursion at these types in walk_calls - function_boundary_types: frozenset = frozenset() - - # Import handler: called for import nodes instead of generic handling - import_handler: Callable | None = None - - # Optional custom name resolver for functions (C, C++ declarator unwrapping) - resolve_function_name_fn: Callable | None = None - - # Extra label formatting for functions: if True, functions get "name()" label - function_label_parens: bool = True - - # Extra walk hook called after generic dispatch (for JS arrow functions, C# namespaces, etc.) - extra_walk_fn: Callable | None = None - # ── Generic helpers ─────────────────────────────────────────────────────────── @@ -1742,34 +1330,6 @@ def _import_python(node, source: bytes, file_nid: str, stem: str, edges: list, s }) -def _resolve_js_import_target(raw: str, str_path: str) -> "tuple[str, Path | None] | None": - """Resolve a JS/TS import path string to (target_nid, resolved_path). - - Handles relative paths, tsconfig path aliases, workspace packages, and - bare/scoped imports. - Returns None if `raw` is empty. - """ - if not raw: - return None - resolved_path = _resolve_js_module_path(raw, Path(str_path).parent) - if resolved_path is not None: - return _make_id(str(resolved_path)), resolved_path - module_name = raw.split("/")[-1] - if not module_name: - return None - # Unresolved: relative/absolute, tsconfig-alias and workspace resolution have - # all run and failed, so this is an external package (or a dangling local - # path). Namespace the id with the "ref" prefix — the J-4 convention already - # used for tsconfig `extends`/`$ref` externals — so it can NEVER collapse to - # the same _make_id as a local file/symbol node. Without it, the bare - # last-segment id (e.g. "tailwindcss/colors" -> "colors") collides with any - # unrelated local file of that stem via build.py's pre-migration alias index, - # producing a confident (EXTRACTED) cross-language phantom imports_from edge - # (#1638). The ref-namespaced target has no node, so build drops it as an - # external reference — the correct outcome for a third-party import. - return _make_id("ref", raw), None - - def _import_js(node, source: bytes, file_nid: str, stem: str, edges: list, str_path: str, scope_stack: list[str] | None = None) -> None: is_reexport = node.type == "export_statement" # Only handle export_statement if it has a `from` clause (re-export). @@ -1973,20 +1533,6 @@ def _walk_scoped(n) -> str: break -def _resolve_c_include_path(raw: str, str_path: str) -> "Path | None": - """Resolve a quoted #include path to a real file on disk. - - Searches relative to the including file's directory. Returns None for - system headers (<...>) or paths that don't exist on disk. - """ - if not raw: - return None - candidate = (Path(str_path).parent / raw).resolve() - if candidate.is_file(): - return candidate - return None - - def _import_c(node, source: bytes, file_nid: str, stem: str, edges: list, str_path: str, scope_stack: list[str] | None = None) -> None: for child in node.children: if child.type in ("string_literal", "system_lib_string", "string"): @@ -3101,44 +2647,6 @@ def _java_extra_walk(node, source: bytes, file_nid: str, stem: str, str_path: st ) -def _resolve_lua_import_target(raw_module: str, str_path: str) -> str: - """Resolve a Lua require() module name to a node id. - - Lua module names use dots as path separators: `require("pkg.b")` looks for - `pkg/b.lua` (or `pkg/b/init.lua`) relative to a package root. We probe the - importing file's directory and walk upward looking for a matching file on - disk; if found, the returned id matches the file node id `_extract_generic` - assigns to that file (`_make_id(str(path))`), so the edge lands on a real - node. When nothing matches, fall back to `_make_id` of the full dotted - module name so cross-file resolution can still complete via the symbol - resolution pass instead of dropping the edge entirely (#1075). - """ - if not raw_module: - return "" - rel = raw_module.replace(".", "/") - try: - start_dir = Path(str_path).parent - except Exception: - start_dir = None - if start_dir is not None: - probe = start_dir - # Walk up a few levels so requires from nested files still resolve when - # the package root is above the importing file. - for _ in range(6): - for suffix in (".lua", ".luau"): - cand = probe / f"{rel}{suffix}" - if cand.is_file(): - return _make_id(str(cand)) - for suffix in (".lua", ".luau"): - cand = probe / rel / f"init{suffix}" - if cand.is_file(): - return _make_id(str(cand)) - if probe.parent == probe: - break - probe = probe.parent - return _make_id(raw_module) - - def _import_lua(node, source: bytes, file_nid: str, stem: str, edges: list, str_path: str, scope_stack: list[str] | None = None) -> None: """Extract require('module') from Lua variable_declaration nodes.""" text = _read_text(node, source) @@ -6143,40 +5651,6 @@ def extract_astro(path: Path) -> dict: # The open-tag matcher skips over quoted attribute values so a `>` inside one # (e.g. Vue 3.3+ generic components: ` close tag - pos = m.end() - if lang is None: - lang_m = _VUE_SCRIPT_LANG_RE.search(m.group(1)) - if lang_m: - lang = lang_m.group(1).lower() - out.append(_blank(src[pos:])) - return "".join(out), lang def extract_vue(path: Path) -> dict: @@ -6716,145 +6190,6 @@ def _emit_import(name): # ── Cross-file import resolution ────────────────────────────────────────────── -def _source_key(source_file: str, root: Path) -> str: - if not source_file: - return "" - source_path = Path(source_file) - try: - return str(source_path.resolve().relative_to(root)) - except Exception: - return str(source_path) - - -def _node_disambiguation_source_key(node: dict, root: Path) -> str: - source_file = str(node.get("source_file", "")) - if source_file: - return _source_key(source_file, root) - return _source_key(str(node.get("origin_file", "")), root) - - -def _disambiguate_colliding_node_ids( - nodes: list[dict], - edges: list[dict], - raw_calls: list[dict], - root: Path, -) -> None: - """Rewrite only colliding node IDs, using source path as the disambiguator. - - Module anchor nodes (#1327) are exempt: ``import CoreKit`` from three files - yields three ``type=module`` nodes with the same id but different - source_files. Those are the *same* module, not distinct same-named symbols, - so they must collapse to one shared node — disambiguating them by path would - scatter a single module across N file-qualified duplicates. - """ - by_id: dict[str, list[dict]] = {} - for node in nodes: - if node.get("type") in ("module", "namespace"): - continue - nid = node.get("id") - if isinstance(nid, str) and nid: - by_id.setdefault(nid, []).append(node) - - remap: dict[tuple[str, str], str] = {} - ambiguous_ids: set[str] = set() - for old_id, group in by_id.items(): - source_keys = {_node_disambiguation_source_key(node, root) for node in group} - if len(group) < 2 or len(source_keys) < 2: - continue - ambiguous_ids.add(old_id) - # Salt the colliding id with the *path* it came from. The naive salt is - # ``_make_id(source_key, old_id)`` — source_key is the raw repo-relative - # path. But _make_id collapses every separator, so two DISTINCT paths - # whose only difference is a separator-vs-inner-punctuation swap - # (``a/b/c.md`` vs ``a.b/c.md``, ``foo/bar_baz.md`` vs ``foo_bar/baz.md``) - # normalize to the SAME salted id and still collide (#1522 — the residual - # of #1504 the 0.9.0 full-path stem didn't reach). When that happens, - # append a short stable hash of the *raw* source_key, which IS injective - # over distinct paths, so the colliders separate. Computed in code from - # source_file (never trusted from the LLM), so AST↔semantic parity holds. - naive: dict[str, str] = {} # source_key -> _make_id(source_key, old_id) - for source_key in source_keys: - if source_key: - naive[source_key] = _make_id(source_key, old_id) - # source_keys that, after normalization, are not unique among themselves. - seen: dict[str, int] = {} - for nid in naive.values(): - seen[nid] = seen.get(nid, 0) + 1 - needs_hash = {sk for sk, nid in naive.items() if seen.get(nid, 0) > 1} - for node in group: - source_key = _node_disambiguation_source_key(node, root) - if not source_key: - continue - if source_key in needs_hash: - salt = hashlib.sha1(source_key.encode("utf-8")).hexdigest()[:6] - new_id = _make_id(source_key, old_id, salt) - else: - new_id = naive.get(source_key) or _make_id(source_key, old_id) - remap[(old_id, source_key)] = new_id - if new_id != old_id: - node["id"] = new_id - - if not remap: - return - - unambiguous_remaps: dict[str, str] = {} - for old_id, group in by_id.items(): - if old_id in ambiguous_ids: - continue - candidates = { - node["id"] for node in group - if isinstance(node.get("id"), str) and node["id"] != old_id - } - if len(candidates) == 1: - unambiguous_remaps[old_id] = next(iter(candidates)) - - # A C/ObjC/C++ `#include "foo.h"` / `#import "foo.h"` resolves to the header's - # file node, but `foo.h` and its sibling `foo.c`/`foo.m`/`foo.cpp` collapse to - # the same `foo` file id, so disambiguation salts them apart by path. A - # cross-file import edge from a THIRD file carries neither salt's source_key, so - # the (target, edge_source_key) lookup misses and the edge dangles on the now - # dead `foo` id. Repoint those import edges to the HEADER variant (the include - # always targeted the header), keyed by the original colliding id (#1475). - _HEADER_SUFFIXES = (".h", ".hpp", ".hh", ".hxx") - header_remaps: dict[str, str] = {} - for old_id in ambiguous_ids: - for node in by_id.get(old_id, []): - sk = _node_disambiguation_source_key(node, root) - if sk and Path(sk).suffix.lower() in _HEADER_SUFFIXES: - new_id = remap.get((old_id, sk)) - if new_id: - header_remaps[old_id] = new_id - break - - for edge in edges: - edge_source_key = _source_key(str(edge.get("source_file", "")), root) - source_key = (edge.get("source", ""), edge_source_key) - target_key = (edge.get("target", ""), edge_source_key) - if source_key in remap: - edge["source"] = remap[source_key] - elif edge.get("source") in unambiguous_remaps: - edge["source"] = unambiguous_remaps[str(edge["source"])] - # imports/imports_from always target a header file, so they must resolve to - # the header variant BEFORE the same-source-file salt is considered. Keying - # the import target by the importer's own source file mis-points a `.m` - # importing its own `.h` back at itself (self-loop), and is wrong for any - # cross-file import whose importer shares the colliding id (#1475). - if (edge.get("relation") in ("imports", "imports_from") - and edge.get("target") in header_remaps): - edge["target"] = header_remaps[str(edge["target"])] - elif target_key in remap: - edge["target"] = remap[target_key] - elif edge.get("target") in unambiguous_remaps: - edge["target"] = unambiguous_remaps[str(edge["target"])] - - for raw_call in raw_calls: - call_source_key = _source_key(str(raw_call.get("source_file", "")), root) - caller_key = (raw_call.get("caller_nid", ""), call_source_key) - if caller_key in remap: - raw_call["caller_nid"] = remap[caller_key] - elif raw_call.get("caller_nid") in unambiguous_remaps: - raw_call["caller_nid"] = unambiguous_remaps[str(raw_call["caller_nid"])] - def _canonicalize_csharp_namespace_nodes(all_nodes: list[dict], all_edges: list[dict]) -> None: """Collapse duplicate C# namespace node entries to one canonical node per label.""" @@ -6968,19 +6303,6 @@ def _node_label_key(node: dict, fold: bool = False) -> str: return key.lower() if fold else key -def _is_type_like_definition(node: dict) -> bool: - if node.get("type") == "namespace": - return False - label = str(node.get("label", "")).strip() - if not label: - return False - if label.endswith(")") or label.startswith("."): - return False - if "." in label: - return False - return node.get("file_type") == "code" - - def _rewire_unique_stub_nodes(nodes: list[dict], edges: list[dict]) -> None: """Map unresolved no-source stubs to a unique real definition with the same label.""" real_by_label: dict[str, list[dict]] = {} # exact-case (all languages) @@ -7052,605 +6374,56 @@ def _rewire_unique_stub_nodes(nodes: list[dict], edges: list[dict]) -> None: nodes[:] = [node for node in nodes if node.get("id") not in drop_ids] -def _js_source_path(source_file: str, root: Path) -> Path | None: - if not source_file: - return None - path = Path(source_file) - if not path.is_absolute(): - path = root / path - try: - return path.resolve() - except Exception: - return path - - -@dataclass(frozen=True) -class _SymbolDeclarationFact: - file_path: Path - name: str - line: int - - -@dataclass(frozen=True) -class _SymbolImportFact: - file_path: Path - local_name: str - target_path: Path - imported_name: str - line: int - - -@dataclass(frozen=True) -class _SymbolAliasFact: - file_path: Path - alias: str - target_name: str - line: int - - -@dataclass(frozen=True) -class _SymbolExportFact: - file_path: Path - exported_name: str - line: int - local_name: str | None = None - target_path: Path | None = None - target_name: str | None = None - - -@dataclass(frozen=True) -class _StarExportFact: - file_path: Path - target_path: Path - line: int - - -@dataclass(frozen=True) -class _NamespaceExportFact: - file_path: Path - exported_name: str - target_path: Path - line: int - - -@dataclass(frozen=True) -class _SymbolUseFact: - file_path: Path - source_id: str - local_name: str - relation: str - context: str - line: int - - -@dataclass -class _SymbolResolutionFacts: - declarations: list[_SymbolDeclarationFact] = field(default_factory=list) - imports: list[_SymbolImportFact] = field(default_factory=list) - aliases: list[_SymbolAliasFact] = field(default_factory=list) - exports: list[_SymbolExportFact] = field(default_factory=list) - star_exports: list[_StarExportFact] = field(default_factory=list) - namespace_exports: list[_NamespaceExportFact] = field(default_factory=list) - uses: list[_SymbolUseFact] = field(default_factory=list) - # File-to-file submodule imports from `from pkg import submod` (#1146). - # Each entry is (importing_file, submodule_file, line). - module_imports: list[tuple[Path, Path, int]] = field(default_factory=list) - - -def _apply_symbol_resolution_facts( - paths: list[Path], - nodes: list[dict], - edges: list[dict], - root: Path, - facts: _SymbolResolutionFacts, -) -> None: - """Apply language-provided import/export/use facts to graph edges.""" - if not ( - facts.declarations - or facts.imports - or facts.aliases - or facts.exports - or facts.star_exports - or facts.namespace_exports - or facts.uses - or facts.module_imports - ): - return +def _ts_decorator_name(deco_node, source: bytes) -> str | None: + """Return the head symbol of a TS `decorator` node. - path_by_resolved = {path.resolve(): path for path in paths} - source_file_id = {path.resolve(): _make_id(str(path)) for path in paths} - symbol_nodes: dict[tuple[Path, str], str] = {} - for node in nodes: - source_path = _js_source_path(str(node.get("source_file", "")), root) - if source_path is None: + `@Injectable` -> the identifier; `@Component({...})` / `@Input()` -> the + `function` of the call_expression; `@ng.Component()` / `@core.Injectable` -> + the `property` of the member_expression (the imported symbol, not the + namespace alias). + """ + for child in deco_node.children: + if not child.is_named: continue - label = str(node.get("label", "")).strip().strip("()").lstrip(".") - if label and node.get("id"): - symbol_nodes[(source_path, label)] = str(node["id"]) - - def ensure_symbol_node(path: Path, name: str, line: int) -> str: - resolved_path = path.resolve() - existing = symbol_nodes.get((resolved_path, name)) - if existing is not None: - return existing - node_id = _make_id(_file_stem(path), name) - symbol_nodes[(resolved_path, name)] = node_id - nodes.append({ - "id": node_id, - "label": name, - "file_type": "code", - "source_file": str(path), - "source_location": f"L{line}", - }) - return node_id - - existing_edges = { - ( - str(edge.get("source")), - str(edge.get("target")), - str(edge.get("relation")), - str(edge.get("context") or ""), - ) - for edge in edges - } + target = child + if target.type == "call_expression": + target = target.child_by_field_name("function") or target + if target.type == "member_expression": + prop = target.child_by_field_name("property") + return _read_text(prop, source) if prop else None + if target.type == "identifier": + return _read_text(target, source) + return None + return None - def add_edge(source: str, target: str, relation: str, context: str, line: int, source_path: Path) -> None: - key = (source, target, relation, context or "") - if key in existing_edges: - return - existing_edges.add(key) - edges.append({ - "source": source, - "target": target, - "relation": relation, - "context": context, - "confidence": "EXTRACTED", - "source_file": str(source_path), - "source_location": f"L{line}", - "weight": 1.0, - }) - for declaration in facts.declarations: - ensure_symbol_node(declaration.file_path, declaration.name, declaration.line) +def _ts_method_name(method_node, source: bytes) -> str | None: + """Name of a `method_definition`, matching the id the function-types branch + builds (`_make_id(class_nid, name)`).""" + name_node = method_node.child_by_field_name("name") + return _read_text(name_node, source) if name_node else None - local_aliases_by_file: dict[Path, dict[str, tuple[Path, str]]] = {} - for import_fact in facts.imports: - file_path = import_fact.file_path.resolve() - local_aliases_by_file.setdefault(file_path, {})[import_fact.local_name] = ( - import_fact.target_path.resolve(), - import_fact.imported_name, - ) - pending_aliases_by_file: dict[Path, list[_SymbolAliasFact]] = {} - for alias_fact in facts.aliases: - pending_aliases_by_file.setdefault(alias_fact.file_path.resolve(), []).append(alias_fact) - - for file_path, aliases in pending_aliases_by_file.items(): - local_aliases = local_aliases_by_file.setdefault(file_path, {}) - changed = True - while changed: - changed = False - for alias_fact in aliases: - if alias_fact.alias in local_aliases: - continue - origin = local_aliases.get(alias_fact.target_name) - if origin is not None: - local_aliases[alias_fact.alias] = origin - changed = True - - named_exports_by_file: dict[Path, dict[str, tuple[Path, str]]] = {} - star_exports_by_file: dict[Path, list[Path]] = {} - - for star_fact in facts.star_exports: - source_path = star_fact.file_path.resolve() - target_path = star_fact.target_path.resolve() - star_exports_by_file.setdefault(source_path, []).append(target_path) - source_id = source_file_id.get(source_path) - if source_id is not None: - add_edge( - source_id, - _make_id(str(path_by_resolved.get(target_path, target_path))), - "re_exports", - "export", - star_fact.line, - star_fact.file_path, - ) +def _ts_descendant_decorators(node) -> list: + """Collect `decorator` nodes under `node` (e.g. parameter decorators inside a + method's formal_parameters, or a field's own decorator), without crossing into + a nested class or a nested method, which own their own decorators.""" + out: list = [] - for namespace_fact in facts.namespace_exports: - source_path = namespace_fact.file_path.resolve() - target_path = namespace_fact.target_path.resolve() - namespace_id = ensure_symbol_node( - namespace_fact.file_path, - namespace_fact.exported_name, - namespace_fact.line, - ) - named_exports_by_file.setdefault(source_path, {})[ - namespace_fact.exported_name - ] = (source_path, namespace_fact.exported_name) - source_id = source_file_id.get(source_path) - if source_id is not None: - add_edge( - source_id, - namespace_id, - "contains", - "namespace_export", - namespace_fact.line, - namespace_fact.file_path, - ) - add_edge( - source_id, - _make_id(str(path_by_resolved.get(target_path, target_path))), - "re_exports", - "export", - namespace_fact.line, - namespace_fact.file_path, - ) + def rec(n, top: bool) -> None: + for child in n.children: + ct = child.type + if ct == "decorator": + out.append(child) + elif ct in ("class_declaration", "abstract_class_declaration"): + continue + elif ct == "method_definition" and not top: + continue + else: + rec(child, False) - for export_fact in facts.exports: - file_path = export_fact.file_path.resolve() - origin: tuple[Path, str] | None = None - if export_fact.target_path is not None and export_fact.target_name is not None: - origin = (export_fact.target_path.resolve(), export_fact.target_name) - elif export_fact.local_name is not None: - origin = local_aliases_by_file.get(file_path, {}).get(export_fact.local_name) - if origin is None and (file_path, export_fact.local_name) in symbol_nodes: - origin = (file_path, export_fact.local_name) - if origin is None: - continue - named_exports_by_file.setdefault(file_path, {})[export_fact.exported_name] = origin - if origin[0] != file_path: - source_id = source_file_id.get(file_path) - if source_id is not None: - add_edge( - source_id, - _make_id(str(path_by_resolved.get(origin[0], origin[0]))), - "re_exports", - "export", - export_fact.line, - export_fact.file_path, - ) - - def resolve_exported_origin(target_path: Path, imported_name: str, seen: set[tuple[Path, str]] | None = None) -> tuple[Path, str]: - target_path = target_path.resolve() - key = (target_path, imported_name) - if seen is None: - seen = set() - if key in seen: - return key - seen.add(key) - origin = named_exports_by_file.get(target_path, {}).get(imported_name) - if origin is not None: - return resolve_exported_origin(origin[0], origin[1], seen) - for star_target in star_exports_by_file.get(target_path, []): - star_key = (star_target, imported_name) - if star_key in symbol_nodes: - return star_key - resolved = resolve_exported_origin(star_target, imported_name, seen) - if resolved in symbol_nodes: - return resolved - return key - - for import_fact in facts.imports: - source_id = source_file_id.get(import_fact.file_path.resolve()) - if source_id is None: - continue - origin_path, origin_symbol = resolve_exported_origin( - import_fact.target_path, - import_fact.imported_name, - ) - target_id = symbol_nodes.get((origin_path, origin_symbol)) - if target_id is None: - continue - add_edge( - source_id, - target_id, - "imports", - "import", - import_fact.line, - import_fact.file_path, - ) - - # #1146: emit file-to-file imports_from edges for package-form submodule imports. - for from_path, to_path, line in facts.module_imports: - try: - from_rel = from_path.relative_to(root) - to_rel = to_path.relative_to(root) - except ValueError: - continue - source_id = _make_id(_file_stem(from_rel)) - target_id = _make_id(_file_stem(to_rel)) - add_edge(source_id, target_id, "imports_from", "submodule_import", line, from_path) - - for use_fact in facts.uses: - file_path = use_fact.file_path.resolve() - target_id = None - unresolved_origin = local_aliases_by_file.get(file_path, {}).get(use_fact.local_name) - if unresolved_origin is not None: - origin_path, origin_symbol = resolve_exported_origin(*unresolved_origin) - target_id = symbol_nodes.get((origin_path, origin_symbol)) - if target_id is None and use_fact.relation in ("inherits", "implements"): - # Same-file fallback for HERITAGE only: a base declared in the same - # file (`class X extends Y`, `interface A extends B`) has no import - # alias, so resolve it directly against the file's own symbol nodes. - # Scoped to heritage because same-file calls/uses already resolve via - # the dedicated call-graph pass; widening this would duplicate those - # edges. Import resolution still takes precedence (#1095). - target_id = symbol_nodes.get((file_path, use_fact.local_name)) - if target_id is None: - continue - add_edge( - use_fact.source_id, - target_id, - use_fact.relation, - use_fact.context, - use_fact.line, - use_fact.file_path, - ) - - -def _parse_js_tree(path: Path): - try: - from tree_sitter import Language, Parser - # .vue embeds the script in non-JS markup; mask it out and parse the - # close tag + pos = m.end() + if lang is None: + lang_m = _VUE_SCRIPT_LANG_RE.search(m.group(1)) + if lang_m: + lang = lang_m.group(1).lower() + out.append(_blank(src[pos:])) + return "".join(out), lang + +def _source_key(source_file: str, root: Path) -> str: + if not source_file: + return "" + source_path = Path(source_file) + try: + return str(source_path.resolve().relative_to(root)) + except Exception: + return str(source_path) + +def _node_disambiguation_source_key(node: dict, root: Path) -> str: + source_file = str(node.get("source_file", "")) + if source_file: + return _source_key(source_file, root) + return _source_key(str(node.get("origin_file", "")), root) + +def _disambiguate_colliding_node_ids( + nodes: list[dict], + edges: list[dict], + raw_calls: list[dict], + root: Path, +) -> None: + """Rewrite only colliding node IDs, using source path as the disambiguator. + + Module anchor nodes (#1327) are exempt: ``import CoreKit`` from three files + yields three ``type=module`` nodes with the same id but different + source_files. Those are the *same* module, not distinct same-named symbols, + so they must collapse to one shared node — disambiguating them by path would + scatter a single module across N file-qualified duplicates. + """ + by_id: dict[str, list[dict]] = {} + for node in nodes: + if node.get("type") in ("module", "namespace"): + continue + nid = node.get("id") + if isinstance(nid, str) and nid: + by_id.setdefault(nid, []).append(node) + + remap: dict[tuple[str, str], str] = {} + ambiguous_ids: set[str] = set() + for old_id, group in by_id.items(): + source_keys = {_node_disambiguation_source_key(node, root) for node in group} + if len(group) < 2 or len(source_keys) < 2: + continue + ambiguous_ids.add(old_id) + # Salt the colliding id with the *path* it came from. The naive salt is + # ``_make_id(source_key, old_id)`` — source_key is the raw repo-relative + # path. But _make_id collapses every separator, so two DISTINCT paths + # whose only difference is a separator-vs-inner-punctuation swap + # (``a/b/c.md`` vs ``a.b/c.md``, ``foo/bar_baz.md`` vs ``foo_bar/baz.md``) + # normalize to the SAME salted id and still collide (#1522 — the residual + # of #1504 the 0.9.0 full-path stem didn't reach). When that happens, + # append a short stable hash of the *raw* source_key, which IS injective + # over distinct paths, so the colliders separate. Computed in code from + # source_file (never trusted from the LLM), so AST↔semantic parity holds. + naive: dict[str, str] = {} # source_key -> _make_id(source_key, old_id) + for source_key in source_keys: + if source_key: + naive[source_key] = _make_id(source_key, old_id) + # source_keys that, after normalization, are not unique among themselves. + seen: dict[str, int] = {} + for nid in naive.values(): + seen[nid] = seen.get(nid, 0) + 1 + needs_hash = {sk for sk, nid in naive.items() if seen.get(nid, 0) > 1} + for node in group: + source_key = _node_disambiguation_source_key(node, root) + if not source_key: + continue + if source_key in needs_hash: + salt = hashlib.sha1(source_key.encode("utf-8")).hexdigest()[:6] + new_id = _make_id(source_key, old_id, salt) + else: + new_id = naive.get(source_key) or _make_id(source_key, old_id) + remap[(old_id, source_key)] = new_id + if new_id != old_id: + node["id"] = new_id + + if not remap: + return + + unambiguous_remaps: dict[str, str] = {} + for old_id, group in by_id.items(): + if old_id in ambiguous_ids: + continue + candidates = { + node["id"] for node in group + if isinstance(node.get("id"), str) and node["id"] != old_id + } + if len(candidates) == 1: + unambiguous_remaps[old_id] = next(iter(candidates)) + + # A C/ObjC/C++ `#include "foo.h"` / `#import "foo.h"` resolves to the header's + # file node, but `foo.h` and its sibling `foo.c`/`foo.m`/`foo.cpp` collapse to + # the same `foo` file id, so disambiguation salts them apart by path. A + # cross-file import edge from a THIRD file carries neither salt's source_key, so + # the (target, edge_source_key) lookup misses and the edge dangles on the now + # dead `foo` id. Repoint those import edges to the HEADER variant (the include + # always targeted the header), keyed by the original colliding id (#1475). + _HEADER_SUFFIXES = (".h", ".hpp", ".hh", ".hxx") + header_remaps: dict[str, str] = {} + for old_id in ambiguous_ids: + for node in by_id.get(old_id, []): + sk = _node_disambiguation_source_key(node, root) + if sk and Path(sk).suffix.lower() in _HEADER_SUFFIXES: + new_id = remap.get((old_id, sk)) + if new_id: + header_remaps[old_id] = new_id + break + + for edge in edges: + edge_source_key = _source_key(str(edge.get("source_file", "")), root) + source_key = (edge.get("source", ""), edge_source_key) + target_key = (edge.get("target", ""), edge_source_key) + if source_key in remap: + edge["source"] = remap[source_key] + elif edge.get("source") in unambiguous_remaps: + edge["source"] = unambiguous_remaps[str(edge["source"])] + # imports/imports_from always target a header file, so they must resolve to + # the header variant BEFORE the same-source-file salt is considered. Keying + # the import target by the importer's own source file mis-points a `.m` + # importing its own `.h` back at itself (self-loop), and is wrong for any + # cross-file import whose importer shares the colliding id (#1475). + if (edge.get("relation") in ("imports", "imports_from") + and edge.get("target") in header_remaps): + edge["target"] = header_remaps[str(edge["target"])] + elif target_key in remap: + edge["target"] = remap[target_key] + elif edge.get("target") in unambiguous_remaps: + edge["target"] = unambiguous_remaps[str(edge["target"])] + + for raw_call in raw_calls: + call_source_key = _source_key(str(raw_call.get("source_file", "")), root) + caller_key = (raw_call.get("caller_nid", ""), call_source_key) + if caller_key in remap: + raw_call["caller_nid"] = remap[caller_key] + elif raw_call.get("caller_nid") in unambiguous_remaps: + raw_call["caller_nid"] = unambiguous_remaps[str(raw_call["caller_nid"])] + +def _is_type_like_definition(node: dict) -> bool: + if node.get("type") == "namespace": + return False + label = str(node.get("label", "")).strip() + if not label: + return False + if label.endswith(")") or label.startswith("."): + return False + if "." in label: + return False + return node.get("file_type") == "code" + +def _js_source_path(source_file: str, root: Path) -> Path | None: + if not source_file: + return None + path = Path(source_file) + if not path.is_absolute(): + path = root / path + try: + return path.resolve() + except Exception: + return path + +def _apply_symbol_resolution_facts( + paths: list[Path], + nodes: list[dict], + edges: list[dict], + root: Path, + facts: _SymbolResolutionFacts, +) -> None: + """Apply language-provided import/export/use facts to graph edges.""" + if not ( + facts.declarations + or facts.imports + or facts.aliases + or facts.exports + or facts.star_exports + or facts.namespace_exports + or facts.uses + or facts.module_imports + ): + return + + path_by_resolved = {path.resolve(): path for path in paths} + source_file_id = {path.resolve(): _make_id(str(path)) for path in paths} + symbol_nodes: dict[tuple[Path, str], str] = {} + for node in nodes: + source_path = _js_source_path(str(node.get("source_file", "")), root) + if source_path is None: + continue + label = str(node.get("label", "")).strip().strip("()").lstrip(".") + if label and node.get("id"): + symbol_nodes[(source_path, label)] = str(node["id"]) + + def ensure_symbol_node(path: Path, name: str, line: int) -> str: + resolved_path = path.resolve() + existing = symbol_nodes.get((resolved_path, name)) + if existing is not None: + return existing + node_id = _make_id(_file_stem(path), name) + symbol_nodes[(resolved_path, name)] = node_id + nodes.append({ + "id": node_id, + "label": name, + "file_type": "code", + "source_file": str(path), + "source_location": f"L{line}", + }) + return node_id + + existing_edges = { + ( + str(edge.get("source")), + str(edge.get("target")), + str(edge.get("relation")), + str(edge.get("context") or ""), + ) + for edge in edges + } + + def add_edge(source: str, target: str, relation: str, context: str, line: int, source_path: Path) -> None: + key = (source, target, relation, context or "") + if key in existing_edges: + return + existing_edges.add(key) + edges.append({ + "source": source, + "target": target, + "relation": relation, + "context": context, + "confidence": "EXTRACTED", + "source_file": str(source_path), + "source_location": f"L{line}", + "weight": 1.0, + }) + + for declaration in facts.declarations: + ensure_symbol_node(declaration.file_path, declaration.name, declaration.line) + + local_aliases_by_file: dict[Path, dict[str, tuple[Path, str]]] = {} + for import_fact in facts.imports: + file_path = import_fact.file_path.resolve() + local_aliases_by_file.setdefault(file_path, {})[import_fact.local_name] = ( + import_fact.target_path.resolve(), + import_fact.imported_name, + ) + + pending_aliases_by_file: dict[Path, list[_SymbolAliasFact]] = {} + for alias_fact in facts.aliases: + pending_aliases_by_file.setdefault(alias_fact.file_path.resolve(), []).append(alias_fact) + + for file_path, aliases in pending_aliases_by_file.items(): + local_aliases = local_aliases_by_file.setdefault(file_path, {}) + changed = True + while changed: + changed = False + for alias_fact in aliases: + if alias_fact.alias in local_aliases: + continue + origin = local_aliases.get(alias_fact.target_name) + if origin is not None: + local_aliases[alias_fact.alias] = origin + changed = True + + named_exports_by_file: dict[Path, dict[str, tuple[Path, str]]] = {} + star_exports_by_file: dict[Path, list[Path]] = {} + + for star_fact in facts.star_exports: + source_path = star_fact.file_path.resolve() + target_path = star_fact.target_path.resolve() + star_exports_by_file.setdefault(source_path, []).append(target_path) + source_id = source_file_id.get(source_path) + if source_id is not None: + add_edge( + source_id, + _make_id(str(path_by_resolved.get(target_path, target_path))), + "re_exports", + "export", + star_fact.line, + star_fact.file_path, + ) + + for namespace_fact in facts.namespace_exports: + source_path = namespace_fact.file_path.resolve() + target_path = namespace_fact.target_path.resolve() + namespace_id = ensure_symbol_node( + namespace_fact.file_path, + namespace_fact.exported_name, + namespace_fact.line, + ) + named_exports_by_file.setdefault(source_path, {})[ + namespace_fact.exported_name + ] = (source_path, namespace_fact.exported_name) + source_id = source_file_id.get(source_path) + if source_id is not None: + add_edge( + source_id, + namespace_id, + "contains", + "namespace_export", + namespace_fact.line, + namespace_fact.file_path, + ) + add_edge( + source_id, + _make_id(str(path_by_resolved.get(target_path, target_path))), + "re_exports", + "export", + namespace_fact.line, + namespace_fact.file_path, + ) + + for export_fact in facts.exports: + file_path = export_fact.file_path.resolve() + origin: tuple[Path, str] | None = None + if export_fact.target_path is not None and export_fact.target_name is not None: + origin = (export_fact.target_path.resolve(), export_fact.target_name) + elif export_fact.local_name is not None: + origin = local_aliases_by_file.get(file_path, {}).get(export_fact.local_name) + if origin is None and (file_path, export_fact.local_name) in symbol_nodes: + origin = (file_path, export_fact.local_name) + if origin is None: + continue + named_exports_by_file.setdefault(file_path, {})[export_fact.exported_name] = origin + if origin[0] != file_path: + source_id = source_file_id.get(file_path) + if source_id is not None: + add_edge( + source_id, + _make_id(str(path_by_resolved.get(origin[0], origin[0]))), + "re_exports", + "export", + export_fact.line, + export_fact.file_path, + ) + + def resolve_exported_origin(target_path: Path, imported_name: str, seen: set[tuple[Path, str]] | None = None) -> tuple[Path, str]: + target_path = target_path.resolve() + key = (target_path, imported_name) + if seen is None: + seen = set() + if key in seen: + return key + seen.add(key) + origin = named_exports_by_file.get(target_path, {}).get(imported_name) + if origin is not None: + return resolve_exported_origin(origin[0], origin[1], seen) + for star_target in star_exports_by_file.get(target_path, []): + star_key = (star_target, imported_name) + if star_key in symbol_nodes: + return star_key + resolved = resolve_exported_origin(star_target, imported_name, seen) + if resolved in symbol_nodes: + return resolved + return key + + for import_fact in facts.imports: + source_id = source_file_id.get(import_fact.file_path.resolve()) + if source_id is None: + continue + origin_path, origin_symbol = resolve_exported_origin( + import_fact.target_path, + import_fact.imported_name, + ) + target_id = symbol_nodes.get((origin_path, origin_symbol)) + if target_id is None: + continue + add_edge( + source_id, + target_id, + "imports", + "import", + import_fact.line, + import_fact.file_path, + ) + + # #1146: emit file-to-file imports_from edges for package-form submodule imports. + for from_path, to_path, line in facts.module_imports: + try: + from_rel = from_path.relative_to(root) + to_rel = to_path.relative_to(root) + except ValueError: + continue + source_id = _make_id(_file_stem(from_rel)) + target_id = _make_id(_file_stem(to_rel)) + add_edge(source_id, target_id, "imports_from", "submodule_import", line, from_path) + + for use_fact in facts.uses: + file_path = use_fact.file_path.resolve() + target_id = None + unresolved_origin = local_aliases_by_file.get(file_path, {}).get(use_fact.local_name) + if unresolved_origin is not None: + origin_path, origin_symbol = resolve_exported_origin(*unresolved_origin) + target_id = symbol_nodes.get((origin_path, origin_symbol)) + if target_id is None and use_fact.relation in ("inherits", "implements"): + # Same-file fallback for HERITAGE only: a base declared in the same + # file (`class X extends Y`, `interface A extends B`) has no import + # alias, so resolve it directly against the file's own symbol nodes. + # Scoped to heritage because same-file calls/uses already resolve via + # the dedicated call-graph pass; widening this would duplicate those + # edges. Import resolution still takes precedence (#1095). + target_id = symbol_nodes.get((file_path, use_fact.local_name)) + if target_id is None: + continue + add_edge( + use_fact.source_id, + target_id, + use_fact.relation, + use_fact.context, + use_fact.line, + use_fact.file_path, + ) + +def _parse_js_tree(path: Path): + try: + from tree_sitter import Language, Parser + # .vue embeds the script in non-JS markup; mask it out and parse the + # +

{projects.length}

+""", + ) + _write(root / "src/lib/content.ts", "export const projects = [1];\n") + + result = extract( + [root / "src/routes/page.svelte", root / "src/lib/content.ts"], + cache_root=root, + ) + + _assert_no_root_slug(result, root) + + content_file_nodes = [ + n for n in result["nodes"] if n["id"] == "src_lib_content" + ] + assert len(content_file_nodes) == 1 + assert content_file_nodes[0]["source_file"] == "src/lib/content.ts" + + page_id = _file_node_id(Path("src/routes/page.svelte")) + edge_pairs = { + (e["source"], e["target"], e["relation"]) for e in result["edges"] + } + assert (page_id, "src_lib_content", "imports_from") in edge_pairs + assert (page_id, "src_lib_content", "dynamic_import") in edge_pairs + + +def test_astro_unresolved_relative_import_id_still_portable(tmp_path): + """A rescued import whose target does NOT exist still mints a stub (so the + edge survives as a hint), but the stub id must be the repo-relative form, + not the absolute-path slug — the belt-and-braces remap in the + relativization pass (#2195).""" + root = Path(os.path.realpath(tmp_path)) + _write( + root / "src/pages/index.astro", + """--- +import { gone } from '../missing/nowhere'; +--- +

x

+""", + ) + result = extract([root / "src/pages/index.astro"], cache_root=root) + + _assert_no_root_slug(result, root) + stub = [n for n in result["nodes"] if n.get("label") == "../missing/nowhere"] + assert len(stub) == 1 + assert stub[0]["id"] == "src_missing_nowhere" From e395ff9b433a441da3848a18bd4c82c0c9b888fc Mon Sep 17 00:00:00 2001 From: safishamsi Date: Sun, 26 Jul 2026 11:52:20 +0100 Subject: [PATCH 275/292] fix(dedup): merge cross-file concept nodes with identical normalized labels (#2182) Pass 1 deferred cross-file exact matches to Pass 2, but Pass 2's candidate filter keeps only the first node per normalized label, so identical-label cross-file concept pairs could never merge (while fuzzy pairs did). Pass 1 now unions the cross-file residue of each label group, gated to concept nodes with provenance and above the entropy floor, so code/rationale/ document/image/empty-source and cross-repo guards are all preserved. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/dedup.py | 38 ++++++++-- tests/test_dedup.py | 177 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 209 insertions(+), 6 deletions(-) diff --git a/graphify/dedup.py b/graphify/dedup.py index 0e1341e7c..bc2f1c418 100644 --- a/graphify/dedup.py +++ b/graphify/dedup.py @@ -411,8 +411,10 @@ def deduplicate_entities( for key, group in norm_to_nodes.items(): if len(group) <= 1: continue - # Partition by source_file — only merge within the same file in Pass 1. - # Cross-file matches fall through to Pass 2 fuzzy matching. + # Partition by source_file — same-file exact matches always merge here. + # Cross-file exact matches are handled just below, gated to `concept` + # nodes only: Pass 2 cannot form them because its candidate list keeps a + # single node per normalized label (#2182). by_file: dict[str, list[dict]] = defaultdict(list) for node in group: sf = node.get("source_file") or "" @@ -427,6 +429,26 @@ def deduplicate_entities( for node in file_group: uf.union(winner["id"], node["id"]) exact_merges += len(file_group) - 1 + # Cross-file residue: union exact matches across files, but only where + # it is provably safe (#2182). `concept` is the one file_type meant to + # unify across files (#1284) — code is keyed by ID (#1205), rationale/ + # document are file-anchored (#1284), and image/paper labels are often + # shared basenames (logo.png). Provenance is required (#1178), and the + # entropy gate mirrors Pass 2 so short generic labels ("API") stay + # distinct. Sorting by id keeps the winner order-independent. + mergeable = sorted( + (n for n in group + if n.get("file_type") == "concept" + and (n.get("source_file") or "") + and _entropy(n.get("label", "")) >= _ENTROPY_THRESHOLD), + key=lambda n: n["id"], + ) + if len(mergeable) > 1: + winner = _pick_winner(mergeable) + for node in mergeable: + if uf.find(winner["id"]) != uf.find(node["id"]): + uf.union(winner["id"], node["id"]) + exact_merges += 1 # ── pass 2: MinHash/LSH + Jaro-Winkler (high-entropy nodes only) ───────── candidates: list[dict] = [] @@ -522,10 +544,14 @@ def deduplicate_entities( score += _COMMUNITY_BOOST if score >= _MERGE_THRESHOLD: - # Identical labels across different source files almost always - # means same-named-but-different symbols (trait impls, wrapper - # methods, common type names). Mirror Pass 1's source_file - # partition for this sub-case. (#1046, leaks #895's fix) + # Belt-and-braces (#1046, narrowed by #2182): candidates are + # norm-unique (`seen_norms` above), so two candidates can + # never share a normalized label and this branch is + # unreachable today. Retained in case candidate selection + # changes. Equal-norm cross-file pairs are handled in Pass 1 + # instead, gated to `concept` nodes — the original #1046 + # rationale (same-named code symbols) was obsoleted by code + # being excluded from label matching entirely (#1205, #1247). if norm_label == neighbor_norm: sf_a = node.get("source_file") or "" sf_b = neighbor.get("source_file") or "" diff --git a/tests/test_dedup.py b/tests/test_dedup.py index ab38b0f48..e1370fc7a 100644 --- a/tests/test_dedup.py +++ b/tests/test_dedup.py @@ -692,3 +692,180 @@ def test_dedup_fills_explicit_none_attribute(): result, _ = deduplicate_entities([dict(n) for n in nodes], [], communities={}) assert len(result) == 1 assert result[0].get("source_location") == "L7", "explicit-None must be filled from the loser" + + +# ── #2182: cross-file exact-duplicate concepts must merge ───────────────────── + +def test_crossfile_identical_concepts_merge_and_rewire(): + """Two `concept` nodes whose labels are byte-identical after _norm() but + live in different files must merge (#2182). Pass 1 used to defer them to + Pass 2, whose norm-unique candidate filter (`seen_norms`) structurally + cannot form an equal-norm pair — so exact cross-file duplicates were the + one class of duplicate that never merged, while a one-char-different + fuzzy pair did.""" + nodes = [ + {"id": "sz_intl", "label": "SHENZHEN INTERNATIONAL", + "file_type": "concept", "source_file": "doc1.md"}, + {"id": "shenzhen_international_holdings", "label": "Shenzhen international", + "file_type": "concept", "source_file": "doc2.md"}, + {"id": "port_ops", "label": "Port Operations", + "file_type": "concept", "source_file": "doc2.md"}, + ] + edges = [{"source": "shenzhen_international_holdings", "target": "port_ops", + "relation": "operates"}] + result_nodes, result_edges = deduplicate_entities(nodes, edges, communities={}) + ids = {n["id"] for n in result_nodes} + assert len(result_nodes) == 2 + # _pick_winner prefers the shorter, non-chunk-suffixed id. + assert "sz_intl" in ids + assert "shenzhen_international_holdings" not in ids + # The loser's edge is rewired to the winner. + assert result_edges == [ + {"source": "sz_intl", "target": "port_ops", "relation": "operates"}] + + +def test_crossfile_one_char_typo_concepts_still_merge(): + """Non-regression: the near-identical (one-char-different) cross-file pair + that already merged via Pass 2 fuzzy matching must keep merging (#2182).""" + nodes = [ + {"id": "g1", "label": "Authentication Manager", + "file_type": "concept", "source_file": "a.md"}, + {"id": "g2", "label": "Authentication Managr", + "file_type": "concept", "source_file": "b.md"}, + ] + result_nodes, _ = deduplicate_entities(nodes, [], communities={}) + assert len(result_nodes) == 1 + + +_RATIONALE_BOILER = ("Django app config for apps.platform.cards. No business " + "logic here. Domain services live in services.py.") + + +@pytest.mark.parametrize("a,b", [ + ({"id": "d1", "label": "Getting Started Installation Guide", + "file_type": "document", "source_file": "docs/a.md"}, + {"id": "d2", "label": "Getting Started Installation Guide", + "file_type": "document", "source_file": "docs/b.md"}), + ({"id": "r1", "label": _RATIONALE_BOILER, + "file_type": "rationale", "source_file": "apps/platform/cards/apps.py"}, + {"id": "r2", "label": _RATIONALE_BOILER, + "file_type": "rationale", "source_file": "apps/platform/cores/apps.py"}), + ({"id": "backend_a_render_frame", "label": "render_frame", + "file_type": "code", "source_file": "backend_a.py"}, + {"id": "backend_b_render_frame", "label": "render_frame", + "file_type": "code", "source_file": "backend_b.py"}), + ({"id": "web_logo", "label": "logo.png", + "file_type": "image", "source_file": "web/assets/logo.png"}, + {"id": "docs_logo", "label": "logo.png", + "file_type": "image", "source_file": "docs/img/logo.png"}), + ({"id": "logo_concept", "label": "logo.png", + "file_type": "concept", "source_file": "doc1.md"}, + {"id": "logo_image", "label": "logo.png", + "file_type": "image", "source_file": "assets/logo.png"}), + ({"id": "shenzhen_a", "label": "Shenzhen International", + "file_type": "concept", "source_file": ""}, + {"id": "shenzhen_b", "label": "Shenzhen International", + "file_type": "concept", "source_file": ""}), + ({"id": "api_a", "label": "API", + "file_type": "concept", "source_file": "doc1.md"}, + {"id": "api_b", "label": "API", + "file_type": "concept", "source_file": "doc2.md"}), +], ids=["document", "rationale", "code", "image-basename", "concept-image-mixed", + "empty-source-file", "low-entropy-concept"]) +def test_crossfile_identical_labels_stay_distinct_for_guarded_types(a, b): + """The #2182 fix is gated to high-entropy `concept` nodes with provenance + on BOTH sides. Identical labels must NOT merge for: file-anchored types + (document/rationale, #1284), code (#1205), images sharing a basename in + different dirs, mixed concept+image pairs, provenance-less nodes (#1178), + and low-entropy generic labels.""" + result_nodes, _ = deduplicate_entities([dict(a), dict(b)], [], communities={}) + assert len(result_nodes) == 2, ( + f"guarded pair ({a['id']}, {b['id']}) was merged — #2182 fix leaked " + f"past its concept-only gate" + ) + + +def test_cross_repo_guard_still_raises(): + """The cross-repo guard is untouched by #2182: identical concepts from + different repos must still raise, never merge.""" + nodes = [ + {"id": "c1", "label": "Shenzhen International", "file_type": "concept", + "source_file": "doc1.md", "repo": "repo-a"}, + {"id": "c2", "label": "Shenzhen International", "file_type": "concept", + "source_file": "doc2.md", "repo": "repo-b"}, + ] + with pytest.raises(ValueError, match="multiple repos"): + deduplicate_entities(nodes, [], communities={}) + + +def test_crossfile_concept_merge_is_order_independent(): + """Three identical-norm concepts across three files: every input order must + yield the same single survivor (#2182). Winner ids differ in length so + _pick_winner has a unique minimum.""" + import itertools + base = [ + {"id": "shenzhen", "label": "SHENZHEN INTERNATIONAL", + "file_type": "concept", "source_file": "doc1.md"}, + {"id": "shenzhen_intl", "label": "Shenzhen international", + "file_type": "concept", "source_file": "doc2.md"}, + {"id": "shenzhen_international", "label": "shenzhen-international", + "file_type": "concept", "source_file": "doc3.md"}, + ] + survivors = set() + for perm in itertools.permutations(base): + out, _ = deduplicate_entities([dict(n) for n in perm], [], communities={}) + assert len(out) == 1 + survivors.add(out[0]["id"]) + assert survivors == {"shenzhen"}, f"non-deterministic survivor: {survivors}" + + +def test_crossfile_concept_merge_deterministic_across_hash_seeds(): + """#2182 determinism, #1753/#2074 precedent: the survivor must not depend on + PYTHONHASHSEED. pytest fixes the seed per process, so run out-of-process + with shuffled input.""" + import os + import subprocess + import sys + script = ( + "import random, sys\n" + "from graphify.dedup import deduplicate_entities\n" + "nodes = [\n" + " {'id': 'shenzhen', 'label': 'SHENZHEN INTERNATIONAL',\n" + " 'file_type': 'concept', 'source_file': 'doc1.md'},\n" + " {'id': 'shenzhen_intl', 'label': 'Shenzhen international',\n" + " 'file_type': 'concept', 'source_file': 'doc2.md'},\n" + " {'id': 'shenzhen_international', 'label': 'shenzhen-international',\n" + " 'file_type': 'concept', 'source_file': 'doc3.md'},\n" + "]\n" + "random.Random(int(sys.argv[1])).shuffle(nodes)\n" + "out, _ = deduplicate_entities(nodes, [], communities={})\n" + "print(len(out), sorted(n['id'] for n in out)[0])\n" + ) + results = set() + for seed in ("0", "1", "2", "3"): + env = {**os.environ, "PYTHONHASHSEED": seed} + r = subprocess.run( + [sys.executable, "-c", script, seed], + capture_output=True, text=True, env=env, + ) + assert r.returncode == 0, r.stderr + results.add(r.stdout.strip().splitlines()[-1]) + assert results == {"1 shenzhen"}, ( + f"non-deterministic dedup across hash seeds: {results}" + ) + + +def test_crossfile_concept_merge_is_transitive(): + """Exact cross-file matches and a punctuation variant collapse to one + survivor: {'Acme Corp' doc1, 'Acme Corp' doc2, 'Acme Corp.' doc3} all + normalize to 'acme corp' and must transitively union (#2182).""" + nodes = [ + {"id": "acme_corp_one", "label": "Acme Corp", + "file_type": "concept", "source_file": "doc1.md"}, + {"id": "acme_corp_two", "label": "Acme Corp", + "file_type": "concept", "source_file": "doc2.md"}, + {"id": "acme_corp_three", "label": "Acme Corp.", + "file_type": "concept", "source_file": "doc3.md"}, + ] + result_nodes, _ = deduplicate_entities(nodes, [], communities={}) + assert len(result_nodes) == 1 From 334cff6172ab8684eb45327ec76bc8df70e0cbd3 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Sun, 26 Jul 2026 11:52:20 +0100 Subject: [PATCH 276/292] fix(cache): portable stat-index keys + normalized semantic source_file (#2199, #2197) stat-index.json keyed entries by absolute path and never pruned, so a moved/cloned corpus got 0% cache hits and the index grew unbounded. Keys are now stored root-relative and re-anchored on load (mirroring the manifest.json portability fix), and dead-file entries are pruned on flush. save_semantic_cache also normalizes source_file to root-relative before persisting so an absolute/backslash fragment can't poison later updates. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/cache.py | 134 +++++++++++++-- tests/test_stat_index_portability.py | 243 +++++++++++++++++++++++++++ 2 files changed, 366 insertions(+), 11 deletions(-) create mode 100644 tests/test_stat_index_portability.py diff --git a/graphify/cache.py b/graphify/cache.py index 0a86a7e80..7257aca72 100644 --- a/graphify/cache.py +++ b/graphify/cache.py @@ -187,9 +187,48 @@ def _body_content(content: bytes) -> bytes: # skip the cache reads and re-dispatch everything when needed (#1894). _stat_index: dict[str, dict] = {} _stat_index_root: Path | None = None +# Key anchor for the ON-DISK index (#2199): the first caller's key-root, i.e. +# the corpus. Distinct from _stat_index_root, which is the cache-FILE location +# (cache_root, #1774) — the two differ under --out and must not be conflated. +_stat_index_anchor: Path | None = None _stat_index_dirty: bool = False +def _stat_key_to_relative(key: str, anchor: Path) -> str: + """Return ``key`` as a forward-slash relative path from ``anchor``. + + Local duplicate of :func:`graphify.detect._to_relative_for_storage` — + detect imports cache, so cache cannot import detect without a cycle + (and pulling detect in during the atexit flush would be fragile). + Out-of-anchor and already-relative keys pass through unchanged, and + ``..``-escaping relpaths are rejected (kept absolute), mirroring the + manifest's portability rules. + """ + p = Path(key) + if not p.is_absolute(): + return key + try: + rel = os.path.relpath(p, anchor) + except (ValueError, OSError): + return key # outside anchor (e.g. Windows cross-drive) + if rel == ".." or rel.startswith(".." + os.sep) or rel.startswith("../"): + return key # escaped anchor — keep absolute + return rel.replace(os.sep, "/") + + +def _stat_key_to_absolute(key: str, anchor: Path) -> str: + """Inverse of :func:`_stat_key_to_relative`. + + Re-anchor a stored relative key against ``anchor``. Already-absolute keys + (legacy indexes, out-of-anchor entries) pass through unchanged so an index + written by an older graphify remains readable. + """ + p = Path(key) + if p.is_absolute(): + return str(p) + return str(anchor / p) + + def _stat_index_file(root: Path) -> Path: _out = Path(_GRAPHIFY_OUT) base = _out if _out.is_absolute() else Path(root).resolve() / _out @@ -197,22 +236,36 @@ def _stat_index_file(root: Path) -> Path: def _ensure_stat_index(root: Path, cache_root: "Path | None" = None) -> None: - global _stat_index, _stat_index_root, _stat_index_dirty + global _stat_index, _stat_index_root, _stat_index_anchor, _stat_index_dirty if _stat_index_root is not None: return - # The stat index only determines the cache FILE location (entry keys are - # absolute paths), so honoring an explicit cache_root keeps detect()'s - # word-count cache under the requested --out dir instead of polluting the - # scanned corpus with a stray graphify-out/ (#1747). + # _stat_index_root determines the cache FILE location, so honoring an + # explicit cache_root keeps detect()'s word-count cache under the requested + # --out dir instead of polluting the scanned corpus with a stray + # graphify-out/ (#1747). _stat_index_anchor is the separate KEY anchor: + # in-memory keys stay absolute, but the on-disk index stores in-anchor keys + # relative so a moved/cloned corpus still hits (#2199) — same load/save + # re-anchoring the detect manifest uses. _stat_index_root = Path(cache_root if cache_root is not None else root).resolve() + _stat_index_anchor = Path(root).resolve() p = _stat_index_file(_stat_index_root) + _stat_index = {} if p.exists(): try: - _stat_index = json.loads(p.read_text(encoding="utf-8")) + raw = json.loads(p.read_text(encoding="utf-8")) + if isinstance(raw, dict): + for k, v in raw.items(): + if not isinstance(k, str): + continue + if Path(k).is_absolute(): + # Legacy/out-of-anchor key: pass through, but never + # clobber a re-anchored relative (new-format) entry + # that resolved to the same absolute path. + _stat_index.setdefault(k, v) + else: + _stat_index[_stat_key_to_absolute(k, _stat_index_anchor)] = v except (json.JSONDecodeError, OSError): _stat_index = {} - else: - _stat_index = {} atexit.register(_flush_stat_index) @@ -221,11 +274,26 @@ def _flush_stat_index() -> None: if not _stat_index_dirty or _stat_index_root is None: return p = _stat_index_file(_stat_index_root) + # Build the on-disk form (#2199): prune entries whose file is gone (the + # index otherwise grows without bound), then store in-anchor keys as + # forward-slash relative paths so the index survives a corpus move/clone. + # Out-of-anchor keys stay absolute (same rule as the detect manifest); a + # reader tells the formats apart by absoluteness, so no version marker is + # needed. In-memory keys are untouched — only the serialization changes. + on_disk: dict[str, dict] = {} + for k, v in _stat_index.items(): + try: + if not os.path.exists(k): + continue + except OSError: + continue + dk = _stat_key_to_relative(k, _stat_index_anchor) if _stat_index_anchor is not None else k + on_disk[dk] = v try: p.parent.mkdir(parents=True, exist_ok=True) fd, tmp = tempfile.mkstemp(dir=p.parent, prefix="stat-index.", suffix=".tmp") try: - os.write(fd, json.dumps(_stat_index, separators=(",", ":")).encode()) + os.write(fd, json.dumps(on_disk, separators=(",", ":")).encode()) os.close(fd) os.replace(tmp, p) except Exception: @@ -423,6 +491,29 @@ def _relativize_source_files_in(payload: dict, root: Path) -> None: item["source_file"] = rel.replace(os.sep, "/") +def _normalize_source_file_value(src: "str | Path", root_resolved: Path) -> str: + """Return ``src`` in portable form: backslashes flipped to forward slashes, + then relativized against ``root_resolved`` when the path is in-root. + + Windows ``detect()`` emits absolute backslash paths, and a semantic + fragment carrying one verbatim used to be persisted as-is — poisoning later + ``graphify update`` runs with a machine-specific ``source_file`` (#2197). + Out-of-root absolute paths pass through (slash-normalized only), same + in/out rule and ``..``-rejection as :func:`_relativize_source_files_in`. + """ + s = str(src).replace("\\", "/") + p = Path(s) + if not p.is_absolute(): + return s + try: + rel = os.path.relpath(p, root_resolved) + except (ValueError, OSError): + return s # out-of-root (e.g. Windows cross-drive) + if rel == ".." or rel.startswith(".." + os.sep) or rel.startswith("../"): + return s # escaped root — keep absolute + return rel.replace(os.sep, "/") + + def _absolutize_source_files_in(payload: dict, root: Path) -> None: """Inverse of :func:`_relativize_source_files_in`. @@ -869,22 +960,43 @@ def save_semantic_cache( from collections import defaultdict kind = "semantic" if mode is None else f"semantic-{mode}" + root_path = Path(root).resolve() + + def _normalized(item: dict) -> dict: + """Copy of ``item`` with a portable ``source_file`` (#2197). + + Normalizing BEFORE grouping means both the group key and the persisted + item carry the relative forward-slash form, so a fragment whose + source_file arrived absolute (Windows detect() output) can never be + cached verbatim. A shallow copy keeps the caller's dicts untouched — + downstream steps may still rely on the original absolute shape (same + reasoning as :func:`save_cached`'s on-disk deepcopy). + """ + src = item.get("source_file") + if not src: + return item + norm = _normalize_source_file_value(src, root_path) + if norm != src: + item = {**item, "source_file": norm} + return item + by_file: dict[str, dict] = defaultdict(lambda: {"nodes": [], "edges": [], "hyperedges": []}) for n in nodes: + n = _normalized(n) src = n.get("source_file", "") if src: by_file[src]["nodes"].append(n) for e in edges: + e = _normalized(e) src = e.get("source_file", "") if src: by_file[src]["edges"].append(e) for h in (hyperedges or []): + h = _normalized(h) src = h.get("source_file", "") if src: by_file[src]["hyperedges"].append(h) - root_path = Path(root).resolve() - def resolved_source_path(value: str | Path) -> Path: path = Path(value) if not path.is_absolute(): diff --git a/tests/test_stat_index_portability.py b/tests/test_stat_index_portability.py new file mode 100644 index 000000000..51ae36a67 --- /dev/null +++ b/tests/test_stat_index_portability.py @@ -0,0 +1,243 @@ +"""#2199 — stat-index.json must be portable and self-pruning. + +The on-disk stat index used to key entries by resolved ABSOLUTE path, so a +corpus reached via a different absolute path (clone, move, second mount) got +0% cache hits (100% re-extraction), and entries for deleted files were never +pruned (unbounded growth). In-memory keys stay absolute; only the on-disk +form is relativized against the key anchor — mirroring the detect manifest's +_to_relative_for_storage/_to_absolute_from_storage round-trip. + +Also covers #2197 (cache.py portion): save_semantic_cache must normalize each +item's source_file (backslashes -> forward slashes, relativize when in-root) +before persisting, so a fragment carrying an absolute path (Windows detect() +output) cannot poison the cache. +""" +from __future__ import annotations + +import hashlib +import json +import os +import shutil +from pathlib import Path + +from graphify import cache + + +def _reset_stat_index(): + """The stat-index location/anchor are chosen once per process via module + globals (#1747/#2199). Reset them so each test sees a fresh-process + decision — same pattern as tests/test_extract_cache_location.py.""" + cache._stat_index_root = None + cache._stat_index_anchor = None + cache._stat_index = {} + cache._stat_index_dirty = False + + +def _stat_index_path(root: Path) -> Path: + return root / "graphify-out" / "cache" / "stat-index.json" + + +def _read_index(root: Path) -> dict: + return json.loads(_stat_index_path(root).read_text(encoding="utf-8")) + + +def _count_read_bytes(monkeypatch): + """Wrap Path.read_bytes with a call counter (file_hash's content read).""" + calls = {"n": 0} + orig = Path.read_bytes + + def counting(self): + calls["n"] += 1 + return orig(self) + + monkeypatch.setattr(Path, "read_bytes", counting) + return calls + + +def _fail_compute(p: Path) -> int: + raise AssertionError(f"word-count compute invoked for {p}; expected a warm stat hit") + + +def test_cache_hits_survive_corpus_move(tmp_path, monkeypatch): + """Run A under tmp/a, copy the corpus (with graphify-out/) to tmp/b: run B + must be 100% warm — zero content reads, zero word-count computes, digests + identical to run A.""" + _reset_stat_index() + a = tmp_path / "a" + a.mkdir() + (a / "f1.py").write_text("x = 1\n") + sub = a / "sub" + sub.mkdir() + (sub / "f2.md").write_text("hello world one two\n") + + digests_a = { + "f1.py": cache.file_hash(a / "f1.py", a), + "sub/f2.md": cache.file_hash(sub / "f2.md", a), + } + wc_a = cache.cached_word_count(a / "f1.py", a, lambda p: len(p.read_text().split())) + cache._flush_stat_index() + + on_disk = _read_index(a) + assert on_disk, "flush should have written entries" + for k in on_disk: + assert not os.path.isabs(k), f"absolute key leaked to disk: {k}" + assert "\\" not in k, f"non-portable separator in key: {k}" + assert set(on_disk) == {"f1.py", "sub/f2.md"} + + # Move the corpus (graphify-out/ rides along; copy2 preserves mtime_ns). + b = tmp_path / "b" + shutil.copytree(a, b, copy_function=shutil.copy2) + + _reset_stat_index() + reads = _count_read_bytes(monkeypatch) + assert cache.file_hash(b / "f1.py", b) == digests_a["f1.py"] + assert cache.file_hash(b / "sub" / "f2.md", b) == digests_a["sub/f2.md"] + assert cache.cached_word_count(b / "f1.py", b, _fail_compute) == wc_a + assert reads["n"] == 0, "moved corpus should be served entirely from the stat index" + + +def test_deleted_entries_are_pruned_on_flush(tmp_path): + _reset_stat_index() + a = tmp_path / "a" + a.mkdir() + f1 = a / "f1.py" + f1.write_text("x = 1\n") + f2 = a / "f2.py" + f2.write_text("y = 2\n") + cache.file_hash(f1, a) + cache.file_hash(f2, a) + cache._flush_stat_index() + assert set(_read_index(a)) == {"f1.py", "f2.py"} + + f2.unlink() + _reset_stat_index() + # Bump f1's mtime so the re-hash dirties the index and a flush is written. + os.utime(f1, ns=(f1.stat().st_atime_ns, f1.stat().st_mtime_ns + 1_000_000)) + cache.file_hash(f1, a) + cache._flush_stat_index() + + on_disk = _read_index(a) + assert set(on_disk) == {"f1.py"}, "deleted f2.py should have been pruned" + assert not os.path.isabs(next(iter(on_disk))) + + +def test_legacy_absolute_index_migrates_gracefully(tmp_path, monkeypatch): + """A pre-#2199 index keyed by absolute paths still HITS on the unmoved + root, and the first flush prunes dead entries and rewrites live keys + relative (self-heals).""" + _reset_stat_index() + a = tmp_path / "a" + a.mkdir() + f1 = a / "f1.py" + f1.write_text("x = 1\n") + st = f1.stat() + salt = "f1.py" + digest = hashlib.sha256(f1.read_bytes() + b"\x00" + salt.encode()).hexdigest() + + dead = tmp_path / "dead" # never created + legacy = { + str(f1.resolve()): {"size": st.st_size, "mtime_ns": st.st_mtime_ns, + "hashes": {salt: digest}}, + str(dead / "x.py"): {"size": 1, "mtime_ns": 1, "hashes": {"x.py": "aa"}}, + str(dead / "y.py"): {"size": 2, "mtime_ns": 2, "hashes": {"y.py": "bb"}}, + } + p = _stat_index_path(a) + p.parent.mkdir(parents=True) + p.write_text(json.dumps(legacy), encoding="utf-8") + + reads = _count_read_bytes(monkeypatch) + assert cache.file_hash(f1, a) == digest + assert reads["n"] == 0, "legacy absolute key should still serve a warm hit" + + # Force a write so the self-heal is observable (a pure warm run leaves the + # index clean and flush is a no-op by design). + cache._stat_index_dirty = True + cache._flush_stat_index() + + on_disk = _read_index(a) + assert set(on_disk) == {"f1.py"}, "dead absolute keys should be pruned" + assert on_disk["f1.py"]["hashes"][salt] == digest + + +def test_out_of_root_key_round_trips_absolute(tmp_path, monkeypatch): + _reset_stat_index() + a = tmp_path / "a" + a.mkdir() + outside = tmp_path / "outside.txt" + outside.write_text("out of root\n") + + d1 = cache.file_hash(outside, a) + cache._flush_stat_index() + + on_disk = _read_index(a) + assert set(on_disk) == {str(outside.resolve())}, "out-of-root key must stay absolute" + + _reset_stat_index() + reads = _count_read_bytes(monkeypatch) + assert cache.file_hash(outside, a) == d1 + assert reads["n"] == 0, "second call should be a stat hit" + + +def test_relative_key_wins_over_colliding_legacy_absolute(tmp_path): + """When an old absolute key and a new relative key resolve to the same + file, the relative (new-format) entry wins on load.""" + _reset_stat_index() + a = tmp_path / "a" + a.mkdir() + f1 = a / "f1.py" + f1.write_text("x = 1\n") + p = _stat_index_path(a) + p.parent.mkdir(parents=True) + p.write_text(json.dumps({ + str(f1.resolve()): {"size": 1, "mtime_ns": 1, "hashes": {"f1.py": "legacy"}}, + "f1.py": {"size": 2, "mtime_ns": 2, "hashes": {"f1.py": "fresh"}}, + }), encoding="utf-8") + + cache._ensure_stat_index(a) + assert cache._stat_index[str(f1.resolve())]["hashes"]["f1.py"] == "fresh" + + +def test_semantic_cache_normalizes_absolute_source_file(tmp_path): + """#2197: an item whose source_file is absolute is persisted root-relative + posix, and the caller's dict is not mutated.""" + _reset_stat_index() + root = tmp_path / "corpus" + root.mkdir() + f = root / "m.py" + f.write_text("x = 1\n") + + node = {"id": "m.x", "type": "variable", "source_file": str(f.resolve())} + saved = cache.save_semantic_cache([node], [], root=root) + assert saved == 1 + assert node["source_file"] == str(f.resolve()), "caller's dict must not be mutated" + + entries = list((root / "graphify-out" / "cache" / "semantic").glob("*.json")) + assert len(entries) == 1 + persisted = json.loads(entries[0].read_text(encoding="utf-8")) + assert persisted["nodes"][0]["source_file"] == "m.py" + + # Replay resolves back to the same absolute shape a fresh extraction has. + _, _, _, uncached = cache.check_semantic_cache([str(f)], root=root) + assert uncached == [] + + +def test_semantic_cache_normalizes_backslash_poisoned_source_file(tmp_path): + """A Windows-shaped absolute source_file (backslash separators) must be + slash-normalized and relativized instead of being skipped/persisted raw.""" + _reset_stat_index() + root = tmp_path / "corpus" + root.mkdir() + sub = root / "sub" + sub.mkdir() + f = sub / "n.py" + f.write_text("y = 2\n") + + poisoned = str(root.resolve()) + "\\sub\\n.py" + node = {"id": "n.y", "type": "variable", "source_file": poisoned} + saved = cache.save_semantic_cache([node], [], root=root) + assert saved == 1 + + entries = list((root / "graphify-out" / "cache" / "semantic").glob("*.json")) + assert len(entries) == 1 + persisted = json.loads(entries[0].read_text(encoding="utf-8")) + assert persisted["nodes"][0]["source_file"] == "sub/n.py" From 8adb261d16d6c801066578fff72ec281608dadc2 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Sun, 26 Jul 2026 11:52:20 +0100 Subject: [PATCH 277/292] fix(build): fold legacy node/edge aliases and re-key absolute-derived semantic ids (#2194, #2197) build_from_json now folds name->label, path->source_file, edge type->relation, and confidence_score->confidence=INFERRED before validation, so alias-carrying nodes stop entering the graph without label/source_file (invisible, unmergeable ghosts); the same folds run before dedup. _semantic_id_remap now also learns the absolute-path stem form, so a Windows absolute-derived semantic id re-keys to the canonical root-relative id. The extraction warning now breaks errors down by cause. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/build.py | 87 +++++++++++++++++++++++++- tests/test_build.py | 139 +++++++++++++++++++++++++++++++++++++++++ tests/test_validate.py | 21 +++++++ 3 files changed, 245 insertions(+), 2 deletions(-) diff --git a/graphify/build.py b/graphify/build.py index bfe8fe958..7e7ed6157 100644 --- a/graphify/build.py +++ b/graphify/build.py @@ -124,6 +124,39 @@ def _normalize_hyperedge_members(he: object) -> None: he.pop(alias, None) +def _fold_node_aliases(node: dict) -> None: + """Fold legacy node field aliases onto canonical keys, in place (#2194). + + ``name`` -> ``label`` and ``path`` -> ``source_file``. Uses an empty-check + (not mere key presence) so a node carrying ``label: ""``/``None`` next to a + real ``name`` is healed too. When the canonical field already holds a value + it wins and the alias key is left untouched. Without this fold an alias-only + node enters the graph with no label/source_file: it fails validation, gets + ``norm_label == ""`` (invisible to query/explain), and is excluded from every + label-keyed merge/dedup — a permanent ghost that ``graphify update`` + re-feeds through build_from_json forever. + """ + if not node.get("label") and isinstance(node.get("name"), str) and node["name"]: + node["label"] = node.pop("name") + if not node.get("source_file") and isinstance(node.get("path"), str) and node["path"]: + node["source_file"] = node.pop("path") + + +def _fold_edge_aliases(edge: dict) -> None: + """Fold legacy edge field aliases onto canonical keys, in place (#2194). + + ``type`` -> ``relation``. A ``confidence_score`` float with no ``confidence`` + enum backfills ``confidence: "INFERRED"`` — never EXTRACTED (alias recovery + is not provenance) and never a threshold mapping of the float. The + ``confidence_score`` key itself is NOT popped: it is a legitimate companion + field that the edge loop sanitizes and to_json round-trips. + """ + if not edge.get("relation") and isinstance(edge.get("type"), str) and edge["type"]: + edge["relation"] = edge.pop("type") + if not edge.get("confidence") and edge.get("confidence_score") is not None: + edge["confidence"] = "INFERRED" + + def _norm_source_file(p: str | None, root: str | None = None) -> str | None: """Normalize path separators and relativize absolute paths. @@ -395,7 +428,23 @@ def _semantic_id_remap(nodes: list, root: str | None) -> dict: if norm_nid == new_stem or norm_nid.startswith(new_stem + "_"): continue new_id: str | None = None - for old_stem in _old_file_stems(rel): + old_forms = _old_file_stems(rel) + # #2197: on Windows, detect() can emit an ABSOLUTE source_file, and a + # semantic fragment's id derived from that absolute path (e.g. + # d_projects_myrepo_docs_dataflow) matches neither the canonical + # relative stem nor the legacy short forms above — so while source_file + # itself is healed by _norm_source_file, the id would ghost against the + # existing graph's docs_dataflow. When the raw path was absolute and + # relativized under root, treat the raw-absolute stem as one more + # old-stem form — the semantic-side twin of extract.py's absolute-form + # id registration. It is the longest form, so it goes first (greedy + # prefix stripping, same ordering rule as _old_file_stems). + sf_raw = str(sf).replace("\\", "/") + if sf_raw != sf_norm and os.path.isabs(sf_raw): + abs_stem = make_id(_file_stem(Path(sf_raw))) + if abs_stem and abs_stem != new_stem and abs_stem not in old_forms: + old_forms.insert(0, abs_stem) + for old_stem in old_forms: if old_stem == new_stem: continue # already canonical for this form if norm_nid == old_stem: @@ -518,6 +567,11 @@ def build_from_json(extraction: dict, *, directed: bool = False, root: str | Pat file=sys.stderr, ) node["source_file"] = node.pop("source") + # Fold the remaining legacy node aliases (`name`->`label`, + # `path`->`source_file`, #2194) before validation and before the + # semantic-rekey / ghost-merge passes below, all of which key on + # label/source_file and would otherwise skip the node entirely. + _fold_node_aliases(node) # Default missing/None file_type to "concept" so legacy graph.json # entries (and stub nodes preserved by `_rebuild_code` from older # graphify versions that didn't always populate file_type) don't @@ -536,11 +590,33 @@ def build_from_json(extraction: dict, *, directed: bool = False, root: str | Pat for he in extraction.get("hyperedges", []) or []: _normalize_hyperedge_members(he) + # Fold legacy edge field aliases (`type`->`relation`, + # `confidence_score`->`confidence`, #2194) BEFORE validation. The existing + # from/to endpoint fold lives in the edge loop further down, which runs + # after validate_extraction — too late for fields the validator requires. + for edge in extraction.get("edges", []): + if isinstance(edge, dict): + _fold_edge_aliases(edge) + errors = validate_extraction(extraction) # Dangling edges (stdlib/external imports) are expected - only warn about real schema errors. real_errors = [e for e in errors if "does not match any node id" not in e] if real_errors: - print(f"[graphify] Extraction warning ({len(real_errors)} issues): {real_errors[0]}", file=sys.stderr) + # Break the warning down by cause (#2194): a mixed batch used to surface + # only real_errors[0], hiding every other failure mode. Group on the + # "missing required field 'X'" suffix and report per-cause counts plus + # one example each, so the operator sees the full shape of the damage. + by_cause: dict[str, list[str]] = {} + for err in real_errors: + m = re.search(r"missing required field '[^']*'", err) + by_cause.setdefault(m.group(0) if m else "other schema issue", []).append(err) + breakdown = "; ".join( + f"{len(errs)}x {cause} (e.g. {errs[0]})" for cause, errs in by_cause.items() + ) + print( + f"[graphify] Extraction warning ({len(real_errors)} issues): {breakdown}", + file=sys.stderr, + ) # Deterministic semantic re-key (#1504/#1509): the node-ID stem is now the # full repo-relative path (docs/v1/api/README.md -> docs_v1_api_readme), but # the semantic cache is UNVERSIONED, so a cached/LLM fragment can still carry @@ -972,6 +1048,13 @@ def build( combined["input_tokens"] += ext.get("input_tokens", 0) combined["output_tokens"] += ext.get("output_tokens", 0) if dedup and combined["nodes"]: + # Fold legacy node field aliases before dedup (#2194): dedup runs BEFORE + # build_from_json and keys on `label`, so a `name`/`path` alias node + # would be invisible to it and only label-dedup one build later, after + # build_from_json's own fold has healed the persisted graph.json. + for n in combined["nodes"]: + if isinstance(n, dict): + _fold_node_aliases(n) combined["nodes"], combined["edges"] = deduplicate_entities( combined["nodes"], combined["edges"], communities={}, dedup_llm_backend=dedup_llm_backend, diff --git a/tests/test_build.py b/tests/test_build.py index 3d8582c35..0851ba75e 100644 --- a/tests/test_build.py +++ b/tests/test_build.py @@ -135,6 +135,145 @@ def test_legacy_edge_from_to_canonicalized(): assert G.number_of_edges() == 1 +def test_legacy_node_name_path_aliases_folded(): + """#2194: nodes carrying `name`/`path` instead of `label`/`source_file` must + be canonicalized before validation, not enter the graph as label-less + ghosts. After build the canonicalized dict also passes validation.""" + from graphify.validate import validate_extraction + ext = {"nodes": [{"id": "n1", "name": "Foo", "path": "a/b.md", "file_type": "concept"}], + "edges": [], "input_tokens": 0, "output_tokens": 0} + G = build_from_json(ext) + attrs = G.nodes["n1"] + assert attrs["label"] == "Foo" + assert attrs["source_file"] == "a/b.md" + assert "name" not in attrs + assert "path" not in attrs + # build_from_json canonicalizes in place; the extraction dict must now be + # schema-valid (no missing-field errors for the alias node). + assert not [e for e in validate_extraction(ext) if "missing required field" in e] + + +def test_legacy_edge_type_confidence_score_aliases_folded(): + """#2194: edges carrying `type`/`confidence_score` instead of + `relation`/`confidence` fold to canonical fields. Recovery confidence is + INFERRED (never EXTRACTED — alias recovery is not provenance) and the + companion confidence_score float is retained, not popped.""" + ext = {"nodes": [{"id": "n1", "label": "A", "file_type": "code", "source_file": "a.py"}, + {"id": "n2", "label": "B", "file_type": "code", "source_file": "b.py"}], + "edges": [{"source": "n1", "target": "n2", "type": "references", + "confidence_score": 0.9, "source_file": "a.py"}], + "input_tokens": 0, "output_tokens": 0} + G = build_from_json(ext) + data = edge_data(G, "n1", "n2") + assert data["relation"] == "references" + assert data["confidence"] == "INFERRED" + assert data["confidence_score"] == 0.9 + assert "type" not in data + + +def test_node_alias_canonical_field_wins(): + """#2194: when both the canonical field and its alias are present, the + canonical value wins and the alias key is left untouched.""" + ext = {"nodes": [{"id": "n1", "label": "Real", "name": "Alias", + "file_type": "code", "source_file": "a.py"}], + "edges": [], "input_tokens": 0, "output_tokens": 0} + G = build_from_json(ext) + assert G.nodes["n1"]["label"] == "Real" + assert G.nodes["n1"]["name"] == "Alias" # preserved, not consumed + + +def test_alias_node_ghost_merges_into_ast_twin(): + """#2194: an alias-only semantic node (name/path) must participate in the + AST/LLM ghost merge once folded — same label and file as an AST node with a + different id collapses into the AST node instead of surviving as a ghost.""" + ext = {"nodes": [ + {"id": "src_foo_helper", "label": "helper", "file_type": "code", + "source_file": "src/foo.py", "_origin": "ast", "source_location": "L10"}, + {"id": "helper_ghost", "name": "helper", "path": "src/foo.py", + "file_type": "code"}, + ], "edges": [], "input_tokens": 0, "output_tokens": 0} + G = build_from_json(ext) + assert "src_foo_helper" in G.nodes + assert "helper_ghost" not in G.nodes + assert G.number_of_nodes() == 1 + + +def test_alias_node_gets_nonempty_norm_label(tmp_path): + """#2194: a recovered alias node must serialize with a non-empty norm_label + so query/explain can find it.""" + from graphify.export import to_json + ext = {"nodes": [{"id": "n1", "name": "Foo", "path": "a/b.md", "file_type": "concept"}], + "edges": [], "input_tokens": 0, "output_tokens": 0} + G = build_from_json(ext) + out = tmp_path / "graph.json" + assert to_json(G, {}, str(out)) + data = json.loads(out.read_text()) + node = next(n for n in data["nodes"] if n["id"] == "n1") + assert node["norm_label"] == "foo" + + +def test_extraction_warning_breakdown_by_cause(capsys): + """#2194: a mixed batch of schema errors must report per-cause counts, not + just the first error.""" + ext = {"nodes": [ + {"id": "n1", "label": "A", "file_type": "code", "source_file": "a.py"}, + {"id": "n2", "label": "B", "file_type": "code", "source_file": "b.py"}, + # two nodes missing label (and carrying no name alias) + {"id": "x1", "file_type": "code", "source_file": "x.py"}, + {"id": "x2", "file_type": "code", "source_file": "x.py"}, + ], "edges": [ + # three edges missing relation (and carrying no type alias) + {"source": "n1", "target": "n2", "confidence": "EXTRACTED", "source_file": "a.py"}, + {"source": "n2", "target": "n1", "confidence": "EXTRACTED", "source_file": "a.py"}, + {"source": "n1", "target": "x1", "confidence": "EXTRACTED", "source_file": "a.py"}, + ], "input_tokens": 0, "output_tokens": 0} + build_from_json(ext) + err = capsys.readouterr().err + assert "2x missing required field 'label'" in err + assert "3x missing required field 'relation'" in err + + +def test_absolute_derived_semantic_ids_rekeyed(tmp_path): + """#2197: a semantic fragment whose ids were derived from an ABSOLUTE + source_file (Windows detect() emits them) must re-key to the canonical + repo-relative stem instead of ghosting against the existing graph.""" + from graphify.ids import make_id + (tmp_path / "docs").mkdir() + abs_sf = str(tmp_path / "docs" / "DATAFLOW.md") + abs_stem = make_id(str(tmp_path / "docs" / "DATAFLOW")) + ext = {"nodes": [ + {"id": abs_stem, "label": "DATAFLOW.md", "file_type": "document", + "source_file": abs_sf}, + {"id": f"{abs_stem}_pipeline", "label": "Pipeline", "file_type": "concept", + "source_file": abs_sf}, + ], "edges": [ + {"source": abs_stem, "target": f"{abs_stem}_pipeline", "relation": "describes", + "confidence": "INFERRED", "source_file": abs_sf, "weight": 1.0}, + ], "input_tokens": 0, "output_tokens": 0} + G = build_from_json(ext, root=tmp_path) + assert "docs_dataflow" in G.nodes + assert "docs_dataflow_pipeline" in G.nodes + assert abs_stem not in G.nodes + assert G.nodes["docs_dataflow"]["source_file"] == "docs/DATAFLOW.md" + assert G.has_edge("docs_dataflow", "docs_dataflow_pipeline") + + +def test_absolute_derived_semantic_ids_rekeyed_backslash(tmp_path): + """#2197 (separator variant): the same absolute-derived-id fragment with + backslash separators in source_file re-keys identically.""" + from graphify.ids import make_id + (tmp_path / "docs").mkdir() + abs_sf = str(tmp_path / "docs" / "DATAFLOW.md").replace("/", "\\") + abs_stem = make_id(str(tmp_path / "docs" / "DATAFLOW")) + ext = {"nodes": [ + {"id": f"{abs_stem}_pipeline", "label": "Pipeline", "file_type": "concept", + "source_file": abs_sf}, + ], "edges": [], "input_tokens": 0, "output_tokens": 0} + G = build_from_json(ext, root=tmp_path) + assert "docs_dataflow_pipeline" in G.nodes + assert G.nodes["docs_dataflow_pipeline"]["source_file"] == "docs/DATAFLOW.md" + + def test_source_file_backslash_normalized(): """Windows backslash paths and POSIX paths for the same file must produce one node.""" extraction = { diff --git a/tests/test_validate.py b/tests/test_validate.py index ea865faeb..013fa8721 100644 --- a/tests/test_validate.py +++ b/tests/test_validate.py @@ -87,6 +87,27 @@ def test_assert_valid_passes_silently(): assert_valid(VALID) # should not raise +def test_legacy_aliases_valid_after_build_canonicalization(): + # #2194: build_from_json folds legacy aliases (name->label, + # path->source_file, type->relation, confidence_score->confidence) in + # place BEFORE validation, so an alias-only extraction that fails + # validation raw is fully schema-valid after canonicalization. + from graphify.build import build_from_json + data = { + "nodes": [ + {"id": "n1", "name": "Foo", "path": "a/b.md", "file_type": "concept"}, + {"id": "n2", "label": "Bar", "file_type": "code", "source_file": "bar.py"}, + ], + "edges": [ + {"source": "n1", "target": "n2", "type": "references", + "confidence_score": 0.9, "source_file": "a/b.md"}, + ], + } + assert any("missing required field" in e for e in validate_extraction(data)) + build_from_json(data) + assert validate_extraction(data) == [] + + def test_non_hashable_node_id_reported_not_raised(): # A malformed LLM extraction can emit a list-valued id. The validator must # report it as an error string (its documented contract) rather than crash From 0858954db94c39e6431f8c3ec504d0e135fd3561 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Sun, 26 Jul 2026 12:43:17 +0100 Subject: [PATCH 278/292] feat(csharp): namespace-aware member-call resolution + shadow poisoning (#1609) Adapted from #1620 by @TheFedaikin, reworked onto v8 as a focused change (without the module split or the references-fallback behavior change). Builds on the shipped #1609 resolver: instead of bailing when a receiver's class name is ambiguous corpus-wide, the declared type is resolved with a shared CsharpNameResolver (same-namespace, using-directive, and alias aware) against the caller's namespace/scope, falling back to the unique bare match only when scoping is non-decisive. Adds base./this.field receivers and inherited-member lookup through the inherits chain (an out-of-corpus base poisons the lookup, so no wrong edge). The per-file type table now poisons a name on any conflicting rebinding, killing the wrong-edge class where a local shadows a field of a different type. C#-gated; never emits a wrong edge. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/extract.py | 126 +++++++++++++++-- graphify/extractors/csharp.py | 228 ++++++++++++++++++------------ graphify/extractors/engine.py | 68 ++++++--- tests/test_csharp_member_calls.py | 184 ++++++++++++++++++++++++ 4 files changed, 481 insertions(+), 125 deletions(-) diff --git a/graphify/extract.py b/graphify/extract.py index d55f45664..0835a9795 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -34,6 +34,7 @@ from graphify.extractors.bash import extract_bash # noqa: F401 from graphify.extractors.blade import extract_blade # noqa: F401 from graphify.extractors.csharp import ( + CsharpNameResolver, _resolve_cross_file_csharp_imports, _resolve_csharp_type_references, ) @@ -2517,20 +2518,32 @@ def _resolve_csharp_member_calls( all_edges: list[dict], ) -> None: """Resolve C# member calls (``recv.Method()``) to the receiver's declared type - (#1609). + (#1609), namespace-aware (#1620). The shared cross-file pass drops every ``is_member_call`` because a bare method name collides across the corpus — and for C# an in-file bare match silently mis-bound ``_server.Save()`` to an unrelated ``Cache.Save()``. The C# extractor now records each member call's receiver plus a per-file ``name -> Type`` table - (``csharp_type_table``) of fields/properties/params/locals. This pass types the - receiver, then emits an edge ONLY when that type resolves to exactly ONE - definition (the god-node guard); an untypable receiver is skipped (no guess). + (``csharp_type_table``) of fields/properties/params/locals (with conflicting + rebindings POISONED out, so a shadowing local of a different type produces no + edge rather than a wrong one). This pass types the receiver, then resolves the + declared type name with the same namespace/using/alias scoping machinery the + type-reference pass uses (``CsharpNameResolver``), so a class name duplicated + across namespaces still binds to the one in scope; only when scoping knows + nothing about the name does it fall back to the corpus-wide unique bare-name + match (the god-node guard). An untypable/ambiguous receiver is skipped — never + a guess. Receiver typing, by precision tier: * ``this.M()`` — receiver is the caller's own enclosing class -> EXTRACTED. + * ``base.M()`` — the caller's single resolvable base class -> EXTRACTED. * ``Type.M()`` (capitalized) — the type is named explicitly in source -> EXTRACTED. - * ``recv.M()`` — ``recv`` typed via the file's field/param/local table -> INFERRED. + * ``recv.M()`` / ``this.recv.M()`` — ``recv`` typed via the file's + field/param/local table -> INFERRED. + + A method not declared on the receiver's type is looked up through its + ``inherits`` chain; a chain containing an unresolvable (out-of-corpus) base + poisons the lookup — the method may live there, so no edge is emitted. Must run after id-disambiguation so node ids and caller_nids are final. """ @@ -2552,6 +2565,11 @@ def _key(label: str) -> str: if n.get("source_file") and n.get("id") in contained and _is_type_like_definition(n): type_def_nids.setdefault(_key(n.get("label", "")), []).append(n["id"]) + # Namespace/using/alias-aware simple-name resolution, shared with the C# + # type-reference pass (which has already arbitrated inherits/implements/ + # references targets by the time the resolver registry runs). + resolver = CsharpNameResolver(all_nodes, all_edges) + # (type_node_id, method_key) -> method_node_id, and caller -> enclosing type. # C# owns its methods via `method` edges. method_index: dict[tuple[str, str], str] = {} @@ -2566,6 +2584,78 @@ def _key(label: str) -> str: enclosing_type.setdefault(tgt, src) method_index[(src, _key(tnode.get("label", "")))] = tgt + # Base-class chain from `inherits` edges (C# files only). The type-reference + # pass has already re-pointed each resolvable base to its real definition and + # left unresolvable ones on dangling sourceless stubs — a stub target marks + # the derived type's base chain as UNRESOLVED (poison: an inherited-member + # lookup through it must bail, the member may be declared out of corpus). + bases_of: dict[str, list[str]] = {} + unresolved_base: set[str] = set() + for e in all_edges: + if e.get("relation") != "inherits": + continue + src_file = e.get("source_file") + if not (isinstance(src_file, str) and src_file.endswith(".cs")): + continue + src, tgt = e.get("source"), e.get("target") + if not (isinstance(src, str) and isinstance(tgt, str)): + continue + tnode = node_by_id.get(tgt) + if tnode is None or not tnode.get("source_file"): + unresolved_base.add(src) + else: + bucket = bases_of.setdefault(src, []) + if tgt not in bucket: + bucket.append(tgt) + + def _method_on_type_or_bases(type_nid: str, callee_key: str) -> str | None: + """The method's definition on the type or its resolvable base chain. + + A type that declares the method directly wins (overrides shadow the + base). Otherwise walk `inherits` upward; an unresolved base anywhere the + walk actually reaches poisons the lookup (no edge), as does anything + other than exactly one declaration found. + """ + hits: set[str] = set() + seen: set[str] = set() + frontier = [type_nid] + while frontier: + nid = frontier.pop() + if nid in seen: + continue + seen.add(nid) + method_nid = method_index.get((nid, callee_key)) + if method_nid: + hits.add(method_nid) + continue # an override shadows anything above it + if nid in unresolved_base: + return None # the method may live on the out-of-corpus base + frontier.extend(bases_of.get(nid, [])) + return next(iter(hits)) if len(hits) == 1 else None + + def _resolve_type_name_nid(type_name: str | None, caller_node: dict | None, + src_file: str) -> str | None: + """Resolve a declared type name to exactly one definition node id. + + Namespace/using/alias scoping first (so `Svc` duplicated across + namespaces binds to the one in scope); when scoping is decisive but + ambiguous, bail. Only when scoping knows nothing about the name fall + back to the corpus-wide unique bare-name match (which also covers + nested types, absent from the scoped index). + """ + if not type_name: + return None + if caller_node is not None: + resolved, decisive = resolver.resolve_type_name( + type_name, caller_node, src_file + ) + if resolved: + return resolved + if decisive: + return None + type_defs = type_def_nids.get(_key(type_name), []) + return type_defs[0] if len(type_defs) == 1 else None + all_raw_calls: list[dict] = [] for result in per_file: all_raw_calls.extend(result.get("raw_calls", [])) @@ -2580,33 +2670,41 @@ def _key(label: str) -> str: if not receiver or not callee or not caller: continue src_file = rc.get("source_file", "") + caller_node = node_by_id.get(caller) if receiver == "this": type_nid = enclosing_type.get(caller) if not type_nid: continue type_qualified = True + elif receiver == "base": + enclosing = enclosing_type.get(caller) + if not enclosing or enclosing in unresolved_base: + continue + bases = bases_of.get(enclosing, []) + if len(bases) != 1: # no base, or can't tell which — bail + continue + type_nid = bases[0] + type_qualified = True elif receiver[:1].isupper(): # Type.M() — the type is named explicitly (also covers a Pascal-cased # local whose name equals its type, resolved via the table below if the # explicit-type lookup misses). - type_defs = type_def_nids.get(_key(receiver), []) - if len(type_defs) != 1: + type_nid = _resolve_type_name_nid(receiver, caller_node, src_file) + if not type_nid: type_name = type_table_by_file.get(src_file, {}).get(receiver) - type_defs = type_def_nids.get(_key(type_name), []) if type_name else [] - if len(type_defs) != 1: + type_nid = _resolve_type_name_nid(type_name, caller_node, src_file) + if not type_nid: continue - type_nid = type_defs[0] type_qualified = True else: type_name = type_table_by_file.get(src_file, {}).get(receiver) if not type_name: continue - type_defs = type_def_nids.get(_key(type_name), []) - if len(type_defs) != 1: # ambiguous or absent -> bail (god-node guard) + type_nid = _resolve_type_name_nid(type_name, caller_node, src_file) + if not type_nid: # ambiguous or absent -> bail (god-node guard) continue - type_nid = type_defs[0] type_qualified = False - method_nid = method_index.get((type_nid, _key(callee))) + method_nid = _method_on_type_or_bases(type_nid, _key(callee)) if not method_nid: continue # receiver typed, but the type has no such method — skip if method_nid == caller or (caller, method_nid) in existing_pairs: diff --git a/graphify/extractors/csharp.py b/graphify/extractors/csharp.py index 2cda37263..3aa34e0ed 100644 --- a/graphify/extractors/csharp.py +++ b/graphify/extractors/csharp.py @@ -149,108 +149,110 @@ def _resolve_cross_file_csharp_imports( ] -def _resolve_csharp_type_references( - per_file: list[dict], - paths: list[Path], - all_nodes: list[dict], - all_edges: list[dict], -) -> None: - """Arbitrate all C# ``inherits``/``implements``/``references`` targets. +def _is_cs_file(value: object) -> bool: + return isinstance(value, str) and value.endswith(".cs") - The extractor emits provisional same-file bindings and sourceless stubs. This - pass is the single soundness gate: it uses only graph-stamped namespace/import - facts, keeps a binding only when the referenced simple name resolves to one - in-scope real type definition, and otherwise leaves the edge on a dangling stub. - """ - _ = (per_file, paths) - def _is_cs_file(value: object) -> bool: - return isinstance(value, str) and value.endswith(".cs") +def _metadata(value: object) -> dict: + return value if isinstance(value, dict) else {} - def _metadata(value: object) -> dict: - return value if isinstance(value, dict) else {} - def _namespace(node: dict | None) -> str: - metadata = _metadata((node or {}).get("metadata")) - namespace = metadata.get("namespace", "") - return namespace if isinstance(namespace, str) else "" +class CsharpNameResolver: + """Namespace/using/alias-aware C# simple-name resolution. - def _append_unique(items: list[str], value: str) -> None: - if value not in items: - items.append(value) + Factored out of ``_resolve_csharp_type_references`` so the member-call pass + (``_resolve_csharp_member_calls`` in ``graphify/extract.py``) can type a + receiver with the same scoping rules the type-reference side already applies + (the core of #1620), instead of bailing on any corpus-wide bare-name clash. - node_by_id = { - node["id"]: node - for node in all_nodes - if isinstance(node.get("id"), str) and node.get("id") - } - type_def_index = _build_csharp_type_def_index(all_nodes) - known_namespaces = { - node.get("label") - for node in all_nodes - if node.get("type") == "namespace" and isinstance(node.get("label"), str) - } + Built from graph-stamped facts only: the ``(namespace, name)`` type-definition + index plus the per-file ``using``/alias import edges (each carrying its lexical + scope — ``("file", None)`` applies file-wide; ``("namespace", scope_id)`` + applies only where ``scope_id`` is in the referencing node's ``scope_chain``). + """ - # Each using carries its lexical scope: ("file", None) applies file-wide; - # ("namespace", scope_id) applies only where scope_id is in the ref's scope_chain. - namespace_usings_by_file: dict[str, list[tuple[str, str, str | None]]] = {} - aliases_by_file: dict[str, dict[str, list[tuple[str, str, str | None]]]] = {} + def __init__(self, all_nodes: list[dict], all_edges: list[dict]) -> None: + self.node_by_id: dict[str, dict] = { + node["id"]: node + for node in all_nodes + if isinstance(node.get("id"), str) and node.get("id") + } + self.type_def_index = _build_csharp_type_def_index(all_nodes) + self.known_namespaces = { + node.get("label") + for node in all_nodes + if node.get("type") == "namespace" and isinstance(node.get("label"), str) + } - for edge in all_edges: - if edge.get("relation") != "imports": - continue - source_node = node_by_id.get(edge.get("source")) - if not ( - source_node - and isinstance(source_node.get("label"), str) - and source_node.get("label", "").endswith(".cs") - ): - continue - source_file = source_node.get("source_file") - if not _is_cs_file(source_file): - continue - metadata = _metadata(edge.get("metadata")) - target_fqn = metadata.get("target_fqn") - if not isinstance(target_fqn, str) or not target_fqn: - continue - scope_kind = metadata.get("scope_kind") or "file" - scope_id = metadata.get("scope_id") - using_kind = metadata.get("using_kind") - if using_kind == "namespace": - entry = (target_fqn, scope_kind, scope_id) - bucket = namespace_usings_by_file.setdefault(source_file, []) - if entry not in bucket: - bucket.append(entry) - elif using_kind == "alias": - alias = metadata.get("alias") - if isinstance(alias, str) and alias: + self.namespace_usings_by_file: dict[str, list[tuple[str, str, str | None]]] = {} + self.aliases_by_file: dict[str, dict[str, list[tuple[str, str, str | None]]]] = {} + + for edge in all_edges: + if edge.get("relation") != "imports": + continue + source_node = self.node_by_id.get(edge.get("source")) + if not ( + source_node + and isinstance(source_node.get("label"), str) + and source_node.get("label", "").endswith(".cs") + ): + continue + source_file = source_node.get("source_file") + if not _is_cs_file(source_file): + continue + metadata = _metadata(edge.get("metadata")) + target_fqn = metadata.get("target_fqn") + if not isinstance(target_fqn, str) or not target_fqn: + continue + scope_kind = metadata.get("scope_kind") or "file" + scope_id = metadata.get("scope_id") + using_kind = metadata.get("using_kind") + if using_kind == "namespace": entry = (target_fqn, scope_kind, scope_id) - bucket = aliases_by_file.setdefault(source_file, {}).setdefault(alias, []) + bucket = self.namespace_usings_by_file.setdefault(source_file, []) if entry not in bucket: bucket.append(entry) + elif using_kind == "alias": + alias = metadata.get("alias") + if isinstance(alias, str) and alias: + entry = (target_fqn, scope_kind, scope_id) + bucket = self.aliases_by_file.setdefault(source_file, {}).setdefault(alias, []) + if entry not in bucket: + bucket.append(entry) + + @staticmethod + def _namespace(node: dict | None) -> str: + metadata = _metadata((node or {}).get("metadata")) + namespace = metadata.get("namespace", "") + return namespace if isinstance(namespace, str) else "" + @staticmethod def _scope_chain(node: dict) -> list[str]: chain = _metadata(node.get("metadata")).get("scope_chain") return chain if isinstance(chain, list) else [] - def _using_in_scope(scope_kind: str, scope_id: str | None, source_node: dict) -> bool: + def _using_in_scope(self, scope_kind: str, scope_id: str | None, source_node: dict) -> bool: if scope_kind == "file": return True - return scope_id is not None and scope_id in _scope_chain(source_node) + return scope_id is not None and scope_id in self._scope_chain(source_node) + + def _scopes_for(self, source_node: dict, source_file: str) -> list[str]: + def _append_unique(items: list[str], value: str) -> None: + if value not in items: + items.append(value) - def _scopes_for(source_node: dict, source_file: str) -> list[str]: scopes: list[str] = [] - _append_unique(scopes, _namespace(source_node)) + _append_unique(scopes, self._namespace(source_node)) _append_unique(scopes, "") - for namespace, scope_kind, scope_id in namespace_usings_by_file.get(source_file, []): - if _using_in_scope(scope_kind, scope_id, source_node): + for namespace, scope_kind, scope_id in self.namespace_usings_by_file.get(source_file, []): + if self._using_in_scope(scope_kind, scope_id, source_node): _append_unique(scopes, namespace) return scopes - def _resolve_alias(label: str, source_node: dict, source_file: str) -> str | None: + def _resolve_alias(self, label: str, source_node: dict, source_file: str) -> str | None: hits = set() - for target_fqn, scope_kind, scope_id in aliases_by_file.get(source_file, {}).get(label, []): - if not _using_in_scope(scope_kind, scope_id, source_node): + for target_fqn, scope_kind, scope_id in self.aliases_by_file.get(source_file, {}).get(label, []): + if not self._using_in_scope(scope_kind, scope_id, source_node): continue base_fqn = _strip_trailing_csharp_generic_args(html.unescape(target_fqn)) namespace, sep, simple_name = base_fqn.rpartition(".") @@ -259,43 +261,89 @@ def _resolve_alias(label: str, source_node: dict, source_file: str) -> str | Non namespace = "" if not simple_name: continue - hit = type_def_index.get((namespace, simple_name)) + hit = self.type_def_index.get((namespace, simple_name)) if hit: hits.add(hit) return next(iter(hits)) if len(hits) == 1 else None - def _resolve_label(label: str, source_node: dict, source_file: str) -> str | None: - if label in aliases_by_file.get(source_file, {}): - return _resolve_alias(label, source_node, source_file) + def resolve_type_name( + self, label: str, source_node: dict, source_file: str + ) -> tuple[str | None, bool]: + """Resolve a simple type name to a definition node id, with a verdict. + + Returns ``(node_id, decisive)``: + * ``(nid, True)`` — exactly one in-scope definition (or alias target). + * ``(None, True)`` — the name is claimed by this file's scoping (an + alias, or >1 in-scope candidates): genuinely ambiguous/unresolvable, + the caller must NOT fall back to a looser corpus-wide match. + * ``(None, False)`` — scoping knows nothing about the name; a caller + may fall back (e.g. to the corpus-wide unique bare-name match). + """ + if label in self.aliases_by_file.get(source_file, {}): + return self._resolve_alias(label, source_node, source_file), True candidates: list[str] = [] - for namespace in _scopes_for(source_node, source_file): - hit = type_def_index.get((namespace, label)) + for namespace in self._scopes_for(source_node, source_file): + hit = self.type_def_index.get((namespace, label)) if hit and hit not in candidates: candidates.append(hit) - return candidates[0] if len(candidates) == 1 else None + if len(candidates) == 1: + return candidates[0], True + return None, bool(candidates) - def _resolve_qualified(label: str, qualifier: object, source_node: dict, source_file: str) -> str | None: + def resolve_label(self, label: str, source_node: dict, source_file: str) -> str | None: + resolved, _decisive = self.resolve_type_name(label, source_node, source_file) + return resolved + + def resolve_qualified( + self, label: str, qualifier: object, source_node: dict, source_file: str + ) -> str | None: # Sound qualified resolution: an in-scope alias for Q shadows the namespace Q. For a qualified # ref Q.label, look up (alias_target_namespace, label). If no in-scope alias, fall through to an # exact known namespace. Dangle on ambiguity / no hit / unknown qualifier. if not isinstance(qualifier, str) or not qualifier: return None in_scope = [ - entry for entry in aliases_by_file.get(source_file, {}).get(qualifier, []) - if _using_in_scope(entry[1], entry[2], source_node) + entry for entry in self.aliases_by_file.get(source_file, {}).get(qualifier, []) + if self._using_in_scope(entry[1], entry[2], source_node) ] if in_scope: hits = set() for target_fqn, _scope_kind, _scope_id in in_scope: alias_ns = _strip_trailing_csharp_generic_args(html.unescape(target_fqn)) - hit = type_def_index.get((alias_ns, label)) + hit = self.type_def_index.get((alias_ns, label)) if hit: hits.add(hit) return next(iter(hits)) if len(hits) == 1 else None - if qualifier in known_namespaces: - return type_def_index.get((qualifier, label)) + if qualifier in self.known_namespaces: + return self.type_def_index.get((qualifier, label)) return None + +def _resolve_csharp_type_references( + per_file: list[dict], + paths: list[Path], + all_nodes: list[dict], + all_edges: list[dict], +) -> None: + """Arbitrate all C# ``inherits``/``implements``/``references`` targets. + + The extractor emits provisional same-file bindings and sourceless stubs. This + pass is the single soundness gate: it uses only graph-stamped namespace/import + facts, keeps a binding only when the referenced simple name resolves to one + in-scope real type definition, and otherwise leaves the edge on a dangling stub. + """ + _ = (per_file, paths) + + resolver = CsharpNameResolver(all_nodes, all_edges) + node_by_id = resolver.node_by_id + aliases_by_file = resolver.aliases_by_file + + def _resolve_label(label: str, source_node: dict, source_file: str) -> str | None: + return resolver.resolve_label(label, source_node, source_file) + + def _resolve_qualified(label: str, qualifier: object, source_node: dict, source_file: str) -> str | None: + return resolver.resolve_qualified(label, qualifier, source_node, source_file) + def _is_placeholder(node: dict | None) -> bool: return bool(node) and not node.get("source_file") diff --git a/graphify/extractors/engine.py b/graphify/extractors/engine.py index 39c698791..8099c85e8 100644 --- a/graphify/extractors/engine.py +++ b/graphify/extractors/engine.py @@ -1424,15 +1424,29 @@ def _csharp_member_type_table(root, source: bytes) -> dict[str, str]: """Collect ``name -> TypeName`` for C# receiver typing (#1609): class fields, properties, method parameters, and local variable declarations. - File-scoped, first-binding-wins (like the C++ table): a field declared once at - class scope is visible to every method's `field.Method()`, and a param/local - shadowing the same name is a conservative approximation graphify already accepts - for receiver typing. Only a resolvable, non-`var` type name is recorded; `var` - without a `new T()` initializer, and predefined/lower-cased primitives, are - skipped (precision over recall — an untypable receiver is left for the resolver - to drop rather than guess). `var v = new T()` is typed from the object-creation. + File-scoped with conflict POISONING (#1620): a name bound to two different + resolvable types anywhere in the file — or bound once to a resolvable type and + redeclared with an unresolvable one (``var x = Compute();``, a primitive, a + ``dynamic``) — is dropped from the table entirely, so a local shadowing a field + of a DIFFERENT type can never produce a wrong edge (the resolver simply emits + none). Consistent rebindings (the same resolved type) keep the single entry. + Only a resolvable, non-`var` type name is recorded; `var` without a `new T()` + initializer, and predefined/lower-cased primitives, are unresolvable (precision + over recall — an untypable receiver is left for the resolver to drop rather + than guess). `var v = new T()` is typed from the object-creation. """ table: dict[str, str] = {} + poisoned: set[str] = set() + + def _bind(name: str | None, resolved: str | None) -> None: + if not name: + return + if resolved is None or table.get(name, resolved) != resolved: + # An unresolvable redeclaration, or a second binding with a different + # type: the name is scope-ambiguous at file granularity — poison it. + poisoned.add(name) + else: + table[name] = resolved def _typed(type_node) -> str | None: info = _read_csharp_type_name(type_node, source) @@ -1468,25 +1482,19 @@ def _new_type(declarator) -> str | None: type_node = vd.child_by_field_name("type") declared = _typed(type_node) for name, decl in _decl_names(vd): - resolved = declared or _new_type(decl) - if name and resolved and name not in table: - table[name] = resolved + _bind(name, declared or _new_type(decl)) elif t == "property_declaration": nm = n.child_by_field_name("name") - resolved = _typed(n.child_by_field_name("type")) - if nm is not None and resolved: - pname = _read_text(nm, source) - if pname not in table: - table[pname] = resolved + if nm is not None: + _bind(_read_text(nm, source), _typed(n.child_by_field_name("type"))) elif t == "parameter": nm = n.child_by_field_name("name") - resolved = _typed(n.child_by_field_name("type")) - if nm is not None and resolved: - pname = _read_text(nm, source) - if pname not in table: - table[pname] = resolved + if nm is not None: + _bind(_read_text(nm, source), _typed(n.child_by_field_name("type"))) for c in n.children: stack.append(c) + for name in poisoned: + table.pop(name, None) return table def _ts_receiver_type_table(root, source: bytes, table: dict[str, str]) -> None: @@ -3928,8 +3936,26 @@ def walk_calls( is_member_call = True if recv is not None and recv.type == "identifier": member_receiver = _read_text(recv, source) - elif recv is not None and recv.type == "this_expression": + elif recv is not None and recv.type in ("this", "this_expression"): member_receiver = "this" + elif recv is not None and recv.type in ("base", "base_expression"): + # base.M(): resolved against the caller's single + # resolvable base class in the cross-file pass. + member_receiver = "base" + elif recv is not None and recv.type == "member_access_expression": + # this.field.M(): the explicit-`this` field access is + # typed exactly like a bare `field.M()` via the file + # table; any other chained receiver stays untyped + # (the resolver bails rather than guessing). + inner = recv.child_by_field_name("expression") + fname = recv.child_by_field_name("name") + if ( + inner is not None + and inner.type in ("this", "this_expression") + and fname is not None + and fname.type == "identifier" + ): + member_receiver = _read_text(fname, source) elif fn_node is not None and fn_node.type == "identifier": callee_name = _read_text(fn_node, source) else: diff --git a/tests/test_csharp_member_calls.py b/tests/test_csharp_member_calls.py index 0dba3ed3a..5487939c1 100644 --- a/tests/test_csharp_member_calls.py +++ b/tests/test_csharp_member_calls.py @@ -143,6 +143,190 @@ def test_unqualified_call_still_resolves(tmp_path): assert any("_r_a" in s and "helper" in t for s, t in calls), "no regression on unqualified calls" +# ── Namespace-aware receiver typing + shadow poisoning (#1620) ──────────────── + +_NS_AB = { + "A.cs": "namespace A { public class Svc { public bool Do() => true; } }\n", + "B.cs": "namespace B { public class Svc { public bool Do() => false; } }\n", +} + + +def test_namespace_using_directive_disambiguates_receiver_type(tmp_path): + """`Svc` exists in namespaces A and B; a caller file `using A;` must bind an + `A.Svc`-typed receiver to A.Svc.Do — before #1620 the corpus-wide bare-name + ambiguity made the resolver bail (missing edge).""" + calls, r = _calls(tmp_path, { + **_NS_AB, + "Caller.cs": ( + "using A;\n" + "namespace App {\n" + " public class Runner { public bool Go(Svc s) { return s.Do(); } }\n" + "}\n" + ), + }) + a_do = _find(r, ".Do()", "a_a_svc") + b_do = _find(r, ".Do()", "b_b_svc") + runner_go = _find(r, ".Go()", "runner") + assert (runner_go, a_do) in calls, "using A; must resolve Svc to A.Svc" + assert (runner_go, b_do) not in calls, "must NOT bind to the same-named B.Svc" + + +def test_namespace_using_directive_resolves_to_other_namespace(tmp_path): + calls, r = _calls(tmp_path, { + **_NS_AB, + "Caller.cs": ( + "using B;\n" + "namespace App {\n" + " public class Runner { public bool Go(Svc s) { return s.Do(); } }\n" + "}\n" + ), + }) + a_do = _find(r, ".Do()", "a_a_svc") + b_do = _find(r, ".Do()", "b_b_svc") + runner_go = _find(r, ".Go()", "runner") + assert (runner_go, b_do) in calls, "using B; must resolve Svc to B.Svc" + assert (runner_go, a_do) not in calls + + +def test_namespace_ambiguous_without_using_bails(tmp_path): + """No using directive and `Svc` in two foreign namespaces: genuinely + ambiguous — no edge to either candidate (never a guess).""" + calls, r = _calls(tmp_path, { + **_NS_AB, + "Caller.cs": ( + "namespace App {\n" + " public class Runner { public bool Go(Svc s) { return s.Do(); } }\n" + "}\n" + ), + }) + assert not any("runner" in s and "svc_do" in t for s, t in calls), \ + "ambiguous cross-namespace type must produce no edge" + + +def test_same_namespace_receiver_resolves_without_using(tmp_path): + """A caller in namespace A resolves `Svc` to A.Svc even though B.Svc also + exists — same-namespace visibility needs no using directive.""" + calls, r = _calls(tmp_path, { + **_NS_AB, + "A2.cs": ( + "namespace A {\n" + " public class Client { public bool Go(Svc s) { return s.Do(); } }\n" + "}\n" + ), + }) + a_do = _find(r, ".Do()", "a_a_svc") + b_do = _find(r, ".Do()", "b_b_svc") + client_go = _find(r, ".Go()", "client") + assert (client_go, a_do) in calls + assert (client_go, b_do) not in calls + + +def test_local_shadowing_field_of_different_type_poisons_name(tmp_path): + """A local `Other x` shadowing a field `Server x` makes the name's type + conflicting — the binding is poisoned and `x.Run()` emits NO edge, instead + of first-binding-wins mis-binding to Server.Run (a wrong edge).""" + calls, r = _calls(tmp_path, { + "S.cs": ( + "public class Server { public bool Run() => true; }\n" + "public class Other { public bool Run() => false; }\n" + "public class Holder {\n" + " private Server x = new Server();\n" + " public bool A() { Other x = new Other(); return x.Run(); }\n" + "}\n" + ) + }) + holder_a = _find(r, ".A()", "holder") + server_run = _find(r, ".Run()", "server") + other_run = _find(r, ".Run()", "other") + assert (holder_a, server_run) not in calls, \ + "shadowed field's type must not win (wrong edge)" + assert (holder_a, other_run) not in calls, \ + "conflicting bindings poison the name entirely (conservative: no edge)" + + +def test_untyped_redeclaration_poisons_typed_field(tmp_path): + """`var x = Compute();` (untypable) redeclaring a typed field poisons the + name: `x.Run()` must not bind to the field's type.""" + calls, r = _calls(tmp_path, { + "S.cs": ( + "public class Server { public bool Run() => true; }\n" + "public class Holder {\n" + " private Server x = new Server();\n" + " public object Compute() => new object();\n" + " public bool A() { var x = Compute(); return x.Run(); }\n" + "}\n" + ) + }) + assert not any("holder_a" in s and "run" in t.lower() for s, t in calls) + + +def test_this_field_receiver_resolves(tmp_path): + """`this._s.Save()` types the field exactly like a bare `_s.Save()`.""" + calls, r = _calls(tmp_path, { + "S.cs": ( + "public class Server { public bool Save() => true; }\n" + "public class Cache { public bool Save() => false; }\n" + "public class Repo {\n" + " private Server _s = new Server();\n" + " public bool Commit() { return this._s.Save(); }\n" + "}\n" + ) + }) + commit = _find(r, ".Commit()", "commit") + server_save = _find(r, ".Save()", "server") + cache_save = _find(r, ".Save()", "cache") + assert (commit, server_save) in calls + assert (commit, cache_save) not in calls + + +def test_base_receiver_resolves_to_base_class_method(tmp_path): + calls, r = _calls(tmp_path, { + "Base.cs": "public class BaseSvc { public bool Ping() => true; }\n", + "Sub.cs": ( + "public class Sub : BaseSvc {\n" + " public bool Go() { return base.Ping(); }\n" + "}\n" + ), + }) + sub_go = _find(r, ".Go()", "sub") + ping = _find(r, ".Ping()", "basesvc") + assert (sub_go, ping) in calls, "base.Ping() must resolve to the base class method" + + +def test_inherited_method_resolves_through_base_chain(tmp_path): + """A method not declared on the receiver's type but inherited from a + resolvable in-corpus base resolves to the base's declaration.""" + calls, r = _calls(tmp_path, { + "S.cs": ( + "public class BaseSvc { public bool Ping() => true; }\n" + "public class Derived : BaseSvc { }\n" + "public class User {\n" + " public bool Use(Derived d) { return d.Ping(); }\n" + "}\n" + ) + }) + use = _find(r, ".Use()", "user") + ping = _find(r, ".Ping()", "basesvc") + assert (use, ping) in calls + + +def test_unresolved_base_poisons_inherited_member_lookup(tmp_path): + """The receiver's type inherits from an out-of-corpus base: a method missing + on the type may live on that base, so the lookup is poisoned — and it must + NOT fall back to an unrelated same-named in-corpus method.""" + calls, r = _calls(tmp_path, { + "S.cs": ( + "public class Server { public bool Save() => true; }\n" + "public class Ext : NotInCorpus { }\n" + "public class User {\n" + " public bool U(Ext e) { return e.Save(); }\n" + "}\n" + ) + }) + assert not any("user_u" in s and "save" in t.lower() for s, t in calls), \ + "unresolved base chain must bail, not mis-bind to Server.Save" + + def test_method_chained_off_new_expression_resolves(tmp_path): """#1770: a method invoked directly on a `new X(...)` object-creation expression (no intermediate variable) must still emit a calls edge to the From b4dd6d7b25bb703383cb739c680642e407988fa6 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Sun, 26 Jul 2026 13:00:06 +0100 Subject: [PATCH 279/292] docs: changelog for the Track A / node-identity / C# batch (0.9.27) Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1cffd45c0..d183643ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,18 @@ Full release notes with details on each version: [GitHub Releases](https://githu - Fix: JavaScript/TypeScript non-relative imports resolve through `jsconfig.json`/`tsconfig.json` `baseUrl` and `paths` (#2153, thanks @Rishet11), so imports like `import x from 'src/utils'` are no longer left dangling. - Fix: Swift/Foundation/SwiftUI builtins (`Data`, `URL`, `Sendable`, `View`, ...) are filtered from call resolution and god-node ranking (#2147, thanks @MasterFede5), so they no longer fabricate cross-file edges to user symbols or dominate the graph's high-degree nodes. - Fix: running the test suite no longer touches the developer's real `~/.claude`/`~/.gemini`/`~/.codebuddy`/`~/.copilot` (#2168). An autouse fixture sandboxes HOME for every test. +- Fix: bash calls into functions from a `source`d file now resolve for extensionless shebang scripts and bare `source lib.sh` (no `./` prefix) too (#2171, thanks @Souptik96). The bare-name binding is marked INFERRED since it resolves via `$PATH` at runtime. +- Fix: a `source "${VAR}/lib.sh"` whose variable is a tracked top-level assignment (or the `dirname "${BASH_SOURCE[0]}"` idiom) now resolves against the variable's real directory instead of guessing the script's own, so it no longer binds to a same-named decoy under the script dir (#2172, thanks @Souptik96). +- Fix: SQL `CREATE FUNCTION`/`PROCEDURE` routines with PL/pgSQL-only bodies that tree-sitter cannot parse are now recovered by a raw-text scan, gated on a failed parse so a cleanly-parsing file cannot fabricate routines from commented-out DDL or `EXECUTE` strings (#2180, thanks @Souptik96). +- Fix: the extraction process pool is skipped when only one worker is available, falling back to in-process sequential extraction (#2173, thanks @Souptik96), which also removes the last case where a hung Windows hook rebuild could orphan a worker. +- Fix: the git-hook interpreter pin now handles a Python path containing a space (#2166, thanks @Souptik96); the merge-driver command is quoted and the pin allowlist admits a space while still rejecting shell metacharacters. +- Docs: the Codex `PreToolUse` hook (`graphify hook-check`) is documented as an intentional no-op (#2165, thanks @Souptik96); Codex Desktop rejects `additionalContext` there, so graph guidance comes from `AGENTS.md`. +- Fix: JavaScript/TypeScript regex-rescued imports (Svelte/Astro/Vue) no longer create ghost target nodes with absolute-path ids; the target resolves to the canonical root-relative file node and the dangling absolute-id edges are gone (#2195). +- Fix: cross-file `concept` nodes whose labels normalize identically are now merged, matching the behavior already applied to near-identical labels (#2182). Gated to `concept` nodes with provenance, so code/rationale/document/image and cross-repo guards are preserved. +- Fix: `graphify-out/cache/stat-index.json` is now stored with root-relative keys (re-anchored on load, mirroring `manifest.json`) and pruned of deleted-file entries, so a moved or cloned corpus keeps its cache hits instead of re-extracting everything, and the index stops growing unbounded (#2199). +- Fix: absolute `source_file` paths (for example from a Windows scan) no longer break node identity: `build_from_json` re-keys absolute-derived semantic ids to the canonical root-relative id and the semantic cache stores normalized paths (#2197). +- Fix: `build_from_json` now folds legacy field aliases (`name`->`label`, `path`->`source_file`, edge `type`->`relation`, `confidence_score`->`confidence`), so nodes carrying them stop entering the graph without a label or source_file where they are invisible and unmergeable (#2194). The extraction warning also breaks issues down by cause. +- Feat: C# member calls on a typed receiver now resolve namespace-aware (a class name used in two namespaces resolves via `using`/scope instead of bailing), with `base.`/`this.field` receivers, inherited-member lookup through the `inherits` chain, and shadow-poisoning so a local shadowing a field of a different type no longer produces a wrong edge (#1609, adapted from #1620 by @TheFedaikin). ## 0.9.26 (2026-07-25) From f5a3592882ad54e5394c5cd5391786a589110bd1 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Sun, 26 Jul 2026 13:01:17 +0100 Subject: [PATCH 280/292] chore(release): 0.9.27 Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d183643ae..e490e6d73 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ Full release notes with details on each version: [GitHub Releases](https://github.com/safishamsi/graphify/releases) -## 0.9.27 (unreleased) +## 0.9.27 (2026-07-26) - Fix: `claude`/`gemini`/`codex`/`codebuddy install` no longer overwrite an existing settings/hooks file they cannot parse (#2167). The installers fell back to an empty config on any JSON parse error and then rewrote the whole file, destroying the user's settings (the likely trigger is a UTF-8 BOM, the same class as #2163). They now read `utf-8-sig`, refuse to modify a file that is not a JSON object (naming the path) instead of clobbering it, and back up to `.graphify-bak` before any modifying write. - Fix: incremental `extract --no-cluster` no longer overwrites the full graph with just the changed files (#2169). The raw path wrote only the current run's extraction over `graph.json`, dropping every node and edge owned by an unchanged file, and the changed file's cross-file edges dangled on absolute-path ids. It now merges the existing graph forward with the same replace/prune semantics as the clustered path and canonicalizes cross-file edge targets; a corrupt existing graph is refused rather than overwritten. From 62b1d4370b8f8d61dea9a1d952393ae63ad53fa4 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 27 Jul 2026 00:19:11 +0100 Subject: [PATCH 281/292] docs(readme): link the YC badge to the company page Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7df3b01c8..a4f188f46 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ Downloads Discord LinkedIn - YC S26 + YC S26

Type `/graphify` in your AI coding assistant and it maps your entire project (code, docs, PDFs, images, videos) into a **knowledge graph** you can **query instead of grepping** through files. From d1f303e23752203de55fcdfabc08e663a01deb56 Mon Sep 17 00:00:00 2001 From: ozdemirsarman Date: Sun, 26 Jul 2026 22:08:38 +0300 Subject: [PATCH 282/292] fix(swift): extract computed & observed properties (#2181) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit function_types only recognised func/init/deinit/subscript, so computed properties (var body: some View { ... }) and willSet/didSet observers produced no node and their bodies were never walked — erasing the whole SwiftUI view layer. Emit a function-like member node for them and defer the body to the call-walk via function_bodies; stored properties are unchanged. Adds tests. --- graphify/extractors/engine.py | 16 +++++ tests/test_swift_computed_properties.py | 86 +++++++++++++++++++++++++ 2 files changed, 102 insertions(+) create mode 100644 tests/test_swift_computed_properties.py diff --git a/graphify/extractors/engine.py b/graphify/extractors/engine.py index 8099c85e8..136085e23 100644 --- a/graphify/extractors/engine.py +++ b/graphify/extractors/engine.py @@ -3101,6 +3101,22 @@ def _emit_java_parent_type(type_node, rel: str, at_line: int) -> None: prop_name = _swift_property_name(node, source) if prop_name and prop_type: type_table[prop_name] = prop_type + # #2181: a computed property (`var body: some View { … }`) or an + # observed one (`willSet`/`didSet`) carries a body that the branches + # above never emitted — so the property node AND every call inside it + # were dropped. For SwiftUI this erases the whole view layer, since + # `body` is a computed property. Emit a function-like member node and + # defer its body to the call-walk via function_bodies (mirroring how + # methods register their bodies). Stored properties have no such body + # child, so their behaviour is unchanged (no regression). + comp_bodies = [c for c in node.children + if c.type in ("computed_property", "willset_didset_block")] + if comp_bodies and prop_name: + prop_nid = _make_id(parent_class_nid, prop_name) + add_node(prop_nid, f".{prop_name}", line) + add_edge(parent_class_nid, prop_nid, "method", line) + for body_block in comp_bodies: + function_bodies.append((prop_nid, body_block)) return if (config.ts_module == "tree_sitter_scala" diff --git a/tests/test_swift_computed_properties.py b/tests/test_swift_computed_properties.py new file mode 100644 index 000000000..dd78d7482 --- /dev/null +++ b/tests/test_swift_computed_properties.py @@ -0,0 +1,86 @@ +"""Regression tests for #2181. + +Swift computed properties (`var body: some View { … }`) and observed +properties (`willSet` / `didSet`) carry a body. Before the fix the Swift +extractor only recognised function/init/deinit/subscript as callables, so +those properties produced no node and their bodies were never walked -- which +for SwiftUI erased the entire view layer (`body` is a computed property). +""" +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path + +from graphify.extract import extract_swift + + +def _labels(result): + return [n["label"] for n in result["nodes"]] + + +def _rel(result, relation): + return [e for e in result["edges"] if e["relation"] == relation] + + +class TestSwiftComputedProperties(unittest.TestCase): + def _extract(self, src: str) -> dict: + with tempfile.TemporaryDirectory() as d: + p = Path(d) / "View.swift" + p.write_text(src, encoding="utf-8") + return extract_swift(p) + + def test_computed_property_emits_node_and_walks_body(self): + r = self._extract( + "struct PlayerScrubber: View {\n" + " var body: some View {\n" + " VStack { doTap() }\n" + " }\n" + " var toggled: Int { 1 }\n" + " func doTap() {}\n" + "}\n" + ) + labels = _labels(r) + # Both computed properties become nodes... + self.assertIn(".body", labels) + self.assertIn(".toggled", labels) + # ...and the call inside `body` is attributed to the body node, not lost. + call_pairs = {(e["source"], e["target"]) for e in _rel(r, "calls")} + body_nid = next(n["id"] for n in r["nodes"] if n["label"] == ".body") + dotap_nid = next(n["id"] for n in r["nodes"] if n["label"] == ".doTap()") + self.assertIn((body_nid, dotap_nid), call_pairs, + "call inside computed `body` was not captured") + + def test_stored_property_not_emitted_as_member_but_keeps_type_ref(self): + # A plain stored property has no body block: it must NOT become a + # function-like node, but its type must still produce a references edge. + r = self._extract( + "struct S {\n" + " var vm: ViewModel\n" + "}\n" + ) + self.assertNotIn(".vm", _labels(r)) + ref_targets = {n["label"] + for e in _rel(r, "references") + for n in r["nodes"] if n["id"] == e["target"]} + self.assertIn("ViewModel", ref_targets) + + def test_observed_property_body_is_walked(self): + # willSet/didSet observers also carry a body whose calls used to vanish. + r = self._extract( + "class M {\n" + " var score: Int = 0 {\n" + " didSet { react() }\n" + " }\n" + " func react() {}\n" + "}\n" + ) + self.assertIn(".score", _labels(r)) + call_pairs = {(e["source"], e["target"]) for e in _rel(r, "calls")} + score_nid = next(n["id"] for n in r["nodes"] if n["label"] == ".score") + react_nid = next(n["id"] for n in r["nodes"] if n["label"] == ".react()") + self.assertIn((score_nid, react_nid), call_pairs) + + +if __name__ == "__main__": + unittest.main() From 06e5978309a8a1b9d9ea39321a04422896aad042 Mon Sep 17 00:00:00 2001 From: Sfahad7 Date: Mon, 27 Jul 2026 03:05:26 +0530 Subject: [PATCH 283/292] fix(detect): NFC-normalize manifest keys so --update survives macOS path forms macOS yields NFD paths from os.walk/getcwd while skill path literals are often NFC. Raw string compare treated every file as deleted+new and forced a full re-extract. Canonicalize at the manifest boundary (same idea as #1226). Fixes #2221 --- graphify/detect.py | 64 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 50 insertions(+), 14 deletions(-) diff --git a/graphify/detect.py b/graphify/detect.py index 338b1695c..8fc676635 100644 --- a/graphify/detect.py +++ b/graphify/detect.py @@ -1460,6 +1460,18 @@ def _stat_and_hash(path_str: str) -> tuple[str, float, str] | None: return None +def _nfc(s: str) -> str: + """NFC-normalize a path string used as a manifest key. + + On macOS, ``os.walk`` / ``getcwd`` yield NFD paths while path literals + and many skill-substituted roots are NFC. Raw string compare then treats + every file as both deleted and new, forcing a full re-extract (#2221). + Same boundary as the Office sidecar hash fix (#1226). + """ + import unicodedata + return unicodedata.normalize("NFC", s) + + def _to_relative_for_storage(key: str, root: Path) -> str: """Return ``key`` as a forward-slash relative path from ``root``. @@ -1517,14 +1529,19 @@ def load_manifest( manifests with absolute keys pass through unchanged, so a graphify-out/ written by an older version (or by a caller that didn't supply ``root`` to :func:`save_manifest`) remains readable. + + Keys are NFC-normalized on load so a manifest written under one Unicode + form still matches a scan that yields the other (#2221). """ try: raw = json.loads(Path(manifest_path).read_text(encoding="utf-8")) except Exception: return {} - if root is None or not isinstance(raw, dict): + if not isinstance(raw, dict): return raw - return {_to_absolute_from_storage(k, root): v for k, v in raw.items()} + if root is None: + return {_nfc(k): v for k, v in raw.items()} + return {_nfc(_to_absolute_from_storage(k, root)): v for k, v in raw.items()} def save_manifest( @@ -1572,26 +1589,39 @@ def save_manifest( """ existing = load_manifest(manifest_path, root=root) - scan_set: set[str] | None = set(scan_corpus) if scan_corpus is not None else None - clear_set: set[str] | None = set(clear_semantic) if clear_semantic is not None else None + # Index both raw and NFC forms so scan/clear membership survives the + # same NFC/NFD mismatch that breaks manifest lookups (#2221). + def _path_index(paths: set[str] | list[str] | None) -> set[str] | None: + if paths is None: + return None + indexed: set[str] = set() + for p in paths: + indexed.add(p) + indexed.add(_nfc(p)) + return indexed + + scan_set = _path_index(scan_corpus) + clear_set = _path_index(clear_semantic) try: root_res: Path | None = Path(root).resolve() if root is not None else None except (OSError, RuntimeError): root_res = Path(root) if root is not None else None def _in_scan(path_str: str) -> bool: - if path_str in scan_set: + if path_str in scan_set or _nfc(path_str) in scan_set: return True try: - return str(Path(path_str).resolve()) in scan_set + resolved = str(Path(path_str).resolve()) + return resolved in scan_set or _nfc(resolved) in scan_set except (OSError, RuntimeError): return False def _in_clear(path_str: str) -> bool: - if path_str in clear_set: + if path_str in clear_set or _nfc(path_str) in clear_set: return True try: - return str(Path(path_str).resolve()) in clear_set + resolved = str(Path(path_str).resolve()) + return resolved in clear_set or _nfc(resolved) in clear_set except (OSError, RuntimeError): return False @@ -1657,7 +1687,8 @@ def _normalise_entry(entry): if f not in hashed: continue # file deleted between detect() and manifest write mtime, h = hashed[f] - prev = _normalise_entry(existing.get(f, {})) or {} + key = _nfc(f) + prev = _normalise_entry(existing.get(key, {})) or {} entry: dict = {"mtime": mtime} if kind in ("ast", "both"): entry["ast_hash"] = h @@ -1668,13 +1699,17 @@ def _normalise_entry(entry): else: # Preserve semantic_hash only when content is unchanged entry["semantic_hash"] = prev.get("semantic_hash", "") if h == prev.get("ast_hash", "") else "" - manifest[f] = entry + manifest[key] = entry if root is not None: # Persist in portable form: forward-slash relative paths. Keys outside # ``root`` (out-of-tree symlinked corpora, --include sources) keep # their absolute form so the manifest round-trips on the saving # machine even when not every entry can be portably encoded. - manifest = {_to_relative_for_storage(k, root): v for k, v in manifest.items()} + # NFC after relativize so on-disk keys match what load_manifest + # re-anchors and compares against (#2221). + manifest = {_nfc(_to_relative_for_storage(k, root)): v for k, v in manifest.items()} + else: + manifest = {_nfc(k): v for k, v in manifest.items()} from graphify.paths import write_json_atomic # Atomic write: a crash mid-write must not leave a truncated manifest that # detect_incremental then fails to parse. @@ -1740,7 +1775,8 @@ def detect_incremental( for ftype, file_list in full["files"].items(): for f in file_list: - stored = manifest.get(f) + # Manifest keys are NFC; scan paths may arrive NFD (#2221). + stored = manifest.get(_nfc(f)) try: current_mtime = os.stat(_os_path(Path(f))).st_mtime except Exception: @@ -1791,11 +1827,11 @@ def detect_incremental( # current scan was EXCLUDED (ignore rules / --exclude changed) and must # not be reported as deleted. Mirrors the watch-side excluded-vs-deleted # distinction (#1795). - current_files = {f for flist in full["files"].values() for f in flist} + current_files = {_nfc(f) for flist in full["files"].values() for f in flist} deleted_files: list[str] = [] excluded_files: list[str] = [] for f in manifest: - if f in current_files: + if _nfc(f) in current_files: continue try: alive = Path(f).exists() From 072698a1762b240f4d2c51b1e9c134647a1da11a Mon Sep 17 00:00:00 2001 From: Sfahad7 Date: Mon, 27 Jul 2026 03:33:08 +0530 Subject: [PATCH 284/292] fix(detect): NFC both sides of relpath for portable manifests on macOS Stamped keys can already be NFC while Path(root).resolve() is NFD, so os.path.relpath treated in-root files as ../ and kept them absolute. Normalize both operands inside _to_relative_for_storage (and the join in _to_absolute_from_storage). --- graphify/detect.py | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/graphify/detect.py b/graphify/detect.py index 8fc676635..d243641e2 100644 --- a/graphify/detect.py +++ b/graphify/detect.py @@ -1486,12 +1486,18 @@ def _to_relative_for_storage(key: str, root: Path) -> str: under its own name. Resolving the key would point the stored entry at the symlink target, and the original key would then miss on reload and re-extract on every incremental run. + + Both sides of ``relpath`` are NFC'd first: stamped keys may already be + NFC while ``Path(root).resolve()`` is NFD on macOS, and a mixed-form + compare would mark an in-root file as ``../…`` and keep it absolute + (#2221 / #777). """ p = Path(key) if not p.is_absolute(): return key try: - rel = os.path.relpath(p, Path(root).resolve()) + base = _nfc(str(Path(root).resolve())) + rel = os.path.relpath(_nfc(str(p)), base) except (ValueError, OSError): return key # outside root (e.g. Windows cross-drive) # ``os.path.relpath`` happily produces ``../foo`` for paths outside @@ -1510,11 +1516,15 @@ def _to_absolute_from_storage(key: str, root: Path) -> str: that newly-loaded manifests from before this change remain readable. Uses ``Path(root).resolve()`` so the produced absolute path matches what :func:`detect` returns (which also resolves the scan root). + NFC both sides so a relative key and an NFD-resolved root still join + to the same string form the rest of the manifest path uses (#2221). """ p = Path(key) if p.is_absolute(): return str(p) - return str(Path(root).resolve() / p) + # NFC the joined result so an NFD-resolved root + relative key lands on + # the same form load_manifest / detect_incremental compare against. + return _nfc(str(Path(root).resolve() / p)) def load_manifest( From 6d42c48f4de6773970982b332c1acbc92016ff6a Mon Sep 17 00:00:00 2001 From: Bob Spryn Date: Sun, 26 Jul 2026 14:29:13 -0400 Subject: [PATCH 285/292] fix(watch): stop incremental rebuilds from reusing stale community labels MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Community labels are saved keyed by community id, but re-clustering reassigns those ids: after a rebuild that adds nodes, cid 30 can cover a completely different community and its saved name is then simply wrong. cluster-only already guards this — it validates each reused label against the `.graphify_labels.json.sig` membership fingerprints and re-hubs any community that changed (the case community_member_sigs() was written for). _rebuild_code() skipped that check entirely: it reused every label whose cid was present and hub-filled only the *missing* ones, so stale names survived — and then wrote them back to labels.json, laundering them as current. It also never refreshed the .sig sidecar, so the signatures kept describing an older clustering and drifted out of step with the labels they sit beside, leaving the cluster-only guard nothing accurate to check. Adding ~3.5k nodes to a real graph re-clustered 463 -> 515 communities and mislabeled 162 of them this way: a `domain.audit` namespace reading "ACH / bank payments", a `domain.auth` namespace reading "document-sensitivity.up.sql". Node and edge data stayed correct, so nothing failed loudly — only the names lied. Apply the same signature check in the incremental path, write the sidecar in step with the labels, and print the same "run `graphify label`" notice cluster-only emits so a drifted community set is visible rather than silent. --- graphify/watch.py | 45 ++++++++++++++++++++++++++ tests/test_watch.py | 79 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 124 insertions(+) diff --git a/graphify/watch.py b/graphify/watch.py index 1ef1ebd4d..08b6fdd43 100644 --- a/graphify/watch.py +++ b/graphify/watch.py @@ -1240,6 +1240,7 @@ def _add_deleted_source(path: Path) -> None: gods = god_nodes(G) surprises = surprising_connections(G, communities) labels_file = out / ".graphify_labels.json" + sig_file = out / (".graphify_labels.json" + ".sig") try: raw = json.loads(labels_file.read_text(encoding="utf-8")) if labels_file.exists() else {} # Skip persisted "Community N" placeholders so the hub-fill below @@ -1251,12 +1252,50 @@ def _add_deleted_source(path: Path) -> None: except Exception: raw = {} labels = {} + # A saved label belongs to a cid, but re-clustering reassigns cids: after a + # rebuild that adds nodes, cid 30 can cover a completely different community + # and its old name is then simply wrong. Validate every reused label against + # the membership signature saved beside the labels — the same guard the + # cluster-only path applies — and drop any whose community changed so the + # hub-fill below renames it, deterministically and correct-by-construction. + # Without this, an incremental `graphify update` launders stale names into + # labels.json as though they were current (#label-stale). + from graphify.cluster import community_member_sigs + cur_sigs = community_member_sigs(communities) + saved_sigs: dict[int, str] = {} + if sig_file.exists(): + try: + saved_sigs = { + int(k): v for k, v in + json.loads(sig_file.read_text(encoding="utf-8")).items() + if isinstance(v, str) + } + except Exception: + saved_sigs = {} + if saved_sigs: + # Precise: the signature tells us exactly which communities changed. + stale = {cid for cid in labels if saved_sigs.get(cid) != cur_sigs.get(cid)} + else: + # No sidecar (labels predate it). A differing community COUNT means the + # labels describe a different clustering, so no cid's label is trustworthy; + # an equal count is the best available "unchanged" signal. + stale = set(labels) if len(raw) != len(communities) else set() + for cid in stale: + del labels[cid] missing = {cid: members for cid, members in communities.items() if cid not in labels} if missing: # Deterministic hub name (highest-degree member) beats a bare "Community N" # placeholder for any community without a saved label. from graphify.cluster import label_communities_by_hub labels.update(label_communities_by_hub(G, missing)) + if stale: + print( + f"[graphify watch] community set changed since labeling " + f"({len(raw)} saved labels, {len(communities)} communities now; " + f"renamed {len(stale)} community(ies) by their hub). " + f"Run `graphify label` to refresh names with the LLM.", + file=sys.stderr, + ) questions = suggest_questions(G, communities, labels) from graphify.report import load_learning_for_report as _llfr report = generate(G, communities, cohesion, labels, gods, surprises, detection, @@ -1301,6 +1340,12 @@ def _add_deleted_source(path: Path) -> None: graph_tmp.replace(existing_graph) report_path.write_text(report, encoding="utf-8") labels_file.write_text(labels_json, encoding="utf-8") + # Keep the membership signatures in step with the labels we just wrote. + # Skipping this was the other half of the stale-label bug: labels.json + # advanced every rebuild while the sidecar kept describing an older + # clustering, so the guard above had nothing accurate to check against. + sig_file.write_text( + json.dumps({str(k): v for k, v in cur_sigs.items()}), encoding="utf-8") (out / ".graphify_root").write_text(str(watch_path), encoding="utf-8") diff --git a/tests/test_watch.py b/tests/test_watch.py index b60c0f608..a9b3ebe89 100644 --- a/tests/test_watch.py +++ b/tests/test_watch.py @@ -188,6 +188,85 @@ def test_rebuild_code_writes_community_name(tmp_path): ) +def test_rebuild_code_drops_labels_whose_community_changed(tmp_path): + """An incremental rebuild must not reuse a saved label for a community whose + membership changed. Labels are keyed by cid, but re-clustering reassigns cids, + so after new files land cid N can cover a different community and its old name + is then simply wrong. cluster-only guards this with the `.sig` membership + fingerprints; _rebuild_code ignored them and hub-filled only *missing* labels, + so stale names survived and were written back to labels.json as if current.""" + import json + from graphify.watch import _rebuild_code + + corpus = tmp_path / "corpus" + corpus.mkdir() + (corpus / "a.py").write_text( + "def alpha():\n return beta()\n\ndef beta():\n return 1\n", encoding="utf-8" + ) + assert _rebuild_code(corpus, acquire_lock=False) is True + + out = corpus / "graphify-out" + labels_file = out / ".graphify_labels.json" + sig_file = out / ".graphify_labels.json.sig" + assert sig_file.exists(), "rebuild must persist membership signatures beside labels" + + # Stand in for an LLM naming pass: give every community a distinctive name, + # leaving the signatures untouched so they still describe THIS clustering. + labels = json.loads(labels_file.read_text(encoding="utf-8")) + assert labels, "expected the first rebuild to write community labels" + labels_file.write_text( + json.dumps({cid: f"Named-{cid}" for cid in labels}), encoding="utf-8" + ) + + # Grow the corpus so clustering changes, then rebuild incrementally. + for name in ("b.py", "c.py", "d.py"): + (corpus / name).write_text( + f"def {name[0]}_one():\n return {name[0]}_two()\n\n" + f"def {name[0]}_two():\n return 2\n", + encoding="utf-8", + ) + assert _rebuild_code(corpus, acquire_lock=False) is True + + graph = json.loads((out / "graph.json").read_text(encoding="utf-8")) + after = json.loads(labels_file.read_text(encoding="utf-8")) + sigs = json.loads(sig_file.read_text(encoding="utf-8")) + + communities = {} + for node in graph["nodes"]: + cid = node.get("community") + if cid is not None: + communities.setdefault(str(cid), []).append(node["id"]) + + from graphify.cluster import community_member_sigs + expected = { + str(cid): sig + for cid, sig in community_member_sigs( + {int(c): m for c, m in communities.items()} + ).items() + } + assert sigs == expected, ( + "signatures must be rewritten in step with the labels; a drifting sidecar " + "leaves the staleness guard nothing accurate to check against" + ) + + # Any surviving "Named-N" must sit on a community that genuinely did not change. + for cid, name in after.items(): + if name.startswith("Named-"): + assert cid in communities, f"label kept for vanished community {cid}" + assert sigs[cid] == expected[cid], ( + f"community {cid} kept the stale label {name!r} after its " + f"membership changed" + ) + + for node in graph["nodes"]: + cid = node.get("community") + if cid is not None and node.get("community_name", "").startswith("Named-"): + assert sigs[str(cid)] == expected[str(cid)], ( + f"node {node['id']} carries stale community_name " + f"{node['community_name']!r}" + ) + + def test_update_rebuilds_with_nested_star_gitignore(tmp_path): """#1880: `graphify update` must not emit 0 nodes (and then refuse to overwrite) just because the source tree has a nested `.gitignore` with a From 498b76ba32aede9bc540d4178f5d0d96fe9991dd Mon Sep 17 00:00:00 2001 From: Bob Spryn Date: Sun, 26 Jul 2026 15:04:42 -0400 Subject: [PATCH 286/292] fix(watch): keep a visualization when the graph outgrows the viz cap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Crossing MAX_NODES_FOR_VIZ left the project with no graph.html at all: _rebuild_code unlinked the existing file and wrote nothing in its place. The delete on its own is defensible — a kept graph.html would describe an older, smaller graph — but the file is gone before the user reads the message, and the next incremental rebuild silently removes it again, so a repo that grows past the threshold just loses its visualization with no way to keep one. The export path already solved this (#1019): over the cap it re-renders the community-aggregation view rather than going without. Do the same here, so the artifact is current AND present instead of current OR present. GRAPHIFY_VIZ_NODE_LIMIT=0 still means "no HTML viz" (CI runners) rather than "aggregate", and if the aggregated render also fails the old skip-and-remove behaviour stands. --- graphify/watch.py | 34 ++++++++++++++++++++++++----- tests/test_watch.py | 53 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+), 6 deletions(-) diff --git a/graphify/watch.py b/graphify/watch.py index 08b6fdd43..45bd97b66 100644 --- a/graphify/watch.py +++ b/graphify/watch.py @@ -1359,18 +1359,40 @@ def _add_deleted_source(path: Path) -> None: except Exception: pass - # to_html raises ValueError for graphs > MAX_NODES_FOR_VIZ (5000). + # to_html raises ValueError for graphs > the viz node limit. # Wrap so core outputs (graph.json + GRAPH_REPORT.md) always land. html_written = False if not no_change: + html_target = out / "graph.html" try: - to_html(G, communities, str(out / "graph.html"), community_labels=labels or None) + to_html(G, communities, str(html_target), community_labels=labels or None) html_written = True except ValueError as viz_err: - print(f"[graphify watch] Skipped graph.html: {viz_err}") - stale = out / "graph.html" - if stale.exists(): - stale.unlink() + # Over the cap. Deleting was defensible on its own — a kept + # graph.html would describe an older, smaller graph — but it + # leaves a project that crossed the threshold with no + # visualization at all, and the file is gone before the user + # sees the message. The export path (#1019) already re-renders + # the community-aggregation view in exactly this case, so do + # the same here: current AND present beats current OR present. + from graphify.exporters.html import _viz_node_limit + if html_target.exists(): + html_target.unlink() + limit = _viz_node_limit() + if limit <= 0: + # GRAPHIFY_VIZ_NODE_LIMIT=0 means "no HTML viz" (CI runners), + # so honour it rather than aggregating around it. + print(f"[graphify watch] Skipped graph.html: {viz_err}") + else: + try: + to_html(G, communities, str(html_target), + community_labels=labels or None, node_limit=limit) + # The aggregator declines to write a single-community + # graph, so trust the file rather than the call. + html_written = html_target.exists() + except Exception as fallback_err: + print(f"[graphify watch] Skipped graph.html: {viz_err} " + f"(aggregated view also failed: {fallback_err})") # Regenerate callflow HTML if the user previously generated one — # opt-in by existence so users who never ran callflow-html aren't affected. diff --git a/tests/test_watch.py b/tests/test_watch.py index a9b3ebe89..f2bec415f 100644 --- a/tests/test_watch.py +++ b/tests/test_watch.py @@ -267,6 +267,59 @@ def test_rebuild_code_drops_labels_whose_community_changed(tmp_path): ) +def test_rebuild_code_keeps_a_visualization_when_over_the_viz_cap(tmp_path, monkeypatch): + """Crossing the viz node limit must not leave the project with no graph.html. + _rebuild_code used to unlink the existing file and write nothing, so a repo + that grew past the cap silently lost its visualization — and the file was + already gone by the time the user read the message. The export path falls + back to the community-aggregation view in exactly this case; the incremental + path should too, so the artifact stays both current and present.""" + import json + from graphify.watch import _rebuild_code + + corpus = tmp_path / "corpus" + corpus.mkdir() + # Several *disconnected* clusters: the aggregator declines to render a + # single-community meta-graph, so a ring of mutually-importing modules + # would collapse to one community and exercise the wrong path. + for g in range(4): + for i in range(3): + other = (i + 1) % 3 + (corpus / f"g{g}_m{i}.py").write_text( + f"import g{g}_m{other}\n\n" + + "".join(f"def g{g}_f{i}_{j}():\n return {j}\n\n" for j in range(4)), + encoding="utf-8", + ) + assert _rebuild_code(corpus, acquire_lock=False) is True + html = corpus / "graphify-out" / "graph.html" + assert html.exists(), "expected a normal (under-cap) rebuild to write graph.html" + before = html.read_text(encoding="utf-8") + + # Drop the cap below the graph's size but above its community count — the + # real shape of this bug. (A cap under the community count would make the + # aggregated meta-graph breach it too, which is a different situation.) + graph = json.loads((corpus / "graphify-out" / "graph.json").read_text(encoding="utf-8")) + communities = {n.get("community") for n in graph["nodes"] if n.get("community") is not None} + cap = (len(communities) + len(graph["nodes"])) // 2 + assert len(communities) < cap < len(graph["nodes"]), "test corpus cannot exercise the cap" + monkeypatch.setenv("GRAPHIFY_VIZ_NODE_LIMIT", str(cap)) + (corpus / "g9_extra.py").write_text("def extra():\n return 1\n", encoding="utf-8") + assert _rebuild_code(corpus, acquire_lock=False) is True + + assert html.exists(), ( + "graph.html was deleted when the graph exceeded the viz cap — the " + "incremental rebuild must fall back to the aggregated view like export does" + ) + after = html.read_text(encoding="utf-8") + assert after != before, "graph.html must be re-rendered, not left stale" + + # And the documented kill switch still means "no viz", not "aggregate". + monkeypatch.setenv("GRAPHIFY_VIZ_NODE_LIMIT", "0") + (corpus / "g9_extra2.py").write_text("def extra2():\n return 2\n", encoding="utf-8") + assert _rebuild_code(corpus, acquire_lock=False) is True + assert not html.exists(), "GRAPHIFY_VIZ_NODE_LIMIT=0 must disable the HTML viz outright" + + def test_update_rebuilds_with_nested_star_gitignore(tmp_path): """#1880: `graphify update` must not emit 0 nodes (and then refuse to overwrite) just because the source tree has a nested `.gitignore` with a From d91a987aa57063ac3705fa2e4b903523433f2ec0 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 27 Jul 2026 10:46:47 +0100 Subject: [PATCH 287/292] fix(extract): stamp target_file on Python imports and markdown refs so incremental targets canonicalize (#2211, #2213) The #2169 incremental canonicalization only rewrites edge targets that carry a target_file stamp. Python relative imports and markdown reference links emitted absolute-path-derived target ids without one, so on an incremental/subset extraction they dangled on an absolute id instead of resolving to the canonical root-relative node (dropping md->md references and leaving a dangling imports_from on --no-cluster). Both now stamp the resolved target (existence-gated); the stamp is popped before graph.json ships. Also register the unresolved target form in the remap loop so a symlinked root (macOS /tmp) can't cause an id-form mismatch. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/extract.py | 42 ++++++++++++-- graphify/extractors/markdown.py | 28 ++++++++-- tests/test_incremental.py | 97 +++++++++++++++++++++++++++++++++ 3 files changed, 158 insertions(+), 9 deletions(-) diff --git a/graphify/extract.py b/graphify/extract.py index 0835a9795..9de0c7622 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -339,6 +339,7 @@ def _import_python(node, source: bytes, file_nid: str, stem: str, edges: list, s module_node = node.child_by_field_name("module_name") if module_node: raw = _read_text(module_node, source) + target_path: "Path | None" = None if raw.startswith("."): # Relative import - resolve to full path so IDs match file node IDs dots = len(raw) - len(raw.lstrip(".")) @@ -347,10 +348,11 @@ def _import_python(node, source: bytes, file_nid: str, stem: str, edges: list, s for _ in range(dots - 1): base = base.parent rel = (module_name.replace(".", "/") + ".py") if module_name else "__init__.py" - tgt_nid = _make_id(str(base / rel)) + target_path = base / rel + tgt_nid = _make_id(str(target_path)) else: tgt_nid = _make_id(raw) - edges.append({ + edge = { "source": file_nid, "target": tgt_nid, "relation": "imports_from", @@ -359,7 +361,22 @@ def _import_python(node, source: bytes, file_nid: str, stem: str, edges: list, s "source_file": str_path, "source_location": f"L{node.start_point[0] + 1}", "weight": 1.0, - }) + } + # Stamp the resolved target file (mirroring _import_js, #1814) so + # the #2169 remap pass can canonicalize this edge's target on an + # incremental run where the target file itself is not in the + # batch — without it the target keeps an absolute-path-derived id + # that matches no node in the merged graph and dangles (#2213). + # Existence-gated: a speculative import of a nonexistent sibling + # must stay dangling, exactly as before. The stamp is transient + # and popped before graph.json ships. + if target_path is not None: + try: + if target_path.is_file(): + edge["target_file"] = str(target_path) + except OSError: + pass + edges.append(edge) def _import_js(node, source: bytes, file_nid: str, stem: str, edges: list, str_path: str, scope_stack: list[str] | None = None) -> None: @@ -4698,11 +4715,18 @@ def extract( _tf = _e.get("target_file") if not _tf: continue + _raw_tp = Path(_tf) try: - _tp = Path(_tf).resolve() + _tp = _raw_tp.resolve() except (OSError, RuntimeError): continue if _tp in _remap_seen: + # Already covered: either the target is in this batch (its input + # form is the same form the extractors minted ids from, and the + # per-path loop registers both that and the resolved form) or an + # earlier stamped edge registered it. Re-appending it here would + # re-run its per-path iteration AFTER later batch files and could + # flip the last-writer of a colliding old-id key. continue _remap_seen.add(_tp) try: @@ -4719,6 +4743,16 @@ def extract( except OSError: continue remap_paths.append(_tp) + # Also register the AS-STAMPED (unresolved) form. The edge target id + # was minted from the stamped path exactly as written (e.g. + # ``str(base / rel)`` for a Python relative import), which under a + # symlinked root (macOS /tmp -> /private/tmp) or relative inputs + # differs from the resolved form; the per-path loop below derives the + # old ids from whichever Path it is given, so a missing form would + # leave the edge target unmapped and dangling. + if _raw_tp != _tp: + _remap_seen.add(_raw_tp) + remap_paths.append(_raw_tp) for path in remap_paths: old_id = _make_id(str(path)) try: diff --git a/graphify/extractors/markdown.py b/graphify/extractors/markdown.py index d6d45ac8f..e1b24409a 100644 --- a/graphify/extractors/markdown.py +++ b/graphify/extractors/markdown.py @@ -94,10 +94,14 @@ def add_node(nid: str, label: str, line: int, file_type: str = "document") -> No "source_file": str_path, "source_location": f"L{line}"}) def add_edge(src: str, tgt: str, relation: str, line: int, - confidence: str = "EXTRACTED", weight: float = 1.0) -> None: - edges.append({"source": src, "target": tgt, "relation": relation, - "confidence": confidence, "source_file": str_path, - "source_location": f"L{line}", "weight": weight}) + confidence: str = "EXTRACTED", weight: float = 1.0, + target_file: "str | None" = None) -> None: + edge = {"source": src, "target": tgt, "relation": relation, + "confidence": confidence, "source_file": str_path, + "source_location": f"L{line}", "weight": weight} + if target_file is not None: + edge["target_file"] = target_file + edges.append(edge) file_nid = _make_id(str(path)) add_node(file_nid, path.name, 1) @@ -120,7 +124,21 @@ def add_link(raw: str, line: int) -> None: if tgt_nid == file_nid or tgt_nid in linked_targets: return linked_targets.add(tgt_nid) - add_edge(file_nid, tgt_nid, "references", line) + # Stamp the resolved target file (mirroring the JS/Python import + # stamps, #1814/#2213) so the #2169 remap pass can canonicalize this + # edge's target on an incremental run where the linked doc is not in + # the batch — without it the target keeps an absolute-path-derived id + # that matches no node in the merged graph and the md->md reference + # silently drops (#2211). Existence-gated: a link to a nonexistent + # doc must stay dangling, exactly as before. The stamp is transient + # and popped before graph.json ships. + target_file = None + try: + if resolved.is_file(): + target_file = str(resolved) + except OSError: + pass + add_edge(file_nid, tgt_nid, "references", line, target_file=target_file) # Track heading stack for nesting: [(level, nid), ...] heading_stack: list[tuple[int, str]] = [] diff --git a/tests/test_incremental.py b/tests/test_incremental.py index 91bf8e57c..62c120325 100644 --- a/tests/test_incremental.py +++ b/tests/test_incremental.py @@ -211,6 +211,103 @@ def test_extract_no_cluster_incremental_code_only_preserves_doc_nodes(tmp_path): assert any("beta" in i for i in after_by_id), sorted(after_by_id) +def test_incremental_python_relative_import_target_canonicalizes(tmp_path): + """#2213 (defect 1, shared root with #2211): a Python relative import's + imports_from edge must stamp target_file so the #2169 remap canonicalizes + its target on an incremental extraction where the target file is NOT in + the batch — instead of leaving an absolute-path-derived dangling id.""" + from graphify.extract import extract, _make_id + + # realpath: on macOS the pytest tmp dir can sit behind a symlink + # (/tmp -> /private/tmp); anchor everything on the resolved form so the + # canonical-id assertions are deterministic. + tmp = Path(os.path.realpath(tmp_path)) + pkg = tmp / "pkg" + pkg.mkdir() + (pkg / "b.py").write_text( + "class Thing:\n def go(self):\n return 1\n", encoding="utf-8" + ) + a = pkg / "a.py" + a.write_text( + "from .b import Thing\n\n\ndef use():\n return Thing().go()\n", + encoding="utf-8", + ) + + full = extract([a, pkg / "b.py"], cache_root=tmp) + full_imports = [ + e for e in full["edges"] + if e.get("relation") == "imports_from" + and str(e.get("source_file", "")).endswith("a.py") + ] + assert full_imports, full["edges"] + canonical = full_imports[0]["target"] + assert canonical == "pkg_b", canonical + + # Incremental: only the importer is in the batch (b.py unchanged, so the + # #2169 merge path re-extracts a.py alone). Same cache/scan root. + inc = extract([a], cache_root=tmp) + inc_imports = [ + e for e in inc["edges"] + if e.get("relation") == "imports_from" + and str(e.get("source_file", "")).endswith("a.py") + ] + assert inc_imports, inc["edges"] + assert inc_imports[0]["target"] == canonical, inc_imports + # Not an absolute-path-shaped ghost id (…_pkg_b would end "_b", but the + # pre-fix dangling form was the full path with the extension folded in). + assert not inc_imports[0]["target"].endswith("_py"), inc_imports + + root_slug = _make_id(str(tmp)) + for e in inc["edges"]: + assert root_slug not in str(e.get("target", "")), e + # The target_file hint is transient and must never ship. + assert "target_file" not in e, e + + +def test_incremental_md_reference_target_canonicalizes(tmp_path): + """#2211: a markdown [link](docs/setup.md) references edge must stamp + target_file so the #2169 remap canonicalizes its target on an incremental + extraction where the linked doc is NOT in the batch — instead of the + md->md reference dangling on an absolute-path-derived id and dropping.""" + from graphify.extract import extract, _make_id + + tmp = Path(os.path.realpath(tmp_path)) + docs = tmp / "docs" + docs.mkdir() + setup = docs / "setup.md" + setup.write_text("# Setup\nInstall the thing.\n", encoding="utf-8") + claude = tmp / "CLAUDE.md" + claude.write_text( + "# Overview\nSee [setup](docs/setup.md) for install steps.\n", + encoding="utf-8", + ) + + full = extract([claude, setup], cache_root=tmp) + full_refs = [ + e for e in full["edges"] + if e.get("relation") == "references" + and str(e.get("source_file", "")).endswith("CLAUDE.md") + ] + assert full_refs, full["edges"] + canonical = full_refs[0]["target"] + assert canonical == "docs_setup", canonical + + # Incremental: only the linking doc is in the batch. + inc = extract([claude], cache_root=tmp) + inc_refs = [ + e for e in inc["edges"] + if e.get("relation") == "references" + and str(e.get("source_file", "")).endswith("CLAUDE.md") + ] + assert inc_refs, inc["edges"] + assert inc_refs[0]["target"] == canonical, inc_refs + + root_slug = _make_id(str(tmp)) + for e in inc["edges"]: + assert root_slug not in str(e.get("target", "")), e + assert "target_file" not in e, e + + def test_update_prunes_a_removed_imports_edge(tmp_path): """#1521: when an import is deleted from a file, `graphify update` must prune the edge it produced — preserving it (keyed only on endpoint membership) left a From 2f78439ffd526bb05dd3d3c58eb237e8f916a527 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 27 Jul 2026 10:46:47 +0100 Subject: [PATCH 288/292] fix: load raw edges-keyed graphs + stop pruning alive files as deleted (#2212, #2210) #2212: benchmark, the merge-driver graph load, and callflow_html crashed or silently failed on a --no-cluster graph.json (edges stored under 'edges', not 'links'). A shared load_node_link_graph helper normalizes links/edges before node_link_graph and is used at all three sites. #2210: _stale_graph_sources compared graph source_file spellings to the scan with a raw string test (no NFC), and pruned any non-match with no liveness check, so alive files (macOS NFD paths, legacy basenames) were pruned as 'deleted'. It now compares NFC-on-both-sides and is fail-closed: a corpus-missing source whose file still exists is pruned only when the exclusion is provable, else kept with a warning. Prune message corrected to 'deleted or excluded'. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/benchmark.py | 15 ++-- graphify/build.py | 10 ++- graphify/callflow_html.py | 11 +-- graphify/cli.py | 102 ++++++++++++++++++++++++-- graphify/paths.py | 41 +++++++++++ tests/test_benchmark_raw_graph.py | 103 ++++++++++++++++++++++++++ tests/test_stale_prune.py | 118 ++++++++++++++++++++++++++++++ 7 files changed, 374 insertions(+), 26 deletions(-) create mode 100644 tests/test_benchmark_raw_graph.py create mode 100644 tests/test_stale_prune.py diff --git a/graphify/benchmark.py b/graphify/benchmark.py index ee9c3925e..cf2bc6ba2 100644 --- a/graphify/benchmark.py +++ b/graphify/benchmark.py @@ -1,10 +1,7 @@ """Token-reduction benchmark - measures how much context graphify saves vs naive full-corpus approach.""" from __future__ import annotations -import json import sys -from pathlib import Path import networkx as nx -from networkx.readwrite import json_graph from graphify.build import edge_data from graphify.serve import _query_terms @@ -100,13 +97,11 @@ def run_benchmark( Returns dict with: corpus_tokens, avg_query_tokens, reduction_ratio, per_question """ graph_path = graph_path or _default_graph_json() - from graphify.security import check_graph_file_size_cap - check_graph_file_size_cap(Path(graph_path)) - data = json.loads(Path(graph_path).read_text(encoding="utf-8")) - try: - G = json_graph.node_link_graph(data, edges="links") - except TypeError: - G = json_graph.node_link_graph(data) + # Size-cap check + links/edges normalization + node-link parse. A raw + # --no-cluster graph stores edges under "edges" and used to KeyError + # here (#2212). + from graphify.paths import load_node_link_graph + G = load_node_link_graph(graph_path) if corpus_words is None: # Rough estimate: each node label is ~3 words, plus source context diff --git a/graphify/build.py b/graphify/build.py index 7e7ed6157..1f840bbc0 100644 --- a/graphify/build.py +++ b/graphify/build.py @@ -1408,7 +1408,8 @@ def _prune_match(sf: "str | None") -> bool: n_nodes = len(to_remove) if n_nodes: print( - f"[graphify] Pruned {n_nodes} node(s) from {n_files} deleted source file(s).", + f"[graphify] Pruned {n_nodes} node(s) from {n_files} deleted or " + f"excluded source file(s).", file=sys.stderr, ) @@ -1419,14 +1420,15 @@ def _prune_match(sf: "str | None") -> bool: if edges_to_remove: G.remove_edges_from(edges_to_remove) print( - f"[graphify] Pruned {len(edges_to_remove)} edge(s) from deleted source file(s).", + f"[graphify] Pruned {len(edges_to_remove)} edge(s) from deleted or " + f"excluded source file(s).", file=sys.stderr, ) if not n_nodes and not edges_to_remove: print( - f"[graphify] {n_files} source file(s) deleted since last run — " - f"no matching nodes or edges in graph, already clean.", + f"[graphify] {n_files} source file(s) deleted or excluded since " + f"last run — no matching nodes or edges in graph, already clean.", file=sys.stderr, ) diff --git a/graphify/callflow_html.py b/graphify/callflow_html.py index 181e7493b..dabb2997c 100644 --- a/graphify/callflow_html.py +++ b/graphify/callflow_html.py @@ -227,12 +227,13 @@ def _node_link_payload(data: dict) -> tuple[list, list] | None: return None try: - from networkx.readwrite import json_graph + # Shared loader normalizes the raw writer's "edges" key to "links" + # before parsing; without it an edges-keyed payload raised + # KeyError: 'links' and this function silently returned None even + # though the shape check above accepts "edges" (#2212). + from graphify.paths import load_node_link_graph - try: - graph = json_graph.node_link_graph(data, edges="links") - except TypeError: - graph = json_graph.node_link_graph(data) + graph = load_node_link_graph(data) except Exception: return None diff --git a/graphify/cli.py b/graphify/cli.py index 88f5b51ed..bd8f12bca 100644 --- a/graphify/cli.py +++ b/graphify/cli.py @@ -149,6 +149,7 @@ def _stale_graph_sources( graph_path: Path, scan_root: Path, seen_files: set[str], + detection: dict | None = None, ) -> list[str]: """Source files graph.json still references but the current scan no longer contains (#1909). @@ -173,7 +174,22 @@ def _stale_graph_sources( ``seen_files`` must be the FULL detect output including unclassified files, so nodes from walked-but-unsupported sources (e.g. introspected Cargo.toml manifests) are not misread as stale. + + Paths are compared NFC-normalized on both sides: macOS reports NFD + filenames while graph ``source_file`` entries are typically NFC, and a + raw-string membership test misread every accented live file as stale + (#2210; same class as the manifest-layer #2221/#2224). + + Fail-closed liveness guard (#2210, mirrors watch.py's excluded-vs-deleted + distinction): a source missing from the scan corpus is only pruned when + the file is gone from disk, or when its exclusion is PROVABLE from the + same scan that produced ``seen_files`` — ``detection``'s ``ignored`` / + ``pruned_noise_dirs`` / ``skipped_sensitive`` output, or detect's + sensitivity predicate. An alive file that merely failed the membership + test (path-spelling drift the normalization didn't cover, walk errors, + …) is KEPT and reported, never mass-evicted. """ + from graphify.paths import nfc try: data = json.loads(graph_path.read_text(encoding="utf-8")) except Exception: @@ -203,15 +219,57 @@ def _within_root(p: Path) -> bool: except (ValueError, OSError, RuntimeError): return False + seen_nfc = {nfc(s) for s in seen_files} + seen_basenames = {nfc(os.path.basename(s)) for s in seen_files} + def _in_seen(p: Path) -> bool: - if str(p) in seen_files: + if nfc(str(p)) in seen_nfc: return True try: - return str(p.resolve()) in seen_files + return nfc(str(p.resolve())) in seen_nfc except (OSError, RuntimeError): return False + # Provable-exclusion evidence from the scan that produced seen_files: + # individually ignored files are exact entries; ignored/noise-pruned + # directories are recorded once with a trailing separator and cover + # their whole subtree. skipped_sensitive entries may carry a + # " [reason]" suffix. + excluded_exact: set[str] = set() + excluded_prefixes: list[str] = [] + if detection: + for entry in list(detection.get("ignored", [])) + list( + detection.get("pruned_noise_dirs", []) + ): + e = nfc(str(entry)) + if e.endswith(os.sep) or e.endswith("/"): + excluded_prefixes.append(e) + else: + excluded_exact.add(e) + for entry in detection.get("skipped_sensitive", []): + excluded_exact.add(nfc(str(entry).split(" [", 1)[0])) + + def _provably_excluded(c: Path) -> bool: + spellings = [nfc(str(c))] + try: + spellings.append(nfc(str(c.resolve()))) + except (OSError, RuntimeError): + pass + for s in spellings: + if s in excluded_exact: + return True + if any(s.startswith(pref) for pref in excluded_prefixes): + return True + try: + from graphify.detect import _is_sensitive as _det_sensitive + if _det_sensitive(c): + return True + except Exception: + pass + return False + stale: list[str] = [] + kept_alive: list[str] = [] checked: set[str] = set() for n in data.get("nodes", []): if not isinstance(n, dict): @@ -238,7 +296,37 @@ def _in_seen(p: Path) -> bool: continue # out-of-root under every anchor: never prune if any(_in_seen(c) for c in in_root): continue # still part of the scan corpus + # Fail-closed liveness guard (#2210): absence from the corpus is + # only deletion evidence when the file is actually gone from disk. + alive = [] + for c in in_root: + try: + if c.exists(): + alive.append(c) + except OSError: + pass + if alive: + if all(_provably_excluded(c) for c in alive): + stale.append(sf) # alive but excluded under current rules (#1909) + else: + kept_alive.append(sf) + continue + # No anchored candidate exists, but a legacy bare-basename spelling + # can't be anchored reliably — a live corpus file with the same name + # means deletion is unproven; keep. + rel_sf = sf.replace("\\", "/") + if "/" not in rel_sf and nfc(rel_sf) in seen_basenames: + kept_alive.append(sf) + continue stale.append(sf) + if kept_alive: + print( + f"[graphify] fail-closed: kept node(s) from {len(kept_alive)} " + "source file(s) that left the scan corpus but still exist on disk " + "(ignore rules or filters changed?). Run a full re-extraction to " + "purge them if the exclusion is intentional.", + file=sys.stderr, + ) return stale @@ -1946,10 +2034,10 @@ def _load_graph(p: str): f"graph.json {p} is {size} bytes, exceeds {_MERGE_MAX_BYTES}-byte cap" ) data = json.loads(path_obj.read_text(encoding="utf-8")) - try: - return _jg.node_link_graph(data, edges="links"), data - except TypeError: - return _jg.node_link_graph(data), data + # A committed raw (--no-cluster) graph stores edges under "edges"; + # parse via the shared links/edges-normalizing loader (#2212). + from graphify.paths import load_node_link_graph as _lnlg + return _lnlg(data), data try: G_cur, _ = _load_graph(_current_path) G_oth, _ = _load_graph(_other_path) @@ -2729,7 +2817,7 @@ def _parse_float(name: str, raw: str) -> float: _seen_files = {f for _fl in files_by_type.values() for f in _fl} _seen_files.update(detection.get("unclassified", [])) graph_stale_sources = _stale_graph_sources( - existing_graph_path, target, _seen_files + existing_graph_path, target, _seen_files, detection=detection ) else: print(f"[graphify extract] scanning {target}") diff --git a/graphify/paths.py b/graphify/paths.py index d43bc7766..a1adaf9f2 100644 --- a/graphify/paths.py +++ b/graphify/paths.py @@ -302,3 +302,44 @@ def default_graph_json() -> str: the path is passed explicitly (#1423). """ return str(out_path("graph.json")) + + +def nfc(s: str) -> str: + """NFC-normalize a path string. + + macOS (HFS+/APFS) reports filenames in NFD while manifests, graph + ``source_file`` entries and user input are typically NFC. Comparing raw + strings makes the same file look like two different paths, so any path + membership test must normalize BOTH sides (#2210, #2221/#2224). + """ + import unicodedata + return unicodedata.normalize("NFC", s) + + +def load_node_link_graph(path_or_data): + """Load a graphify graph.json into a networkx graph, accepting both writers. + + The clustered writer stores edges under ``links`` (networkx's node-link + default); the raw ``--no-cluster`` writer stores them under ``edges``. + Consumers that call ``node_link_graph(data, edges="links")`` directly + raise ``KeyError: 'links'`` on a raw graph (#2212) — the ``except + TypeError`` fallback only covers old networkx without the ``edges`` + kwarg, not the missing key. Normalize before parsing, same idiom as + affected.py/serve.py. + + Accepts a path (size-cap-checked via the security module, then parsed) + or an already-parsed dict (no size check — the caller owns any cap). + """ + from networkx.readwrite import json_graph + data = path_or_data + if not isinstance(data, dict): + p = Path(data) + from graphify.security import check_graph_file_size_cap # lazy: security imports paths + check_graph_file_size_cap(p) + data = json.loads(p.read_text(encoding="utf-8")) + if isinstance(data, dict) and "links" not in data and "edges" in data: + data = dict(data, links=data["edges"]) + try: + return json_graph.node_link_graph(data, edges="links") + except TypeError: # networkx too old for the edges kwarg; default is "links" + return json_graph.node_link_graph(data) diff --git a/tests/test_benchmark_raw_graph.py b/tests/test_benchmark_raw_graph.py new file mode 100644 index 000000000..51da5fe02 --- /dev/null +++ b/tests/test_benchmark_raw_graph.py @@ -0,0 +1,103 @@ +"""#2212: run_benchmark must accept a raw --no-cluster graph.json. + +The clustered writer stores edges under "links" (networkx node-link default); +the raw --no-cluster writer stores them under "edges". Consumers calling +node_link_graph(data, edges="links") raised KeyError: 'links' on the raw +shape — the `except TypeError` fallback only covered old networkx versions, +not the missing key. +""" +from __future__ import annotations +import json + +from graphify.benchmark import run_benchmark + + +def _graph_payload(edges_key: str) -> dict: + # Raw extract shape: top-level nodes/edges/hyperedges + token counters. + nodes = [ + { + "id": "auth_flow", + "label": "authentication flow", + "source_file": "auth.py", + "source_location": "L1", + }, + { + "id": "login_handler", + "label": "user login authentication handler", + "source_file": "auth.py", + "source_location": "L10", + }, + { + "id": "main_entry", + "label": "main entry point", + "source_file": "main.py", + "source_location": "L1", + }, + ] + edges = [ + { + "id": "edge_1", + "source": "auth_flow", + "target": "login_handler", + "relation": "calls", + "confidence": "EXTRACTED", + "confidence_score": 1.0, + }, + { + "id": "edge_2", + "source": "login_handler", + "target": "main_entry", + "relation": "used_by", + "confidence": "EXTRACTED", + "confidence_score": 1.0, + }, + ] + return { + "nodes": nodes, + edges_key: edges, + "hyperedges": [], + "input_tokens": 0, + "output_tokens": 0, + } + + +def test_run_benchmark_raw_edges_keyed_graph(tmp_path): + """A --no-cluster graph.json ("edges" key) must not raise KeyError.""" + graph_file = tmp_path / "graph.json" + graph_file.write_text(json.dumps(_graph_payload("edges")), encoding="utf-8") + + result = run_benchmark(graph_path=str(graph_file), corpus_words=5_000) + + assert "error" not in result + assert result["nodes"] == 3 + assert result["edges"] == 2 + assert result["reduction_ratio"] > 0 + assert any( + "authentication" in p["question"] for p in result["per_question"] + ) + + +def test_run_benchmark_links_keyed_graph(tmp_path): + """The clustered writer's "links" key keeps working identically.""" + graph_file = tmp_path / "graph.json" + graph_file.write_text(json.dumps(_graph_payload("links")), encoding="utf-8") + + result = run_benchmark(graph_path=str(graph_file), corpus_words=5_000) + + assert "error" not in result + assert result["nodes"] == 3 + assert result["edges"] == 2 + assert result["reduction_ratio"] > 0 + + +def test_raw_and_links_graphs_benchmark_identically(tmp_path): + """Both spellings of the same graph must produce the same stats.""" + raw_file = tmp_path / "raw.json" + raw_file.write_text(json.dumps(_graph_payload("edges")), encoding="utf-8") + links_file = tmp_path / "links.json" + links_file.write_text(json.dumps(_graph_payload("links")), encoding="utf-8") + + raw = run_benchmark(graph_path=str(raw_file), corpus_words=5_000) + links = run_benchmark(graph_path=str(links_file), corpus_words=5_000) + + assert raw == links diff --git a/tests/test_stale_prune.py b/tests/test_stale_prune.py new file mode 100644 index 000000000..b4997b1ce --- /dev/null +++ b/tests/test_stale_prune.py @@ -0,0 +1,118 @@ +"""#2210: incremental extract's graph-layer prune must not evict ALIVE files. + +_stale_graph_sources compared stored graph source_file spellings against the +current scan with a raw string membership test: (a) no NFC/NFD normalization, +so a macOS NFD on-disk path never matched an NFC graph entry; (b) no liveness +check, so any membership miss was declared "deleted" and pruned even though +the file exists on disk and is in the scan. These tests exercise the NFC +normalization, the fail-closed liveness guard, and that genuinely-deleted +sources are still pruned. +""" +from __future__ import annotations +import json +import unicodedata + +from graphify.cli import _stale_graph_sources +from graphify.detect import detect + +NFC_NAME = unicodedata.normalize("NFC", "café.md") # café.md, composed +NFD_NAME = unicodedata.normalize("NFD", "café.md") # cafe + combining accent + + +def _write_graph(tmp_path, source_files: list[str]): + out = tmp_path / "graphify-out" + out.mkdir(exist_ok=True) + graph_path = out / "graph.json" + nodes = [ + {"id": f"n{i}", "label": f"node {i}", "source_file": sf} + for i, sf in enumerate(source_files) + ] + graph_path.write_text( + json.dumps({"nodes": nodes, "links": []}), encoding="utf-8" + ) + return graph_path + + +def _scan(tmp_path): + detection = detect(tmp_path) + seen = {f for flist in detection["files"].values() for f in flist} + seen.update(detection.get("unclassified", [])) + return detection, seen + + +def test_nfd_disk_nfc_graph_source_not_pruned(tmp_path): + """(a) NFD spelling on disk vs NFC spelling in the graph: NOT stale.""" + docs = tmp_path / "docs" + docs.mkdir() + (docs / NFD_NAME).write_text("# cafe notes\n\nhello\n", encoding="utf-8") + + graph_path = _write_graph(tmp_path, ["docs/" + NFC_NAME]) + detection, seen = _scan(tmp_path) + + stale = _stale_graph_sources(graph_path, tmp_path, seen, detection=detection) + assert stale == [] + + +def test_bare_basename_alive_elsewhere_not_pruned(tmp_path, capsys): + """(b) fail-closed: a legacy bare-basename source_file whose file is + alive at docs/café.md cannot be proven deleted — keep it.""" + docs = tmp_path / "docs" + docs.mkdir() + (docs / NFD_NAME).write_text("# cafe notes\n\nhello\n", encoding="utf-8") + + graph_path = _write_graph(tmp_path, [NFC_NAME]) # bare legacy spelling + detection, seen = _scan(tmp_path) + + stale = _stale_graph_sources(graph_path, tmp_path, seen, detection=detection) + assert stale == [] + + +def test_genuinely_deleted_source_still_pruned(tmp_path): + """(c) a source_file with no file on disk anywhere IS pruned.""" + docs = tmp_path / "docs" + docs.mkdir() + (docs / "keep.md").write_text("# keep\n\nstill here\n", encoding="utf-8") + + graph_path = _write_graph(tmp_path, ["docs/keep.md", "docs/gone.md"]) + detection, seen = _scan(tmp_path) + + stale = _stale_graph_sources(graph_path, tmp_path, seen, detection=detection) + assert stale == ["docs/gone.md"] + + +def test_alive_but_ignored_source_is_pruned(tmp_path): + """#1909 must keep working: an alive file excluded by ignore rules is + provably excluded, so its nodes ARE pruned.""" + docs = tmp_path / "docs" + docs.mkdir() + (docs / "keep.md").write_text("# keep\n\nstill here\n", encoding="utf-8") + (docs / "secret.md").write_text("# secret\n\nexcluded\n", encoding="utf-8") + (tmp_path / ".graphifyignore").write_text("docs/secret.md\n", encoding="utf-8") + + graph_path = _write_graph(tmp_path, ["docs/keep.md", "docs/secret.md"]) + detection, seen = _scan(tmp_path) + + stale = _stale_graph_sources(graph_path, tmp_path, seen, detection=detection) + assert stale == ["docs/secret.md"] + + +def test_alive_unproven_exclusion_kept_with_warning(tmp_path, capsys): + """Fail-closed: an alive in-root file missing from the corpus without + provable exclusion evidence is kept, and the keep is reported.""" + docs = tmp_path / "docs" + docs.mkdir() + (docs / "keep.md").write_text("# keep\n\nstill here\n", encoding="utf-8") + (docs / "other.md").write_text("# other\n\nalive\n", encoding="utf-8") + + graph_path = _write_graph(tmp_path, ["docs/keep.md", "docs/other.md"]) + detection, seen = _scan(tmp_path) + # Simulate a scan that lost docs/other.md for a reason that is NOT a + # provable exclusion (e.g. a walk error): drop it from seen and from + # the detection evidence. + seen = {s for s in seen if not s.endswith("other.md")} + detection = dict(detection, ignored=[], pruned_noise_dirs=[], skipped_sensitive=[]) + + stale = _stale_graph_sources(graph_path, tmp_path, seen, detection=detection) + assert stale == [] + err = capsys.readouterr().err + assert "fail-closed" in err From bdb678858b61183c3cf820fd6f82eec9748b7134 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 27 Jul 2026 10:46:47 +0100 Subject: [PATCH 289/292] fix(install): scope uninstall to project_dir instead of always deleting the global skill (#2215) claude_uninstall/gemini_uninstall/codebuddy_uninstall accepted project_dir but, with the default project=False, still deleted the user-global skill tree, so a library/test caller passing a project_dir nuked ~/.claude et al (the API trap behind #2168). They now take remove_user_skill and treat a passed project_dir as authoritative; uninstall_all opts in explicitly to preserve 'graphify uninstall' behavior. Also fixes a live CLI bug where 'uninstall --project' deleted the global codebuddy skill. Co-Authored-By: Claude Opus 4.8 (1M context) --- graphify/install.py | 65 +++++++++++++++---- tests/test_home_sandbox.py | 14 +++-- tests/test_uninstall_scope.py | 113 ++++++++++++++++++++++++++++++++++ 3 files changed, 175 insertions(+), 17 deletions(-) create mode 100644 tests/test_uninstall_scope.py diff --git a/graphify/install.py b/graphify/install.py index c3503c2e8..8c0cfee65 100644 --- a/graphify/install.py +++ b/graphify/install.py @@ -792,10 +792,22 @@ def _uninstall_gemini_hook(project_dir: Path) -> None: settings["hooks"]["BeforeTool"] = filtered settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") print(" .gemini/settings.json -> BeforeTool hook removed") -def gemini_uninstall(project_dir: Path | None = None, *, project: bool = False) -> None: - """Remove the graphify section from GEMINI.md, uninstall hook, and remove skill file.""" +def gemini_uninstall(project_dir: Path | None = None, *, project: bool = False, remove_user_skill: bool | None = None) -> None: + """Remove the graphify section from GEMINI.md, uninstall hook, and remove skill file. + + Scope rules (#2215): a bare call removes the user-global skill; passing + ``project_dir`` (or ``project=True``) scopes skill removal to that project + and leaves the global tree untouched, unless ``remove_user_skill=True`` + explicitly opts back into the global delete (as ``uninstall_all`` does). + """ + explicit_dir = project_dir is not None project_dir = project_dir or Path(".") - _remove_skill_file("gemini", project=project, project_dir=project_dir) + if remove_user_skill is None: + remove_user_skill = not project and not explicit_dir + if project or (explicit_dir and not remove_user_skill): + _remove_skill_file("gemini", project=True, project_dir=project_dir) + if remove_user_skill: + _remove_skill_file("gemini", project=False) target = project_dir / "GEMINI.md" if not target.exists(): @@ -1600,7 +1612,9 @@ def _project_uninstall(platform_name: str, project_dir: Path | None = None) -> N if not removed: print("nothing to remove") elif platform_name == "codebuddy": - codebuddy_uninstall(project_dir) + # project=True keeps `uninstall --project` project-scoped; previously + # this deleted the user-global codebuddy skill (#2215). + codebuddy_uninstall(project_dir, project=True) else: _remove_skill_file(platform_name, project=True, project_dir=project_dir) def _project_uninstall_all(project_dir: Path | None = None) -> None: @@ -1760,10 +1774,12 @@ def uninstall_all(project_dir: Path | None = None, purge: bool = False) -> None: pd = project_dir or Path(".") print("Uninstalling graphify from all detected platforms...\n") - # Skill-file / config-section uninstallers - claude_uninstall(pd) - codebuddy_uninstall(pd) - gemini_uninstall(pd) + # Skill-file / config-section uninstallers. remove_user_skill=True keeps the + # historical `graphify uninstall` behavior: global skill delete plus md/hook + # cleanup at the project dir (#2215). + claude_uninstall(pd, remove_user_skill=True) + codebuddy_uninstall(pd, remove_user_skill=True) + gemini_uninstall(pd, remove_user_skill=True) vscode_uninstall(pd) _cursor_uninstall(pd) _kiro_uninstall(pd) @@ -1798,7 +1814,7 @@ def uninstall_all(project_dir: Path | None = None, purge: bool = False) -> None: print(f"\n {_GRAPHIFY_OUT}/ -> not found (nothing to purge)") print("\nDone. Run 'pip uninstall graphifyy' to remove the package itself.") -def claude_uninstall(project_dir: Path | None = None, *, project: bool = False) -> None: +def claude_uninstall(project_dir: Path | None = None, *, project: bool = False, remove_user_skill: bool | None = None) -> None: """Remove the graphify skill tree (SKILL.md + references/) and the graphify section from CLAUDE.md and its local-only variants, plus the PreToolUse hook. @@ -1809,9 +1825,20 @@ def claude_uninstall(project_dir: Path | None = None, *, project: bool = False) A user may relocate the section/hook into the local-only files Claude Code supports so they are not committed to a shared repo, so uninstall also cleans CLAUDE.local.md, .claude/CLAUDE.local.md and .claude/settings.local.json (#1731). + + Scope rules (#2215): a bare call removes the user-global skill; passing + ``project_dir`` (or ``project=True``) scopes skill removal to that project + and leaves the global tree untouched, unless ``remove_user_skill=True`` + explicitly opts back into the global delete (as ``uninstall_all`` does). """ + explicit_dir = project_dir is not None project_dir = project_dir or Path(".") - _remove_skill_file("claude", project=project, project_dir=project_dir) + if remove_user_skill is None: + remove_user_skill = not project and not explicit_dir + if project or (explicit_dir and not remove_user_skill): + _remove_skill_file("claude", project=True, project_dir=project_dir) + if remove_user_skill: + _remove_skill_file("claude", project=False) md_targets = [ project_dir / "CLAUDE.md", @@ -1914,10 +1941,22 @@ def _uninstall_codebuddy_hook(project_dir: Path) -> None: settings["hooks"]["PreToolUse"] = filtered settings_path.write_text(json.dumps(settings, indent=2), encoding="utf-8") print(f" .codebuddy/settings.json -> PreToolUse hook removed") -def codebuddy_uninstall(project_dir: Path | None = None, *, project: bool = False) -> None: - """Remove the graphify skill tree (SKILL.md + references/) and the CODEBUDDY.md section.""" +def codebuddy_uninstall(project_dir: Path | None = None, *, project: bool = False, remove_user_skill: bool | None = None) -> None: + """Remove the graphify skill tree (SKILL.md + references/) and the CODEBUDDY.md section. + + Scope rules (#2215): a bare call removes the user-global skill; passing + ``project_dir`` (or ``project=True``) scopes skill removal to that project + and leaves the global tree untouched, unless ``remove_user_skill=True`` + explicitly opts back into the global delete (as ``uninstall_all`` does). + """ + explicit_dir = project_dir is not None project_dir = project_dir or Path(".") - _remove_skill_file("codebuddy", project=project, project_dir=project_dir) + if remove_user_skill is None: + remove_user_skill = not project and not explicit_dir + if project or (explicit_dir and not remove_user_skill): + _remove_skill_file("codebuddy", project=True, project_dir=project_dir) + if remove_user_skill: + _remove_skill_file("codebuddy", project=False) target = project_dir / "CODEBUDDY.md" if not target.exists(): diff --git a/tests/test_home_sandbox.py b/tests/test_home_sandbox.py index fc388adcd..2ddb3ffbc 100644 --- a/tests/test_home_sandbox.py +++ b/tests/test_home_sandbox.py @@ -39,11 +39,12 @@ def test_claude_config_dir_escape_hatch_is_cleared(): def test_global_uninstall_is_captured_by_sandbox(tmp_path, tmp_path_factory): - """claude_uninstall deletes the *global* ~/.claude/skills/graphify tree. + """Global skill deletes land inside the sandbox home, never the real one. - Plant that tree inside the sandbox home, run uninstall against an - unrelated project dir, and prove the delete landed in the sandbox - (and therefore not in the developer's real home). + Since #2215, `claude_uninstall(project_dir)` is project-scoped and must NOT + touch the global tree; the global delete now requires either a bare call or + an explicit `remove_user_skill=True`. Both scopes are exercised here so the + sandbox (#2168) is still proven to capture the global delete. """ skill = Path.home() / ".claude" / "skills" / "graphify" / "SKILL.md" skill.parent.mkdir(parents=True) @@ -51,8 +52,13 @@ def test_global_uninstall_is_captured_by_sandbox(tmp_path, tmp_path_factory): project_dir = tmp_path / "some-project" project_dir.mkdir() + + # Project-scoped call: the global tree in the sandbox home must survive. claude_uninstall(project_dir) + assert skill.exists(), "project-scoped uninstall deleted the global skill (#2215 trap)" + # Explicit global opt-in: the delete happens, and lands in the sandbox. + claude_uninstall(project_dir, remove_user_skill=True) assert not skill.exists(), "global skill delete was not captured by the sandbox" # And the sandbox home itself is still inside pytest's tmp area. assert Path.home().is_relative_to(tmp_path_factory.getbasetemp()) diff --git a/tests/test_uninstall_scope.py b/tests/test_uninstall_scope.py new file mode 100644 index 000000000..9102cdfab --- /dev/null +++ b/tests/test_uninstall_scope.py @@ -0,0 +1,113 @@ +"""Scope regression tests for the uninstall API trap (issue #2215). + +`X_uninstall(project_dir)` used to delete the USER-GLOBAL skill tree because +`_platform_skill_destination` honors ``project_dir`` only when ``project=True``. +These tests pin the fixed contract: + +- bare call -> global skill removed (CLI behavior unchanged) +- fn(pd) -> project-scoped, global untouched (trap closed) +- fn(pd, project=True) -> project only +- fn(pd, remove_user_skill=True) -> global removed, project tree untouched +- `graphify uninstall --project` for codebuddy no longer nukes the global skill +""" +from __future__ import annotations + +from pathlib import Path + +import pytest + +from graphify.install import ( + _project_uninstall, + claude_uninstall, + codebuddy_uninstall, + gemini_uninstall, +) + +PLATFORMS = [ + pytest.param(claude_uninstall, "claude", ".claude", id="claude"), + pytest.param(gemini_uninstall, "gemini", ".gemini", id="gemini"), + pytest.param(codebuddy_uninstall, "codebuddy", ".codebuddy", id="codebuddy"), +] + + +def _plant_skill_tree(root: Path, dot_dir: str) -> Path: + """Create //skills/graphify/{SKILL.md, references/x.md, .graphify_version}.""" + skill_dir = root / dot_dir / "skills" / "graphify" + (skill_dir / "references").mkdir(parents=True) + (skill_dir / "SKILL.md").write_text("# graphify skill\n", encoding="utf-8") + (skill_dir / "references" / "x.md").write_text("ref\n", encoding="utf-8") + (skill_dir / ".graphify_version").write_text("0.0.0-test", encoding="utf-8") + return skill_dir + + +@pytest.mark.parametrize("uninstall_fn,platform,dot_dir", PLATFORMS) +def test_project_dir_call_never_touches_global(uninstall_fn, platform, dot_dir, tmp_path): + """fn(project_dir) removes only the project skill tree (#2215 trap closed).""" + global_tree = _plant_skill_tree(Path.home(), dot_dir) + proj_dir = tmp_path / "proj" + project_tree = _plant_skill_tree(proj_dir, dot_dir) + + uninstall_fn(proj_dir) + + assert (global_tree / "SKILL.md").exists(), "global skill deleted by project-scoped uninstall" + assert (global_tree / "references" / "x.md").exists() + assert (global_tree / ".graphify_version").exists() + assert not (project_tree / "SKILL.md").exists() + assert not project_tree.exists() + + +@pytest.mark.parametrize("uninstall_fn,platform,dot_dir", PLATFORMS) +def test_bare_call_still_removes_global(uninstall_fn, platform, dot_dir, tmp_path, monkeypatch): + """fn() with no args keeps the historical CLI behavior: global skill removed.""" + global_tree = _plant_skill_tree(Path.home(), dot_dir) + cwd = tmp_path / "empty-cwd" + cwd.mkdir() + monkeypatch.chdir(cwd) + + uninstall_fn() + + assert not (global_tree / "SKILL.md").exists() + assert not global_tree.exists() + + +@pytest.mark.parametrize("uninstall_fn,platform,dot_dir", PLATFORMS) +def test_remove_user_skill_opt_in_with_project_dir(uninstall_fn, platform, dot_dir, tmp_path): + """fn(pd, remove_user_skill=True) removes the global skill, leaves the project tree.""" + global_tree = _plant_skill_tree(Path.home(), dot_dir) + proj_dir = tmp_path / "proj" + project_tree = _plant_skill_tree(proj_dir, dot_dir) + + uninstall_fn(proj_dir, remove_user_skill=True) + + assert not (global_tree / "SKILL.md").exists() + assert not global_tree.exists() + assert (project_tree / "SKILL.md").exists() + assert (project_tree / "references" / "x.md").exists() + + +@pytest.mark.parametrize("uninstall_fn,platform,dot_dir", PLATFORMS) +def test_project_true_removes_only_project_tree(uninstall_fn, platform, dot_dir, tmp_path): + """fn(pd, project=True) removes only the project skill tree.""" + global_tree = _plant_skill_tree(Path.home(), dot_dir) + proj_dir = tmp_path / "proj" + project_tree = _plant_skill_tree(proj_dir, dot_dir) + + uninstall_fn(proj_dir, project=True) + + assert (global_tree / "SKILL.md").exists() + assert not (project_tree / "SKILL.md").exists() + assert not project_tree.exists() + + +def test_project_uninstall_codebuddy_spares_global(tmp_path): + """`graphify uninstall --project` (codebuddy branch) must not delete ~/.codebuddy (#2215).""" + global_tree = _plant_skill_tree(Path.home(), ".codebuddy") + proj_dir = tmp_path / "proj" + project_tree = _plant_skill_tree(proj_dir, ".codebuddy") + + _project_uninstall("codebuddy", proj_dir) + + assert (global_tree / "SKILL.md").exists(), "CLI --project uninstall deleted the global codebuddy skill" + assert (global_tree / ".graphify_version").exists() + assert not (project_tree / "SKILL.md").exists() + assert not project_tree.exists() From 753a6ac2cc3a12d71daba4c337b8f23490014dd3 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 27 Jul 2026 10:46:47 +0100 Subject: [PATCH 290/292] test(detect): regression for NFC manifest keys on macOS --update (#2221) Adds the regression test PR #2224 shipped without: an NFD-keyed manifest (portable and legacy-absolute) must match an NFC scan so --update is a no-op instead of re-extracting everything. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/test_detect.py | 85 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) diff --git a/tests/test_detect.py b/tests/test_detect.py index e97ff5d4b..15fb570e4 100644 --- a/tests/test_detect.py +++ b/tests/test_detect.py @@ -1921,6 +1921,91 @@ def test_detect_incremental_portable_across_paths(tmp_path): ) +def _rewrite_manifest_keys_nfd(manifest_path): + """Rewrite a saved manifest so every key is in NFD form, simulating a + manifest written by a macOS run where os.walk/getcwd yielded decomposed + paths (#2221). Returns the rewritten key list for sanity checks.""" + import json + p = Path(manifest_path) + raw = json.loads(p.read_text(encoding="utf-8")) + nfd = {unicodedata.normalize("NFD", k): v for k, v in raw.items()} + p.write_text(json.dumps(nfd), encoding="utf-8") + return list(nfd) + + +def test_manifest_nfc_keys_survive_macos_path_forms(tmp_path): + """#2221 (portable/relative-key manifest): a manifest whose keys were + written in NFD (macOS os.walk form) must still match an NFC scan, so + --update reports nothing new/changed/deleted instead of re-extracting + the whole corpus. + + NOTE: the fixture filename must contain a character that actually + decomposes under NFD ("é" in "café" does, as do "ä" and "й"). Plain + Cyrillic like "заметка" has no decomposition, so NFC == NFD and the + test would pass vacuously even without the fix. Keep the byte-wise + inequality assertion below when changing the fixture name. + """ + corpus = tmp_path / "corpus" + (corpus / "docs").mkdir(parents=True) + nfc_name = unicodedata.normalize("NFC", "café.md") + assert nfc_name != unicodedata.normalize("NFD", nfc_name) # must decompose + (corpus / "docs" / nfc_name).write_text("hello unicode\n") + + # Manifest lives OUTSIDE the corpus so it never enters the scan. + manifest_path = str(tmp_path / "out" / "manifest.json") + full = detect(corpus) + assert full["total_files"] == 1 # sanity: the café file was scanned + save_manifest(full["files"], manifest_path, root=corpus) + + # Simulate the macOS-written manifest: keys stored in NFD form. + nfd_keys = _rewrite_manifest_keys_nfd(manifest_path) + # Sanity: the on-disk keys are genuinely decomposed, not silently NFC. + assert any(unicodedata.normalize("NFC", k) != k for k in nfd_keys) + + inc = detect_incremental(corpus, manifest_path) + assert inc["new_total"] == 0, ( + f"NFD manifest keys must match NFC scan paths (#2221); " + f"new_files={inc['new_files']}" + ) + assert all(v == [] for v in inc["new_files"].values()) + assert inc["deleted_files"] == [], ( + f"NFD keys misreported as deletions: {inc['deleted_files']}" + ) + assert inc["excluded_files"] == [] + + +def test_manifest_nfc_keys_legacy_absolute(tmp_path): + """#2221 exact repro: legacy manifest saved WITHOUT root (absolute keys), + then rewritten to NFD. Before the load_manifest/detect_incremental NFC + normalization, every file looked simultaneously new AND deleted on + --update. + + NOTE: as above, the filename must contain an NFD-decomposable character + ("é"); a non-decomposing name would make this test vacuous. + """ + corpus = tmp_path / "corpus" + (corpus / "docs").mkdir(parents=True) + nfc_name = unicodedata.normalize("NFC", "café.md") + assert nfc_name != unicodedata.normalize("NFD", nfc_name) # must decompose + (corpus / "docs" / nfc_name).write_text("hello unicode\n") + + manifest_path = str(tmp_path / "out" / "manifest.json") + full = detect(corpus) + assert full["total_files"] == 1 + # No root= -> legacy absolute-keyed manifest format. + save_manifest(full["files"], manifest_path) + + _rewrite_manifest_keys_nfd(manifest_path) + + inc = detect_incremental(corpus, manifest_path) + assert inc["new_total"] == 0, ( + f"legacy absolute NFD keys must match NFC scan (#2221); " + f"new_files={inc['new_files']}" + ) + assert inc["deleted_files"] == [] + assert inc["excluded_files"] == [] + + def test_save_manifest_in_root_symlink_roundtrips(tmp_path): """In-root symlinks must store under the symlink's own name, not the resolved target. Resolving the key when relativizing pointed the stored From bbc4e5167c0ac9e41ff8d42c963d59131f14571b Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 27 Jul 2026 10:47:30 +0100 Subject: [PATCH 291/292] chore: bump to 0.9.28; changelog for #2211/#2213/#2210/#2212/#2215/#2221/#2181/#2218 Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 10 ++++++++++ pyproject.toml | 2 +- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e490e6d73..e7ca8c75b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,16 @@ Full release notes with details on each version: [GitHub Releases](https://github.com/safishamsi/graphify/releases) +## 0.9.28 (unreleased) + +- Fix: incremental extraction no longer drops cross-file edges whose target file wasn't in the batch (#2211, #2213). Python relative imports and markdown reference links emitted absolute-path-derived target ids without the `target_file` stamp the incremental canonicalization needs, so a re-extracted file's imports/references dangled or vanished; both now stamp the resolved target and canonicalize to the root-relative node. +- Fix: incremental extraction no longer prunes alive files as "deleted" (#2210). The stale-source check compared graph paths to the scan with a raw string test (no Unicode NFC normalization) and pruned any non-match without checking whether the file still exists, so macOS NFD paths and legacy basename spellings lost their nodes. It now compares NFC on both sides and is fail-closed: a source missing from the scan is pruned only when its exclusion is provable, otherwise kept with a warning. +- Fix: `graphify benchmark`, the graph merge-driver, and the call-flow HTML export no longer crash or silently fail on a `--no-cluster` `graph.json` (#2212). Those graphs store edges under `edges` rather than `links`; a shared loader now normalizes both. +- Fix: `claude`/`gemini`/`codebuddy` uninstall no longer delete the user-global skill when called with a `project_dir` (#2215). A passed `project_dir` now scopes the removal; this also fixes a CLI bug where `graphify uninstall --project` deleted the global codebuddy skill. +- Fix: `--update` on macOS no longer re-extracts everything when the corpus path or a filename contains non-ASCII characters (#2221, thanks @SyedFahad7). Manifest keys are NFC-normalized so NFD and NFC path forms match. +- Fix: Swift computed and observed properties (`var body: some View { ... }`, `get`/`set`, `willSet`/`didSet`) now emit graph nodes, so SwiftUI views are no longer erased (#2181, thanks @ozdemirsarman). +- Fix: incremental rebuilds no longer reuse stale community labels, and a graph that outgrows the visualization cap now keeps an aggregated view instead of deleting `graph.html` (#2218, thanks @bobspryn). + ## 0.9.27 (2026-07-26) - Fix: `claude`/`gemini`/`codex`/`codebuddy install` no longer overwrite an existing settings/hooks file they cannot parse (#2167). The installers fell back to an empty config on any JSON parse error and then rewrote the whole file, destroying the user's settings (the likely trigger is a UTF-8 BOM, the same class as #2163). They now read `utf-8-sig`, refuse to modify a file that is not a JSON object (naming the path) instead of clobbering it, and back up to `.graphify-bak` before any modifying write. diff --git a/pyproject.toml b/pyproject.toml index f7993fc69..68dfbe048 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "graphifyy" -version = "0.9.27" +version = "0.9.28" description = "AI coding assistant skill (Claude Code, CodeBuddy, Codex, OpenCode, Kilo Code, Cursor, Gemini CLI, Aider, OpenClaw, Factory Droid, Trae, Hermes, Kiro, Pi, Devin CLI, Google Antigravity) - turn any folder of code, docs, papers, images, or videos into a queryable knowledge graph" readme = "README.md" license = "Apache-2.0" From 16442303e197650255f6dbf77c6448500bd91e51 Mon Sep 17 00:00:00 2001 From: safishamsi Date: Mon, 27 Jul 2026 10:57:57 +0100 Subject: [PATCH 292/292] chore(release): 0.9.28 Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 2 +- uv.lock | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e7ca8c75b..0ce3fdb7e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ Full release notes with details on each version: [GitHub Releases](https://github.com/safishamsi/graphify/releases) -## 0.9.28 (unreleased) +## 0.9.28 (2026-07-27) - Fix: incremental extraction no longer drops cross-file edges whose target file wasn't in the batch (#2211, #2213). Python relative imports and markdown reference links emitted absolute-path-derived target ids without the `target_file` stamp the incremental canonicalization needs, so a re-extracted file's imports/references dangled or vanished; both now stamp the resolved target and canonicalize to the root-relative node. - Fix: incremental extraction no longer prunes alive files as "deleted" (#2210). The stale-source check compared graph paths to the scan with a raw string test (no Unicode NFC normalization) and pruned any non-match without checking whether the file still exists, so macOS NFD paths and legacy basename spellings lost their nodes. It now compares NFC on both sides and is fail-closed: a source missing from the scan is pruned only when its exclusion is provable, otherwise kept with a warning. diff --git a/uv.lock b/uv.lock index 6ef863263..671c304f9 100644 --- a/uv.lock +++ b/uv.lock @@ -1090,7 +1090,7 @@ wheels = [ [[package]] name = "graphifyy" -version = "0.9.27" +version = "0.9.28" source = { editable = "." } dependencies = [ { name = "networkx", version = "3.4.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },