diff --git a/.gitignore b/.gitignore index d08b165..fa43513 100644 --- a/.gitignore +++ b/.gitignore @@ -1,24 +1,21 @@ -# macOS -.DS_Store -# Node -node_modules/ - -# Dev tool / AI assistant state (track committed skills, ignore local agent state) -.claude/* !.claude/skills/ !.claude/skills/** +.claude/* .codegraph/ -.opencode/ .codex/ +.DS_Store .gemini/ - -# Local caches / build output .gocache/ .gomodcache/ +.opencode/ .tmp/ -dist/ -build/ - -# Logs *.log +# Dev tool / AI assistant state (track committed skills, ignore local agent state) +# Local caches / build output +# Logs +# macOS +# Node +build/ +dist/ +node_modules/ diff --git a/README.md b/README.md index 2be6af1..45d4327 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # hawk-graph — Unified Architecture Platform -Single platform for the hawk-eco ecosystem (14 repos). Architecture trees, call flows, live queries — all from one place. +Single platform for the hawk-eco ecosystem (15 source repos). Architecture trees, call flows, live queries — all from one place. ## Quick Start @@ -50,4 +50,4 @@ hawk-graph/ - Node.js 22+ - `graphify` — `uv tool install graphify` - `gitnexus` — `npm install -g gitnexus` -- 14 repos in parent directory (`../`) +- 15 source repos in parent directory (`../`), plus this visualization repo diff --git a/global/ecosystem-agentic-graph-architecture.md b/global/ecosystem-agentic-graph-architecture.md new file mode 100644 index 0000000..7e95c05 --- /dev/null +++ b/global/ecosystem-agentic-graph-architecture.md @@ -0,0 +1,432 @@ +# Ecosystem Agentic Graph Architecture + +## Status + +Implementation in progress. This document distinguishes shipped slices from +the target architecture and preserves the ecosystem's local-first execution +model and repository boundaries. + +Implemented in the current working branches: + +- Graph Contract v1 in `hawk-core-contracts`, mirrored in Go and protobuf; +- a read-only Trace execution-graph projection for sessions, checkpoints, + `produced` relationships, and lifecycle events; +- `trace graph export`, also composed into Hawk as + `hawk trace graph export`; +- a Hawk execution-graph projection for sessions, task requests, structured + tasks, runtime tasks, tool calls, policy verdicts, verification reports, and + explicit Trace checkpoint references; +- `hawk graph export`, with metadata-only payloads and strict dangling-edge + rejection; +- bounded validation of Hawk blocking-task dependencies with deterministic + topological waves and fail-closed ready-work selection; +- append-only, privacy-safe capture of Hawk tool permission outcomes, enabled + approval-gate outcomes, and built-in plan-verification summaries; +- a Yaad `yaad.graph/v1` portable knowledge projection and automatic Hawk + capture of retrieved Yaad memory subgraphs and Hawk code-index chunks; +- an Inspect `inspect.graph/v1` bounded quality projection and opt-in Hawk + capture/composition of report and finding subgraphs; +- a Sight `sight.graph/v1` bounded quality projection and opt-in Hawk + capture/composition of review and finding subgraphs; +- neutral, privacy-safe verification summaries emitted alongside observed + Inspect and Sight quality graphs for `validated_by` composition; +- a Tok `tok.graph/v1` mixed operations/policy/quality projection and automatic + Hawk capture of persisted-session context-compression statistics; +- an Eyrie `eyrie.graph/v1` operations projection and automatic Hawk capture + of model route and normalized usage for persisted-session turns; +- a Hawk Cloud-owned, project-scoped D1 graph ledger with authenticated + ingestion, deterministic privacy normalization, transactional idempotency, + stable-ID collision rejection, and bounded self-contained reads; +- a GrayCode Core BFF projection that validates cloud responses, removes + provenance and non-display metadata, and serves a read-only dashboard + node-link view; +- portable data-only Graph v1 models in the Go, Python, and TypeScript SDKs, + including topology validation and runtime boundary parsing where supported; +- an authenticated Hawk daemon session-graph endpoint, consumed through + matching Go, Python sync/async, and TypeScript SDK methods; +- an opt-in Hawk task-graph execution path that consumes validated scheduling + waves, dispatches bounded parallel mission workers per wave, and persists + deterministic wave joins while leaving downstream tasks pending after an + upstream failure; +- a reusable staged-mission graph execution option (`WithExecutionWaves`) so + library callers can combine dependency waves with PRD, verification, and + fix phases; +- a portable Hawk mission graph artifact written beside `mission.json`, with + mission, feature, and wave-join facts in the shared `hawk.graph/v1` envelope; +- an additive MCPKit helper for exposing producer-owned graph documents as + typed, read-only MCP resources, with explicit whole-surface auth guidance; +- a deterministic Community Skills registry projection linking the registry, + categories, skills, and cross-cutting tags without changing registry truth; +- an explicit Hawk-to-Cloud graph sync command with deterministic privacy + preparation, local preflight limits, device authentication, and fail-open + separation from local execution; +- first-class export and explicit Cloud synchronization of the portable Hawk + mission graph artifact, with local schema/topology validation before upload; +- schema validation, stable IDs, repository scope, provenance, correlation, + idempotency keys, deterministic ordering, filtering, limits, and tests. + +Not yet implemented: + +- Full graph-fact export streaming and user-requested deletion workflows + remain planned follow-on capabilities; current graph reads are bounded, + cursor-pageable, project-scoped, and governed by admin retention purge and + project-delete cleanup. + +## Architectural model + +Every agentic operation is designed through five layers: + +```text +prompt -> context -> harness -> loop -> graph +``` + +The graph layer contains six typed, temporal views over a compatible event and +provenance vocabulary. These views remain owned by the repositories that +produce them; there is no central operational graph database in `hawk-graph`: + +```text +system | knowledge | execution | policy | quality | operations +``` + +`hawk-core-contracts` owns the portable vocabulary. Individual repositories +own their facts, storage, and runtime behavior; they publish only contracts +and events that other repositories genuinely need. + +## Current versus proposed graph techniques + +The ecosystem uses several graph methods, but each has a different authority +and maturity. “Graph everywhere” means every important relationship can be +projected and inspected; it does not mean every repository needs a graph +database or every task needs multi-agent fan-out. + +| Technique | Best use | Current state | Proposed next state | +|---|---|---|---| +| Typed property graph | Portable entities, relationships, and metadata | Graph v1 has six node views, six edge types, and string-bounded attributes | Additive vocabulary only when two or more producers need the same concept | +| Temporal/event graph | Lifecycle, replay, audit, and idempotency | Created/updated/transitioned/observed/deleted events; Trace checkpoints; Cloud append ledger | Retention tiers and explicit supersession/version links | +| Execution DAG | Task dependencies and tool provenance | Hawk projects task/runtime/tool relationships, validates blocking dependencies, can execute validated waves through `hawk mission --from-tasks`, and persists mission-scoped graph artifacts with deterministic joins | Expand only where genuine dependency/parallel work benefits from the existing bounded mission controls | +| Knowledge graph / GraphRAG | Evidence-backed context and memory | Yaad memory subgraphs and Hawk code chunks are retrieved, bounded, and journaled | Retrieval evaluation, freshness scoring, and supersession-aware ranking | +| Heterogeneous multilayer graph | Join execution, knowledge, policy, quality, and operations | Hawk composes Yaad, Inspect, Sight, Tok, Eyrie, and Trace facts through shared references | Cross-view queries with explicit authorization and cost budgets | +| Policy graph | Explain what governed an action | Hawk journals permission and approval outcomes as `governed_by` facts | Policy dependency/conflict analysis without moving enforcement out of current owners | +| Quality/security graph | Findings, evidence, remediation, and regressions | Inspect and Sight publish bounded report/finding projections | Finding lifecycle and regression links after stable identity rules are agreed | +| Operations graph | Routing, usage, budgets, compression, and health | Eyrie route/usage plus Tok compression, redaction, usage, and budget decisions are captured | Add cost/latency SLO queries and durable aggregate windows | +| Capability graph | Skills, tags, tools, and compatibility | Community Skills projects registry/category/skill/tag topology; MCPKit exposes read-only resources | Signed capability and required-tool/version relationships | +| Provenance graph | Attribution and immutable evidence | Every portable fact carries producer/version/source/evidence fields | Evidence retention policies and verifiable artifact digests | +| Cloud project graph | Authorized durable coordination and dashboards | Device-auth sync, privacy normalization, D1 ledger, cursor-pageable BFF, node-link UI, graph retention purge, export summary, and project-delete cleanup | Full fact export streaming and aggregate analytics | +| Multi-agent orchestration graph | Parallel specialists with join/approval gates | Used for the opt-in task-graph mission path, not as a universal scheduler | Keep expansion behind typed state, bounded fan-out, cancellation, budgets, and deterministic joins | +| Causal graph | Explain why a later event occurred | Correlation, causation, and idempotency fields are stored, privacy-normalized, and queryable through bounded Cloud/BFF filters | Causal analysis and lineage expansion after producers emit reliable causation links | +| Code/dependency graph | Impact analysis and architecture navigation | Hawk code intelligence and visualization artifacts are read-only inputs | Freshness metadata and automated drift checks; never an execution authority | + +## Shared contract requirements + +Every externally visible graph node, edge, and event must have: + +- a stable, opaque ID and a type; +- tenant/project/repository scope; +- creation and effective timestamps; +- source, evidence, and producer version; +- access classification and policy decision where relevant; +- idempotency key for side-effecting execution nodes; +- bounded payloads, with large artifacts represented by immutable references. + +## Repository specifications + +### `hawk-core-contracts` — cross-repository vocabulary + +**Owns:** common graph DTOs, event envelopes, provenance references, execution +states, policy decisions, approvals, and checkpoint metadata. + +**Publishes:** additive, versioned Go and protobuf contracts only. It must not +own a graph database, orchestrator, CLI, or product behavior. + +**Consumes:** nothing outside the Go standard library. + +**Robustness:** additive schema evolution, strict type parsing, contract tests, +and cross-language generated schema validation. + +### `hawk` — local execution graph runtime + +**Owns:** the task execution graph for local coding work: task, context plan, +agent loop, worktree, tool invocation, validation, decision, and completion. + +**Publishes:** lifecycle events, requested/approved capabilities, evidence +references, result summaries, and checkpoint references. + +**Consumes:** scoped code/knowledge context, policy verdicts, provider results, +review/inspection findings, and shared contracts. + +**Robustness:** bounded retries, explicit cancellation, worktree isolation, +idempotent mutating tools, budget limits, and a mandatory verify gate before +integration. + +### `eyrie` — provider and model operations graph + +**Owns:** provider, model, credential scope, request, fallback, cache, health, +latency, token use, and cost relationships. + +**Publishes:** provider health, route choice, retry/fallback reason, usage, and +redacted request provenance. + +**Consumes:** model-selection policy and request budgets from Hawk/Cloud. + +**Robustness:** provider isolation, typed errors, circuit breaking, bounded +retries, cache correctness, and no credential material in graph payloads. + +### `tok` — context quality and safety signals + +**Owns:** token budgets, compaction snapshots, redaction findings, context +segments, compression lineage, and rate-limit signals. + +**Publishes:** safe context references, token/cost accounting, redaction +outcomes, and compaction provenance. + +**Consumes:** phase budgets and content-classification policy. + +**Robustness:** deterministic redaction before persistence/export, immutable +anchors during compaction, and budget exhaustion as a controlled state. + +**Implemented runtime:** Hawk feeds Tok from the central deduplicated provider +usage seam. Tok publishes a usage operations node and a dependent budget-policy +node for persisted sessions, while its decision participates in Hawk's +pre-turn guard. A thread-safe limit snapshot API keeps runtime configuration +and graph observations consistent. + +### `yaad` — temporal knowledge graph + +**Owns:** memories, entities, conventions, decisions, source evidence, +supersession, communities, and retention/decay state. + +**Publishes:** scoped, ranked, evidence-backed knowledge subgraphs and memory +write events. + +**Consumes:** validated task outcomes, repository scope, privacy policy, and +optional code/entity links. + +**Robustness:** source attribution, confidence, temporal supersession, privacy +filtering before ingestion, deterministic IDs, and a review path for disputed +memory. + +### `trace` — provenance and replay graph + +**Owns:** sessions, agent turns, checkpoints, Git commits, branch/fork links, +artifact references, and replay lineage. + +**Publishes:** immutable checkpoint and provenance events. + +**Consumes:** execution lifecycle events and repository/Git metadata. + +**Robustness:** append-only checkpoint records, secret redaction, resumable +session state, fork-safe identity, and retention policies. + +### `sight` — code quality graph + +**Owns:** diffs, findings, comments, review runs, evidence locations, +remediation status, and regression links. + +**Publishes:** typed review findings tied to commit, file, symbol, and execution +node references. + +**Consumes:** diffs, code graph context, review policy, and optional prior +finding history. + +**Robustness:** deterministic finding identity, severity/confidence contracts, +deduplication, and evidence required for blocking verdicts. + +### `inspect` — security and external-surface graph + +**Owns:** targets, domains, endpoints, TLS/cookie/header observations, +vulnerabilities, attack-surface findings, and remediation state. + +**Publishes:** timestamped, evidence-backed security findings and target +relationships. + +**Consumes:** authorized target scope, policy, and application/deployment +metadata. + +**Robustness:** read-only default, explicit target authorization, scan limits, +reproducible evidence, and clear distinction between observation and inference. + +### `hawk-mcpkit` — graph-aware capability gateway + +**Owns:** shared MCP transport and handler construction patterns. + +**Publishes:** standardized MCP tool metadata, capability declarations, schemas, +and graph query/result envelopes. + +**Consumes:** tool contracts, permission verdicts, and individual engine +handlers. + +**Robustness:** strict schemas, capability-scoped tool exposure, redacted error +messages, and correlation IDs propagated to tracing. + +**Implemented foundation:** `AddGraphResource` registers producer-supplied JSON +as a read-only `application/vnd.hawk.graph+json` MCP resource. MCPKit remains +contract-agnostic and has no ecosystem dependency; producers still own schema +validation, redaction, authorization, and size bounds. Sensitive graph +resources must use the whole-HTTP-surface token gate because the tool-only +bearer mode does not protect resources. + +### `hawk-graph` — ecosystem architecture visualization + +**Owns:** the static dashboard, generated architecture trees/call-flow pages, +and documentation that visually presents all repositories. + +**Publishes:** visualization pages and non-authoritative navigation links only. + +**Consumes:** generated, read-only analysis artifacts from each repository. +It does not consume runtime events, perform agent retrieval, or make policy or +execution decisions. + +**Robustness:** rebuildable generated pages, source-version labels, stale-index +visibility, and no authority to mutate or serve as a source of truth for any +other repository. + +### `hawk-cloud` — durable cloud coordination graph + +**Owns:** organization, tenant, project, device, policy, approval, audit, usage, +deployment, and cloud execution state. + +**Publishes:** authoritative tenancy/policy/audit/usage events and approval +decisions. + +**Consumes:** user-approved synchronization from local Hawk, GrayCode identity, +and provider/deployment facts. + +**Robustness:** durable workflow instances for long-running cloud operations, +strong per-entity coordination, idempotent ledger writes, event archival, and +fail-open local execution where cloud sync is optional. + +**Implemented foundation:** Hawk Cloud independently validates `*.graph/v1` +payloads without importing another ecosystem repository. Device-authenticated +syncs are capped at 1 MiB and 900 facts, scoped to one project/session, and +stored transactionally in D1. Local identifiers and provenance pointers are +hashed before persistence. Authorized reads return a bounded topology whose +edges and events cannot dangle outside the returned node set. + +Hawk's explicit `cloud graph sync` command builds a stable completed-session +snapshot, hashes sensitive attribute values before transport, enforces Cloud's +fact and body limits locally, and derives a deterministic sync ID. The adapter +returns errors to this user-invoked command but is not part of local execution, +so Cloud remains optional and fail-open. + +### `graycode-core` — product, identity, and dashboard graph + +**Owns:** web UX, authenticated API surface, account identity, dashboard views, +and product-facing integrations. + +**Publishes:** authenticated requests, user/organization/project references, +and UI-safe graph projections. + +**Consumes:** Hawk Cloud control-plane data and read-only ecosystem graph views. + +**Robustness:** tenant isolation, authorization on every graph query, secure +headers/CORS, D1 migration discipline, and no client-side trust in policy +verdicts. + +**Implemented foundation:** the authenticated BFF validates Hawk Cloud's +response and emits a smaller UI contract containing counts, display-safe node +details, relationships, and lifecycle events. The dashboard visualizes that +projection but owns no graph facts, policy decisions, or runtime behavior. + +### `hawk-community-skills` — skill and capability graph + +**Owns:** skills, manifests, source/provenance, tool requirements, compatibility, +audit results, and trust status. + +**Publishes:** installable capability metadata and audit evidence. + +**Consumes:** policy/risk classification, tool contract versions, and registry +validation results. + +**Robustness:** signed/versioned manifests where available, quarantine for +untrusted skills, dependency scanning, and explicit capability disclosure. + +**Implemented foundation:** `tools/skill_graph.py` projects the validated +registry into a `community-skills.graph/v1` heterogeneous graph. The registry +contains categories and tags, categories contain skills, and skills reference +their tags. Stable opaque IDs, a canonical input digest, one idempotent snapshot +event, optional bounds, and SDK compatibility tests make it reproducible while +leaving `registry.json` authoritative and generated by its existing workflow. + +### `hawk-sdk-go` — Go graph/runtime client + +**Owns:** idiomatic Go bindings for Hawk task, execution, trace, and graph APIs. + +**Publishes:** no source-of-truth graph facts; it is a typed client. + +**Consumes:** versioned Hawk and shared contract APIs. + +**Robustness:** context-aware cancellation, typed errors, retries only for +idempotent operations, SSE reconnect safety, and contract parity tests. + +### `hawk-sdk-python` — Python graph/runtime client + +**Owns:** Python bindings and models for the same API surface. + +**Publishes:** no source-of-truth graph facts. + +**Consumes:** versioned Hawk and shared contract APIs. + +**Robustness:** async cancellation, Pydantic validation, safe retries, and +contract compatibility tests. + +### `hawk-sdk-typescript` — TypeScript graph/runtime client + +**Owns:** TypeScript bindings, streaming clients, and typed tool/agent access. + +**Publishes:** no source-of-truth graph facts. + +**Consumes:** versioned Hawk and shared contract APIs. + +**Robustness:** runtime schema validation at API boundaries, abort support, +streaming recovery, and compatibility tests. + +## Cross-repository execution flow + +```text +user task + -> Hawk creates execution graph + -> Yaad + Hawk's local code intelligence provide scoped context subgraph + -> Eyrie routes model request within budget + -> Hawk invokes approved tools in an isolated worktree + -> Tok measures/redacts/compacts context + -> Sight + Inspect validate in parallel where applicable + -> Trace records checkpoints and evidence + -> Hawk resolves outcome or requests approval + -> Yaad learns validated knowledge; Hawk Cloud receives approved audit/usage sync +``` + +## Rollout order + +1. Add Graph Contract v1 and compatibility tests in `hawk-core-contracts`. + **Implemented.** +2. Implement a single Hawk execution-graph vertical slice with Trace checkpoints. + **Implemented as read-only Hawk and Trace projections plus Hawk's append-only + runtime observation journal. Trace's exact-match correlation lookup and + Hawk-side automatic session/checkpoint reference composition are + implemented with bounded, fail-open validation.** +3. Connect Yaad and Hawk's local code intelligence as read-only context sources. + **Implemented for retrieved Yaad memory subgraphs and Yaad-backed Hawk code + index chunks, journaled and linked to Hawk sessions.** +4. Attach Sight, Inspect, and Tok as validation, safety, and budget nodes. + **Inspect and Sight quality projections plus Hawk composition implemented; + Tok projection plus context-compression and response-redaction capture + implemented. Tok usage/budget capture and next-turn enforcement now run from + Hawk's deduplicated provider-accounting seam.** +5. Project approved cloud events into Hawk Cloud and GrayCode Core. + **Cloud ingestion/ledger/query and GrayCode's UI-safe BFF/dashboard are + implemented, including Hawk's explicit opt-in upload client.** +6. Regenerate `hawk-graph` visualizations from approved repository artifacts. +7. Expand SDKs and MCP tools only after the API contracts stabilize. + **Portable Graph v1 SDK models and the generic read-only MCP resource helper + are implemented; HTTP client methods remain intentionally deferred until + Hawk exposes a stable graph endpoint.** + +## Acceptance criteria + +- Every side effect is attributable, policy-checked, and idempotent. +- Failed execution resumes without repeating completed side effects. +- Context is scoped, redacted, evidence-backed, and budgeted. +- Graph queries cannot cross tenant or repository authorization boundaries. +- Every graph topology has deterministic tests and telemetry. +- A simple task uses a simple loop; graph fan-out is reserved for real parallel or + branching dependencies. diff --git a/global/global-arch.html b/global/global-arch.html index 454912d..6b43e13 100644 --- a/global/global-arch.html +++ b/global/global-arch.html @@ -85,7 +85,7 @@