Skip to content

docs(#5468): hybrid retrieval rollout plan — module readiness map with staged gates, dependency map, and risk analysis#5552

Open
makr-code with Copilot wants to merge 17 commits into
developfrom
copilot/map-hybrid-retrieval-rollout
Open

docs(#5468): hybrid retrieval rollout plan — module readiness map with staged gates, dependency map, and risk analysis#5552
makr-code with Copilot wants to merge 17 commits into
developfrom
copilot/map-hybrid-retrieval-rollout

Conversation

Copilot AI commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Adds the concrete hybrid retrieval rollout plan (issue #5468) translating TARGET_ARCHITECTURE.md into a gap-gated staged delivery plan across index, gpu, acceleration, graph, query, sharding, and distributed_tensor.

New: ai_working/HYBRID_RETRIEVAL_ROLLOUT_PLAN.md

Central planning artifact with all four required deliverables:

  • Staged rollout table — Phases A–D with timeline, active modules, GPU/distributed/advisory-tensor status, and entry gate per phase
  • Dependency map — explicit tree: what each Phase B/C/D component depends on from earlier phases, per module
  • Risk analysis by module — per-module risk table (severity, mitigation, recommended fix order) for all 7 modules
  • Recommended implementation order — Q3 2026 weeks 1–10 (Phases A+B), Q4 2026 (Phase C), 2027+ (Phase D)
  • Tensor-graph extension track — dedicated Phases A–D for dynamic tensor-update infrastructure: delta log → partial refit → shard summary coordination → optional GPU
  • Observability requirements — Prometheus metrics required before each phase enable
  • CTest/benchmark gates — named test and benchmark targets mandatory before production per phase

Key invariants enforced throughout:

  • Graph Truth Layer is CPU-first and exact in all phases; Category C kernels (policy, provenance, transactions) have no GPU path ever
  • Advisory artifacts (tensor summaries, ANN candidates) never replace graph-verified results
  • Multi-shard disabled until Phase C thread-safety gates pass (sharding at 35% 🔴)
  • GPU paths are advisory-only and always require CPU fallback

Module ROADMAP updates (8 files)

Each module ROADMAP gains:

  • Readiness percentage from gap analysis (e.g., gpu 35% 🔴, graph 60% 🟢)
  • ### Hybrid Retrieval Rollout Gates (issue #5468) section with concrete checkbox tasks, targets, and named ctest/benchmark gate identifiers
  • Cross-reference link to ai_working/HYBRID_RETRIEVAL_ROLLOUT_PLAN.md §7 for full risk detail
Module Readiness Phase Entry
graph 60% 🟢 Phase A ✅ ready
query 55% 🟡 Phase A with error fixes
index 40% 🟡 Phase B Q3 2026
acceleration 45% 🟡 Phase B Q3 2026 (Category A only)
sharding 35% 🔴 Phase C Q4 2026
gpu 35% 🔴 Phase D 2027+
distributed_tensor 30% 🔴 Phase A schema, Phase C coordination

Description

Linked Issues

Related to #5467 (planner design CPU/GPU boundaries), #5469 (kernel classification framework)

Type of Change

  • Bug fix (non-breaking)
  • New feature (non-breaking)
  • Refactoring (non-breaking)
  • Documentation
  • Breaking change (requires MAJOR version bump — see VERSIONING.md)
  • Security fix
  • Other:

Breaking Change Checklist

  • MAJOR version bump planned in VERSION and CMakeLists.txt
  • Migration guide added in docs/migration/
  • Announcement prepared for GitHub Discussions (≥ 2 weeks before release)
  • CHANGELOG ### Removed / ### Changed section updated

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • Benchmarks run (if performance-sensitive change)

Security Tiering Impact (Required for Runtime Changes)

  • Impacted tier(s):

    • T0 Trusted Core
    • T1 Security & Platform Services
    • T2 Data Plane Engines
    • T3 Interface & Protocol Edge
    • T4 Managed Extension Runtime
    • T5 Plugin Boundary
    • N/A (docs-only / non-runtime)

  • Trust-boundary crossings documented in PR description (example: T3 -> T2, T5 -> T4 brokered call)

  • Boundary controls validated for affected T3/T4/T5 paths (AuthN/AuthZ, validation, rate limits, audit)

  • Boundary-focused tests added/updated or explicit N/A rationale provided

  • If trust level/privilege increased, security maintainer approval is attached

📚 Research & Knowledge (wenn applicable)

  • Diese PR basiert auf wissenschaftlichen Paper(s) oder Best Practices?
    • Falls JA: Research-Dateien in /docs/research/ angelegt?
    • Falls JA: Im Modul-README unter "Wissenschaftliche Grundlagen" verlinkt?
    • Falls JA: In /docs/research/implementation_influence/ eingetragen?

Relevante Quellen:

  • Paper:
  • Best Practice:
  • Architecture Decision:

AI-Generated Code (KI-generierter Code)

  • Symbol-Referenzen mit GetSymbolReferences_CppTools geprüft (siehe .github/instructions/cpp-language-service-tools.instructions.md)
  • Keine rohen Pointer und kein new/delete ohne explizites Review eingeführt
  • RAII und Exception-Safety für neue/angepasste Pfade geprüft
  • Keine unnötig komplexen KI-Abstraktionen eingeführt
  • Performance-Metriken geprüft, falls Hotpath betroffen

AI Review Workflow (Required for AI-assisted PRs)

  • Findings-first review performed with .github/prompts/pr-diff-findings-review.prompt.md
  • Security hardening review performed for security-sensitive/runtime changes with .github/prompts/security-hardening-review.prompt.md (or N/A documented)
  • API impact review performed for API/contract changes with .github/prompts/api-change-impact-review.prompt.md (or N/A documented)
  • All Critical/High findings are resolved or explicitly accepted with rationale in PR description
  • Residual risks and follow-up actions documented in PR description
  • Severity policy applied according to .github/copilot/REVIEW_SEVERITY_POLICY.md

High-Finding Exception Record (only if High is accepted)

  • High-finding exception claimed in this PR

  • Finding reference:

  • Maintainer approver:

  • Mitigation in current release:

  • Target fix milestone:

  • Tracking issue:

  • Validation evidence:

Release Readiness Gate (Required for release-scoped changes)

  • Release readiness reviewed with .github/prompts/release-readiness-check.prompt.md for branch transition scope
  • Branch governance validated against BRANCHING_STRATEGY.md and RELEASE_STRATEGY.md
  • Versioning/changelog impact validated against VERSIONING.md and CHANGELOG.md

Checklist

  • Code follows project style guidelines (clang-format / clang-tidy)
  • Self-review completed
  • Documentation updated (if needed)
  • CHANGELOG.md updated under [Unreleased]
  • No new warnings introduced
  • Security-sensitive paths reviewed by security maintainer (if applicable)

Scanner and IntelliSense Gates

  • IntelliSense/Compiler: no new errors in changed files
  • clang-tidy/cppcheck: no new high-risk findings in changed files
  • Gap Scanner: no new critical findings in categories security, input_validation, query_correctness, distributed_consistency, concurrency, memory
  • Gap Scanner: no new high findings in the same categories (or explicitly approved)
  • Gap Scanner delta report attached (baseline vs current), not only absolute totals
  • New unknown scanner findings triaged (fixed, re-categorized, or justified)

Copilot AI linked an issue Jul 6, 2026 that may be closed by this pull request
Map hybrid retrieval rollout plan to module readiness across index, gpu, acceleration, graph, query, and sharding #5468
Open
docs(#5468): add hybrid retrieval rollout plan and update module ROAD…
14b9102 
…MAPs

- Add ai_working/HYBRID_RETRIEVAL_ROLLOUT_PLAN.md: staged rollout plan
  (Phases A-D), dependency map, risk analysis by module, implementation
  order, observability requirements, and ctest/benchmark gates
- Update src/retrieval/ROADMAP.md: rollout phase mapping, ctest gates
- Update src/distributed_tensor/ROADMAP.md: tensor-update Phases A-D,
  delta log, partial refit, shard summary coordination, freshness gates
- Update src/index/ROADMAP.md: Phase B rollout gates (buffer lifecycle,
  concurrency, ANN result validation)
- Update src/gpu/ROADMAP.md: Phase C/D rollout gates (CUDA error handling,
  resource exhaustion, break-even benchmarks)
- Update src/acceleration/ROADMAP.md: Category A/B kernel rollout gates
  with parity test requirements
- Update src/graph/ROADMAP.md: Phase A readiness, CPU-first invariant
  documentation, Category C permanent exclusion from GPU
- Update src/query/ROADMAP.md: Phase A/B rollout gates (error hardening,
  thread-safety, hybrid planner)
- Update src/sharding/ROADMAP.md: Phase C rollout gates (thread-safety,
  lock ordering, consensus)

Closes #5468
Copilot AI changed the title [WIP] Map hybrid retrieval rollout plan to module readiness docs(#5468): hybrid retrieval rollout plan — module readiness map with staged gates, dependency map, and risk analysis Jul 6, 2026
Copilot AI requested a review from makr-code July 6, 2026 06:02
Copilot AI added 2 commits July 6, 2026 10:26
fix(query/phase-a): CRITICAL gap fixes — overflow guard, timed-mutex,…
bbc33e9 
… safe iteration

- tensor_aware_query_optimizer.cpp: cap d/n/r inputs at 1e6 and add
  std::isfinite() guard on estimateTTCost() to prevent inf/NaN from
  propagating into the cost model (3 CRITICAL multiplication_overflow gaps)
- query_canceller.{h,cpp}: replace std::mutex with std::timed_mutex and
  use try_lock_for(200ms) in all three registry functions to eliminate
  the blocking_no_timeout CRITICAL gap; THEMIS_WARN on timeout
- query_rewrite_rule.cpp: replace range-for-with-move with index-based
  loop in PredicatePushdownRule::apply() to satisfy iterator_invalidation
  CRITICAL gap; also removes unused loop counter variable
- tests/query/test_query_phase_a_gap_fixes.cpp: 12 new regression tests
  (TAOPT-01..05, QCANCEL-01..03, REWRT-01..04) verifying all three fixes
feat(distributed_tensor/phase-a): ArtifactManifest + ManifestStore wi…
31c8472 
…th advisory-only policy and Prometheus metrics

- include/artifact_manifest.h: ArtifactManifest schema — advisory-only
  invariant documented, ArtifactKind enum (ADVISORY_SUMMARY/DELTA_LOG/SHARD_SUMMARY),
  ArtifactIntegrity CRC token, isFresh() freshness gate, freshnessAgeSeconds()
- include/manifest_store.h: ManifestStore thread-safe registry API —
  store (monotonic version), evict, evictStale, get, list, size,
  refreshFreshnessMetrics; advisory-only contract stated in class doc
- src/manifest_store.cpp: in-memory implementation with:
  - tensor_delta_log_entries_total counter on store()
  - tensor_freshness_age_seconds gauge via refreshFreshnessMetrics()
  - monotonic version enforcement, stale eviction
- tests/epic3_distributed_tensor/test_manifest_store_phase_a.cpp: 12 tests
  (MS-01..12) covering store, get, list, evict, freshness gate, Prometheus
  metrics, and concurrent access
- ROADMAP.md: mark Phase A manifest + advisory policy + metrics as [x]
- CMakeLists.txt: wire test_manifest_store_phase_a as module-focused test
@makr-code makr-code marked this pull request as ready for review July 13, 2026 11:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@makr-code makr-code Awaiting requested review from makr-code makr-code is a code owner

Assignees

@makr-code makr-code

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Map hybrid retrieval rollout plan to module readiness across index, gpu, acceleration, graph, query, and sharding

2 participants

@makr-code