From 07a7cd51d52efa9e1231de1e344e6cf7a3c88f29 Mon Sep 17 00:00:00 2001 From: Jiayao Song Date: Tue, 28 Jul 2026 15:58:06 +0800 Subject: [PATCH 1/3] refactor(config): make [embedding] and [rerank] soft dependencies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make [embedding] and [rerank] soft runtime dependencies so a freshly-onboarded user can run EverOS end-to-end with only [llm] configured. Previously the server refused to start without embedding, locking out anyone who just wanted keyword-only search. ## Capability tiers - Tier 1 ([llm] only) KEYWORD search, add/flush, md writes, cascade sync - Tier 2 ([llm] + [embedding]) + VECTOR / HYBRID search, reflection, skill extraction, backfill - Tier 3 ([llm] + [embedding] + rerank) + AGENTIC search, knowledge Tier upgrades require a server restart (capability accessors cache for the process lifetime). Tier downgrades are read-safe: a Tier-3 user who drops [rerank] can still read/rename/delete existing knowledge documents; only write/search endpoints return 422. ## What changed - Component accessors — component/{embedding,rerank,llm}/accessor.py are the single process-wide provider singletons. service/* never maintains parallel singletons; it consumes get_embedding_capability() / get_rerank_capability() / get_llm_client() directly. Build-time ValueError from the factory is logged as capability_build_failed (was silently swallowed). - Error mapping — ProviderNotConfiguredError -> 422 with everos.toml section hints (never EVEROS_* env-var strings). LanceDBMigrationError fails loud with escalating recovery guidance (restart -> wipe index). LLMNotConfiguredError in search maps to None for KEYWORD degradation. - Nullable-vector LanceDB migration — schema v2 makes the vector column nullable so Tier-1 rows can land without embeddings. Migration is guarded by a cross-process memory_root_lock (fcntl.flock + anyio.to_thread) and runs optimize() per table after Phase-1 backfill to reclaim manifest bloat. - Cascade — knowledge handlers register unconditionally (Tier-3 -> Tier-2/1 downgrade no longer strands DELETE); embed-requiring strategies use body-guards that check capability.available at execution time. _TABLE_SPECS has an import-time drift assertion against BUSINESS_SCHEMAS_WITH_VECTOR. - `everos cascade backfill` CLI — Phase-1 (embed missing vectors) / Phase-2 (emit synthetic events for cascaded processing) / Phase-3 (sync new skill files). Exit codes: 0 / 1 / 2 / 3 (server running preflight) / 4 (COMPLETED_WITH_FAILURES — per-row failures rolled up) / 130 (SIGINT). OMEConfig.crash_recovery_enabled=False in backfill engines prevents stale-RUNNING rows re-enqueuing into a smaller strategy registry. - /health — reports capabilities + disabled_features per tier so ops can distinguish "boots but degraded" from "boots and full". - Presentation split — memory / service / infra never import typer / click. TyperPresenter Protocol + run_backfill() live in entrypoints/cli/commands/_backfill_cmd.py. Enforced by import-linter. - Startup hint — unconditional count_rows(filter="vector IS NULL") sweep emits unbackfilled_memory_rows (event name + hint text pinned) when Tier-1 rows exist. ParserLifespanProvider warms the everalgo.parser import at boot so /health doesn't block on first call. - Knowledge upload UTF-8 short-circuit — _looks_like_utf8_text() routes text/* mime and known plaintext extensions (md/txt/rst) straight to UTF-8 decode instead of the parser. Prevents 503 Multimodal-not-configured when Tier 3 sans [multimodal] uploads a markdown doc. ## Sync history with main (2 merges collapsed into this squash) Merged origin/main at 6dcd3eb (v1.1.4 -> v1.2.0 adds OTel tracing, /api/v2 alias, TracingLifespanProvider, per-cascade-embedding span fix, memory-op instrumentation) and later at 42629df (PR #366 backfills v1.1.4 CWE-22 knowledge path traversal fix + cascade retry-budget rework + errors.py -> core.errors.ExternalServiceError). Key merge decisions: - service/search.py adopts single wrap site — component.llm accessor already applies UsageRecordingClient when observability is on; service layer never keeps a parallel LLM singleton (Round-1 CR rule: "service layer never maintains parallel singletons"). - Knowledge router prefix moved to /knowledge; create_app() mounts it under both /api/v1 and /api/v2. - Cascade retry classification uses ExternalServiceError from core.errors (cascade/errors.py deleted). _MAX_TOTAL_RETRIES=12 cross-cycle budget preserved. - Fixed backport typo: MemoryRoot.default() -> MemoryRoot.resolve() (no .default() classmethod exists — main PR #366 shipped a broken call). ## Verified layering $ git grep -l "^import typer\|^from typer" src/everos/{memory,service,infra} # empty $ git grep -l "^import click\|^from click" src/everos/{memory,service,infra} # empty Memory / service / infra layers clean of CLI presentation libraries. ## Review history Three rounds of Fable 5 (opus) code review across the pre-squash commit history closed 38 findings total: - Round 1: 10 findings (fail-loud migration, backfill hardening, knowledge router gate scoping, SearchManager guards, profile throttle lift) - Round 2: 13 findings (hermetic test env, hot-reload doc drift, knowledge handler registration, Phase 3 sync guarantee, Phase 2 idempotency, profile event-first path, OMEConfig crash-recovery gate, cross-process migration lock, batch embed per-row fallback, LanceDB optimize, typer/click layer split) - Round 3: 15 Minor cleanups (accessor unification, marker revert, episode query hygiene, --verbose subcommand, parser lifespan warm, task-number scrub, temporal-overlap test, real-SIGINT slow mark, 4 design-note back-references) Full per-round context lives in the PR description on GitHub. ## Test plan - make lint (ruff + import-linter 3 contracts + assets + deprecated-names + github-docs + datetime + OpenAPI drift) - Hermetic env full pytest — 2027 passed / 7 deselected (7 = slow + live_llm markers) - Manual e2e across Tier 1/2/3 (21/21 assertions across v1/v2 double-mount and Tier 3 -> Tier 2 downgrade) - /health reports correct capabilities + disabled_features per tier ## Known follow-ups - .superpowers/sdd/followup-http-bridge.md (gitignored) — Path A for spec §10's "backfill 期间 EverOS 完全可用" promise - _TABLE_SCHEMA_VERSION docstring — v3+ migrations need a version dispatch table - extract_user_profile.py throttle-counter block — replace LanceDB count_by_owner with a sqlite memcell count Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/openapi.json | 6 +- scripts/e2e_memorize/run.py | 4 +- src/everos/component/capabilities.py | 74 + src/everos/component/embedding/__init__.py | 17 +- src/everos/component/embedding/accessor.py | 68 +- src/everos/component/embedding/capability.py | 59 + src/everos/component/embedding/factory.py | 14 +- src/everos/component/llm/client.py | 6 +- src/everos/component/llm/factory.py | 11 +- src/everos/component/multimodal/__init__.py | 27 + src/everos/component/multimodal/accessor.py | 41 + src/everos/component/multimodal/capability.py | 40 + src/everos/component/parser/_core.py | 17 +- src/everos/component/rerank/__init__.py | 9 + src/everos/component/rerank/accessor.py | 57 + src/everos/component/rerank/capability.py | 39 + src/everos/component/rerank/factory.py | 20 +- src/everos/component/utils/config_hints.py | 30 + src/everos/core/errors.py | 51 + src/everos/core/persistence/memory_root.py | 4 +- src/everos/entrypoints/api/app.py | 8 +- .../entrypoints/api/exception_handlers.py | 31 +- .../entrypoints/api/lifespans/__init__.py | 3 + .../entrypoints/api/lifespans/cascade.py | 27 +- .../entrypoints/api/lifespans/lancedb.py | 58 +- .../entrypoints/api/lifespans/parser.py | 60 + src/everos/entrypoints/api/routes/health.py | 43 +- .../entrypoints/api/routes/knowledge.py | 128 +- src/everos/entrypoints/cli/_log_setup.py | 29 + .../entrypoints/cli/commands/_backfill_cmd.py | 476 +++++ .../entrypoints/cli/commands/cascade.py | 195 ++- .../entrypoints/cli/commands/config_cmd.py | 15 + src/everos/entrypoints/cli/commands/demo.py | 8 + .../entrypoints/cli/commands/init_cmd.py | 8 + src/everos/infra/ome/_stores/storage.py | 2 +- src/everos/infra/ome/config.py | 23 +- src/everos/infra/ome/engine.py | 10 +- .../infra/persistence/lancedb/__init__.py | 204 ++- .../persistence/lancedb/lancedb_manager.py | 4 +- .../persistence/lancedb/repos/episode.py | 133 ++ .../persistence/lancedb/tables/agent_case.py | 2 +- .../persistence/lancedb/tables/agent_skill.py | 2 +- .../persistence/lancedb/tables/atomic_fact.py | 2 +- .../persistence/lancedb/tables/episode.py | 2 +- .../persistence/lancedb/tables/foresight.py | 2 +- .../lancedb/tables/knowledge_topic.py | 2 +- .../sqlite/repos/md_change_state.py | 42 +- .../infra/persistence/sqlite/repos/memcell.py | 26 +- .../persistence/sqlite/sqlite_manager.py | 6 +- src/everos/memory/cascade/__init__.py | 11 + src/everos/memory/cascade/_backfill.py | 1530 +++++++++++++++++ .../memory/cascade/handlers/agent_case.py | 15 +- .../memory/cascade/handlers/agent_skill.py | 17 +- .../memory/cascade/handlers/atomic_fact.py | 17 +- src/everos/memory/cascade/handlers/base.py | 13 +- src/everos/memory/cascade/handlers/episode.py | 22 +- .../memory/cascade/handlers/foresight.py | 15 +- .../cascade/handlers/knowledge_topic.py | 15 +- src/everos/memory/cascade/orchestrator.py | 36 +- src/everos/memory/cascade/registry.py | 19 + src/everos/memory/cascade/scanner.py | 32 +- src/everos/memory/cascade/worker.py | 29 +- src/everos/memory/reflection/orchestrator.py | 11 +- src/everos/memory/search/manager.py | 152 +- .../memory/strategies/extract_agent_case.py | 2 +- .../memory/strategies/extract_agent_skill.py | 35 +- .../memory/strategies/extract_atomic_facts.py | 2 +- .../memory/strategies/extract_foresight.py | 2 +- .../memory/strategies/extract_user_profile.py | 256 ++- .../memory/strategies/reflect_episodes.py | 22 +- .../strategies/trigger_profile_clustering.py | 39 +- .../strategies/trigger_skill_clustering.py | 26 +- src/everos/service/knowledge.py | 107 +- src/everos/service/memorize.py | 58 +- src/everos/service/search.py | 115 +- tests/conftest.py | 53 + tests/e2e/conftest.py | 4 +- tests/e2e/test_search_endpoint_e2e.py | 41 +- tests/integration/search/conftest.py | 22 +- tests/integration/test_api/__init__.py | 0 .../test_api/test_health_capabilities.py | 327 ++++ .../test_api/test_knowledge_gates.py | 386 +++++ .../test_api/test_provider_error_mapping.py | 70 + .../test_api/test_startup_banner.py | 128 ++ .../test_cascade_all_kinds_consistency.py | 12 +- .../test_cascade_cli_integration.py | 20 +- .../test_cascade_fsevents_repro.py | 12 +- tests/integration/test_cascade_integration.py | 21 +- tests/integration/test_cascade_scenarios.py | 26 +- tests/integration/test_cli/__init__.py | 0 .../test_cli/test_backfill_flags.py | 315 ++++ .../test_cli/test_backfill_phase1.py | 263 +++ .../test_cli/test_backfill_phase2.py | 314 ++++ .../test_cli/test_backfill_phase3.py | 257 +++ tests/integration/test_infra/__init__.py | 0 .../test_lancedb_schema_migration.py | 146 ++ .../integration/test_knowledge_integration.py | 209 ++- tests/integration/test_memorize_agent_mode.py | 2 +- .../test_memorize_concurrent_session_lock.py | 2 +- .../integration/test_memorize_integration.py | 4 +- .../test_memorize_window_segmentation.py | 2 +- .../test_ome_strategies_integration.py | 39 +- .../test_reflection_integration.py | 29 +- tests/integration/test_tiers/__init__.py | 10 + tests/integration/test_tiers/conftest.py | 436 +++++ .../test_tiers/test_tier1_keyword_only.py | 186 ++ .../test_tiers/test_tier2_no_rerank.py | 192 +++ .../integration/test_tiers/test_tier3_full.py | 197 +++ .../test_tiers/test_upgrade_path.py | 178 ++ .../unit/test_component/test_config_hints.py | 34 + .../test_embedding/test_accessor.py | 151 ++ .../test_embedding/test_factory.py | 6 +- .../test_component/test_llm/test_client.py | 4 +- .../test_component/test_llm/test_factory.py | 4 +- .../test_multimodal_capability.py | 126 ++ .../test_parser_available_memoized.py | 111 ++ .../test_rerank/test_accessor.py | 147 ++ .../test_rerank/test_factory.py | 6 +- .../test_lancedb/test_repository.py | 2 +- .../test_persistence/test_memory_root.py | 10 +- tests/unit/test_core/test_provider_error.py | 53 + .../test_lancedb_unbackfilled_hint.py | 128 ++ .../test_api/test_lifespans/test_ome.py | 2 +- .../test_routes/test_knowledge_api.py | 105 +- .../test_search_route_validation.py | 13 +- .../test_cli/test_backfill_command.py | 273 +++ .../test_cli/test_cascade_command.py | 47 +- .../test_cli/test_cascade_root_position.py | 183 ++ .../test_cli/test_cascade_verbose_position.py | 145 ++ .../test_cli/test_cli_default_log_level.py | 98 ++ .../test_cli/test_log_setup.py | 23 + .../test_episode_repo_count_by_owner.py | 156 ++ .../test_episode_repo_list_after_ts.py | 250 +++ .../test_migration_cross_process.py | 275 +++ .../test_repos/test_episode_projection.py | 195 +++ .../test_lancedb_tables_nullability.py | 102 ++ .../test_writers/test_daily_log_writers.py | 13 +- tests/unit/test_infra/test_ome/test_config.py | 4 +- .../test_engine_crash_recovery_gate.py | 77 + .../test_sqlite/test_repos/test_memcell.py | 143 ++ .../test_backfill_batch_concurrency.py | 159 ++ .../test_backfill_engine_isolation.py | 105 ++ ...test_backfill_exit_code_partial_failure.py | 107 ++ .../test_cascade/test_backfill_optimize.py | 208 +++ .../test_backfill_phase2_idempotency.py | 178 ++ .../test_backfill_phase3_sync_recovery.py | 87 + .../test_cascade/test_backfill_preflight.py | 674 ++++++++ .../test_backfill_row_fallback.py | 239 +++ .../test_backfill_skill_idempotency.py | 205 +++ .../test_backfill_subject_null_recovery.py | 247 +++ .../test_backfill_sync_scoping.py | 127 ++ .../test_cascade/test_backfill_table_specs.py | 51 + .../test_cascade/test_handler_agent_skill.py | 21 +- .../test_cascade/test_handler_episode.py | 54 +- .../test_handler_knowledge_document.py | 12 - .../test_handler_knowledge_topic.py | 15 +- .../test_cascade/test_handler_user_profile.py | 18 - .../test_handlers_daily_log_mapping.py | 15 +- .../test_handlers_embed_or_none.py | 232 +++ .../test_cascade/test_lifespan_soft_build.py | 114 ++ .../test_cascade/test_orchestrator.py | 14 +- .../test_registry_knowledge_gate.py | 156 ++ .../test_cascade/test_scanner_kinds.py | 96 ++ .../test_memory/test_cascade/test_worker.py | 23 +- .../test_cascade/test_worker_kinds.py | 179 ++ .../test_memory/test_search/test_manager.py | 72 +- .../test_search/test_validate_components.py | 300 ++++ .../test_extract_agent_skill.py | 220 ++- .../test_extract_user_profile_dual_trigger.py | 770 +++++++++ .../test_strategies/test_reflect_episodes.py | 41 + .../test_strategies/test_registration.py | 2 +- .../test_strategies_persistence.py | 6 +- .../test_strategy_to_handler_contract.py | 23 +- .../test_trigger_profile_clustering.py | 96 +- .../test_trigger_skill_clustering.py | 122 +- .../test_service/test_knowledge_search.py | 8 +- .../test_knowledge_search_degradation.py | 16 +- .../test_service/test_memorize_engine_gate.py | 93 + .../test_service/test_memorize_factories.py | 4 +- .../test_service/test_search_llm_client.py | 71 +- 180 files changed, 15813 insertions(+), 899 deletions(-) create mode 100644 src/everos/component/capabilities.py create mode 100644 src/everos/component/embedding/capability.py create mode 100644 src/everos/component/multimodal/__init__.py create mode 100644 src/everos/component/multimodal/accessor.py create mode 100644 src/everos/component/multimodal/capability.py create mode 100644 src/everos/component/rerank/accessor.py create mode 100644 src/everos/component/rerank/capability.py create mode 100644 src/everos/component/utils/config_hints.py create mode 100644 src/everos/entrypoints/api/lifespans/parser.py create mode 100644 src/everos/entrypoints/cli/_log_setup.py create mode 100644 src/everos/entrypoints/cli/commands/_backfill_cmd.py create mode 100644 src/everos/memory/cascade/_backfill.py create mode 100644 tests/integration/test_api/__init__.py create mode 100644 tests/integration/test_api/test_health_capabilities.py create mode 100644 tests/integration/test_api/test_knowledge_gates.py create mode 100644 tests/integration/test_api/test_provider_error_mapping.py create mode 100644 tests/integration/test_api/test_startup_banner.py create mode 100644 tests/integration/test_cli/__init__.py create mode 100644 tests/integration/test_cli/test_backfill_flags.py create mode 100644 tests/integration/test_cli/test_backfill_phase1.py create mode 100644 tests/integration/test_cli/test_backfill_phase2.py create mode 100644 tests/integration/test_cli/test_backfill_phase3.py create mode 100644 tests/integration/test_infra/__init__.py create mode 100644 tests/integration/test_infra/test_lancedb_schema_migration.py create mode 100644 tests/integration/test_tiers/__init__.py create mode 100644 tests/integration/test_tiers/conftest.py create mode 100644 tests/integration/test_tiers/test_tier1_keyword_only.py create mode 100644 tests/integration/test_tiers/test_tier2_no_rerank.py create mode 100644 tests/integration/test_tiers/test_tier3_full.py create mode 100644 tests/integration/test_tiers/test_upgrade_path.py create mode 100644 tests/unit/test_component/test_config_hints.py create mode 100644 tests/unit/test_component/test_embedding/test_accessor.py create mode 100644 tests/unit/test_component/test_multimodal_capability.py create mode 100644 tests/unit/test_component/test_parser_available_memoized.py create mode 100644 tests/unit/test_component/test_rerank/test_accessor.py create mode 100644 tests/unit/test_core/test_provider_error.py create mode 100644 tests/unit/test_entrypoints/test_api/test_lifespans/test_lancedb_unbackfilled_hint.py create mode 100644 tests/unit/test_entrypoints/test_cli/test_backfill_command.py create mode 100644 tests/unit/test_entrypoints/test_cli/test_cascade_root_position.py create mode 100644 tests/unit/test_entrypoints/test_cli/test_cascade_verbose_position.py create mode 100644 tests/unit/test_entrypoints/test_cli/test_cli_default_log_level.py create mode 100644 tests/unit/test_entrypoints/test_cli/test_log_setup.py create mode 100644 tests/unit/test_infra/test_episode_repo_count_by_owner.py create mode 100644 tests/unit/test_infra/test_episode_repo_list_after_ts.py create mode 100644 tests/unit/test_infra/test_lancedb/test_migration_cross_process.py create mode 100644 tests/unit/test_infra/test_lancedb/test_repos/test_episode_projection.py create mode 100644 tests/unit/test_infra/test_lancedb_tables_nullability.py create mode 100644 tests/unit/test_infra/test_ome/test_engine_crash_recovery_gate.py create mode 100644 tests/unit/test_infra/test_sqlite/test_repos/test_memcell.py create mode 100644 tests/unit/test_memory/test_cascade/test_backfill_batch_concurrency.py create mode 100644 tests/unit/test_memory/test_cascade/test_backfill_engine_isolation.py create mode 100644 tests/unit/test_memory/test_cascade/test_backfill_exit_code_partial_failure.py create mode 100644 tests/unit/test_memory/test_cascade/test_backfill_optimize.py create mode 100644 tests/unit/test_memory/test_cascade/test_backfill_phase2_idempotency.py create mode 100644 tests/unit/test_memory/test_cascade/test_backfill_phase3_sync_recovery.py create mode 100644 tests/unit/test_memory/test_cascade/test_backfill_preflight.py create mode 100644 tests/unit/test_memory/test_cascade/test_backfill_row_fallback.py create mode 100644 tests/unit/test_memory/test_cascade/test_backfill_skill_idempotency.py create mode 100644 tests/unit/test_memory/test_cascade/test_backfill_subject_null_recovery.py create mode 100644 tests/unit/test_memory/test_cascade/test_backfill_sync_scoping.py create mode 100644 tests/unit/test_memory/test_cascade/test_backfill_table_specs.py create mode 100644 tests/unit/test_memory/test_cascade/test_handlers_embed_or_none.py create mode 100644 tests/unit/test_memory/test_cascade/test_lifespan_soft_build.py create mode 100644 tests/unit/test_memory/test_cascade/test_registry_knowledge_gate.py create mode 100644 tests/unit/test_memory/test_cascade/test_scanner_kinds.py create mode 100644 tests/unit/test_memory/test_cascade/test_worker_kinds.py create mode 100644 tests/unit/test_memory/test_search/test_validate_components.py create mode 100644 tests/unit/test_memory/test_strategies/test_extract_user_profile_dual_trigger.py create mode 100644 tests/unit/test_service/test_memorize_engine_gate.py diff --git a/docs/openapi.json b/docs/openapi.json index e19ad97ec..7c223772c 100644 --- a/docs/openapi.json +++ b/docs/openapi.json @@ -12,7 +12,7 @@ "health" ], "summary": "Health", - "description": "Liveness probe — returns ``{\"status\": \"ok\"}`` with HTTP 200.", + "description": "Liveness probe with capabilities and disabled features.\n\nReturns a dict containing:\n- status: \"ok\"\n- version: semantic version string\n- capabilities: dict mapping capability names to availability booleans\n- disabled_features: list of feature names that are disabled due to\n missing capabilities", "operationId": "health_health_get", "responses": { "200": { @@ -20,9 +20,7 @@ "content": { "application/json": { "schema": { - "additionalProperties": { - "type": "string" - }, + "additionalProperties": true, "type": "object", "title": "Response Health Health Get" } diff --git a/scripts/e2e_memorize/run.py b/scripts/e2e_memorize/run.py index 86a4d51fa..032c58a89 100644 --- a/scripts/e2e_memorize/run.py +++ b/scripts/e2e_memorize/run.py @@ -51,7 +51,7 @@ def _print_header(mode: str, fixture_path: Path, session_id: str) -> None: print(f" everos e2e memorize · mode={mode}") print(f" fixture : {fixture_path.name}") print(f" session_id : {session_id}") - print(f" memory root : {MemoryRoot.default().root}") + print(f" memory root : {MemoryRoot.resolve().root}") llm_state = "" if get_llm_client() else "" print(f" llm_client : {llm_state}") print("=" * 72) @@ -59,7 +59,7 @@ def _print_header(mode: str, fixture_path: Path, session_id: str) -> None: def _list_written_files(session_id: str, mode: str) -> None: """Walk memory root and print files touched in this run.""" - root = MemoryRoot.default().root + root = MemoryRoot.resolve().root cutoff = time.time() - 600 # files modified in the last 10 min print() print("─── files modified within the last 10 minutes under memory root ───") diff --git a/src/everos/component/capabilities.py b/src/everos/component/capabilities.py new file mode 100644 index 000000000..fda706455 --- /dev/null +++ b/src/everos/component/capabilities.py @@ -0,0 +1,74 @@ +"""Capabilities inference and feature availability logic. + +Compute which features are disabled based on available capabilities. +Used by the health endpoint, startup banner, and related diagnostics. + +Two distinct feature-name vocabularies exist in the refactored codebase. +They deliberately do NOT share a source of truth: + +- ``ProviderNotConfiguredError.feature`` (raised by + ``SearchManager._validate_components`` and error handlers): per-request + granular tag identifying the specific search mode or endpoint that failed — + ``"vector"`` | ``"user_hybrid"`` | ``"agent_hybrid"`` | ``"agentic_search"`` + | ``"knowledge"`` | ``"skill_extraction_backfill"``. Appears in HTTP 422 + response ``message``. Naming mirrors the ``SearchMethod`` enum values plus + endpoint-scoped tags. + +- ``compute_disabled_features()`` (below): capability-level tag identifying + a whole feature category disabled by the current tier — + ``"vector_search"`` | ``"hybrid_search"`` | ``"agentic_search"`` + | ``"reflection"`` | ``"skill_extraction"`` | ``"knowledge"`` + | ``"multimodal_upload"``. Appears in ``GET /health`` response + ``disabled_features``. + +Callers of either surface should treat these vocabularies as stable client +contracts — do not unify them without a coordinated migration on the client side. +""" + +from __future__ import annotations + + +def compute_disabled_features(caps: dict[str, bool]) -> list[str]: + """Derive the list of disabled features from capability availability. + + Args: + caps: Dictionary with keys "llm", "embed", "rerank", "multimodal_llm", "parser" + and boolean availability values. Note: ``caps["llm"]`` is + accepted for shape symmetry but NOT read here — LLM is a + Tier-1 hard requirement enforced at server startup + (``LLMLifespanProvider``), so any process reaching this + function is guaranteed to have LLM available. If LLM ever + becomes soft, add an ``if not caps["llm"]`` branch here + covering the LLM-dependent features. + + Returns: + List of feature names that are disabled due to missing capabilities. + Possible values: "vector_search", "hybrid_search", "agentic_search", + "reflection", "skill_extraction", "knowledge", "multimodal_upload". + """ + disabled: list[str] = [] + + # Embedding-dependent features + if not caps["embed"]: + disabled.extend( + [ + "vector_search", + "hybrid_search", + "reflection", + "skill_extraction", + ] + ) + + # Rerank-dependent feature + if not caps["rerank"]: + disabled.append("agentic_search") + + # Knowledge requires both embedding and rerank + if not (caps["embed"] and caps["rerank"]): + disabled.append("knowledge") + + # Multimodal upload requires both multimodal_llm and parser + if not (caps["multimodal_llm"] and caps["parser"]): + disabled.append("multimodal_upload") + + return disabled diff --git a/src/everos/component/embedding/__init__.py b/src/everos/component/embedding/__init__.py index f1ec9878b..9a1d4f4e9 100644 --- a/src/everos/component/embedding/__init__.py +++ b/src/everos/component/embedding/__init__.py @@ -6,9 +6,18 @@ - :class:`EmbeddingProvider` — Protocol every provider satisfies. - :class:`EmbeddingServiceError` — provider-side failure. - :class:`EmbeddingError` — backward-compat alias for ``EmbeddingServiceError``. +- :class:`EmbeddingCapability` — soft-dependency wrapper around an + optional :class:`EmbeddingProvider` (``available`` / ``embed_or_none`` + / ``require``). - :class:`OpenAIEmbeddingProvider` — concrete provider for any OpenAI-protocol embeddings endpoint (DeepInfra, vLLM, OpenAI, …). - :func:`build_embedding_provider` — settings-driven factory. +- :func:`get_embedding_capability` — process-wide lazy singleton + accessor for :class:`EmbeddingCapability`. There is no separate + ``get_embedder`` accessor: consumers that need a provider call + ``get_embedding_capability().require()``, which routes every caller + through a single shared provider (and its single ``AsyncOpenAI`` + client + ``asyncio.Semaphore``). External usage:: @@ -19,19 +28,19 @@ from everos.core.errors import EmbeddingServiceError as EmbeddingServiceError -from .accessor import EmbeddingNotConfiguredError as EmbeddingNotConfiguredError -from .accessor import get_embedder as get_embedder +from .accessor import get_embedding_capability as get_embedding_capability +from .capability import EmbeddingCapability as EmbeddingCapability from .factory import build_embedding_provider as build_embedding_provider from .openai_provider import OpenAIEmbeddingProvider as OpenAIEmbeddingProvider from .protocol import EmbeddingError as EmbeddingError from .protocol import EmbeddingProvider as EmbeddingProvider __all__ = [ + "EmbeddingCapability", "EmbeddingError", - "EmbeddingNotConfiguredError", "EmbeddingProvider", "EmbeddingServiceError", "OpenAIEmbeddingProvider", "build_embedding_provider", - "get_embedder", + "get_embedding_capability", ] diff --git a/src/everos/component/embedding/accessor.py b/src/everos/component/embedding/accessor.py index cbe4d1102..aad795d51 100644 --- a/src/everos/component/embedding/accessor.py +++ b/src/everos/component/embedding/accessor.py @@ -1,14 +1,16 @@ -"""Process-wide embedding provider accessor. - -Lazy singleton mirror of :func:`everos.component.llm.get_llm_client`: -first call reads settings and builds the OpenAI-protocol embedding -client; subsequent calls return the cached instance. Strategies and -other components that need a process-wide embedder import this rather -than threading the provider through their constructors. - -Raises :class:`EmbeddingNotConfiguredError` when credentials are missing -so misconfiguration surfaces at the call site (or at app startup via a -lifespan provider) instead of silently degrading. +"""Process-wide embedding capability accessor. + +Lazy singleton for :class:`EmbeddingCapability`. The first call reads +settings and attempts to build an OpenAI-protocol embedding provider; +subsequent calls return the cached wrapper. Consumers that must have an +embedder call ``get_embedding_capability().require()``; consumers that +can degrade gracefully use ``.embed_or_none`` or check ``.available``. + +There is deliberately no separate ``get_embedder()`` accessor: routing +every consumer through the capability keeps a single provider (and its +underlying ``AsyncOpenAI`` client + ``asyncio.Semaphore``) per process, +so the configured ``max_concurrent`` bound holds instead of silently +doubling. """ from __future__ import annotations @@ -16,33 +18,41 @@ from everos.config import load_settings from everos.core.observability.logging import get_logger +from .capability import EmbeddingCapability from .factory import build_embedding_provider -from .protocol import EmbeddingProvider logger = get_logger(__name__) -class EmbeddingNotConfiguredError(RuntimeError): - """Raised when ``settings.embedding`` lacks ``model``/``api_key``/``base_url``.""" - +_capability: EmbeddingCapability | None = None -_embedder: EmbeddingProvider | None = None +def get_embedding_capability() -> EmbeddingCapability: + """Return the process-wide :class:`EmbeddingCapability`. Never raises. -def get_embedder() -> EmbeddingProvider: - """Return the singleton :class:`EmbeddingProvider`. + On the first call, builds and caches a capability from current + settings — ``available`` is ``False`` when the provider cannot be + built (missing fields, unsupported provider name, malformed URL, …). + The build outcome is cached, so a later settings change requires a + process restart to take effect. - Raises: - EmbeddingNotConfiguredError: When required settings fields are - unset. See :func:`build_embedding_provider` for the exact - keys. + Configuration failures (:class:`ValueError` from + :func:`build_embedding_provider`) are logged at ``warning`` level: + the downstream :class:`ProviderNotConfiguredError` message maps both + "user hasn't configured it" and "user configured it wrong" onto the + same HTTP 422, so the log line is the only place an operator can + tell those two states apart. """ - global _embedder - if _embedder is not None: - return _embedder + global _capability + if _capability is not None: + return _capability try: - _embedder = build_embedding_provider(load_settings().embedding) + provider = build_embedding_provider(load_settings().embedding) except ValueError as exc: - raise EmbeddingNotConfiguredError(str(exc)) from exc - logger.info("embedder_built") - return _embedder + logger.warning( + "embedding_capability_build_failed", + reason=str(exc), + ) + provider = None + _capability = EmbeddingCapability(provider=provider) + return _capability diff --git a/src/everos/component/embedding/capability.py b/src/everos/component/embedding/capability.py new file mode 100644 index 000000000..0826bc924 --- /dev/null +++ b/src/everos/component/embedding/capability.py @@ -0,0 +1,59 @@ +"""EmbeddingCapability — soft-dependency wrapper for EmbeddingProvider. + +The wrapper encapsulates ``Optional[EmbeddingProvider]`` so consumers +never see ``None`` directly. Two consumption modes: + +- :meth:`embed_or_none` for soft-degrade write paths (cascade handlers + that write ``vector=NULL`` when embedding is unavailable). +- :meth:`require` for hard-required paths (search VECTOR/HYBRID/AGENTIC) + that raise :class:`ProviderNotConfiguredError` -> HTTP 422 when missing. + +Design rationale: EverOS has multiple code paths that need to know +whether embedding is available. Rather than each callsite reading +settings and re-implementing the check, the check lives in +:func:`everos.component.embedding.accessor.get_embedding_capability` +(module-level singleton); every consumer asks the capability. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from everos.core.errors import ProviderNotConfiguredError + +from .protocol import EmbeddingProvider + + +@dataclass(frozen=True) +class EmbeddingCapability: + """Wraps an optional EmbeddingProvider with soft / strict consumption modes.""" + + provider: EmbeddingProvider | None + + @property + def available(self) -> bool: + """True iff a provider was successfully constructed.""" + return self.provider is not None + + async def embed_or_none(self, text: str) -> list[float] | None: + """Embed ``text`` if available, else return ``None``. + + For write paths that can proceed without a vector (cascade + handlers write ``vector=NULL`` on the LanceDB row). + """ + if self.provider is None: + return None + return list(await self.provider.embed(text)) + + def require(self) -> EmbeddingProvider: + """Return the provider, or raise :class:`ProviderNotConfiguredError`. + + For paths that cannot proceed without embedding (search + VECTOR/HYBRID/AGENTIC methods). Caller supplies its own + ``feature``/``alternative_hint`` context by catching and + re-raising if needed; this base call only identifies the + missing provider as ``"embedding"``. + """ + if self.provider is None: + raise ProviderNotConfiguredError(provider="embedding") + return self.provider diff --git a/src/everos/component/embedding/factory.py b/src/everos/component/embedding/factory.py index f32e7446f..1b1814790 100644 --- a/src/everos/component/embedding/factory.py +++ b/src/everos/component/embedding/factory.py @@ -2,6 +2,7 @@ from __future__ import annotations +from everos.component.utils.config_hints import missing_config_error from everos.config import EmbeddingSettings from .openai_provider import OpenAIEmbeddingProvider @@ -32,18 +33,11 @@ def build_embedding_provider( ValueError: If ``model``, ``api_key`` or ``base_url`` is unset. """ if not settings.model: - raise ValueError( - "Embedding model is not configured " - "(set EVEROS_EMBEDDING__MODEL or [embedding] model in user toml)" - ) + raise ValueError(missing_config_error("Embedding model", "embedding")) if not settings.api_key or not settings.api_key.get_secret_value(): - raise ValueError( - "Embedding api_key is not configured (set EVEROS_EMBEDDING__API_KEY)" - ) + raise ValueError(missing_config_error("Embedding api_key", "embedding")) if not settings.base_url: - raise ValueError( - "Embedding base_url is not configured (set EVEROS_EMBEDDING__BASE_URL)" - ) + raise ValueError(missing_config_error("Embedding base_url", "embedding")) return OpenAIEmbeddingProvider( model=settings.model, api_key=settings.api_key.get_secret_value(), diff --git a/src/everos/component/llm/client.py b/src/everos/component/llm/client.py index dfcbcaff2..6d48804fb 100644 --- a/src/everos/component/llm/client.py +++ b/src/everos/component/llm/client.py @@ -17,6 +17,7 @@ from everalgo.llm.types import ChatMessage, ChatResponse from pydantic import BaseModel +from everos.component.utils.config_hints import missing_config_error from everos.config import load_settings from everos.core.observability.logging import get_logger @@ -93,7 +94,7 @@ def get_llm_client() -> LLMClient: ) if not api_key or not llm_cfg.base_url: raise LLMNotConfiguredError( - "LLM is required; set EVEROS_LLM__API_KEY + EVEROS_LLM__BASE_URL" + missing_config_error("LLM api_key and base_url", "llm") ) client: LLMClient = build_client( LLMConfig( @@ -133,8 +134,7 @@ def get_multimodal_llm_client() -> LLMClient: api_key = cfg.api_key.get_secret_value() if cfg.api_key is not None else None if not api_key or not cfg.base_url: raise LLMNotConfiguredError( - "Multimodal LLM is required for parsing; set " - "EVEROS_MULTIMODAL__API_KEY + EVEROS_MULTIMODAL__BASE_URL" + missing_config_error("Multimodal LLM api_key and base_url", "multimodal") ) _multimodal_client = build_client( LLMConfig( diff --git a/src/everos/component/llm/factory.py b/src/everos/component/llm/factory.py index 5dff064aa..71c08b8c3 100644 --- a/src/everos/component/llm/factory.py +++ b/src/everos/component/llm/factory.py @@ -2,6 +2,7 @@ from __future__ import annotations +from everos.component.utils.config_hints import missing_config_error from everos.config import LLMSettings from .openai_provider import OpenAIProvider @@ -29,15 +30,9 @@ def build_llm_provider(settings: LLMSettings) -> LLMClient: ValueError: If ``api_key`` or ``base_url`` is unset. """ if not settings.api_key or not settings.api_key.get_secret_value(): - raise ValueError( - "LLM api_key is not configured " - "(set EVEROS_LLM__API_KEY or [llm] api_key in user toml)" - ) + raise ValueError(missing_config_error("LLM api_key", "llm")) if not settings.base_url: - raise ValueError( - "LLM base_url is not configured " - "(set EVEROS_LLM__BASE_URL or [llm] base_url in user toml)" - ) + raise ValueError(missing_config_error("LLM base_url", "llm")) return OpenAIProvider( model=settings.model, api_key=settings.api_key.get_secret_value(), diff --git a/src/everos/component/multimodal/__init__.py b/src/everos/component/multimodal/__init__.py new file mode 100644 index 000000000..0c7953a99 --- /dev/null +++ b/src/everos/component/multimodal/__init__.py @@ -0,0 +1,27 @@ +"""Multimodal LLM capability — optional vision/audio support for parsing. + +Public surface: + +- :class:`MultimodalLLMCapability` — soft-dependency wrapper around an + optional multimodal LLMClient (``available`` / ``require``; no soft-degrade + accessor — multimodal parsing is entirely optional). +- :func:`get_multimodal_llm_capability` — process-wide lazy singleton accessor + for :class:`MultimodalLLMCapability`. + +External usage:: + + from everos.component.multimodal import get_multimodal_llm_capability + cap = get_multimodal_llm_capability() + if cap.available: + client = cap.require() +""" + +from __future__ import annotations + +from .accessor import get_multimodal_llm_capability as get_multimodal_llm_capability +from .capability import MultimodalLLMCapability as MultimodalLLMCapability + +__all__ = [ + "MultimodalLLMCapability", + "get_multimodal_llm_capability", +] diff --git a/src/everos/component/multimodal/accessor.py b/src/everos/component/multimodal/accessor.py new file mode 100644 index 000000000..e402722d3 --- /dev/null +++ b/src/everos/component/multimodal/accessor.py @@ -0,0 +1,41 @@ +"""Process-wide multimodal LLM capability accessor. + +Lazy singleton — first call reads settings and attempts to build a multimodal +LLM client. Unlike :func:`everos.component.llm.client.get_multimodal_llm_client` +(which raises when misconfigured), this wraps the outcome in a capability that +reports ``available=False`` when the client cannot be built. + +Subsequent calls return the cached instance. +""" + +from __future__ import annotations + +from everos.component.llm.client import ( + LLMNotConfiguredError, + get_multimodal_llm_client, +) + +from .capability import MultimodalLLMCapability + +_capability: MultimodalLLMCapability | None = None + + +def get_multimodal_llm_capability() -> MultimodalLLMCapability: + """Return the process-wide :class:`MultimodalLLMCapability`. Never raises. + + Lazy singleton: the first call attempts to build a multimodal client from + current settings — ``available`` is ``False`` when the client cannot be + built (e.g. missing model/base_url/api_key). Use this from upload + endpoints, the health endpoint, the startup banner, and any other caller + that needs to check "is multimodal parsing available?" without a hard + dependency. + """ + global _capability + if _capability is not None: + return _capability + try: + provider = get_multimodal_llm_client() + except (ValueError, LLMNotConfiguredError): + provider = None + _capability = MultimodalLLMCapability(provider=provider) + return _capability diff --git a/src/everos/component/multimodal/capability.py b/src/everos/component/multimodal/capability.py new file mode 100644 index 000000000..0805a2c03 --- /dev/null +++ b/src/everos/component/multimodal/capability.py @@ -0,0 +1,40 @@ +"""MultimodalLLMCapability — soft-dependency wrapper for multimodal LLM client. + +Parallel structure to :class:`everos.component.rerank.RerankCapability`, but +for the multimodal LLM used by ``everalgo.parser``. A caller either requires +multimodal (raising :class:`ProviderNotConfiguredError` -> HTTP 422 when +missing) or checks ``available`` to skip parsing when unavailable. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import TYPE_CHECKING + +from everos.core.errors import ProviderNotConfiguredError + +if TYPE_CHECKING: + from everalgo.llm.protocols import LLMClient + + +@dataclass(frozen=True) +class MultimodalLLMCapability: + """Wraps an optional multimodal LLMClient with a hard-require API.""" + + provider: LLMClient | None + + @property + def available(self) -> bool: + """True iff a provider was successfully constructed.""" + return self.provider is not None + + def require(self) -> LLMClient: + """Return the provider, or raise :class:`ProviderNotConfiguredError`. + + Caller supplies its own ``feature``/``alternative_hint`` context by + catching and re-raising if needed; this base call only identifies + the missing provider as ``"multimodal_llm"``. + """ + if self.provider is None: + raise ProviderNotConfiguredError(provider="multimodal_llm") + return self.provider diff --git a/src/everos/component/parser/_core.py b/src/everos/component/parser/_core.py index a93e57d65..d30a35d4f 100644 --- a/src/everos/component/parser/_core.py +++ b/src/everos/component/parser/_core.py @@ -6,6 +6,7 @@ from __future__ import annotations +from functools import lru_cache from typing import TYPE_CHECKING from everos.core.errors import MultimodalNotEnabledError, UnsupportedModalityError @@ -14,8 +15,22 @@ from everalgo.types import ParsedContent, RawFile +@lru_cache(maxsize=1) def parser_available() -> bool: - """Whether ``everalgo.parser`` is importable.""" + """Whether ``everalgo.parser`` is importable. + + Memoised: the underlying ``import everalgo.parser`` pulls in heavy + PDF/Office dependencies (``pypdf``, ``python-docx``, ...) that would + block the event loop for hundreds of ms — unacceptable inside + ``async def health()`` on a liveness probe. First call at startup + (via :class:`ParserLifespanProvider`) pays the import cost off the + request path; subsequent calls hit the cache instantly. + + The cache is process-wide. Tests that patch ``sys.modules`` to + swap the ``everalgo.parser`` module in or out must invoke + :meth:`parser_available.cache_clear` between assertions to avoid + cross-test contamination. + """ try: import everalgo.parser # noqa: F401 except ImportError: diff --git a/src/everos/component/rerank/__init__.py b/src/everos/component/rerank/__init__.py index 1ce9ca7c8..ebc83b282 100644 --- a/src/everos/component/rerank/__init__.py +++ b/src/everos/component/rerank/__init__.py @@ -5,6 +5,9 @@ - :class:`RerankProvider` — Protocol every provider satisfies. - :class:`RerankResult` / :class:`RerankServiceError` — value type + error. - :class:`RerankError` — backward-compat alias for :class:`RerankServiceError`. +- :class:`RerankCapability` — soft-dependency wrapper around an optional + :class:`RerankProvider` (``available`` / ``require``; no soft-degrade + accessor — rerank has no write path to degrade). - :class:`DeepInfraRerankProvider` — DeepInfra inference-API rerank. - :class:`VllmRerankProvider` — OpenAI-compat ``/v1/rerank`` (vLLM, self-hosted, other compatible servers). @@ -12,6 +15,8 @@ ``gte-rerank-v2`` native text-rerank endpoint. - :func:`build_rerank_provider` — settings-driven factory that picks the concrete provider via ``settings.rerank.provider``. +- :func:`get_rerank_capability` — process-wide lazy singleton accessor + for :class:`RerankCapability`. External usage:: @@ -22,6 +27,8 @@ from everos.core.errors import RerankServiceError as RerankServiceError +from .accessor import get_rerank_capability as get_rerank_capability +from .capability import RerankCapability as RerankCapability from .dashscope_provider import DashScopeRerankProvider as DashScopeRerankProvider from .deepinfra_provider import DeepInfraRerankProvider as DeepInfraRerankProvider from .factory import build_rerank_provider as build_rerank_provider @@ -33,10 +40,12 @@ __all__ = [ "DashScopeRerankProvider", "DeepInfraRerankProvider", + "RerankCapability", "RerankError", "RerankProvider", "RerankResult", "RerankServiceError", "VllmRerankProvider", "build_rerank_provider", + "get_rerank_capability", ] diff --git a/src/everos/component/rerank/accessor.py b/src/everos/component/rerank/accessor.py new file mode 100644 index 000000000..f8e0adb40 --- /dev/null +++ b/src/everos/component/rerank/accessor.py @@ -0,0 +1,57 @@ +"""Process-wide rerank capability accessor. + +Lazy singleton mirror of +:func:`everos.component.embedding.accessor.get_embedding_capability`: first +call reads settings, attempts to build a rerank provider, and wraps the +outcome (provider or ``None``) in a :class:`RerankCapability`. Subsequent +calls return the cached instance. + +Rerank is a Tier-3 optional provider — call sites either go through +:func:`get_rerank_capability` and call ``.require()`` when rerank is +mandatory, or check ``.available`` and skip reranking when it is not. +""" + +from __future__ import annotations + +from everos.config import load_settings +from everos.core.observability.logging import get_logger + +from .capability import RerankCapability +from .factory import build_rerank_provider + +logger = get_logger(__name__) + + +_capability: RerankCapability | None = None + + +def get_rerank_capability() -> RerankCapability: + """Return the process-wide :class:`RerankCapability`. Never raises. + + Lazy singleton: the first call builds and caches a capability from + current settings — ``available`` is ``False`` when the provider cannot + be built (e.g. missing model/base_url/api_key, unsupported provider + name, malformed URL, …). Use this from search strategies, the health + endpoint, the startup banner, and any other caller that needs to + check "is rerank available?" without a hard dependency. + + Configuration failures (:class:`ValueError` from + :func:`build_rerank_provider`) are logged at ``warning`` level: the + downstream :class:`ProviderNotConfiguredError` message maps both + "user hasn't configured it" and "user configured it wrong" onto the + same HTTP 422, so the log line is the only place an operator can + tell those two states apart. + """ + global _capability + if _capability is not None: + return _capability + try: + provider = build_rerank_provider(load_settings().rerank) + except ValueError as exc: + logger.warning( + "rerank_capability_build_failed", + reason=str(exc), + ) + provider = None + _capability = RerankCapability(provider=provider) + return _capability diff --git a/src/everos/component/rerank/capability.py b/src/everos/component/rerank/capability.py new file mode 100644 index 000000000..a3d7c7115 --- /dev/null +++ b/src/everos/component/rerank/capability.py @@ -0,0 +1,39 @@ +"""RerankCapability — soft-dependency wrapper for RerankProvider. + +Parallel structure to :class:`everos.component.embedding.EmbeddingCapability`, +but without a soft-degrade accessor: rerank is a query-time enhancement, +not a write path. A caller either hard-requires rerank (raising +:class:`ProviderNotConfiguredError` -> HTTP 422 when missing) or chooses to +skip reranking entirely — there is no equivalent of ``embed_or_none``. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from everos.core.errors import ProviderNotConfiguredError + +from .protocol import RerankProvider + + +@dataclass(frozen=True) +class RerankCapability: + """Wraps an optional RerankProvider with a hard-require API.""" + + provider: RerankProvider | None + + @property + def available(self) -> bool: + """True iff a provider was successfully constructed.""" + return self.provider is not None + + def require(self) -> RerankProvider: + """Return the provider, or raise :class:`ProviderNotConfiguredError`. + + Caller supplies its own ``feature``/``alternative_hint`` context by + catching and re-raising if needed; this base call only identifies + the missing provider as ``"rerank"``. + """ + if self.provider is None: + raise ProviderNotConfiguredError(provider="rerank") + return self.provider diff --git a/src/everos/component/rerank/factory.py b/src/everos/component/rerank/factory.py index 1cf4c1a0f..ce2b8b57a 100644 --- a/src/everos/component/rerank/factory.py +++ b/src/everos/component/rerank/factory.py @@ -13,6 +13,7 @@ from __future__ import annotations +from everos.component.utils.config_hints import missing_config_error from everos.config import RerankSettings from .dashscope_provider import DashScopeRerankProvider @@ -38,22 +39,14 @@ def build_rerank_provider(settings: RerankSettings) -> RerankProvider: string) for ``vllm`` self-hosted endpoints. """ if not settings.model: - raise ValueError( - "Rerank model is not configured " - "(set EVEROS_RERANK__MODEL or [rerank] model in user toml)" - ) + raise ValueError(missing_config_error("Rerank model", "rerank")) if not settings.base_url: - raise ValueError( - "Rerank base_url is not configured (set EVEROS_RERANK__BASE_URL)" - ) + raise ValueError(missing_config_error("Rerank base_url", "rerank")) api_key = settings.api_key.get_secret_value() if settings.api_key else "" if settings.provider == "deepinfra": if not api_key: - raise ValueError( - "DeepInfra rerank api_key is not configured " - "(set EVEROS_RERANK__API_KEY)" - ) + raise ValueError(missing_config_error("DeepInfra rerank api_key", "rerank")) return DeepInfraRerankProvider( model=settings.model, api_key=api_key, @@ -75,10 +68,7 @@ def build_rerank_provider(settings: RerankSettings) -> RerankProvider: ) if settings.provider == "dashscope": if not api_key: - raise ValueError( - "DashScope rerank api_key is not configured " - "(set EVEROS_RERANK__API_KEY)" - ) + raise ValueError(missing_config_error("DashScope rerank api_key", "rerank")) return DashScopeRerankProvider( model=settings.model, api_key=api_key, diff --git a/src/everos/component/utils/config_hints.py b/src/everos/component/utils/config_hints.py new file mode 100644 index 000000000..ef22590bc --- /dev/null +++ b/src/everos/component/utils/config_hints.py @@ -0,0 +1,30 @@ +"""User-facing error message helpers for missing provider configuration. + +Guidance points users at the ``/everos.toml`` file, not at +environment variables. Env vars still work as an override mechanism +(see ``pydantic-settings`` precedence in ``config/settings.py``) but +are not surfaced in onboarding-facing text. +""" + +from __future__ import annotations + +from everos.core.persistence import MemoryRoot + + +def missing_config_error(field_label: str, toml_section: str) -> str: + """Return a uniform error message for a missing config field. + + Args: + field_label: Human-readable label (e.g. ``"LLM api_key"``). + toml_section: TOML section name without brackets (e.g. ``"llm"``). + + Returns: + A single-line message including the resolved memory-root path + and a hint to run ``everos init``. Never mentions env vars. + """ + root = MemoryRoot.resolve().root + return ( + f"{field_label} is not configured. " + f"Edit {root}/everos.toml (run `everos init` to scaffold), " + f"section [{toml_section}]." + ) diff --git a/src/everos/core/errors.py b/src/everos/core/errors.py index ba34c7ed8..4d24aedd6 100644 --- a/src/everos/core/errors.py +++ b/src/everos/core/errors.py @@ -36,6 +36,7 @@ class ErrorCode(StrEnum): CAPABILITY_UNAVAILABLE = "CAPABILITY_UNAVAILABLE" CONFIGURATION_ERROR = "CONFIGURATION_ERROR" INTERNAL_ERROR = "INTERNAL_ERROR" + PROVIDER_NOT_CONFIGURED = "PROVIDER_NOT_CONFIGURED" # --------------------------------------------------------------------------- @@ -175,6 +176,56 @@ class ConfigurationError(AppError): """ +# TOML section name for each provider kind, keyed by the `provider` argument +# passed to `ProviderNotConfiguredError`. +_PROVIDER_SECTIONS: dict[str, str] = { + "llm": "llm", + "embedding": "embedding", + "rerank": "rerank", + "multimodal_llm": "multimodal", +} + + +class ProviderNotConfiguredError(ConfigurationError): + """Raised when a runtime path requires a provider that is not configured. + + Maps to HTTP 422 via the FastAPI exception handler; the message is + directly user-facing and points at the toml file for remediation. + + Args: + provider: Which provider is missing. One of ``"llm"``, + ``"embedding"``, ``"rerank"``, ``"multimodal_llm"``. + feature: Optional user-facing feature that required this + provider (e.g. ``"knowledge"``, ``"agent_hybrid"``). + alternative_hint: Optional alternative workaround the user can + take without configuring the missing provider (e.g. flip + ``enable_llm_rerank=true`` to use the LLM lane). + """ + + def __init__( + self, + provider: str, + feature: str | None = None, + alternative_hint: str | None = None, + ) -> None: + # Lazy import: `config_hints` -> `core.persistence` -> `core.persistence + # .markdown.writer` imports `PathTraversalError` from this module, so a + # module-level import here would be circular. + from everos.component.utils.config_hints import missing_config_error + + self.provider = provider + self.feature = feature + self.alternative_hint = alternative_hint + section = _PROVIDER_SECTIONS.get(provider, provider) + field_label = f"Provider '{provider}'" + if feature: + field_label += f" (required by {feature})" + message = missing_config_error(field_label, section) + if alternative_hint: + message += f" Alternative: {alternative_hint}" + super().__init__(message) + + # --------------------------------------------------------------------------- # Backward compatibility aliases # --------------------------------------------------------------------------- diff --git a/src/everos/core/persistence/memory_root.py b/src/everos/core/persistence/memory_root.py index 7a96bd2f9..6d1775c78 100644 --- a/src/everos/core/persistence/memory_root.py +++ b/src/everos/core/persistence/memory_root.py @@ -23,7 +23,7 @@ The default location and tunables come from :class:`everos.config.Settings` (loaded from ``config/default.toml`` + ``EVEROS_*`` environment variables); -:meth:`MemoryRoot.default` resolves the configured path. +:meth:`MemoryRoot.resolve` resolves the configured path. """ from __future__ import annotations @@ -92,7 +92,7 @@ def __init__(self, root: Path | str) -> None: object.__setattr__(self, "root", resolved) @classmethod - def default(cls, *, explicit_root: str | None = None) -> MemoryRoot: + def resolve(cls, *, explicit_root: str | None = None) -> MemoryRoot: """Return the memory-root resolved from CLI / env / default. Resolution: ``explicit_root`` > ``EVEROS_ROOT`` env > ``~/.everos``. diff --git a/src/everos/entrypoints/api/app.py b/src/everos/entrypoints/api/app.py index 3eae0cd59..eba139a9d 100644 --- a/src/everos/entrypoints/api/app.py +++ b/src/everos/entrypoints/api/app.py @@ -34,6 +34,7 @@ LanceDBLifespanProvider, LLMLifespanProvider, OmeLifespanProvider, + ParserLifespanProvider, SqliteLifespanProvider, ) from .routes import ( @@ -71,9 +72,9 @@ def create_app( cors_allow_headers: Allowed CORS headers (default: ``["*"]``). lifespan_providers: Optional list of LifespanProvider; defaults to ``[TracingLifespanProvider(), MetricsLifespanProvider(), - LLMLifespanProvider(), SqliteLifespanProvider(), - LanceDBLifespanProvider(), CascadeLifespanProvider(), - OmeLifespanProvider()]``. + LLMLifespanProvider(), ParserLifespanProvider(), + SqliteLifespanProvider(), LanceDBLifespanProvider(), + CascadeLifespanProvider(), OmeLifespanProvider()]``. Returns: FastAPI: Configured application instance. @@ -85,6 +86,7 @@ def create_app( TracingLifespanProvider(), MetricsLifespanProvider(), LLMLifespanProvider(), + ParserLifespanProvider(), SqliteLifespanProvider(), LanceDBLifespanProvider(), CascadeLifespanProvider(), diff --git a/src/everos/entrypoints/api/exception_handlers.py b/src/everos/entrypoints/api/exception_handlers.py index da7675a2a..0e3142815 100644 --- a/src/everos/entrypoints/api/exception_handlers.py +++ b/src/everos/entrypoints/api/exception_handlers.py @@ -43,6 +43,7 @@ InvalidInputError, NotFoundError, PathTraversalError, + ProviderNotConfiguredError, UnsupportedModalityError, ) from everos.core.observability.logging import get_logger @@ -241,6 +242,31 @@ async def configuration_handler( ) +async def provider_not_configured_handler( + request: Request, + exc: ProviderNotConfiguredError, +) -> JSONResponse: + """ProviderNotConfiguredError -> 422 (client-actionable, unlike its parent). + + Unlike a generic ``ConfigurationError`` (500 -- an operator bug), a + missing provider is something the caller can fix by editing + ``everos.toml``, so this more-specific subclass gets its own 422 + mapping instead of falling through to ``configuration_handler``. + """ + logger.warning( + "provider_not_configured_error", + path=str(request.url.path), + provider=exc.provider, + feature=exc.feature, + ) + return _error_response( + request, + HTTP_422_UNPROCESSABLE_CONTENT, + ErrorCode.PROVIDER_NOT_CONFIGURED, + str(exc), + ) + + # --------------------------------------------------------------------------- # Pydantic / FastAPI built-in exceptions # --------------------------------------------------------------------------- @@ -350,7 +376,10 @@ def register_handlers(app: FastAPI) -> None: app.add_exception_handler(InfrastructureError, infrastructure_handler) # Capability errors (permanent, not retryable) app.add_exception_handler(CapabilityError, capability_handler) - # Configuration errors + # Configuration errors (specific before parent) + app.add_exception_handler( + ProviderNotConfiguredError, provider_not_configured_handler + ) app.add_exception_handler(ConfigurationError, configuration_handler) # FastAPI built-in exceptions app.add_exception_handler(HTTPException, http_exception_handler) diff --git a/src/everos/entrypoints/api/lifespans/__init__.py b/src/everos/entrypoints/api/lifespans/__init__.py index 262106d35..70fd1cb73 100644 --- a/src/everos/entrypoints/api/lifespans/__init__.py +++ b/src/everos/entrypoints/api/lifespans/__init__.py @@ -13,6 +13,7 @@ from everos.entrypoints.api.lifespans import ( LLMLifespanProvider, + ParserLifespanProvider, SqliteLifespanProvider, LanceDBLifespanProvider, CascadeLifespanProvider, @@ -24,6 +25,7 @@ from .lancedb import LanceDBLifespanProvider as LanceDBLifespanProvider from .llm import LLMLifespanProvider as LLMLifespanProvider from .ome import OmeLifespanProvider as OmeLifespanProvider +from .parser import ParserLifespanProvider as ParserLifespanProvider from .sqlite import SqliteLifespanProvider as SqliteLifespanProvider __all__ = [ @@ -31,5 +33,6 @@ "LLMLifespanProvider", "LanceDBLifespanProvider", "OmeLifespanProvider", + "ParserLifespanProvider", "SqliteLifespanProvider", ] diff --git a/src/everos/entrypoints/api/lifespans/cascade.py b/src/everos/entrypoints/api/lifespans/cascade.py index aa3cf5c52..fc6528920 100644 --- a/src/everos/entrypoints/api/lifespans/cascade.py +++ b/src/everos/entrypoints/api/lifespans/cascade.py @@ -4,9 +4,12 @@ depends on both stores being ready before its watcher / scanner / worker tasks can take the first row. -Construction reads the live :class:`Settings` to build the embedding + -tokenizer providers. If either is misconfigured the lifespan fails -fast — the daemon would be useless without them anyway. +Construction reads the live :class:`Settings` to build the tokenizer +provider, which fails fast if misconfigured. Embedding is a soft +dependency: startup warms the process-wide +:class:`~everos.component.embedding.EmbeddingCapability` singleton via +:func:`get_embedding_capability`, which never raises — the daemon +runs in keyword-only mode when embedding is unavailable. """ from __future__ import annotations @@ -15,9 +18,8 @@ from fastapi import FastAPI -from everos.component.embedding import build_embedding_provider +from everos.component.embedding import get_embedding_capability from everos.component.tokenizer import build_tokenizer -from everos.config import load_settings from everos.core.lifespan import LifespanProvider from everos.core.observability.logging import get_logger from everos.core.persistence import MemoryRoot @@ -34,15 +36,22 @@ def __init__(self, order: int = 12) -> None: self._orchestrator: CascadeOrchestrator | None = None async def startup(self, app: FastAPI) -> Any: - settings = load_settings() - memory_root = MemoryRoot.default() + memory_root = MemoryRoot.resolve() memory_root.ensure() - embedder = build_embedding_provider(settings.embedding) tokenizer = build_tokenizer() + + capability = get_embedding_capability() + if capability.available: + logger.info("cascade_startup_embed_available") + else: + logger.info( + "cascade_startup_embed_unavailable", + reason="embedding not configured; keyword-only mode", + ) + self._orchestrator = CascadeOrchestrator( memory_root=memory_root, - embedder=embedder, tokenizer=tokenizer, ) await self._orchestrator.start() diff --git a/src/everos/entrypoints/api/lifespans/lancedb.py b/src/everos/entrypoints/api/lifespans/lancedb.py index b2a030bdc..57b0f0f0d 100644 --- a/src/everos/entrypoints/api/lifespans/lancedb.py +++ b/src/everos/entrypoints/api/lifespans/lancedb.py @@ -5,9 +5,22 @@ Importing :mod:`everos.infra.persistence.lancedb` also triggers the side-effect import of ``tables`` so business schemas are loaded (future: preflight registration). + Log hint if unbackfilled (vector IS NULL) rows exist. Shutdown: Close the connection (also clears the table cache). + +Unbackfilled hint: + The informational "you have unbackfilled memory rows" banner runs + an unconditional ``count_rows(filter='vector IS NULL')`` against + every business table on startup. An earlier "marker + limit(1) + probe" amortisation was reverted (round-3 finding #3): the vector + column has no scalar index, so ``limit(1)`` on ``vector IS NULL`` + costs the same full scan as ``count_rows``. On a clean state the + probe scanned the entire empty tail before returning, matching the + cost it was meant to avoid; on a dirty state the probe hit early + and then the full ``count_rows`` ran anyway, doubling the scan. + The marker's ``last_seen_count`` field was written but never read. """ from __future__ import annotations @@ -19,19 +32,60 @@ from everos.core.lifespan import LifespanProvider from everos.core.observability.logging import get_logger from everos.infra.persistence.lancedb import ( + BUSINESS_SCHEMAS_WITH_VECTOR, dispose_connection, ensure_business_indexes, get_connection, + get_table, verify_business_schemas, ) logger = get_logger(__name__) +async def _log_unbackfilled_hint() -> None: + """Warn at startup if there are unbackfilled memory rows. + + Runs an unconditional ``count_rows(filter="vector IS NULL")`` per + business table. The vector column has no scalar index, so + ``count_rows`` and any ``limit(1)`` probe cost the same full scan + — a previous "marker + probe" optimisation (removed here) turned + out to be net-zero on clean state and net-negative on dirty state + (probe hits early, then the full count runs anyway = twice the + scan). + + Per-table failures are logged as warnings and don't interrupt + startup. + """ + total_null = 0 + for schema in BUSINESS_SCHEMAS_WITH_VECTOR: + try: + table = await get_table(schema.TABLE_NAME, schema) + count = await table.count_rows(filter="vector IS NULL") + except Exception as exc: + logger.warning( + "unbackfilled_check_failed", + schema=schema.__name__, + error=repr(exc), + ) + continue + if count > 0: + total_null += count + + if total_null > 0: + banner_logger = get_logger("everos.cli.server") + banner_logger.warning( + "unbackfilled_memory_rows", + count=total_null, + hint="Run `everos cascade backfill` to include them in " + "vector/hybrid search (optional).", + ) + + class LanceDBLifespanProvider(LifespanProvider): """Manage the LanceDB connection + table cache for the app lifecycle. - Startup runs three steps: + Startup runs four steps: 1. ``get_connection`` — lazy-open the async connection. 2. ``verify_business_schemas`` — fail loud if an on-disk table's @@ -39,6 +93,7 @@ class LanceDBLifespanProvider(LifespanProvider): online migration; cascade is rebuildable from md so the recovery is documented as ``rm -rf ~/.everos/.index/lancedb``. 3. ``ensure_business_indexes`` — idempotent FTS index creation. + 4. ``_log_unbackfilled_hint`` — warn if unbackfilled rows exist. """ def __init__(self, order: int = 11) -> None: @@ -48,6 +103,7 @@ async def startup(self, app: FastAPI) -> Any: conn = await get_connection() await verify_business_schemas() await ensure_business_indexes() + await _log_unbackfilled_hint() logger.info("lancedb_ready", uri=conn.uri) return conn diff --git a/src/everos/entrypoints/api/lifespans/parser.py b/src/everos/entrypoints/api/lifespans/parser.py new file mode 100644 index 000000000..1a695cddc --- /dev/null +++ b/src/everos/entrypoints/api/lifespans/parser.py @@ -0,0 +1,60 @@ +"""Parser lifespan provider — warms the optional ``everalgo.parser`` import. + +``everalgo.parser`` is an optional dependency (``everos[multimodal]``). +When installed it pulls in ``pypdf`` / ``python-docx`` / ... on first +import — hundreds of milliseconds to seconds of blocking work. That +cost is fine at startup but unacceptable on the request path, where +:func:`everos.entrypoints.api.routes.health.health` calls +:func:`parser_available` from inside ``async def`` — an event-loop +block there stalls liveness probes and any in-flight requests behind +it. + +This provider resolves :func:`parser_available` once at startup, +priming both Python's ``sys.modules`` cache and the +:func:`functools.lru_cache` wrapping ``parser_available`` itself. If +the extra is not installed, the import fails, the cache stores +``False``, and every subsequent probe is a hot dict lookup. + +Ordered between :class:`LLMLifespanProvider` (``order=8``, hard +Tier-1 requirement) and :class:`SqliteLifespanProvider` (``order=10``) +— the warm is best-effort chassis hygiene that must not delay the +storage stack coming up. +""" + +from __future__ import annotations + +from typing import Any + +from fastapi import FastAPI + +from everos.component.parser import parser_available +from everos.core.lifespan import LifespanProvider +from everos.core.observability.logging import get_logger + +logger = get_logger(__name__) + + +class ParserLifespanProvider(LifespanProvider): + """Warm the ``everalgo.parser`` import once at startup. + + Never fails startup: when the optional extra is not installed, + :func:`parser_available` returns ``False`` and this provider logs + the fact at INFO so operators see a single line rather than the + hidden per-request block that the pre-warm eliminates. + """ + + def __init__(self, order: int = 9) -> None: + # Slot picked to sit between LLM (order=8, hard requirement) and + # sqlite (order=10) — verified against every existing provider's + # `order=` default (see PR #361 review notes on M-c). + super().__init__(name="parser", order=order) + + async def startup(self, app: FastAPI) -> Any: + available = parser_available() + logger.info("parser_lifespan_ready", available=available) + return None + + async def shutdown(self, app: FastAPI) -> None: + # Nothing to tear down — the import is process-scoped and lives + # in `sys.modules` for the process lifetime. + return None diff --git a/src/everos/entrypoints/api/routes/health.py b/src/everos/entrypoints/api/routes/health.py index 6a7eeda22..fc4ba0cfc 100644 --- a/src/everos/entrypoints/api/routes/health.py +++ b/src/everos/entrypoints/api/routes/health.py @@ -4,10 +4,47 @@ from fastapi import APIRouter +from everos import __version__ +from everos.component.capabilities import compute_disabled_features +from everos.component.embedding import get_embedding_capability +from everos.component.multimodal import get_multimodal_llm_capability +from everos.component.parser import parser_available +from everos.component.rerank import get_rerank_capability + router = APIRouter(tags=["health"]) @router.get("/health") -async def health() -> dict[str, str]: - """Liveness probe — returns ``{"status": "ok"}`` with HTTP 200.""" - return {"status": "ok"} +async def health() -> dict: + """Liveness probe with capabilities and disabled features. + + Returns a dict containing: + - status: "ok" + - version: semantic version string + - capabilities: dict mapping capability names to availability booleans + - disabled_features: list of feature names that are disabled due to + missing capabilities + """ + # ``llm`` is hardcoded ``True`` — kept for symmetry with the caps + # dict rather than probed live. Rationale: LLM is a Tier-1 hard + # requirement enforced at startup by ``LLMLifespanProvider`` + # (lifespans/llm.py), which eagerly calls ``get_llm_client()`` and + # raises ``LLMNotConfiguredError`` if credentials are missing — + # FastAPI startup then fails, so ``/health`` is unreachable + # without a working LLM. Any code path that reaches this handler + # therefore has ``get_llm_client()`` returning a real client. If + # the LLM capability is ever downgraded to soft (like embed / + # rerank), swap this literal for a real probe. + caps = { + "llm": True, + "embed": get_embedding_capability().available, + "rerank": get_rerank_capability().available, + "multimodal_llm": get_multimodal_llm_capability().available, + "parser": parser_available(), + } + return { + "status": "ok", + "version": __version__, + "capabilities": caps, + "disabled_features": compute_disabled_features(caps), + } diff --git a/src/everos/entrypoints/api/routes/knowledge.py b/src/everos/entrypoints/api/routes/knowledge.py index 58e74f60e..ab3d1f494 100644 --- a/src/everos/entrypoints/api/routes/knowledge.py +++ b/src/everos/entrypoints/api/routes/knowledge.py @@ -25,15 +25,18 @@ from everos.service.knowledge import KnowledgeExtractor from everalgo.types import ParsedContent -from fastapi import APIRouter, Path, Query, Request, Response, UploadFile +from fastapi import APIRouter, Depends, Path, Query, Request, Response, UploadFile from fastapi.params import Form from pydantic import BaseModel, Field +from everos.component.embedding import get_embedding_capability from everos.component.llm import get_llm_client +from everos.component.rerank import get_rerank_capability from everos.component.utils.datetime import to_display_tz from everos.config import load_settings from everos.core.errors import ( InvalidInputError, + ProviderNotConfiguredError, UnsupportedModalityError, ) from everos.core.persistence import MemoryRoot @@ -59,6 +62,49 @@ # a shared module would be cleaner but is out of scope for this PR. from .memorize import PathSafeId, SuccessEnvelope +_KNOWLEDGE_FEATURE = "knowledge" + + +def _require_knowledge_capabilities() -> None: + """Per-endpoint gate: knowledge writes/search are atomic on embed + rerank. + + ``cascade.registry.build_handlers`` gates the ``knowledge_topic`` / + ``knowledge_document`` cascade handlers off as an atomic pair when + either capability is unavailable, so a document + written without both would enqueue into a cascade kind with no + handler and get stuck ``failed(retryable=False)`` — silently + disappearing from search instead of surfacing an error. Gating the + write and search entrypoints turns that into an immediate 422 + instead of a misleading 200 (upload "succeeds") or a 500 further + down a service call that assumed the provider was there. + + Attached per-endpoint (not router-wide) so read / list / delete / + metadata-patch routes stay reachable after a Tier-3 → Tier-2/1 + downgrade: users can still inspect and clean up their existing docs + (rename, recategorize, delete) even when the providers that would + embed or rerank new content are no longer configured. Title / + category patches only rewrite md frontmatter (and move the doc + directory when category changes) — no embed or rerank code runs on + that path. + + Checks both capabilities (not just embedding) up front so a client + missing only rerank gets a rerank-specific message rather than + passing this gate and failing later inside search. + """ + if not get_embedding_capability().available: + raise ProviderNotConfiguredError( + provider="embedding", + feature=_KNOWLEDGE_FEATURE, + ) + if not get_rerank_capability().available: + raise ProviderNotConfiguredError( + provider="rerank", + feature=_KNOWLEDGE_FEATURE, + ) + + +# Router prefix is /knowledge; app.py mounts it under both /api/v1 and +# /api/v2 (see create_app — v1 retained as a permanent alias). router = APIRouter(prefix="/knowledge", tags=["knowledge"]) @@ -276,6 +322,47 @@ def _reject_oversized_upload(file: UploadFile) -> None: raise InvalidInputError(f"Uploaded file exceeds the {limit_mib:.1f} MiB limit.") +_PLAIN_TEXT_EXTENSIONS: frozenset[str] = frozenset( + {"md", "txt", "rst", "markdown", "text"} +) + + +def _looks_like_utf8_text(file: UploadFile) -> bool: + """Decide whether to skip the parser and go straight to UTF-8 decode. + + A file is treated as plain UTF-8 text when its ``content_type`` starts + with ``text/`` (browsers set this for ``.md`` / ``.txt`` uploads), or + when the mime is missing / ``application/octet-stream`` (typical for + ``curl -F file=@x.md`` without an explicit ``type=`` hint) AND the + filename extension is on a small allowlist. This short-circuits the + parser path — which depends on ``[multimodal]`` being configured — + for the common case of uploading a markdown/plaintext knowledge doc. + """ + mime = (file.content_type or "").lower() + if mime.startswith("text/"): + return True + if mime and mime != "application/octet-stream": + return False + if file.filename and "." in file.filename: + return file.filename.rsplit(".", 1)[-1].lower() in _PLAIN_TEXT_EXTENSIONS + return False + + +def _decode_as_utf8(raw_bytes: bytes) -> ParsedContent: + """Decode ``raw_bytes`` as UTF-8 or raise a caller-friendly error.""" + try: + text = raw_bytes.decode("utf-8") + except UnicodeDecodeError as exc: + raise UnsupportedModalityError( + "File is not UTF-8 text. " + "Install everos[multimodal] for PDF/HTML/DOCX support." + ) from exc + parsed = ParsedContent(text=text) + if not parsed.text or not parsed.text.strip(): + raise InvalidInputError("Uploaded file has no valid content.") + return parsed + + async def _parse_upload( file: UploadFile, *, raw_bytes: bytes | None = None ) -> ParsedContent: @@ -294,6 +381,12 @@ async def _parse_upload( if raw_bytes is None: raw_bytes = await file.read() + # Short-circuit plain-text uploads even when everalgo.parser is + # installed — the parser path depends on [multimodal] being + # configured, which markdown/plaintext ingestion doesn't need. + if _looks_like_utf8_text(file): + return _decode_as_utf8(raw_bytes) + if parser_available(): from everalgo.types import RawFile # Deferred: optional dep @@ -313,17 +406,7 @@ async def _parse_upload( raise InvalidInputError("Uploaded file has no valid content.") return parsed - try: - text = raw_bytes.decode("utf-8") - except UnicodeDecodeError as exc: - raise UnsupportedModalityError( - "File is not UTF-8 text. " - "Install everos[multimodal] for PDF/HTML/DOCX support." - ) from exc - parsed = ParsedContent(text=text) - if not parsed.text or not parsed.text.strip(): - raise InvalidInputError("Uploaded file has no valid content.") - return parsed + return _decode_as_utf8(raw_bytes) def _map_create_result(result: CreateDocumentResult) -> DocumentCreateResponse: @@ -432,7 +515,11 @@ def _map_search_result(result: SearchKnowledgeResult) -> KnowledgeSearchResponse # ── Routes ─────────────────────────────────────────────────────────────────── -@router.post("/documents", status_code=201) +@router.post( + "/documents", + status_code=201, + dependencies=[Depends(_require_knowledge_capabilities)], +) # FastAPI requires flat Form/Query params — ≤5 positional rule exempted. async def create_document_route( request: Request, @@ -450,7 +537,7 @@ async def create_document_route( parsed = await _parse_upload(file, raw_bytes=file_content) source_name = file.filename - knowledge_dir = MemoryRoot.default().knowledge_dir(app_id, project_id) + knowledge_dir = MemoryRoot.resolve().knowledge_dir(app_id, project_id) extractor = _build_extractor() result = await create_document( @@ -466,7 +553,11 @@ async def create_document_route( return SuccessEnvelope(request_id=rid, data=_map_create_result(result)) -@router.put("/documents/{doc_id}", status_code=200) +@router.put( + "/documents/{doc_id}", + status_code=200, + dependencies=[Depends(_require_knowledge_capabilities)], +) # FastAPI requires flat Form/Query params — ≤5 positional rule exempted. async def replace_document_route( request: Request, @@ -484,7 +575,7 @@ async def replace_document_route( file_content = await file.read() parsed = await _parse_upload(file, raw_bytes=file_content) - knowledge_dir = MemoryRoot.default().knowledge_dir(app_id, project_id) + knowledge_dir = MemoryRoot.resolve().knowledge_dir(app_id, project_id) extractor = _build_extractor() result = await replace_document( @@ -576,7 +667,10 @@ async def get_topic_route( return SuccessEnvelope(request_id=rid, data=_map_topic_detail(detail)) -@router.post("/search") +@router.post( + "/search", + dependencies=[Depends(_require_knowledge_capabilities)], +) async def search_knowledge_route( request: Request, req: KnowledgeSearchRequest, diff --git a/src/everos/entrypoints/cli/_log_setup.py b/src/everos/entrypoints/cli/_log_setup.py new file mode 100644 index 000000000..466474635 --- /dev/null +++ b/src/everos/entrypoints/cli/_log_setup.py @@ -0,0 +1,29 @@ +"""Interactive-CLI logging setup. + +Interactive commands (``cascade`` / ``init`` / ``config`` / ``demo``) +default to WARNING so lifecycle events — ``sqlite_engine_built``, +``lancedb_connection_opened``, ``lancedb_table_opened`` and friends — +do not leak into the user-facing flow (Phase banners, ``[y/N]`` +prompts, echo output). ``--verbose`` / ``-v`` opens INFO. + +``server start`` is intentionally unaffected: it calls +:func:`everos.core.observability.logging.configure_logging` directly +so production observability still defaults to INFO. This helper is +only for the interactive path. +""" + +from __future__ import annotations + +from everos.core.observability.logging import configure_logging + + +def configure_cli_logging(verbose: bool = False) -> None: + """Configure structlog + stdlib logging for an interactive CLI command. + + Args: + verbose: True to emit INFO-level lifecycle logs; False (default) + keeps stdout limited to WARNING and above so the interactive + flow stays clean. + """ + level = "INFO" if verbose else "WARNING" + configure_logging(level=level) diff --git a/src/everos/entrypoints/cli/commands/_backfill_cmd.py b/src/everos/entrypoints/cli/commands/_backfill_cmd.py new file mode 100644 index 000000000..f20d47649 --- /dev/null +++ b/src/everos/entrypoints/cli/commands/_backfill_cmd.py @@ -0,0 +1,476 @@ +"""CLI-side orchestration for ``everos cascade backfill``. + +Presentation half of PR #361 review finding M11. The memory-layer phase +runners (``memory/cascade/_backfill.py``) are pure — every user-facing +string, colour, and confirmation prompt lives here, behind the +:class:`~everos.memory.cascade._backfill.BackfillPresenter` Protocol. +:class:`TyperPresenter` is the typer-backed implementation; tests can +substitute any structurally-compatible stub. + +:func:`run_backfill` is the top-level entry point the ``cascade`` +subcommand imports. It builds a :class:`TyperPresenter`, dispatches to +the three phase runners in order, catches Ctrl-C / +:class:`click.exceptions.Abort` / :class:`asyncio.CancelledError`, and +maps every terminal state to a CLI exit code: + +- ``0`` — clean success. +- ``1`` — user declined a confirmation prompt. +- ``2`` — unexpected error mid-phase, or preflight found the required + provider not configured. +- ``3`` — Phase 2/3 short-circuited on the OME jobstore lock (another + ``everos`` process is running). +- ``4`` — every phase ran but at least one row failed embedding even + after per-row fallback; automation can requeue. +- ``130`` — Ctrl-C (SIGINT convention: 128 + 2), including Ctrl-C at + the ``[y/N]`` confirmation prompt (typer/click surfaces that as + :class:`click.exceptions.Abort`, a :class:`RuntimeError` subclass). +""" + +from __future__ import annotations + +import asyncio + +import anyio.to_thread +import click +import typer + +from everos.core.errors import ProviderNotConfiguredError +from everos.core.observability.logging import get_logger + +# Module reference (not name-level imports of the phase runners): tests +# monkeypatch the runners on this module to inject Ctrl-C / boundary +# behaviour, and re-binding names on the source module only affects the +# entrypoints layer if we resolve them through the module attribute at +# call time. +from everos.memory.cascade import _backfill as _phase_runners +from everos.memory.cascade._backfill import ( + BackfillPhase, + _BackfillSummary, + _count_failed_rows, + _format_tokens, +) + +logger = get_logger(__name__) + + +PHASES: tuple[BackfillPhase, ...] = ( + BackfillPhase( + number=1, + slug="vectors", + title="Phase 1 — make older memories searchable by meaning", + detail=( + "Memories created before you configured embedding can only be " + "found by keyword right now. This step lets them be found by " + "meaning too." + ), + ), + BackfillPhase( + number=2, + slug="clusters", + title="Phase 2 — group related memories into topics", + detail=( + "Related memories get clustered together so reflection and " + "advanced retrieval can operate on them." + ), + ), + BackfillPhase( + number=3, + slug="skills", + title="Phase 3 — build the agent skill library", + detail=( + "Distill reusable skills from past agent conversations. Uses " + "your LLM — the most expensive step." + ), + ), +) + + +_EXIT_LABELS: dict[int, str] = { + 0: "SUCCESS", + 1: "ABORTED", + 2: "FAILED", + 3: "SERVER_RUNNING", + 4: "COMPLETED_WITH_FAILURES", + 130: "INTERRUPTED", +} +"""Maps :func:`run_backfill`'s exit codes to the summary block's ``Exit:`` +label. ``3`` marks a Phase 2/3 short-circuit because another process +holds the OME jobstore lock (typically ``everos server start``). ``4`` +marks a partial-success run: every phase ran to completion, but at +least one row failed embedding even after the per-row fallback — +operators consult the ``cascade_backfill_row_embed_failed`` / +``..._row_subject_embed_failed`` log events for the failed row ids. +``130`` is the SIGINT convention (128 + signal number 2).""" + + +def _confirm(detail: str, *, auto_yes: bool) -> bool: + """Ask the user to confirm one phase; ``--yes`` skips the prompt. + + Kept as a module-level helper (rather than a bound method of + :class:`TyperPresenter`) so tests can monkeypatch it on this module + the same way the pre-M11 code was patched on ``_backfill``. The + presenter's :meth:`~TyperPresenter.confirm` delegates here. + """ + if auto_yes: + return True + return typer.confirm(f"Proceed with: {detail}", default=False) + + +def _print_phase_header(phase: BackfillPhase) -> None: + typer.secho(phase.title, bold=True) + typer.echo(f" {phase.detail}") + typer.echo("") + + +def _print_aborted() -> None: + typer.secho("Aborted by user.", fg=typer.colors.RED) + + +def _print_interrupted() -> None: + """Print the Ctrl-C resume hint (see :func:`run_backfill`).""" + typer.secho( + "Interrupted — partial progress was written. Resume by running:", + fg=typer.colors.YELLOW, + ) + typer.echo(" everos cascade backfill --phase --yes") + typer.echo("where is one of: vectors, clusters, skills, all.") + + +def _print_capability_missing_hint(message: str) -> None: + """Emit the friendly toml-hint message shown when a phase's required + provider is not configured. + + The formatted :class:`ProviderNotConfiguredError` string is built by + the memory-layer phase runner and passed through the presenter — so + the CLI and API stay in lock-step on the remediation copy without + duplicating it here. + """ + typer.secho(f" {message}", fg=typer.colors.RED) + + +def _print_server_running_hint() -> None: + """Emit the friendly error shown when preflight or ``engine.start()`` + detects another process holding the OME jobstore lock. + + Copy points the user at ``everos server start`` (the usual holder) + and Phase 1 (``--phase=vectors``), which never touches OME and so + stays fully concurrent with a running server — never at + ``EVEROS_*`` env vars. + """ + typer.secho( + " Backfill Phase 2/3 needs exclusive access to the OME jobstore,", + fg=typer.colors.RED, + ) + typer.secho( + " but another everos process is holding it (typically your running", + fg=typer.colors.RED, + ) + typer.secho( + " `everos server start`).", + fg=typer.colors.RED, + ) + typer.echo("") + typer.echo(" Stop your everos server first, then re-run backfill.") + typer.echo(" (Phase 1 --phase=vectors already runs concurrently with the server;") + typer.echo(" only Phase 2/3 need offline mode. See spec §10.)") + + +def _print_vectors_estimate(total_rows: int, total_tokens: int) -> None: + typer.echo(f" memories to process: {total_rows:,}") + typer.echo(f" input tokens: {_format_tokens(total_tokens)}\n") + typer.echo( + " Uses your embedding provider — cost depends on its per-token pricing.\n" + ) + + +def _print_clusters_estimate(episode_count: int, case_count: int) -> None: + typer.echo(f" episodes to cluster: {episode_count:,}") + typer.echo(f" agent cases to cluster: {case_count:,}\n") + typer.echo( + " Uses your embedding provider (and LLM for agent-case merges) — " + "cost depends on provider pricing.\n" + ) + + +def _print_skills_estimate(cases: int, clusters: int) -> None: + typer.echo(f" clusters to extract skills from: {clusters:,}") + typer.echo(f" agent cases to replay: {cases:,}\n") + typer.echo(" Uses your LLM — cost depends on its per-token pricing.\n") + + +def _print_summary(summary: _BackfillSummary, *, exit_code: int) -> None: + """Print the consolidated post-run summary block. + + Field labels are chosen from what each phase's result dataclass + actually tracks — e.g. Phase 3 reports "agent cases processed" + (``events_emitted``) rather than "clusters processed", since the + distinct cluster count isn't retained on + :class:`~everos.memory.cascade._backfill._SkillPhaseResult`. + + On ``exit_code == 4`` (COMPLETED_WITH_FAILURES) also emits a hint + pointing operators at the per-row failure log events so they can + recover the failed row ids. + """ + typer.echo("") + typer.secho("Backfill summary", bold=True) + typer.echo("-" * 44) + if summary.vectors is not None: + r = summary.vectors + typer.echo( + f" Phase 1 (vectors) — {r.rows_processed:,} rows / " + f"{r.rows_failed:,} failed / {_format_tokens(r.tokens_embedded)}" + ) + if summary.clusters is not None: + c = summary.clusters + created = c.clusters_after - c.clusters_before + typer.echo( + f" Phase 2 (clusters) — {c.events_emitted:,} events emitted / " + f"{created:,} clusters created" + ) + if summary.skills is not None: + s = summary.skills + extracted = s.skills_after - s.skills_before + typer.echo( + f" Phase 3 (skills) — {s.events_emitted:,} agent cases " + f"processed / {extracted:,} skills extracted" + ) + typer.echo("-" * 44) + label = _EXIT_LABELS.get(exit_code, "UNKNOWN") + typer.echo(f" Exit: {label} ({exit_code})") + if exit_code == 4: + total_failed = _count_failed_rows(summary) + typer.secho( + f" {total_failed:,} rows failed embedding — see log events " + "`cascade_backfill_row_embed_failed` for row_ids and text prefixes.", + fg=typer.colors.YELLOW, + ) + + +class TyperPresenter: + """typer-backed :class:`BackfillPresenter` — the CLI's user-I/O seam. + + Every method delegates to a module-level ``_print_*`` helper or to + :func:`_confirm`. Kept as a small class (rather than a module of + free functions) so alternative front-ends (e.g. a JSON-emitting + CI-mode presenter) can slot in without touching the memory layer. + """ + + def phase_header(self, phase: BackfillPhase) -> None: + _print_phase_header(phase) + + def nothing_to_backfill(self, message: str) -> None: + colour = ( + typer.colors.YELLOW + if "could not be read" in message + else (typer.colors.GREEN) + ) + typer.secho(message, fg=colour) + + def capability_missing(self, *, provider: str, feature: str, message: str) -> None: + _print_capability_missing_hint(message) + + def server_running(self) -> None: + _print_server_running_hint() + + def estimate_vectors(self, rows: int, tokens: int) -> None: + _print_vectors_estimate(rows, tokens) + + def estimate_clusters(self, episodes: int, cases: int) -> None: + _print_clusters_estimate(episodes, cases) + + def estimate_skills(self, cases: int, clusters: int) -> None: + _print_skills_estimate(cases, clusters) + + async def confirm(self, prompt: str, *, auto_yes: bool) -> bool: + # ``typer.confirm`` is blocking (reads from stdin); offload so a + # slow / interactive user doesn't hold the event loop. Any + # ``click.exceptions.Abort`` raised on Ctrl-C at the prompt + # propagates to :func:`run_backfill` which catches it and maps + # to exit 130. + return await anyio.to_thread.run_sync( + lambda: _confirm(prompt, auto_yes=auto_yes) + ) + + def emit_progress(self, done: int, total: int) -> None: + typer.echo(f" emitted {done:,} / {total:,} event(s)") + + def row_progress(self, table_name: str, done: int, total: int) -> None: + typer.echo(f" {table_name}: processed {done:,} / {total:,} rows") + + def phase_1_complete(self, rows_processed: int, rows_failed: int) -> None: + typer.secho( + f"phase 1 complete — {rows_processed:,} memories now " + "searchable by meaning" + + (f" ({rows_failed:,} failed)" if rows_failed else ""), + fg=typer.colors.GREEN, + ) + + def phase_2_complete(self, before: int, after: int, emitted: int) -> None: + typer.secho( + f"phase 2 complete — clusters {before:,} -> {after:,} " + f"({emitted:,} events processed)", + fg=typer.colors.GREEN, + ) + + def phase_3_complete( + self, skills_before: int, skills_after: int, emitted: int + ) -> None: + typer.secho( + f"phase 3 complete — skills {skills_before:,} -> {skills_after:,} " + f"({emitted:,} events processed)", + fg=typer.colors.GREEN, + ) + + +async def run_backfill(*, phase: str, auto_yes: bool) -> int: + """Run backfill phases interactively; return an exit code. + + ``phase`` is ``"all"`` or one of :data:`PHASES`'s ``slug`` values. + Exit codes: see :data:`_EXIT_LABELS`. + + Every exit path — success, abort, error, or interrupt — ends with a + consolidated summary block (:func:`_print_summary`) covering every + phase that actually ran. The interrupt is only ever caught here, at + the outer loop: a phase body mid-batch must let it propagate rather + than swallow it, so a Ctrl-C during Phase 1 doesn't get silently + absorbed as "phase 1 failed, try phase 2 anyway". + + Both ``KeyboardInterrupt`` and ``asyncio.CancelledError`` are caught: + on Python 3.11+, ``asyncio.Runner`` (which backs ``asyncio.run``) + translates a real SIGINT into ``main_task.cancel()``, which raises + ``CancelledError`` — not ``KeyboardInterrupt`` — at the currently + suspended ``await``. Handling it here lets the task complete + normally with exit code 130 instead of leaking the cancellation out + to ``asyncio.run`` (which would re-synthesize a bare + ``KeyboardInterrupt`` that typer/click turns into ``Abort``, exit 1, + with no resume hint printed). Catching ``CancelledError`` + unconditionally is safe here: nothing in this call tree cancels the + task for reasons other than an interrupt. + """ + presenter = TyperPresenter() + selected = [p for p in PHASES if phase == "all" or p.slug == phase] + summary = _BackfillSummary() + current_phase: BackfillPhase | None = None + try: + for p in selected: + current_phase = p + _print_phase_header(p) + if p.slug == "vectors": + result = await _phase_runners._run_phase_vectors( + auto_yes=auto_yes, presenter=presenter + ) + summary.vectors = result + if result.blocked_by_capability: + logger.warning( + "cascade_backfill_blocked_by_capability", + phase=p.slug, + provider=result.blocked_by_capability, + ) + _print_summary(summary, exit_code=2) + return 2 + if result.aborted: + # Policy (PR #361 round-3 review #11, accepted): + # typing ``n`` at Phase 1's confirmation aborts the + # ENTIRE ``--phase all`` run — not just Phase 1. + # Rationale: ``--phase all`` is opt-in-to-run- + # everything. Explicitly declining Phase 1's cost + # estimate is a decisive "no"; interpreting it as + # "skip Phase 1 but still run Phase 2/3" would spend + # LLM/embed budget on phases the user did not see + # estimates for. Users who want only Phase 2 or 3 + # invoke ``--phase clusters`` or ``--phase skills`` + # explicitly (and see that phase's own estimate). + logger.info("cascade_backfill_aborted", phase=p.slug) + _print_aborted() + _print_summary(summary, exit_code=1) + return 1 + continue + if p.slug == "clusters": + cluster_result = await _phase_runners._run_phase_clusters( + auto_yes=auto_yes, presenter=presenter + ) + summary.clusters = cluster_result + if cluster_result.blocked_by_capability: + logger.warning( + "cascade_backfill_blocked_by_capability", + phase=p.slug, + provider=cluster_result.blocked_by_capability, + ) + _print_summary(summary, exit_code=2) + return 2 + if cluster_result.blocked_by_server: + logger.warning("cascade_backfill_blocked_by_server", phase=p.slug) + _print_summary(summary, exit_code=3) + return 3 + if cluster_result.aborted: + logger.info("cascade_backfill_aborted", phase=p.slug) + _print_aborted() + _print_summary(summary, exit_code=1) + return 1 + continue + if p.slug == "skills": + skill_result = await _phase_runners._run_phase_skills( + auto_yes=auto_yes, presenter=presenter + ) + summary.skills = skill_result + if skill_result.blocked_by_capability: + logger.warning( + "cascade_backfill_blocked_by_capability", + phase=p.slug, + provider=skill_result.blocked_by_capability, + ) + _print_summary(summary, exit_code=2) + return 2 + if skill_result.blocked_by_server: + logger.warning("cascade_backfill_blocked_by_server", phase=p.slug) + _print_summary(summary, exit_code=3) + return 3 + if skill_result.aborted: + logger.info("cascade_backfill_aborted", phase=p.slug) + _print_aborted() + _print_summary(summary, exit_code=1) + return 1 + continue + # ``click.exceptions.Abort`` fires when ``typer.confirm`` (== click.confirm) + # catches KeyboardInterrupt at the y/N prompt and re-raises it as + # ``Abort`` (a RuntimeError subclass, NOT a KeyboardInterrupt). Catching + # it alongside KeyboardInterrupt / CancelledError keeps Ctrl-C at the + # prompt on the interrupt path (exit 130 with resume hint) rather than + # falling through to the generic ``except Exception`` (exit 2). + except (KeyboardInterrupt, asyncio.CancelledError, click.exceptions.Abort): + logger.warning( + "cascade_backfill_interrupted", + phase=current_phase.slug if current_phase else phase, + ) + _print_interrupted() + _print_summary(summary, exit_code=130) + return 130 + except ProviderNotConfiguredError as exc: + # Defensive backstop: phase runners preflight capability and + # return ``blocked_by_capability`` on their own. Reaching this + # branch means a code path bypassed the preflight — surface the + # remediation message instead of letting the broad ``except + # Exception`` below swallow it into "see logs for details". + logger.error( + "cascade_backfill_provider_error_escaped_preflight", + phase=current_phase.slug if current_phase else phase, + provider=exc.provider, + ) + typer.secho(f" {exc}", fg=typer.colors.RED) + _print_summary(summary, exit_code=2) + return 2 + except Exception: + logger.exception("cascade_backfill_phase_failed", phase=phase) + typer.secho("Backfill failed — see logs for details.", fg=typer.colors.RED) + _print_summary(summary, exit_code=2) + return 2 + total_failed = _count_failed_rows(summary) + if total_failed > 0: + logger.warning( + "cascade_backfill_completed_with_failures", + total_failed=total_failed, + ) + _print_summary(summary, exit_code=4) + return 4 + _print_summary(summary, exit_code=0) + return 0 diff --git a/src/everos/entrypoints/cli/commands/cascade.py b/src/everos/entrypoints/cli/commands/cascade.py index 8d27e8aa0..1d2c456d4 100644 --- a/src/everos/entrypoints/cli/commands/cascade.py +++ b/src/everos/entrypoints/cli/commands/cascade.py @@ -11,6 +11,10 @@ - ``cascade fix`` — list every ``failed`` row. With ``--apply``, also reset ``retryable=TRUE`` rows back to ``pending`` and drain the worker once so the retry actually runs before the command returns. +- ``cascade backfill`` — one-shot Tier 1 → Tier 2/3 migration: re-embed + vectors, build clusters, extract skills. See + :func:`everos.entrypoints.cli.commands._backfill_cmd.run_backfill` + for the phase orchestration. CLI is in-process (12 doc §7.1 + 16 doc §9.2): it constructs the same :class:`CascadeOrchestrator` as the daemon but only calls @@ -21,6 +25,7 @@ from __future__ import annotations import asyncio +import enum import os from contextlib import asynccontextmanager from pathlib import Path @@ -29,11 +34,13 @@ import typer from sqlmodel import SQLModel -from everos.component.embedding import build_embedding_provider +from everos.component.embedding import get_embedding_capability from everos.component.tokenizer import build_tokenizer from everos.component.utils.datetime import to_display_tz -from everos.config import load_settings +from everos.core.observability.logging import get_logger from everos.core.persistence import MemoryRoot +from everos.entrypoints.cli._log_setup import configure_cli_logging +from everos.entrypoints.cli.commands._backfill_cmd import run_backfill from everos.infra.persistence.lancedb import ( dispose_connection, ensure_business_indexes, @@ -47,6 +54,8 @@ ) from everos.memory.cascade import CascadeOrchestrator, match_kind +logger = get_logger(__name__) + app = typer.Typer( name="cascade", help="Inspect and operate the md → LanceDB sync queue", @@ -61,12 +70,57 @@ def _cascade_callback( "--root", help="Memory root directory (env: EVEROS_ROOT, default: ~/.everos)", ), + verbose: bool = typer.Option( + False, + "--verbose", + "-v", + help="Emit INFO-level lifecycle logs (default: WARNING only).", + ), ) -> None: - """Set memory root before any cascade subcommand runs.""" + """Set memory root and log level before any cascade subcommand runs.""" + configure_cli_logging(verbose=verbose) + _apply_root_env(root) + + +def _apply_root_env(root: str | None) -> None: + """Set ``EVEROS_ROOT`` env var if a ``--root`` path was passed. + + Called from both the group callback (subcommand-independent path) + and from each subcommand (subcommand-level ``--root``). Later calls + override earlier ones — a subcommand's ``--root`` wins over the + group callback's when both are supplied. + """ if root: os.environ["EVEROS_ROOT"] = root +def _apply_verbose_logging(verbose: bool | None) -> None: + """Re-apply the CLI log level from a subcommand-level ``--verbose``. + + ``None`` means the subcommand did not receive its own ``--verbose`` + flag — leave whatever level the group callback set (default + WARNING, or INFO if the user passed ``cascade --verbose``). A + ``True``/``False`` value came from the subcommand and wins over the + group callback (later configuration overrides earlier). + + Mirrors :func:`_apply_root_env`'s "either position works" pattern + for symmetry — users can write ``everos cascade --verbose status`` + or ``everos cascade status --verbose`` and get the same behaviour. + """ + if verbose is not None: + configure_cli_logging(verbose=verbose) + + +_ROOT_OPTION_HELP = ( + "Memory root directory (alias for `cascade --root`; either position works)." +) + +_VERBOSE_OPTION_HELP = ( + "Emit INFO-level lifecycle logs (alias for `cascade --verbose`; " + "either position works)." +) + + # ── shared runtime context ─────────────────────────────────────────────── @@ -92,14 +146,21 @@ async def _runtime(): # type: ignore[no-untyped-def] def _build_orchestrator() -> CascadeOrchestrator: - settings = load_settings() - memory_root = MemoryRoot.default() + memory_root = MemoryRoot.resolve() memory_root.ensure() - embedder = build_embedding_provider(settings.embedding) tokenizer = build_tokenizer() + + capability = get_embedding_capability() + if capability.available: + logger.info("cli_cascade_embed_available") + else: + logger.info( + "cli_cascade_embed_unavailable", + reason="embedding not configured; keyword-only mode", + ) + return CascadeOrchestrator( memory_root=memory_root, - embedder=embedder, tokenizer=tokenizer, ) @@ -116,8 +177,18 @@ def sync( "If omitted, only the existing queue is drained.", ), ] = None, + root: Annotated[ + str | None, + typer.Option("--root", help=_ROOT_OPTION_HELP), + ] = None, + verbose: Annotated[ + bool | None, + typer.Option("--verbose", "-v", help=_VERBOSE_OPTION_HELP), + ] = None, ) -> None: """Drain the cascade queue (and optionally re-enqueue a path first).""" + _apply_root_env(root) + _apply_verbose_logging(verbose) async def _run() -> None: async with _runtime(): @@ -144,8 +215,19 @@ async def _run() -> None: @app.command("status") -def status() -> None: +def status( + root: Annotated[ + str | None, + typer.Option("--root", help=_ROOT_OPTION_HELP), + ] = None, + verbose: Annotated[ + bool | None, + typer.Option("--verbose", "-v", help=_VERBOSE_OPTION_HELP), + ] = None, +) -> None: """Print the queue / LSN summary.""" + _apply_root_env(root) + _apply_verbose_logging(verbose) async def _run() -> None: async with _runtime(): @@ -190,8 +272,18 @@ def fix( help="Re-enqueue every `retryable=TRUE` row and drain the worker.", ), ] = False, + root: Annotated[ + str | None, + typer.Option("--root", help=_ROOT_OPTION_HELP), + ] = None, + verbose: Annotated[ + bool | None, + typer.Option("--verbose", "-v", help=_VERBOSE_OPTION_HELP), + ] = None, ) -> None: """List failed rows (default) or re-enqueue retryable ones (``--apply``).""" + _apply_root_env(root) + _apply_verbose_logging(verbose) async def _run() -> None: async with _runtime(): @@ -234,6 +326,91 @@ async def _run() -> None: asyncio.run(_run()) +# ── backfill ───────────────────────────────────────────────────────────── + + +class _BackfillPhaseOption(enum.StrEnum): + """CLI-facing ``--phase`` choices — mirrors ``BackfillPhase.slug``.""" + + VECTORS = "vectors" + CLUSTERS = "clusters" + SKILLS = "skills" + ALL = "all" + + +@app.command("backfill") +def backfill( + phase: Annotated[ + _BackfillPhaseOption, + typer.Option( + "--phase", + help=( + "Which phase to run — vectors (re-embed rows so they become " + "searchable by meaning, not just keyword), clusters (group " + "re-embedded episodes/agent cases into topics), skills " + "(extract reusable agent skills from clustered cases), or " + "all (run every phase in order). Default: all." + ), + ), + ] = _BackfillPhaseOption.ALL, + yes: Annotated[ + bool, + typer.Option( + "--yes", + "-y", + help=( + "Auto-confirm every phase's prompt instead of pausing for a " + "y/n answer. Without this flag, each phase shows its " + "token-count estimate and waits for confirmation first." + ), + ), + ] = False, + root: Annotated[ + str | None, + typer.Option("--root", help=_ROOT_OPTION_HELP), + ] = None, + verbose: Annotated[ + bool | None, + typer.Option("--verbose", "-v", help=_VERBOSE_OPTION_HELP), + ] = None, +) -> None: + """Backfill embeddings, clusters, and agent skills after upgrading from + Tier 1 (LLM-only) to Tier 2/3 (embedding configured). + + Runs up to three phases in order (see ``--phase``), each showing a + token-count estimate and confirming before doing any work. The + estimate is an input-token count only (K/M shorthand) — never a + price, since per-token pricing is provider-specific and can change. + + Ctrl-C is safe: interrupting mid-phase prints a resume hint and exits + 130. Every row and phase already written stays written, so + re-running the same ``--phase`` (or ``--phase all``) afterwards only + picks up what's left — nothing is redone from scratch. + """ + _apply_root_env(root) + _apply_verbose_logging(verbose) + + async def _run() -> int: + async with _runtime(): + return await run_backfill(phase=phase.value, auto_yes=yes) + + try: + code = asyncio.run(_run()) + except KeyboardInterrupt: + # Safety net: run_backfill already catches CancelledError and + # returns 130 cleanly in the common case, but asyncio.Runner can + # still re-raise a bare KeyboardInterrupt past asyncio.run() on a + # second Ctrl-C. Without this, typer/click would convert it to + # Abort and exit 1 with no resume hint. + typer.secho( + "Interrupted — partial progress was written. Resume by running:", + fg=typer.colors.YELLOW, + ) + typer.echo(" everos cascade backfill --phase --yes") + raise typer.Exit(code=130) from None + raise typer.Exit(code=code) + + # ── helpers ────────────────────────────────────────────────────────────── @@ -244,7 +421,7 @@ def _resolve_relative(p: Path) -> str: must match that convention before calling :meth:`force_enqueue`. Outside-the-root inputs surface as an error in the caller. """ - memory_root = MemoryRoot.default() + memory_root = MemoryRoot.resolve() absolute = p.expanduser().resolve() try: rel = absolute.relative_to(memory_root.root) diff --git a/src/everos/entrypoints/cli/commands/config_cmd.py b/src/everos/entrypoints/cli/commands/config_cmd.py index 3850a8f24..2bff47132 100644 --- a/src/everos/entrypoints/cli/commands/config_cmd.py +++ b/src/everos/entrypoints/cli/commands/config_cmd.py @@ -7,6 +7,7 @@ import typer from everos.config.settings import resolve_root +from everos.entrypoints.cli._log_setup import configure_cli_logging app = typer.Typer( name="config", @@ -14,6 +15,20 @@ no_args_is_help=True, ) + +@app.callback() +def _config_callback( + verbose: bool = typer.Option( + False, + "--verbose", + "-v", + help="Emit INFO-level lifecycle logs (default: WARNING only).", + ), +) -> None: + """Set log level before any config subcommand runs.""" + configure_cli_logging(verbose=verbose) + + _SECRET_FIELDS = {"api_key"} _SECTION_NAMES = ( diff --git a/src/everos/entrypoints/cli/commands/demo.py b/src/everos/entrypoints/cli/commands/demo.py index 12c3463a7..b3ab57cb7 100644 --- a/src/everos/entrypoints/cli/commands/demo.py +++ b/src/everos/entrypoints/cli/commands/demo.py @@ -15,6 +15,7 @@ from rich.panel import Panel from everos.component.utils.datetime import get_utc_now +from everos.entrypoints.cli._log_setup import configure_cli_logging from everos.entrypoints.tui.demo.data import ( DEFAULT_MEMORY_SEED, DEFAULT_QUERY, @@ -64,8 +65,15 @@ def demo( "--server-url", help="EverOS server URL used by --live.", ), + verbose: bool = typer.Option( + False, + "--verbose", + "-v", + help="Emit INFO-level lifecycle logs (default: WARNING only).", + ), ) -> None: """Launch the EverOS first-memory Textual TUI.""" + configure_cli_logging(verbose=verbose) if live: _run_live_demo( cinematic=cinematic, diff --git a/src/everos/entrypoints/cli/commands/init_cmd.py b/src/everos/entrypoints/cli/commands/init_cmd.py index 092d13e6b..5854e14e0 100644 --- a/src/everos/entrypoints/cli/commands/init_cmd.py +++ b/src/everos/entrypoints/cli/commands/init_cmd.py @@ -17,6 +17,7 @@ import typer from everos.config.settings import resolve_root +from everos.entrypoints.cli._log_setup import configure_cli_logging _EVEROS_TEMPLATE = Path(__file__).resolve().parents[3] / "config" / "default.toml" _OME_TEMPLATE = Path(__file__).resolve().parents[3] / "config" / "default_ome.toml" @@ -42,6 +43,12 @@ def init( "--print", help="Print the everos.toml template to stdout instead of writing to disk.", ), + verbose: bool = typer.Option( + False, + "--verbose", + "-v", + help="Emit INFO-level lifecycle logs (default: WARNING only).", + ), ) -> None: """Generate starter configuration files. @@ -57,6 +64,7 @@ def init( - 0 — files created successfully (or printed to stdout). - 1 — files already exist and ``--force`` was not given. """ + configure_cli_logging(verbose=verbose) if print_: import sys diff --git a/src/everos/infra/ome/_stores/storage.py b/src/everos/infra/ome/_stores/storage.py index ced3f7142..b6635bf42 100644 --- a/src/everos/infra/ome/_stores/storage.py +++ b/src/everos/infra/ome/_stores/storage.py @@ -1,6 +1,6 @@ """OME SQLite storage — schema initialization + connection factory. -Single file (default ``MemoryRoot.default().ome_db`` ≡ +Single file (default ``MemoryRoot.resolve().ome_db`` ≡ ``/.index/sqlite/ome.db``). Holds 3 OME-managed tables (counter_store / idle_store / run_record); APS jobstore table is created by APScheduler itself when its SQLAlchemyJobStore connects. diff --git a/src/everos/infra/ome/config.py b/src/everos/infra/ome/config.py index eec0ef35f..d94671275 100644 --- a/src/everos/infra/ome/config.py +++ b/src/everos/infra/ome/config.py @@ -16,7 +16,7 @@ def _default_jobstore_path() -> Path: - return MemoryRoot.default().ome_db + return MemoryRoot.resolve().ome_db class CounterOverride(BaseModel): @@ -82,7 +82,7 @@ class OMEConfig(BaseModel): default_factory=_default_jobstore_path, description="SQLite DB path holding OME's own state (run records, " "counter store, idle store). Defaults to " - "``MemoryRoot.default().ome_db`` (``/.index/sqlite/ome.db``).", + "``MemoryRoot.resolve().ome_db`` (``/.index/sqlite/ome.db``).", ) aps_jobstore_path: Path | None = Field( default=None, @@ -126,6 +126,25 @@ class OMEConfig(BaseModel): "fresh run_id.", ), ] = 1800 + crash_recovery_enabled: bool = Field( + default=True, + description=( + "Run ``OfflineEngine._run_crash_recovery`` on ``engine.start()``. " + "Default ``True`` — the primary server engine " + "(``service.memorize._get_engine``) needs to resume its own " + "prior sessions' RUNNING work after a crash / restart. " + "Set to ``False`` for one-shot engines that share the jobstore " + "path (e.g. ``memory.cascade._backfill._build_cluster_engine`` " + "and ``_build_skill_engine``). Those engines register only the " + "Phase 2/3 subset of strategies; if they ran crash recovery on " + "startup they would re-enqueue the server's stale RUNNING rows " + "into their own APS scheduler, whose registry doesn't know " + "those strategy names — causing the re-enqueued event to be " + "permanently lost via ``KeyError`` at dispatch time. The " + "server's stale rows stay in ``run_record`` untouched; the " + "next server restart resumes them normally." + ), + ) config_path: Path | None = Field( default=None, description="Path to ome.toml for per-strategy overrides. None " diff --git a/src/everos/infra/ome/engine.py b/src/everos/infra/ome/engine.py index 6c8c8407f..52f2409a0 100644 --- a/src/everos/infra/ome/engine.py +++ b/src/everos/infra/ome/engine.py @@ -290,7 +290,8 @@ async def start(self) -> None: self._idle_event.set() self._launch_scheduler() _ENGINES[self._engine_id] = self - await self._run_crash_recovery() + if self._config.crash_recovery_enabled: + await self._run_crash_recovery() self._register_scheduled_jobs() self._start_config_reloader() self._started = True @@ -346,6 +347,13 @@ def _launch_scheduler(self) -> None: async def _run_crash_recovery(self) -> None: """Scan ``run_record`` for stale RUNNING rows and re-enqueue them. + Runs on :meth:`start` when :attr:`OMEConfig.crash_recovery_enabled` + is ``True`` (default). One-shot engines that register a subset of + strategies (backfill Phase 2/3) opt out via + ``crash_recovery_enabled=False`` to avoid re-enqueueing the + server's stale rows into their own scheduler, whose registry + doesn't contain those strategy names. + Treats rows whose ``started_at`` is older than ``crash_recovery_timeout_seconds`` as crashes from a previous engine session: they are marked CRASHED and re-added to APS with diff --git a/src/everos/infra/persistence/lancedb/__init__.py b/src/everos/infra/persistence/lancedb/__init__.py index 8f8b3c589..d7e2967b1 100644 --- a/src/everos/infra/persistence/lancedb/__init__.py +++ b/src/everos/infra/persistence/lancedb/__init__.py @@ -28,7 +28,7 @@ import datetime as dt from everos.core.observability.logging import get_logger -from everos.core.persistence import MemoryRoot +from everos.core.persistence import BaseLanceTable, MemoryRoot, memory_root_lock # Importing ``tables`` registers every business :class:`BaseLanceTable` # schema so callers can rely on the package alone to surface every schema. @@ -75,6 +75,11 @@ class LanceDBSchemaMismatchError(RuntimeError): """ +class LanceDBMigrationError(RuntimeError): + """Raised when :func:`migrate_table_schemas` cannot complete and + startup must abort rather than continue in a half-migrated state.""" + + _FTS_INDEX_SCHEMA_VERSION = 2 """Bump when the FTS index build config changes so existing on-disk indexes get rebuilt at startup. v2 = ``with_position=False`` (see @@ -96,41 +101,180 @@ async def migrate_fts_indexes() -> None: crashed-optimize churn left behind. Guarded by a version marker in the LanceDB dir so it runs at most once per bump; the rebuild is O(N) but only on the first startup after upgrade. + + Cross-process serialization: wrapped in :func:`memory_root_lock` so a + server startup racing with a concurrent ``everos cascade`` command + (both call this via :func:`ensure_business_indexes`) can't attempt + two concurrent rebuild passes. First process runs the migration and + writes the marker; second waits for the lock, re-checks the marker + on entry, and no-ops. Removes the same TOCTOU window between marker + read and the drop-index/rebuild loop that :issue:`M7` from the + PR #361 review flagged on the sibling schema migration. """ logger = get_logger(__name__) - marker = MemoryRoot.default().lancedb_dir / ".fts_index_version" - try: - current = int(marker.read_text().strip()) if marker.exists() else 0 - except (ValueError, OSError): - current = 0 - if current >= _FTS_INDEX_SCHEMA_VERSION: - return - logger.info("fts_index_migration_started", target=_FTS_INDEX_SCHEMA_VERSION) - for schema in _BUSINESS_SCHEMAS: - if not schema.BM25_FIELDS: - continue - table = await get_table(schema.TABLE_NAME, schema) - # Drop existing indexes (everos only builds FTS here; mirrors - # LanceRepoBase.rebuild_indexes) then rebuild with the new config. - for idx in await table.list_indices(): - await table.drop_index(idx.name) - await schema.ensure_fts_indexes(table) - # Reclaim the orphaned index dirs + data fragments the crashed - # optimize loop piled up. Safe now: the crashing index is gone, - # so compaction no longer decodes a position List. - with contextlib.suppress(Exception): - await table.optimize(cleanup_older_than=dt.timedelta(seconds=0)) - marker.write_text(str(_FTS_INDEX_SCHEMA_VERSION)) - logger.info("fts_index_migration_done", version=_FTS_INDEX_SCHEMA_VERSION) + memory_root = MemoryRoot.resolve() + async with memory_root_lock(memory_root): + marker = memory_root.lancedb_dir / ".fts_index_version" + try: + current = int(marker.read_text().strip()) if marker.exists() else 0 + except (ValueError, OSError): + current = 0 + if current >= _FTS_INDEX_SCHEMA_VERSION: + # Another process finished the migration while we waited + # for the lock. Nothing to do. + return + logger.info("fts_index_migration_started", target=_FTS_INDEX_SCHEMA_VERSION) + for schema in _BUSINESS_SCHEMAS: + if not schema.BM25_FIELDS: + continue + table = await get_table(schema.TABLE_NAME, schema) + # Drop existing indexes (everos only builds FTS here; mirrors + # LanceRepoBase.rebuild_indexes) then rebuild with the new config. + for idx in await table.list_indices(): + await table.drop_index(idx.name) + await schema.ensure_fts_indexes(table) + # Reclaim the orphaned index dirs + data fragments the crashed + # optimize loop piled up. Safe now: the crashing index is gone, + # so compaction no longer decodes a position List. + with contextlib.suppress(Exception): + await table.optimize(cleanup_older_than=dt.timedelta(seconds=0)) + marker.write_text(str(_FTS_INDEX_SCHEMA_VERSION)) + logger.info("fts_index_migration_done", version=_FTS_INDEX_SCHEMA_VERSION) + + +_TABLE_SCHEMA_VERSION = 2 +"""Bump when column nullability or presence changes on business tables. +v2 = ``vector: Vector(_DIM) | None`` on the 6 business tables that carry +a vector column (see :data:`BUSINESS_SCHEMAS_WITH_VECTOR`). + +.. note:: + + :func:`migrate_table_schemas` currently hard-codes the v2 alter — it + does not implement per-version dispatch. Bumping this constant to + ``3`` without first adding a dispatch table will short-circuit the + v3 migration entirely (the current body only compares + ``current >= _TABLE_SCHEMA_VERSION`` and then runs the v2-specific + alter unconditionally). Any future v3 migration MUST: + + 1. Add a ``_MIGRATIONS: dict[int, Callable[[], Awaitable[None]]]`` + registry mapping target version → migration coroutine. + 2. Rewrite :func:`migrate_table_schemas` to run every migration from + ``current + 1`` up to ``_TABLE_SCHEMA_VERSION`` in order, updating + the marker after each step so a mid-migration crash resumes at + the next step instead of replaying earlier ones. + + Accepted as a design gap in PR #361 review finding #5 — not blocking + today because there is no v3. +""" + + +BUSINESS_SCHEMAS_WITH_VECTOR: tuple[type[BaseLanceTable], ...] = ( + Episode, + AtomicFact, + Foresight, + AgentCase, + AgentSkill, + KnowledgeTopic, +) +"""Business schemas whose ``vector`` column needs the v2 nullability +migration. ``Episode.subject_vector`` was already nullable in v1.1.1 and +is intentionally left out of this migration. ``UserProfile`` has no +vector column; ``knowledge_document`` has no LanceDB table.""" + + +async def migrate_table_schemas() -> None: + """One-time ``alter_columns`` making ``vector`` nullable on business tables. + + Soft-dependency embedding mode lets cascade write ``vector=None`` for + rows it cannot embed; the on-disk column must allow that. LanceDB's + ``alter_columns`` is metadata-only (no data movement, no re-embed), + so this runs in milliseconds. Guarded by ``.table_schema_version`` + so it runs at most once per version bump; per-table nullable check + is defense-in-depth so an already-migrated (or freshly-created, + already-nullable) table is a no-op rather than a redundant alter. + + Cross-process serialization: wrapped in :func:`memory_root_lock` so + a server startup racing with a concurrent ``everos cascade`` command + (both call this via :func:`ensure_business_indexes`) doesn't attempt + two concurrent alter passes. First process runs the migration and + writes the marker; second waits for the lock, re-checks the marker + on entry, and no-ops. Removes the TOCTOU window between marker + read and alter that :issue:`M7` from the PR #361 review flagged. + + Fail-loud semantics preserved: if ``alter_columns`` genuinely fails + for any table (LanceDB version mismatch, corrupted index), raises + :class:`LanceDBMigrationError`. Startup aborts rather than + continuing in a half-migrated state — cascade would otherwise write + ``vector=None`` (soft-dependency embedding) into a still NOT-NULL + column and every row would silently fail. Recovery escalates from + a plain restart (transient hiccup) to wiping the LanceDB index + directory (rebuildable from md, the SoT). + """ + logger = get_logger(__name__) + memory_root = MemoryRoot.resolve() + async with memory_root_lock(memory_root): + marker = memory_root.lancedb_dir / ".table_schema_version" + try: + current = int(marker.read_text().strip()) if marker.exists() else 0 + except (ValueError, OSError): + current = 0 + if current >= _TABLE_SCHEMA_VERSION: + # Another process finished the migration while we waited + # for the lock. Nothing to do. + return + logger.info("table_schema_migration_started", target=_TABLE_SCHEMA_VERSION) + failed_schemas: list[str] = [] + for schema in BUSINESS_SCHEMAS_WITH_VECTOR: + table = await get_table(schema.TABLE_NAME, schema) + arrow_schema = await table.schema() + if arrow_schema.field("vector").nullable: + continue + try: + await table.alter_columns({"path": "vector", "nullable": True}) + except Exception as exc: + logger.warning( + "table_schema_migration_alter_failed", + table=schema.TABLE_NAME, + error=str(exc), + ) + failed_schemas.append(schema.TABLE_NAME) + + if failed_schemas: + logger.error( + "table_schema_migration_incomplete", + failed_schemas=failed_schemas, + ) + raise LanceDBMigrationError( + f"LanceDB nullable-vector migration failed for tables " + f"{failed_schemas!r}. Startup aborted to prevent cascade " + f"from writing NULL vectors into NOT-NULL columns. This " + f"typically indicates a LanceDB version mismatch or a " + f"corrupted index. Recovery, in order of least- to " + f"most-destructive: (1) restart the process; a transient " + f"filesystem or LanceDB-side hiccup may resolve. (2) If " + f"the error persists, wipe the index directory " + f"`{memory_root.lancedb_dir}` and restart — cascade will " + f"re-index from source markdown." + ) + + marker.parent.mkdir(parents=True, exist_ok=True) + try: + marker.write_text(str(_TABLE_SCHEMA_VERSION)) + except OSError: + logger.error("table_schema_migration_marker_write_failed", path=str(marker)) + raise + logger.info("table_schema_migration_done", version=_TABLE_SCHEMA_VERSION) async def ensure_business_indexes() -> None: """Ensure FTS (BM25) indexes for every business table (idempotent). Called once at startup by :class:`LanceDBLifespanProvider`. First - runs :func:`migrate_fts_indexes` (one-time, marker-guarded) to - rebuild any pre-fix ``with_position=True`` indexes, then walks the - business schemas (each owns its ``TABLE_NAME`` + ``BM25_FIELDS``), + runs :func:`migrate_table_schemas` (schema first, one-time, + marker-guarded) to make ``vector`` columns nullable, then + :func:`migrate_fts_indexes` (index second, one-time, marker-guarded) + to rebuild any pre-fix ``with_position=True`` indexes, then walks + the business schemas (each owns its ``TABLE_NAME`` + ``BM25_FIELDS``), opens each table via :func:`get_table`, and delegates to ``schema.ensure_fts_indexes(table)``. Already-indexed columns are skipped, so re-runs are no-ops. @@ -139,6 +283,7 @@ async def ensure_business_indexes() -> None: everything else (table name, columns to index) reads off the schema's ClassVars. """ + await migrate_table_schemas() await migrate_fts_indexes() for schema in _BUSINESS_SCHEMAS: table = await get_table(schema.TABLE_NAME, schema) @@ -173,12 +318,14 @@ async def verify_business_schemas() -> None: __all__ = [ + "BUSINESS_SCHEMAS_WITH_VECTOR", "AgentCase", "AgentSkill", "AtomicFact", "Episode", "Foresight", "KnowledgeTopic", + "LanceDBMigrationError", "LanceDBSchemaMismatchError", "ParentType", "UserProfile", @@ -193,6 +340,7 @@ async def verify_business_schemas() -> None: "get_table", "knowledge_topic_repo", "migrate_fts_indexes", + "migrate_table_schemas", "user_profile_repo", "verify_business_schemas", ] diff --git a/src/everos/infra/persistence/lancedb/lancedb_manager.py b/src/everos/infra/persistence/lancedb/lancedb_manager.py index 2099a6705..9c626ad28 100644 --- a/src/everos/infra/persistence/lancedb/lancedb_manager.py +++ b/src/everos/infra/persistence/lancedb/lancedb_manager.py @@ -28,7 +28,7 @@ async def get_connection() -> AsyncConnection: """Return the process-wide async LanceDB connection. - Built on first call from ``MemoryRoot.default().lancedb_dir`` and + Built on first call from ``MemoryRoot.resolve().lancedb_dir`` and ``Settings.lancedb``. Subsequent calls return the same instance. """ async with _lock: @@ -72,7 +72,7 @@ async def _ensure_connection_locked() -> AsyncConnection: global _conn if _conn is None: settings = load_settings() - memory_root = MemoryRoot.default() + memory_root = MemoryRoot.resolve() memory_root.ensure() _conn = await open_lancedb_connection(memory_root.lancedb_dir, settings.lancedb) logger.info( diff --git a/src/everos/infra/persistence/lancedb/repos/episode.py b/src/everos/infra/persistence/lancedb/repos/episode.py index f56d19e74..f63e312c8 100644 --- a/src/everos/infra/persistence/lancedb/repos/episode.py +++ b/src/everos/infra/persistence/lancedb/repos/episode.py @@ -2,9 +2,14 @@ from __future__ import annotations +import datetime as dt +from collections.abc import Sequence +from typing import Any + from lancedb import AsyncTable from everos.core.persistence.lancedb import LanceDailyLogRepoBase +from everos.core.persistence.lancedb.repository import _q from ..lancedb_manager import get_table from ..tables.episode import Episode @@ -16,5 +21,133 @@ class _EpisodeRepo(LanceDailyLogRepoBase[Episode]): async def _table_lookup(self) -> AsyncTable: return await get_table(self.schema.TABLE_NAME, self.schema) + async def count_by_owner( + self, + owner_id: str, + *, + app_id: str = "default", + project_id: str = "default", + parent_type: str | None = None, + ) -> int: + """Count episode rows for one owner within an ``(app, project)`` scope. + + Args: + owner_id: The row's ``owner_id`` field. + app_id: App scope segment (default ``"default"``). + project_id: Project scope segment (default ``"default"``). + parent_type: When non-``None``, restrict the count to rows whose + ``parent_type`` matches (e.g. ``"memcell"`` to exclude + Reflection-merged rows with ``parent_type='cluster'``). Kept + optional so callers without a parent-type requirement rely on + the ``None`` default meaning "count everything". + + Used by :func:`extract_user_profile`'s Tier-1 direct-path throttle + to gate re-extraction frequency. LanceDB ``count_rows`` accepts a + SQL-style ``filter`` predicate; the where-string is built with the + same escape discipline (:func:`_q`) as + :meth:`list_by_owner_after_ts`. + + Excludes rows with ``deprecated_by IS NOT NULL`` (Reflection- + superseded episodes) from the count, matching the filter idiom + used by :func:`memory.search.filters.compile_filters` and the + reflection read paths. Keeps the profile-throttle count aligned + with the "live" episode surface the strategy actually operates on. + """ + filter_str = ( + f"owner_id = '{_q(owner_id)}' " + f"AND app_id = '{_q(app_id)}' " + f"AND project_id = '{_q(project_id)}' " + f"AND deprecated_by IS NULL" + ) + if parent_type is not None: + filter_str += f" AND parent_type = '{_q(parent_type)}'" + table = await self._table() + return await table.count_rows(filter=filter_str) + + async def list_by_owner_after_ts( + self, + *, + owner_id: str, + after_ts: int, + parent_type: str, + app_id: str = "default", + project_id: str = "default", + columns: Sequence[str] | None = None, + limit: int | None = None, + ) -> list[Episode] | list[dict[str, Any]]: + """Scalar filter over owner + timestamp + parent_type + scope. + + Fetches episodes for the given owner after a specific timestamp, + scoped by parent_type and app/project. No vector dependency — + Tier 1-safe. Rows with vector=None are included. + + Used by extract_user_profile direct path (KEYWORD_ONLY mode) to + select memcells to feed the profile extractor without going through + cluster events. + + Excludes rows with ``deprecated_by IS NOT NULL`` (Reflection- + superseded episodes) — matches the filter idiom used by + :func:`memory.search.filters.compile_filters` and reflection + reads. Without this, direct-path profile extraction would feed + stale (superseded) memcell references to the extractor. + + Args: + owner_id: Owner identifier. + after_ts: Timestamp threshold (milliseconds since epoch); rows with + timestamp > after_ts are returned. + parent_type: Parent type filter (e.g. "memcell"). + app_id: Application scope (default "default"). + project_id: Project scope (default "default"). + columns: Optional column projection. Default (``None``) + returns full ``Episode`` model objects. Callers that only + need a small subset (e.g. ``parent_id`` in + ``extract_user_profile``) should pass an explicit list to + skip the 1024-D ``vector`` and ``subject_vector`` payload. + When set, this method returns raw dicts scoped to the + requested columns; otherwise it returns typed ``Episode`` + objects. Callers switch on the projection they asked for. + limit: Optional row cap. Default (``None``) means no cap — + the entire matching set is fetched. Callers with large + historical windows should pass a bound to avoid pulling + every memcell for the owner. + + Returns: + When ``columns`` is ``None``: list of ``Episode`` objects + ordered ascending by timestamp (existing behaviour). + When ``columns`` is a projection: list of raw dicts holding + only the requested columns, ordered ascending by timestamp. + """ + from everos.component.utils.datetime import to_iso_format + + # Convert milliseconds to UTC datetime, then to ISO string for DataFusion + # (stored timestamps are always UTC) + ts_dt = dt.datetime.fromtimestamp(int(after_ts) / 1000.0, tz=dt.UTC) + ts_iso = to_iso_format(ts_dt) + + where = ( + f"owner_id = '{_q(owner_id)}' " + f"AND timestamp > TIMESTAMP '{ts_iso}' " + f"AND parent_type = '{_q(parent_type)}' " + f"AND app_id = '{_q(app_id)}' " + f"AND project_id = '{_q(project_id)}' " + f"AND deprecated_by IS NULL" + ) + table = await self._table() + query = table.query().where(where) + # Projection: caller opted into a raw-dict return by naming columns; + # `timestamp` is force-included so the ascending sort below is stable + # even when the caller forgot to list it. + projected = columns is not None + if projected: + projection = list(dict.fromkeys([*columns, "timestamp"])) # type: ignore[misc] + query = query.select(projection) + if limit is not None: + query = query.limit(limit) + rows = await query.to_list() + rows.sort(key=lambda r: r["timestamp"]) + if projected: + return rows + return [self.schema.model_validate(r) for r in rows] + episode_repo = _EpisodeRepo() diff --git a/src/everos/infra/persistence/lancedb/tables/agent_case.py b/src/everos/infra/persistence/lancedb/tables/agent_case.py index be10131d9..da25dcbf6 100644 --- a/src/everos/infra/persistence/lancedb/tables/agent_case.py +++ b/src/everos/infra/persistence/lancedb/tables/agent_case.py @@ -81,4 +81,4 @@ class AgentCase(BaseLanceTable): session_id / timestamp / parent_id) is NOT in the hash. See :attr:`AgentCaseHandler.content_change_keys`.""" - vector: Vector(_DIM) # type: ignore[valid-type] + vector: Vector(_DIM) | None = None # type: ignore[valid-type] diff --git a/src/everos/infra/persistence/lancedb/tables/agent_skill.py b/src/everos/infra/persistence/lancedb/tables/agent_skill.py index ea249930b..ee32a7b73 100644 --- a/src/everos/infra/persistence/lancedb/tables/agent_skill.py +++ b/src/everos/infra/persistence/lancedb/tables/agent_skill.py @@ -77,4 +77,4 @@ class AgentSkill(BaseLanceTable): changed (e.g. the watcher fires for unrelated stat updates). See :attr:`AgentSkillHandler.content_change_keys`.""" - vector: Vector(_DIM) # type: ignore[valid-type] + vector: Vector(_DIM) | None = None # type: ignore[valid-type] diff --git a/src/everos/infra/persistence/lancedb/tables/atomic_fact.py b/src/everos/infra/persistence/lancedb/tables/atomic_fact.py index 488ff8699..02ba60240 100644 --- a/src/everos/infra/persistence/lancedb/tables/atomic_fact.py +++ b/src/everos/infra/persistence/lancedb/tables/atomic_fact.py @@ -64,4 +64,4 @@ class AtomicFact(BaseLanceTable): consolidated. Value is the cluster entry_id that supersedes this row. ``NULL`` means the row is still active.""" - vector: Vector(_DIM) # type: ignore[valid-type] + vector: Vector(_DIM) | None = None # type: ignore[valid-type] diff --git a/src/everos/infra/persistence/lancedb/tables/episode.py b/src/everos/infra/persistence/lancedb/tables/episode.py index 90de9a2b4..6846f16dd 100644 --- a/src/everos/infra/persistence/lancedb/tables/episode.py +++ b/src/everos/infra/persistence/lancedb/tables/episode.py @@ -80,5 +80,5 @@ class Episode(BaseLanceTable): consolidated into a cluster. Value is the cluster entry_id that supersedes this row. ``NULL`` means the row is still active.""" - vector: Vector(_DIM) # type: ignore[valid-type] + vector: Vector(_DIM) | None = None # type: ignore[valid-type] subject_vector: Vector(_DIM) | None = None # type: ignore[valid-type] diff --git a/src/everos/infra/persistence/lancedb/tables/foresight.py b/src/everos/infra/persistence/lancedb/tables/foresight.py index 8070d2474..26d2c0deb 100644 --- a/src/everos/infra/persistence/lancedb/tables/foresight.py +++ b/src/everos/infra/persistence/lancedb/tables/foresight.py @@ -76,4 +76,4 @@ class Foresight(BaseLanceTable): (owner_id / session_id / timestamp / parent_id / sender_ids) is NOT in the hash. See :attr:`ForesightHandler.content_change_keys`.""" - vector: Vector(_DIM) # type: ignore[valid-type] + vector: Vector(_DIM) | None = None # type: ignore[valid-type] diff --git a/src/everos/infra/persistence/lancedb/tables/knowledge_topic.py b/src/everos/infra/persistence/lancedb/tables/knowledge_topic.py index 31e12ad07..aedd0f77a 100644 --- a/src/everos/infra/persistence/lancedb/tables/knowledge_topic.py +++ b/src/everos/infra/persistence/lancedb/tables/knowledge_topic.py @@ -31,6 +31,6 @@ class KnowledgeTopic(BaseLanceTable): content_labels: list[str] = [] md_path: str content_sha256: str - vector: Vector(_DIM) # type: ignore[valid-type] -- Vector() is runtime-constructed; static analyzers cannot verify + vector: Vector(_DIM) | None = None # type: ignore[valid-type] -- Vector() is runtime-constructed; static analyzers cannot verify created_at: dt.datetime updated_at: dt.datetime diff --git a/src/everos/infra/persistence/sqlite/repos/md_change_state.py b/src/everos/infra/persistence/sqlite/repos/md_change_state.py index f1ac06b90..8eb33ec44 100644 --- a/src/everos/infra/persistence/sqlite/repos/md_change_state.py +++ b/src/everos/infra/persistence/sqlite/repos/md_change_state.py @@ -207,30 +207,48 @@ async def claim_one(self, md_path: str) -> MdChangeState | None: row = await s.get(MdChangeState, md_path) return row - async def claim_pending_batch(self, limit: int = 100) -> list[MdChangeState]: + async def claim_pending_batch( + self, + limit: int = 100, + *, + kinds: set[str] | None = None, + ) -> list[MdChangeState]: """Claim up to ``limit`` pending rows in LSN order. Returns the claimed rows (now ``status='processing'``); empty list if none were pending. Sibling workers / processes may race on the same prefix — the per-row ``WHERE status='pending'`` filter ensures each row lands in exactly one batch. + + Args: + limit: Maximum rows to claim in one batch. + kinds: Optional restriction on which ``kind`` values are + eligible. ``None`` (the default) claims across every + registered kind — the shape the background worker and + the CLI ``cascade sync`` path have always used. Passing + a set restricts the SELECT + UPDATE to ``kind IN (...)`` + via parameter binding (no string interpolation), so a + Phase-3 sync scoped to ``{"agent_skill"}`` cannot touch + unrelated kinds' queue state. Empty set is treated the + same as ``None`` — meaning "no kind filter" would be + surprising, but returning an empty result is what the + caller almost certainly meant, so we short-circuit. """ if limit <= 0: return [] + if kinds is not None and not kinds: + return [] now = get_utc_now() async with session_scope(self._factory) as s: - picks = ( - ( - await s.execute( - select(MdChangeState.md_path) - .where(MdChangeState.status == "pending") - .order_by(MdChangeState.lsn) - .limit(limit) - ) - ) - .scalars() - .all() + pick_stmt = ( + select(MdChangeState.md_path) + .where(MdChangeState.status == "pending") + .order_by(MdChangeState.lsn) + .limit(limit) ) + if kinds is not None: + pick_stmt = pick_stmt.where(MdChangeState.kind.in_(kinds)) + picks = (await s.execute(pick_stmt)).scalars().all() if not picks: return [] update_result = await s.execute( diff --git a/src/everos/infra/persistence/sqlite/repos/memcell.py b/src/everos/infra/persistence/sqlite/repos/memcell.py index 1f8754726..453bff228 100644 --- a/src/everos/infra/persistence/sqlite/repos/memcell.py +++ b/src/everos/infra/persistence/sqlite/repos/memcell.py @@ -17,6 +17,17 @@ from ..sqlite_manager import get_session_factory from ..tables import Memcell +_SQLITE_IN_CHUNK = 500 +"""SQL ``IN (?, ?, ...)`` parameter cap for :meth:`_MemcellRepo.find_by_ids`. + +SQLite's ``SQLITE_MAX_VARIABLE_NUMBER`` defaults to 999 on the shipped +amalgamation and rises to 32766 on ≥3.32; 500 stays well under the older +cap and keeps the query planner's cost estimate cheap. Callers passing +more ``memcell_ids`` than this fan out into multiple SELECTs, merged +in-memory — see the direct-path selector in +``memory/strategies/extract_user_profile.py`` where +``last_profile_ts=0`` can pull an owner's entire episode history.""" + class _MemcellRepo(RepoBase[Memcell]): model = Memcell @@ -36,16 +47,25 @@ async def insert_many(self, rows: list[Memcell]) -> list[Memcell]: async def find_by_ids(self, memcell_ids: list[str]) -> list[Memcell]: """Bulk fetch rows by primary key list — preserves caller order. + Chunked at ``_SQLITE_IN_CHUNK`` to protect against + ``SQLITE_MAX_VARIABLE_NUMBER`` blowups on large owner histories + (e.g. ``extract_user_profile``'s direct path when ``user.md`` + doesn't exist yet — ``last_profile_ts=0`` fetches every memcell + the owner has ever produced). + Used by offline strategies that pull every memcell in a cluster (membership lives in :class:`ClusterMember` and is supplied to the strategy via :class:`everalgo.clustering.Cluster.members`). """ if not memcell_ids: return [] + by_id: dict[str, Memcell] = {} async with session_scope(self._factory) as s: - stmt = select(Memcell).where(Memcell.memcell_id.in_(memcell_ids)) - rows = list((await s.execute(stmt)).scalars().all()) - by_id = {r.memcell_id: r for r in rows} + for start in range(0, len(memcell_ids), _SQLITE_IN_CHUNK): + chunk = memcell_ids[start : start + _SQLITE_IN_CHUNK] + stmt = select(Memcell).where(Memcell.memcell_id.in_(chunk)) + rows = (await s.execute(stmt)).scalars().all() + by_id.update((r.memcell_id, r) for r in rows) return [by_id[mid] for mid in memcell_ids if mid in by_id] diff --git a/src/everos/infra/persistence/sqlite/sqlite_manager.py b/src/everos/infra/persistence/sqlite/sqlite_manager.py index f1ff26163..ead428645 100644 --- a/src/everos/infra/persistence/sqlite/sqlite_manager.py +++ b/src/everos/infra/persistence/sqlite/sqlite_manager.py @@ -3,7 +3,7 @@ The single place that owns the SQLite **runtime state**: the async SQLAlchemy engine and the session factory bound to it. Built lazily on first :func:`get_engine` / :func:`get_session_factory` call from -:func:`everos.config.load_settings` + :meth:`MemoryRoot.default`. The +:func:`everos.config.load_settings` + :meth:`MemoryRoot.resolve`. The :class:`SqliteLifespanProvider` calls :func:`dispose_engine` on shutdown to drain the connection pool; in scripts you can call it manually. """ @@ -29,13 +29,13 @@ def get_engine() -> AsyncEngine: """Return the process-wide async SQLAlchemy engine. - Built on first call from ``MemoryRoot.default()`` and ``Settings.sqlite``. + Built on first call from ``MemoryRoot.resolve()`` and ``Settings.sqlite``. Subsequent calls return the same instance. """ global _engine if _engine is None: settings = load_settings() - memory_root = MemoryRoot.default() + memory_root = MemoryRoot.resolve() memory_root.ensure() _engine = create_system_engine(memory_root.system_db, settings.sqlite) logger.info( diff --git a/src/everos/memory/cascade/__init__.py b/src/everos/memory/cascade/__init__.py index 2a7fef075..894d2abc6 100644 --- a/src/everos/memory/cascade/__init__.py +++ b/src/everos/memory/cascade/__init__.py @@ -11,8 +11,16 @@ - :class:`CascadeConfig` — construction-time tuning knobs. - :data:`KIND_REGISTRY` / :func:`match_kind` — kind dispatch (also used by CLI ``cascade sync --path`` to resolve a single file's kind). +- :class:`BackfillPhase` — dataclass shared between the memory-layer + phase runners and the CLI's ``PHASES`` copy. The concrete ``PHASES`` + tuple and ``run_backfill`` orchestrator live in + ``everos.entrypoints.cli.commands._backfill_cmd`` (M11: the memory + layer must not depend on typer/click). """ +from ._backfill import BackfillPhase as BackfillPhase +from ._backfill import BackfillPresenter as BackfillPresenter +from ._backfill import NullBackfillPresenter as NullBackfillPresenter from .orchestrator import CascadeConfig as CascadeConfig from .orchestrator import CascadeOrchestrator as CascadeOrchestrator from .registry import KIND_REGISTRY as KIND_REGISTRY @@ -21,8 +29,11 @@ __all__ = [ "KIND_REGISTRY", + "BackfillPhase", + "BackfillPresenter", "CascadeConfig", "CascadeOrchestrator", "KindSpec", + "NullBackfillPresenter", "match_kind", ] diff --git a/src/everos/memory/cascade/_backfill.py b/src/everos/memory/cascade/_backfill.py new file mode 100644 index 000000000..53b89b123 --- /dev/null +++ b/src/everos/memory/cascade/_backfill.py @@ -0,0 +1,1530 @@ +"""Backfill phase runners for ``everos cascade backfill``. + +Under soft-dependency (embed-optional), a user who starts on Tier 1 +(LLM only) accumulates memory rows with ``vector IS NULL``. Once they +configure embedding and move to Tier 2/3, those rows stay keyword-only +until backfilled. The three phase runners here upgrade them: + +- Phase 1 (:func:`_run_phase_vectors`) — re-embed missing vectors on + existing rows. +- Phase 2 (:func:`_run_phase_clusters`) — build clusters on the newly- + embedded episodes / agent cases. +- Phase 3 (:func:`_run_phase_skills`) — extract agent skills from + clustered cases. + +Presentation is not the memory layer's job. Every phase runner takes a +:class:`BackfillPresenter` kwarg — the CLI (``entrypoints/cli/commands/ +_backfill_cmd.py``) supplies a typer-backed implementation; tests use a +no-op / recording stub. This is the sole seam through which the +otherwise-pure phase logic surfaces progress and confirmation to the +caller. Fixes PR #361 review finding M11: ``memory`` used to import +``typer`` + ``click`` directly and silently bypass the architecture +layering rule (import-linter does not check third-party imports). +""" + +from __future__ import annotations + +import asyncio +import dataclasses +from collections.abc import Callable +from pathlib import Path +from typing import Any, Protocol +from uuid import uuid4 + +import anyio +import anyio.to_thread +import portalocker + +from everos.component.embedding import EmbeddingProvider, get_embedding_capability +from everos.component.tokenizer import build_tokenizer +from everos.component.utils.datetime import to_timestamp_ms +from everos.core.observability.logging import get_logger +from everos.core.persistence import MarkdownReader, MemoryRoot, SQLModel +from everos.core.persistence.lancedb import BaseLanceTable, LanceRepoBase +from everos.infra.ome.config import OMEConfig +from everos.infra.ome.engine import OfflineEngine +from everos.infra.ome.exceptions import EngineLockHeldError +from everos.infra.persistence.lancedb import ( + BUSINESS_SCHEMAS_WITH_VECTOR, + AgentCase, + AgentSkill, + AtomicFact, + Episode, + Foresight, + KnowledgeTopic, + agent_case_repo, + agent_skill_repo, + atomic_fact_repo, + episode_repo, + foresight_repo, + get_table, + knowledge_topic_repo, +) +from everos.infra.persistence.markdown import AgentSkillFrontmatter +from everos.infra.persistence.sqlite import cluster_repo, get_engine +from everos.memory.events import ( + AgentCaseExtracted, + EpisodeExtracted, + SkillClusterUpdated, +) +from everos.memory.strategies import ( + extract_agent_skill, + trigger_profile_clustering, + trigger_skill_clustering, +) + +from .orchestrator import CascadeOrchestrator + +logger = get_logger(__name__) + + +@dataclasses.dataclass(frozen=True) +class BackfillPhase: + """One backfill phase: identity plus the copy shown at the prompt. + + ``slug`` matches the CLI ``--phase`` value one-to-one (``vectors`` / + ``clusters`` / ``skills``), so phase selection is plain string + equality with no translation layer. The concrete :data:`PHASES` + tuple carrying user-facing ``title`` / ``detail`` strings lives in + ``entrypoints/cli/commands/_backfill_cmd.py`` because those strings + are CLI copy, not domain data. + """ + + number: int + slug: str + title: str + detail: str + + +class BackfillPresenter(Protocol): + """Callback interface phase runners use to surface progress and + confirmation to the caller. + + The CLI (``entrypoints/cli/commands/_backfill_cmd.py``) supplies a + typer-backed implementation (:class:`TyperPresenter`); tests inject + a no-op / recording stub. Kept as a :class:`~typing.Protocol` (not + an ABC) so structural typing lets any object with matching methods + fit without explicit inheritance. + + Contract: + + - Every method may be called from an async context. Sync methods + must not block for meaningful I/O; :meth:`confirm` is async so an + implementation can dispatch the actual (blocking) prompt to a + thread. + - :meth:`confirm` returns ``True`` to proceed, ``False`` to abort + the phase with exit code 1. Ctrl-C at a real prompt is expected + to surface as :class:`click.exceptions.Abort` (a + :class:`RuntimeError` subclass) — the orchestrator in + ``entrypoints/cli/commands/_backfill_cmd.py`` catches it and maps + to exit 130. Memory-layer runners must not catch it here; the + abort is a caller concern. + - ``auto_yes=True`` on :meth:`confirm` MUST short-circuit any + interactive prompt and return ``True``. + """ + + def phase_header(self, phase: BackfillPhase) -> None: ... + def nothing_to_backfill(self, message: str) -> None: ... + def capability_missing( + self, *, provider: str, feature: str, message: str + ) -> None: ... + def server_running(self) -> None: ... + def estimate_vectors(self, rows: int, tokens: int) -> None: ... + def estimate_clusters(self, episodes: int, cases: int) -> None: ... + def estimate_skills(self, cases: int, clusters: int) -> None: ... + async def confirm(self, prompt: str, *, auto_yes: bool) -> bool: ... + def emit_progress(self, done: int, total: int) -> None: ... + def row_progress(self, table_name: str, done: int, total: int) -> None: ... + def phase_1_complete(self, rows_processed: int, rows_failed: int) -> None: ... + def phase_2_complete(self, before: int, after: int, emitted: int) -> None: ... + def phase_3_complete( + self, skills_before: int, skills_after: int, emitted: int + ) -> None: ... + + +class NullBackfillPresenter: + """No-op :class:`BackfillPresenter` for callers that don't need output. + + Kept in the memory layer so tests and non-CLI invocations (offline + scripts, embedded harnesses) can construct one without importing + ``entrypoints``. :meth:`confirm` honours ``auto_yes=True`` and + otherwise returns ``True`` — the phase runners still enforce + correctness gates through their return values, not through the + prompt. + """ + + def phase_header(self, phase: BackfillPhase) -> None: + return None + + def nothing_to_backfill(self, message: str) -> None: + return None + + def capability_missing(self, *, provider: str, feature: str, message: str) -> None: + return None + + def server_running(self) -> None: + return None + + def estimate_vectors(self, rows: int, tokens: int) -> None: + return None + + def estimate_clusters(self, episodes: int, cases: int) -> None: + return None + + def estimate_skills(self, cases: int, clusters: int) -> None: + return None + + async def confirm(self, prompt: str, *, auto_yes: bool) -> bool: + return True + + def emit_progress(self, done: int, total: int) -> None: + return None + + def row_progress(self, table_name: str, done: int, total: int) -> None: + return None + + def phase_1_complete(self, rows_processed: int, rows_failed: int) -> None: + return None + + def phase_2_complete(self, before: int, after: int, emitted: int) -> None: + return None + + def phase_3_complete( + self, skills_before: int, skills_after: int, emitted: int + ) -> None: + return None + + +# ── Phase 1 (vectors) — table wiring ───────────────────────────────────────── + + +@dataclasses.dataclass(frozen=True) +class _TableSpec: + """One business table's Phase 1 wiring. + + ``text_of`` pulls the same source text each cascade handler feeds to + the embedder out of a raw LanceDB row dict (see ``memory/cascade/ + handlers/*.py``). ``subject_of`` is only set for ``episode``, whose + ``subject_vector`` is a second, independent embed of the row's + ``subject`` column — mirrors ``EpisodeHandler._build_row``. + """ + + schema: type[BaseLanceTable] + repo: LanceRepoBase[Any] + text_of: Callable[[dict[str, Any]], str] + subject_of: Callable[[dict[str, Any]], str | None] | None = None + + +def _agent_skill_embed_text(row: dict[str, Any]) -> str: + """Mirrors ``AgentSkillHandler``: embed source is name + description.""" + return "\n".join(s for s in (row.get("name"), row.get("description")) if s) + + +_TABLE_SPECS: tuple[_TableSpec, ...] = ( + _TableSpec( + Episode, + episode_repo, + lambda r: r["episode"], + subject_of=lambda r: r.get("subject") or None, + ), + _TableSpec(AtomicFact, atomic_fact_repo, lambda r: r["fact"]), + _TableSpec(Foresight, foresight_repo, lambda r: r["foresight"]), + _TableSpec(AgentCase, agent_case_repo, lambda r: r["task_intent"]), + _TableSpec(AgentSkill, agent_skill_repo, _agent_skill_embed_text), + _TableSpec(KnowledgeTopic, knowledge_topic_repo, lambda r: r["summary"]), +) + +# Load-bearing invariant: ``_TABLE_SPECS`` must cover exactly the same +# tables as :data:`BUSINESS_SCHEMAS_WITH_VECTOR`. The infra list is the +# public source of truth (consumed by the LanceDB lifespan's unbackfilled +# hint AND by ``migrate_table_schemas``). Adding a new business table +# there without adding a matching ``_TableSpec`` here silently drops it +# from backfill — the hint would count rows the backfill CLI never +# touches. Fail loud at import time so any drift shows up before a test +# even starts. +_TABLE_SPEC_NAMES = {spec.schema.TABLE_NAME for spec in _TABLE_SPECS} +_BUSINESS_SCHEMA_NAMES = {s.TABLE_NAME for s in BUSINESS_SCHEMAS_WITH_VECTOR} +if _TABLE_SPEC_NAMES != _BUSINESS_SCHEMA_NAMES: + raise RuntimeError( + f"_TABLE_SPECS drift: BUSINESS_SCHEMAS_WITH_VECTOR has " + f"{_BUSINESS_SCHEMA_NAMES}, _TABLE_SPECS covers " + f"{_TABLE_SPEC_NAMES}. Update _TABLE_SPECS to include every " + f"business table with a nullable vector column." + ) + +_EMBED_BATCH_SIZE = 32 +"""Rows grouped per ``embed_batch`` call (one for the primary text, one +more only if the group has subject rows) — not the provider's own +request chunk size. The provider's configured ``batch_size`` / +``max_concurrent`` (``embedding.batch_size`` / ``embedding.max_concurrent`` +in settings) govern the actual in-flight request shape from there; this +just bounds per-batch failure isolation + progress feedback granularity +for a bulk migration.""" + + +@dataclasses.dataclass(frozen=True) +class _NullVectorRow: + """One row missing (at least one side of) its vector. + + ``tokens`` is the real token count (via the same tokenizer cascade + uses for BM25), not an estimate — computed once here and never + recomputed before the embed call. + + ``needs_primary`` / ``needs_subject`` mark which side(s) of the + row's embedding are still NULL. Round-1 assumed "in the backlog ⇒ + primary vector is NULL"; round-2 widened the scan filter to also + pick up rows where only ``subject_vector`` is NULL (Episode's + orthogonal second embed can fail on its own — see + :func:`_null_filter`). ``_backfill_table`` reads these flags to + embed only the side that actually needs work, so a re-run after a + partial failure doesn't waste an ``embed_batch`` call on the side + that already succeeded. + + Both default to ``True`` so pre-existing test fixtures that build + :class:`_NullVectorRow` directly keep their round-1 semantics + (embed everything). + """ + + id: str + text: str + subject_text: str | None + tokens: int + needs_primary: bool = True + needs_subject: bool = True + + +@dataclasses.dataclass(frozen=True) +class _TableBacklog: + spec: _TableSpec + rows: list[_NullVectorRow] + + @property + def table_name(self) -> str: + return self.spec.schema.TABLE_NAME + + +@dataclasses.dataclass +class _PhaseResult: + """Phase 1 outcome. + + ``aborted`` distinguishes "user declined the confirmation prompt" + (caller exits 1) from "there was nothing to backfill" (caller exits + 0) — both otherwise leave every counter at zero. + ``blocked_by_capability`` names the missing provider (currently only + ``"embedding"``) when a preflight short-circuits the phase before it + can start; the CLI orchestrator maps that to exit code 2. + """ + + rows_processed: int = 0 + rows_failed: int = 0 + tokens_embedded: int = 0 + aborted: bool = False + blocked_by_capability: str | None = None + + +def _q(value: str) -> str: + """Defensive SQL-quote escape (mirrors the lancedb repo convention).""" + return value.replace("'", "''") + + +def _null_filter(spec: _TableSpec) -> str: + """Where-clause the scan uses to spot rows that need (some) embedding. + + Episode carries a ``subject_vector`` alongside ``vector`` — the two + are orthogonal embeds (see :class:`Episode`), and either being NULL + means the row is incompletely embedded. Round-1 filtered only on + ``vector IS NULL``, which silently missed rows where the primary + embed succeeded but ``_embed_subject_batch`` failed on its own + (``subject_vector`` left NULL). Round-2 widens the filter to + ``vector IS NULL OR subject_vector IS NULL`` so those rows + re-enter the backlog on the next scan. + + Other tables have no subject column; adding the OR clause there + would reference an unknown field and LanceDB would reject the + query, so they keep the round-1 filter. + """ + if spec.schema.TABLE_NAME == Episode.TABLE_NAME: + return "vector IS NULL OR subject_vector IS NULL" + return "vector IS NULL" + + +class _RowSkipped: + """Sentinel returned by :func:`_extract_row` when the row shape is + fine but nothing actually needs embedding (widened Episode filter + matched on a legitimately-NULL ``subject_vector``). Distinct from + ``None`` — which the caller tallies as a scan-failure — so the + two paths never conflate.""" + + +_ROW_SKIPPED = _RowSkipped() + + +def _extract_row( + raw: dict[str, Any], spec: _TableSpec, tokenizer: Any +) -> _NullVectorRow | _RowSkipped | None: + """Pull one raw LanceDB row's embed-source text, token count, and + which side(s) still need embedding. + + Return values: + + - :class:`_NullVectorRow` — the row has real work; carries the + ``needs_primary`` / ``needs_subject`` flags read off the raw + vector columns. + - :data:`_ROW_SKIPPED` — the row's shape is fine but nothing needs + embedding (widened Episode filter matched on a legitimately-NULL + ``subject_vector``). Silent; caller drops it. + - ``None`` — the row's shape is broken (schema drift / bad row). + Logged; caller tallies as scan-failed. + """ + try: + text = spec.text_of(raw) + subject_text = spec.subject_of(raw) if spec.subject_of else None + row_id = raw["id"] + except (KeyError, TypeError) as exc: + logger.warning( + "cascade_backfill_scan_extract_failed", + table=spec.schema.TABLE_NAME, + row_id=raw.get("id"), + error=repr(exc), + ) + return None + needs_primary = raw.get("vector") is None + # Subject-side is only meaningful when the spec has a subject + # projection AND this row actually carries subject text. A NULL + # ``subject_vector`` on a row whose subject was legitimately + # absent is not something to fix — it will always be NULL. + needs_subject = ( + spec.subject_of is not None + and subject_text is not None + and raw.get("subject_vector") is None + ) + if not needs_primary and not needs_subject: + return _ROW_SKIPPED + token_source = f"{text} {subject_text}" if needs_subject else text + return _NullVectorRow( + id=row_id, + text=text, + subject_text=subject_text, + tokens=len(tokenizer.tokenize(token_source)), + needs_primary=needs_primary, + needs_subject=needs_subject, + ) + + +async def _scan_null_vector_backlog() -> tuple[list[_TableBacklog], int]: + """Fetch every row across the 6 business tables that still needs + (some) embedding. + + One query per table, tokenising each row's embed-source text on the + way so the pre-run estimate and the later embed call read off the + exact same text. Returns the backlog plus a count of rows whose + extraction failed (bad row shape) — tallied by the caller into the + phase's ``rows_failed``, never raised. + + Round-2 changes: + + - The Episode filter is widened to + ``vector IS NULL OR subject_vector IS NULL`` via + :func:`_null_filter` so a subject-only failure doesn't hide the + row from subsequent backfill runs. + - The prior ``count_rows`` + ``.limit(null_count)`` split is + dropped. Under concurrent server writes (Phase 1 is documented + as concurrent-safe), a fresh NULL-vector row inserted between + the count and the query would push a legitimate row past the + cap. LanceDB's ``.to_list()`` streams the whole predicate result; + the ``where`` filter bounds it naturally. + """ + tokenizer = build_tokenizer() + backlog: list[_TableBacklog] = [] + scan_failed = 0 + for spec in _TABLE_SPECS: + table = await get_table(spec.schema.TABLE_NAME, spec.schema) + raw_rows = await table.query().where(_null_filter(spec)).to_list() + rows: list[_NullVectorRow] = [] + for raw in raw_rows: + row = _extract_row(raw, spec, tokenizer) + if row is None: + scan_failed += 1 + continue + if isinstance(row, _RowSkipped): + continue + rows.append(row) + backlog.append(_TableBacklog(spec, rows)) + return backlog, scan_failed + + +def _truncated_text_prefix(text: str, *, limit: int = 100) -> str: + """Return the first ``limit`` chars of ``text``, with an ellipsis when + truncated. Used for per-row failure logs — never log full text, it + may be user data / PII.""" + if len(text) <= limit: + return text + return text[:limit] + "…" + + +async def _embed_primary_batch( + provider: EmbeddingProvider, batch: list[_NullVectorRow], table_name: str +) -> list[list[float] | None]: + """Embed the batch's primary text with per-row fallback. + + First attempts the batched ``provider.embed_batch(...)`` call — the + happy path. On any exception (network glitch, one poison row whose + text exceeds the provider's context limit and 400s the whole batch, + etc.), logs the batch failure and falls back to per-row + ``provider.embed(...)`` calls. Each per-row failure is logged with + the row's id and a truncated text prefix (100 chars) so operators + can identify poison rows and quarantine them. + + Returns a same-length list aligned with the input ``batch``: each + entry is either the row's vector, or ``None`` for a row that failed + even the per-row retry. The caller writes only rows with non-None + vectors — failed rows keep ``vector IS NULL`` on disk and re-enter + the next scan's NULL-vector backlog. + """ + try: + vectors = await provider.embed_batch([row.text for row in batch]) + return list(vectors) + except Exception as exc: + logger.warning( + "cascade_backfill_batch_embed_failed_falling_back_per_row", + table=table_name, + batch_size=len(batch), + error=repr(exc), + ) + # Per-row fallback: isolate the poison row(s) so the rest advance. + results: list[list[float] | None] = [] + for row in batch: + try: + vec = await provider.embed(row.text) + except Exception as exc: + logger.warning( + "cascade_backfill_row_embed_failed", + table=table_name, + row_id=row.id, + text_prefix=_truncated_text_prefix(row.text), + error=repr(exc), + ) + results.append(None) + continue + results.append(vec) + return results + + +async def _embed_subject_batch( + provider: EmbeddingProvider, batch: list[_NullVectorRow], table_name: str +) -> dict[str, list[float]]: + """Embed the batch's ``subject_text`` with per-row fallback (episode's + second, independent embed — see ``EpisodeHandler._build_row``). + + A separate batch call because not every row has a subject; folding + it into the primary call would either skip absent subjects (index + mismatch) or force a dense-but-wrong text list. + + Returns an ``{id: vector}`` map. Rows whose per-row retry also failed + are absent from the map, and their ``subject_vector`` stays NULL — + the widened ``vector IS NULL OR subject_vector IS NULL`` scan filter + (M1) picks them up on the next backfill run. + """ + subject_rows = [row for row in batch if row.subject_text is not None] + if not subject_rows: + return {} + try: + vectors = await provider.embed_batch( + [row.subject_text for row in subject_rows] # type: ignore[misc] + ) + return {row.id: vec for row, vec in zip(subject_rows, vectors, strict=True)} + except Exception as exc: + logger.warning( + "cascade_backfill_subject_batch_embed_failed_falling_back_per_row", + table=table_name, + batch_size=len(subject_rows), + error=repr(exc), + ) + results: dict[str, list[float]] = {} + for row in subject_rows: + text = row.subject_text + assert text is not None # guarded by the filter above + try: + vec = await provider.embed(text) + except Exception as exc: + logger.warning( + "cascade_backfill_row_subject_embed_failed", + table=table_name, + row_id=row.id, + text_prefix=_truncated_text_prefix(text), + error=repr(exc), + ) + continue + results[row.id] = vec + return results + + +async def _backfill_table( + backlog: _TableBacklog, + provider: EmbeddingProvider, + *, + presenter: BackfillPresenter, +) -> _PhaseResult: + """Embed + write back every row in one table's backlog, in batches. + + Each batch is split by which side each row still needs — the + orthogonal side (Episode's ``subject_vector``) is a separate + ``embed_batch`` call, and a row that already has one side filled + (e.g. primary succeeded on a prior run, subject failed) only pays + the cost of the missing side. When both sides have work in the + same batch the two calls run concurrently under ``asyncio.gather`` + (round-1 concurrency invariant preserved). + + Failure isolation: + + - Whole primary batch failing → those rows keep ``vector IS NULL`` + on disk; the widened scan filter picks them up on the next run. + Counted as ``rows_failed`` on this pass. + - Whole subject batch failing → the touched rows keep + ``subject_vector IS NULL`` on disk; the widened scan filter + picks them up on the next run (round-1 fix #10's silent + corruption is closed here). + - Per-row write failing → tally the row as ``rows_failed`` and + leave it for the next scan. + + Write-back stays per-row: LanceDB's partial-column ``update`` + takes a single-row predicate, so a failed write only fails that + one row. + """ + result = _PhaseResult() + rows = backlog.rows + for start in range(0, len(rows), _EMBED_BATCH_SIZE): + batch = rows[start : start + _EMBED_BATCH_SIZE] + primary_rows = [r for r in batch if r.needs_primary] + subject_rows = [ + r for r in batch if r.needs_subject and r.subject_text is not None + ] + + # Concurrent embed only for the sides that have work this + # batch. ``asyncio.gather`` under both-need shape preserves the + # round-1 wall-clock invariant; one-sided batches skip the + # gather to avoid a wasted ``embed_batch({})`` round-trip. + primary_vectors: list[list[float] | None] + subject_vectors: dict[str, list[float]] + if primary_rows and subject_rows: + primary_vectors, subject_vectors = await asyncio.gather( + _embed_primary_batch(provider, primary_rows, backlog.table_name), + _embed_subject_batch(provider, subject_rows, backlog.table_name), + ) + elif primary_rows: + primary_vectors = await _embed_primary_batch( + provider, primary_rows, backlog.table_name + ) + subject_vectors = {} + elif subject_rows: + primary_vectors = [] + subject_vectors = await _embed_subject_batch( + provider, subject_rows, backlog.table_name + ) + else: + # Purely defensive: the scan filters no-op rows out via + # ``_ROW_SKIPPED``, so an empty split shouldn't happen in + # practice. Still emit progress so the readout stays + # monotonic. + done = min(start + _EMBED_BATCH_SIZE, len(rows)) + presenter.row_progress(backlog.table_name, done, len(rows)) + continue + + # ``primary_vectors`` is a same-length list aligned with + # ``primary_rows``: successful rows carry the vector, poison + # rows (whose per-row retry also failed) carry ``None``. Count + # the poison rows as ``rows_failed`` — they keep ``vector IS + # NULL`` on disk and re-enter the next scan. + primary_map: dict[str, list[float]] = {} + for r, v in zip(primary_rows, primary_vectors, strict=True): + if v is None: + result.rows_failed += 1 + else: + primary_map[r.id] = v + # ``subject_vectors`` is already an ``{id: vec}`` dict; ids + # absent mean the per-row subject retry also failed and the + # row keeps ``subject_vector IS NULL``, re-entering the backlog + # via the widened NULL scan next run. + + for row in batch: + updates: dict[str, Any] = {} + if row.id in primary_map: + updates["vector"] = primary_map[row.id] + if row.id in subject_vectors: + updates["subject_vector"] = subject_vectors[row.id] + if not updates: + continue + try: + await backlog.spec.repo.update(updates, where=f"id = '{_q(row.id)}'") + except Exception: + result.rows_failed += 1 + logger.warning( + "cascade_backfill_row_write_failed", + table=backlog.table_name, + row_id=row.id, + exc_info=True, + ) + continue + # A row is "processed" once at least one side advanced + # this pass. If both sides advanced (or if only the needed + # side did), the row is fully embedded. + result.rows_processed += 1 + result.tokens_embedded += row.tokens + + done = min(start + _EMBED_BATCH_SIZE, len(rows)) + presenter.row_progress(backlog.table_name, done, len(rows)) + + # Compact fragments accumulated during backfill. Every per-row + # update opens a fresh LanceDB manifest version; without periodic + # optimize() the on-disk directory grows unbounded (same failure + # mode as v1.1.3's FTS optimize gap, #336 / + # lance-format/lance#7653). Gate on rows_processed > 0 so a no-op + # backlog doesn't pay the compact cost. Best-effort maintenance: + # a failure here does not invalidate the writes. + if result.rows_processed > 0: + try: + await backlog.spec.repo.optimize() + logger.info( + "cascade_backfill_table_optimized", + table=backlog.table_name, + rows_processed=result.rows_processed, + ) + except Exception as exc: + logger.warning( + "cascade_backfill_table_optimize_failed", + table=backlog.table_name, + error=repr(exc), + ) + return result + + +async def _run_phase_vectors( + *, auto_yes: bool, presenter: BackfillPresenter +) -> _PhaseResult: + """Re-embed rows with a NULL primary vector (and, for Episode, a + NULL subject vector) across the 6 business tables. + + Preflight capability → scan + tokenize → surface the estimate via + ``presenter`` → confirm once → embed in batches, writing each row + back via ``LanceRepoBase.update`` (a partial-column update, not a + full-row ``merge_insert`` — Phase 1 only ever changes ``vector`` / + ``subject_vector``). Per-row failures (including scan-time + extraction failures) are tallied and logged; the phase runs to + completion regardless. + + Preflight runs FIRST — before scan + tokenize + confirm — so a + user with a large accumulated backlog and no embedding configured + exits fast with a toml-hint instead of paying an O(N) scan they + can't act on. The empty-backlog path also preflights, so a fresh + install still gets the hint rather than a bare "nothing to + backfill" green message. + + After each table's backfill loop, :func:`_backfill_table` calls + ``optimize()`` to compact the manifest / fragment accumulation + from per-row updates. Same treatment as v1.1.3's FTS index + optimize (#336) — LanceDB per-row writes open fragments that would + otherwise grow unbounded. + """ + # ── preflight first ──────────────────────────────────────────── + # Only ``.available`` is inspected here; the actual provider is + # resolved after the confirm prompt so a declined y/N never + # touches ``require()``. + capability = get_embedding_capability() + if not capability.available: + from everos.core.errors import ProviderNotConfiguredError + + err = ProviderNotConfiguredError(provider="embedding", feature="backfill") + presenter.capability_missing( + provider="embedding", feature="backfill", message=str(err) + ) + return _PhaseResult(blocked_by_capability="embedding") + + backlog, scan_failed = await _scan_null_vector_backlog() + total_rows = sum(len(tb.rows) for tb in backlog) + if total_rows == 0: + if scan_failed: + presenter.nothing_to_backfill( + f"Nothing to backfill — {scan_failed:,} row(s) could not be " + "read and were skipped (see logs)." + ) + else: + presenter.nothing_to_backfill( + "Nothing to backfill. All rows already have vectors." + ) + return _PhaseResult(rows_failed=scan_failed) + + total_tokens = sum(row.tokens for tb in backlog for row in tb.rows) + presenter.estimate_vectors(total_rows, total_tokens) + if not await presenter.confirm( + f"proceed with {total_rows:,} rows / ~{_format_tokens(total_tokens)} tokens", + auto_yes=auto_yes, + ): + return _PhaseResult(aborted=True) + + # Resolve the provider only now — a declined y/N above should not + # touch ``capability.require()`` (tests inspect that boundary). + provider = capability.require() + + result = _PhaseResult(rows_failed=scan_failed) + for tb in backlog: + if not tb.rows: + continue + table_result = await _backfill_table(tb, provider, presenter=presenter) + result.rows_processed += table_result.rows_processed + result.rows_failed += table_result.rows_failed + result.tokens_embedded += table_result.tokens_embedded + + presenter.phase_1_complete(result.rows_processed, result.rows_failed) + return result + + +def _format_tokens(n: int) -> str: + """Render a token count in K/M shorthand (provider per-M-token pricing + convention) — never a currency estimate; see the embed-soft-dependency + design doc §10 cost policy. Exposed as a module-level helper because + both the memory-layer confirm-prompt string and the entrypoints + summary reader consume the same format.""" + if n >= 1_000_000: + return f"{n / 1_000_000:.1f}M" + if n >= 1_000: + return f"{n / 1_000:.1f}K" + return f"{n} tokens" + + +# ── Phase 2 (clusters) — synthetic-event replay ────────────────────────────── + + +_CLUSTER_WAIT_TIMEOUT_SECONDS = 300.0 +"""Cap on waiting for the ephemeral engine to drain (see +:meth:`OfflineEngine.wait_idle`). A backfill that never converges must not +hang the CLI forever; on timeout the phase still reports whatever cluster +count it observes and logs a warning rather than raising.""" + +_PROGRESS_EVERY = 100 +"""Emit-progress cadence for the presenter — one call per row would flood +the presenter's output for a large backlog; a per-event log line would +flood the logs.""" + + +@dataclasses.dataclass +class _ClusterPhaseResult: + """Phase 2 outcome. + + ``events_emitted`` counts every synthesized ``EpisodeExtracted`` + + ``AgentCaseExtracted`` event fanned into the ephemeral engine. + ``clusters_before`` / ``clusters_after`` are the total ``cluster`` + row count (:meth:`_ClusterRepo.count`, across every owner/kind) + taken immediately before dispatch and after the engine drains, so + the caller can report growth. ``drained`` is ``False`` only if + :meth:`OfflineEngine.wait_idle` hit :data:`_CLUSTER_WAIT_TIMEOUT_SECONDS` + with runs still in flight. ``aborted`` mirrors :class:`_PhaseResult`. + """ + + events_emitted: int = 0 + clusters_before: int = 0 + clusters_after: int = 0 + drained: bool = True + aborted: bool = False + blocked_by_server: bool = False + blocked_by_capability: str | None = None + + +async def _scan_all_rows(schema: type[BaseLanceTable]) -> list[dict[str, Any]]: + """Fetch every row of ``schema``'s table, no filter. + + Phase 2 doesn't care whether a row already carries a vector — the + cluster strategies re-embed the row's text themselves (see + ``trigger_profile_clustering`` / ``trigger_skill_clustering``); it + just needs every existing episode / agent case so it can + synthesize the trigger event Tier 1's gated-off strategies never + emitted (embed-requiring strategies are body-guarded off when + ``get_embedding_capability().available`` is false — see + :mod:`everos.memory.strategies`). + + Callers that need a ``parent_type`` filter (e.g. the Episode scan in + :func:`_run_phase_clusters`, which excludes Reflection-merged rows) + apply it themselves on the returned rows — this helper stays a plain + unfiltered fetch shared by every business table. + """ + table = await get_table(schema.TABLE_NAME, schema) + total = await table.count_rows() + if total == 0: + return [] + return await table.query().limit(total).to_list() + + +def _episode_row_to_event(raw: dict[str, Any]) -> EpisodeExtracted: + """Synthesize the ``EpisodeExtracted(source="pipeline")`` this row's + original pipeline run never emitted — Tier 1 gated + ``trigger_profile_clustering`` off before it could fire (its + per-dispatch body-guard returns early when embedding is + unavailable). + + ``event_id`` carries a ``backfill_`` prefix so ops can tell a + synthesized run apart from a real one in logs / run records. + """ + return EpisodeExtracted( + event_id=f"backfill_{uuid4().hex}", + memcell_id=raw["parent_id"], + episode_entry_id=raw["entry_id"], + episode_text=raw["episode"], + episode_timestamp_ms=to_timestamp_ms(raw["timestamp"]), + owner_id=raw["owner_id"], + session_id=raw.get("session_id"), + app_id=raw.get("app_id", "default"), + project_id=raw.get("project_id", "default"), + source="pipeline", + ) + + +def _agent_case_row_to_event(raw: dict[str, Any]) -> AgentCaseExtracted: + """Synthesize the ``AgentCaseExtracted`` this row's original pipeline + run never emitted, mirroring :func:`_episode_row_to_event`.""" + return AgentCaseExtracted( + event_id=f"backfill_{uuid4().hex}", + memcell_id=raw["parent_id"], + case_entry_id=raw["entry_id"], + task_intent=raw["task_intent"], + quality_score=raw["quality_score"], + case_timestamp_ms=to_timestamp_ms(raw["timestamp"]), + agent_id=raw["owner_id"], + app_id=raw.get("app_id", "default"), + project_id=raw.get("project_id", "default"), + ) + + +def _report_emit_progress(presenter: BackfillPresenter, done: int, total: int) -> None: + if done % _PROGRESS_EVERY == 0 or done == total: + presenter.emit_progress(done, total) + + +async def _emit_synthetic_events( + engine: OfflineEngine, + episodes: list[dict[str, Any]], + cases: list[dict[str, Any]], + *, + presenter: BackfillPresenter, +) -> int: + """Fan every row into ``engine`` as its own synthetic trigger event. + + Skip rows that a prior Phase-2 run already clustered — otherwise a + rerun (after Ctrl-C, or ``everos cascade backfill --phase all --yes`` + on a partially-completed root) would re-cluster the same rows and + grow cluster counts spuriously. ``cluster_repo.find_cluster_id_for_member`` + is O(log N) via the reverse index and is the exact primitive for + this dedup. ``member_type`` values (``"episode"`` / ``"case"``) match + what :func:`trigger_profile_clustering` and + :func:`trigger_skill_clustering` insert on the write path — a mismatch + here would silently disable the skip and re-open the double-cluster + window. + + Episodes first, then agent cases — order doesn't affect correctness + (each event routes to its own strategy independently); it only + keeps the progress readout monotonic. The progress counter advances + for skipped rows too so the readout matches the pre-scan estimate; + ``_ClusterPhaseResult.events_emitted`` reflects "rows processed" + (real emits + already-clustered skips), not "engine.emit calls". + """ + total = len(episodes) + len(cases) + emitted = 0 + for raw in episodes: + existing = await cluster_repo.find_cluster_id_for_member( + member_type="episode", + member_id=raw["entry_id"], + ) + if existing is None: + await engine.emit(_episode_row_to_event(raw)) + emitted += 1 + _report_emit_progress(presenter, emitted, total) + for raw in cases: + existing = await cluster_repo.find_cluster_id_for_member( + member_type="case", + member_id=raw["entry_id"], + ) + if existing is None: + await engine.emit(_agent_case_row_to_event(raw)) + emitted += 1 + _report_emit_progress(presenter, emitted, total) + return emitted + + +def _probe_ome_lock_available() -> bool: + """Probe whether the OME jobstore file lock is free. + + Backfill Phase 2/3 need exclusive access to + ``/.index/sqlite/ome.db.lock``. If another OfflineEngine + (typically ``everos server start``) already holds it, the phase + cannot proceed — better to detect that before printing the phase + header and asking the user to confirm. + + Uses ``LOCK_EX | LOCK_NB`` briefly, then releases. Returns ``True`` + when acquire succeeded (lock was free at probe time) and ``False`` + when :class:`portalocker.LockException` fired (someone else holds it). + + Best-effort UX polish, not a correctness gate: the lock could become + held between the probe and the actual :meth:`OfflineEngine.start`. + That race path still surfaces :class:`EngineLockHeldError`, caught in + the phase runners and reported through the same friendly path. + """ + root = MemoryRoot.resolve() + lock_path = Path(str(root.ome_db) + ".lock") + lock_path.parent.mkdir(parents=True, exist_ok=True) + handle = open(lock_path, "a+") # noqa: SIM115 + try: + try: + portalocker.lock(handle, portalocker.LOCK_EX | portalocker.LOCK_NB) + except portalocker.LockException: + return False + portalocker.unlock(handle) + return True + finally: + handle.close() + + +def _build_cluster_engine() -> OfflineEngine: + """Construct (but do not start) the throw-away OME engine Phase 2 drives. + + ``memory`` may not import ``service`` (the layering rule forbids it — + see ``.claude/rules/architecture.md``), so this cannot reuse + ``service.memorize``'s process-wide engine singleton; it builds its + own instance and registers only the two clustering strategies whose + body-guards short-circuit under Tier 1 (no embedding provider). + It shares the live engine's ``ome_db`` + jobstore path, so if a server is already running against the same + memory root, ``engine.start()`` fails fast on the file lock instead + of racing it — stop the server before running backfill. + ``config_path`` is deliberately left unset: a one-shot migration has + no use for hot-reloadable per-strategy overrides, and skipping it + means this never depends on ``ome.toml`` existing. + + ``crash_recovery_enabled=False`` prevents this backfill-scoped engine + from re-enqueueing the server's stale RUNNING rows into its own APS + scheduler. The backfill engine only registers the Phase-2 clustering + strategies, so a stale row for any other strategy (e.g. + ``extract_atomic_facts``, ``extract_agent_case``) would hit + ``StrategyRegistry.get`` with an unknown name and raise on dispatch, + permanently marking the event CRASHED-but-not-recovered. Those rows + stay put and are resumed on the next server restart, from an engine + that actually knows those strategy names. See PR #361 review + finding M10. + """ + root = MemoryRoot.resolve() + root.ome_db.parent.mkdir(parents=True, exist_ok=True) + engine = OfflineEngine( + config=OMEConfig( + jobstore_path=root.ome_db, + crash_recovery_enabled=False, + ) + ) + engine.register(trigger_profile_clustering) + engine.register(trigger_skill_clustering) + return engine + + +async def _ensure_cluster_schema() -> None: + """Create every sqlite table (incl. ``cluster``/``cluster_member``) if + this is the first touch of this memory root's system db. + + Phase 1 only touches LanceDB, whose tables auto-create on + ``get_table``. Phase 2 is the first backfill phase to write + through ``cluster_repo``. The CLI ``backfill`` command wraps this + coroutine in the cascade ``_runtime()`` context (same as the + ``sync`` / ``status`` / ``fix`` siblings), which already runs + ``metadata.create_all``; this call is defense-in-depth for other + entry points that invoke the phase runners directly. Idempotent + (``create_all`` no-ops on existing tables). + """ + engine = get_engine() + async with engine.begin() as conn: + await conn.run_sync(SQLModel.metadata.create_all) + + +async def _run_phase_clusters( + *, auto_yes: bool, presenter: BackfillPresenter +) -> _ClusterPhaseResult: + """Rebuild clusters for every episode / agent case via synthetic events. + + Scan every episode + agent case row → surface the estimate → + confirm once → synthesize the ``EpisodeExtracted`` / + ``AgentCaseExtracted`` event each row's original pipeline run + would have emitted had the clustering strategies not been gated + off under Tier 1 (their embed-requiring body-guards short-circuited + the dispatch) → replay them through a dedicated + engine that registers only those two (now-eligible, since embed + is available) strategies → wait for the engine to drain. + + Idempotent: :func:`_emit_synthetic_events` filters each row through + :meth:`cluster_repo.find_cluster_id_for_member` before emitting, so + rerunning the phase (or triggering it from ``--phase all`` after a + Ctrl-C interruption) skips rows already attached to a cluster. + Cluster counts stop growing spuriously across reruns. + + Episode rows carrying ``parent_type == "cluster"`` are Reflection's + merged episodes (``orchestrator._write_merged_episode``), not source + pipeline events, and are excluded — mirrors the same + ``parent_type == "memcell"`` filter idiom used by + ``extract_user_profile._select_via_cluster``. Synthesizing an event + for one would carry a bogus ``memcell_id`` (actually a cluster id) + and defeat ``trigger_profile_clustering``'s own + ``applies_to=lambda e: e.source == "pipeline"`` exclusion of + Reflection output. + """ + # Preflight capability + OME lock BEFORE any collection work. Both + # are cheap checks; either failing makes the run impossible, so + # doing them before ``_scan_all_rows`` (which is bounded but not + # free) matches the Phase-1 defense-in-depth invariant. Capability + # first: it's a permanent config gap, while the OME lock is + # transient (stop the server and try again). + capability = get_embedding_capability() + if not capability.available: + from everos.core.errors import ProviderNotConfiguredError + + err = ProviderNotConfiguredError(provider="embedding", feature="backfill") + presenter.capability_missing( + provider="embedding", feature="backfill", message=str(err) + ) + return _ClusterPhaseResult(blocked_by_capability="embedding") + + if not await anyio.to_thread.run_sync(_probe_ome_lock_available): + presenter.server_running() + return _ClusterPhaseResult(aborted=True, blocked_by_server=True) + + episodes = [ + row + for row in await _scan_all_rows(Episode) + if row.get("parent_type") == "memcell" + ] + cases = await _scan_all_rows(AgentCase) + total = len(episodes) + len(cases) + if total == 0: + presenter.nothing_to_backfill( + "Nothing to backfill. No episodes or agent cases found." + ) + return _ClusterPhaseResult() + + presenter.estimate_clusters(len(episodes), len(cases)) + if not await presenter.confirm( + f"proceed with {total:,} memories " + f"({len(episodes):,} episodes, {len(cases):,} agent cases)", + auto_yes=auto_yes, + ): + return _ClusterPhaseResult(aborted=True) + + await _ensure_cluster_schema() + clusters_before = await cluster_repo.count() + + engine = _build_cluster_engine() + try: + await engine.start() + except EngineLockHeldError: + # Race window: preflight succeeded, but a server started between + # probe and engine.start(). Same friendly path — no traceback. + logger.warning("cascade_backfill_clusters_lock_race") + presenter.server_running() + return _ClusterPhaseResult(aborted=True, blocked_by_server=True) + try: + emitted = await _emit_synthetic_events( + engine, episodes, cases, presenter=presenter + ) + drained = await engine.wait_idle(timeout=_CLUSTER_WAIT_TIMEOUT_SECONDS) + if not drained: + logger.warning( + "cascade_backfill_clusters_wait_timeout", + timeout=_CLUSTER_WAIT_TIMEOUT_SECONDS, + ) + finally: + await engine.stop() + + clusters_after = await cluster_repo.count() + presenter.phase_2_complete(clusters_before, clusters_after, emitted) + return _ClusterPhaseResult( + events_emitted=emitted, + clusters_before=clusters_before, + clusters_after=clusters_after, + drained=drained, + ) + + +# ── Phase 3 (skills) — synthetic-event replay + cascade sync ──────────────── + +_SKILLS_WAIT_TIMEOUT_SECONDS = 300.0 +"""Same rationale as :data:`_CLUSTER_WAIT_TIMEOUT_SECONDS`: cap on waiting +for the ephemeral engine to drain.""" + + +@dataclasses.dataclass(frozen=True) +class _SkillSourceRow: + """One (case, cluster) pairing Phase 3 replays as a synthetic + ``SkillClusterUpdated`` event. + + Mirrors one row of the agent-case-kind cluster's membership + (``ClusterMember`` via :meth:`cluster_repo.list_for_owner`) — the + same shape :func:`trigger_skill_clustering` itself emits per fresh + case, just synthesized after the fact for a case Phase 3 finds + already clustered but not yet skill-extracted. + """ + + case_entry_id: str + cluster_id: str + agent_id: str + app_id: str + project_id: str + + +@dataclasses.dataclass +class _SkillPhaseResult: + """Phase 3 outcome. + + ``events_emitted`` counts every synthesized ``SkillClusterUpdated`` + event fanned into the ephemeral engine — one per + :class:`_SkillSourceRow`. ``skills_before`` / ``skills_after`` are + the total ``agent_skill`` row count (:meth:`agent_skill_repo.count`) + taken immediately before dispatch and after both the engine drains + and the follow-up cascade sync runs (see :func:`_sync_new_skill_files`), + so the caller reports real, already-searchable growth rather than a + number that only becomes true once some future cascade run catches + up. ``drained`` / ``aborted`` mirror :class:`_ClusterPhaseResult`. + """ + + events_emitted: int = 0 + skills_before: int = 0 + skills_after: int = 0 + drained: bool = True + aborted: bool = False + blocked_by_server: bool = False + blocked_by_capability: str | None = None + + +async def _skill_md_exists_for_cluster( + *, + cluster_id: str, + agent_id: str, + app_id: str, + project_id: str, + memory_root: MemoryRoot, +) -> bool: + """Check whether any ``SKILL.md`` under this agent's skills dir + already carries ``cluster_id`` in its frontmatter. + + ``extract_agent_skill`` writes ``SKILL.md`` first and only reaches + LanceDB via the cascade sync afterwards + (see :func:`_sync_new_skill_files`). If a Phase-3 run is + interrupted (Ctrl-C during ``engine.wait_idle`` or between + ``engine.stop`` and the sync), some clusters will have their + ``SKILL.md`` on disk but no LanceDB row. Round-1's idempotency + probe checked only ``agent_skill_repo.count_in_cluster``, so a + later Phase-3 would re-fire ``SkillClusterUpdated`` and produce a + duplicate skill on the next drain. The disk check closes that + window. + + ``skill_name`` is LLM-generated (see + :class:`AgentSkillFrontmatter.name`) so the cluster→md-path + mapping isn't stable — we scan the agent's ``skills/`` directory + for any ``skill_*/SKILL.md`` whose frontmatter names this + ``cluster_id``. This is O(existing skills for the agent), which + is bounded by the agent's own history; Phase 3 already runs a + per-cluster preflight, so an extra frontmatter parse per cluster + is proportionate. + """ + skills_dir = ( + memory_root.agents_dir(app_id, project_id) + / agent_id + / AgentSkillFrontmatter.SKILLS_CONTAINER_NAME + ) + apath = anyio.Path(skills_dir) + if not await apath.is_dir(): + return False + async for skill_dir in apath.iterdir(): + if not await skill_dir.is_dir(): + continue + if not skill_dir.name.startswith(AgentSkillFrontmatter.SKILL_DIR_PREFIX): + continue + md_path = skill_dir / AgentSkillFrontmatter.SKILL_MAIN_FILENAME + if not await md_path.is_file(): + continue + try: + parsed = await MarkdownReader.read(Path(str(md_path))) + except Exception as exc: + # A malformed SKILL.md is not fatal — leave the cluster + # ineligible for the disk-based skip so the scan falls + # back to the LanceDB probe. + logger.warning( + "cascade_backfill_skill_md_parse_failed", + md_path=str(md_path), + error=repr(exc), + ) + continue + if parsed.frontmatter.get("cluster_id") == cluster_id: + return True + return False + + +async def _scan_skill_source() -> list[_SkillSourceRow]: + """Enumerate every clustered agent case Phase 3 should (re-)replay. + + ``trigger_skill_clustering`` runs on the ``agent_case`` track + (``owner_type == "agent"``, ``kind == "agent_case"`` — see + ``Cluster.kind`` docstring); Phase 2's throwaway engine + (:func:`_build_cluster_engine`) registers that strategy but not + ``extract_agent_skill``, so every ``SkillClusterUpdated`` it emitted + during Phase 2 had no listener and was dropped. This scan walks + every agent-owned cluster :meth:`cluster_repo.list_distinct_owners` + has ever seen and fans each cluster's members out into one row — + Phase 3's job is to replay the event those members never got + handled for. + + Idempotent: a cluster is skipped when EITHER + ``agent_skill_repo.count_in_cluster`` > 0 (LanceDB knows about it) + OR :func:`_skill_md_exists_for_cluster` returns ``True`` (a + ``SKILL.md`` on disk names this cluster). The disk check closes + the failure window where ``extract_agent_skill`` wrote the md but + Phase 3 was interrupted before ``_sync_new_skill_files`` ran — a + later re-run would otherwise re-extract every such cluster and + double the skill count. + """ + memory_root = MemoryRoot.resolve() + rows: list[_SkillSourceRow] = [] + owners = await cluster_repo.list_distinct_owners() + for owner_id, owner_type, app_id, project_id in owners: + if owner_type != "agent": + continue + clusters = await cluster_repo.list_for_owner( + owner_id, "agent_case", app_id=app_id, project_id=project_id + ) + for cluster in clusters: + assert cluster.id is not None # persisted clusters always carry an id + already_extracted = await agent_skill_repo.count_in_cluster( + owner_id=owner_id, cluster_id=cluster.id + ) + if already_extracted: + continue + if await _skill_md_exists_for_cluster( + cluster_id=cluster.id, + agent_id=owner_id, + app_id=app_id, + project_id=project_id, + memory_root=memory_root, + ): + logger.info( + "cascade_backfill_skill_md_present_skip_recluster", + cluster_id=cluster.id, + agent_id=owner_id, + ) + continue + rows.extend( + _SkillSourceRow( + case_entry_id=case_entry_id, + cluster_id=cluster.id, + agent_id=owner_id, + app_id=app_id, + project_id=project_id, + ) + for case_entry_id in cluster.members + ) + return rows + + +def _skill_source_to_event(row: _SkillSourceRow) -> SkillClusterUpdated: + """Synthesize the ``SkillClusterUpdated`` Phase 2's clustering pass + emitted into a void — see :func:`_scan_skill_source`.""" + return SkillClusterUpdated( + event_id=f"backfill_{uuid4().hex}", + case_entry_id=row.case_entry_id, + cluster_id=row.cluster_id, + agent_id=row.agent_id, + app_id=row.app_id, + project_id=row.project_id, + ) + + +def _build_skill_engine() -> OfflineEngine: + """Construct (but do not start) the throw-away OME engine Phase 3 drives. + + Mirrors :func:`_build_cluster_engine`, registering only + ``extract_agent_skill`` — the strategy whose body-guard short-circuits + under Tier 1 (no embedding provider) and the one Phase 2's own + throwaway engine never registered. + + ``crash_recovery_enabled=False`` for the same reason as + :func:`_build_cluster_engine`: this Phase-3 engine registers only + ``extract_agent_skill``, so re-enqueueing a server's stale RUNNING + row for any other strategy would raise on dispatch and permanently + lose the event. See PR #361 review finding M10. + """ + root = MemoryRoot.resolve() + root.ome_db.parent.mkdir(parents=True, exist_ok=True) + engine = OfflineEngine( + config=OMEConfig( + jobstore_path=root.ome_db, + crash_recovery_enabled=False, + ) + ) + engine.register(extract_agent_skill) + return engine + + +async def _sync_new_skill_files() -> None: + """Drain the cascade queue once so the ``SKILL.md`` files + ``extract_agent_skill`` just wrote land in LanceDB. + + Phases 1 and 2 write their own storage directly (LanceDB ``vector`` + column; the sqlite ``cluster`` table) and are immediately consistent + once their own call returns. Phase 3's strategy only writes markdown + (:class:`AgentSkillWriter`) and relies on cascade for the LanceDB + side — so, unlike the other two phases, it has to explicitly ask + cascade to catch up rather than being done as soon as the engine + drains, or ``skills_after`` would under-report until some later, + unrelated cascade run happens to pick the files up. + """ + # Scope the sync to ``agent_skill`` only: an unscoped sweep would + # walk every registered kind including knowledge_document / + # knowledge_topic, and if the process has embedding but not rerank, + # the knowledge handlers are gated off — cascade would then mark + # every unseen knowledge md as permanently failed. A backfill run + # must not pollute unrelated kinds' queue state. + orchestrator = CascadeOrchestrator( + memory_root=MemoryRoot.resolve(), tokenizer=build_tokenizer() + ) + await orchestrator.sync_once(kinds={"agent_skill"}) + + +async def _run_phase_skills( + *, auto_yes: bool, presenter: BackfillPresenter +) -> _SkillPhaseResult: + """Extract agent skills from Phase 2's clustered agent cases. + + Preflight → sync orphan ``SKILL.md`` files → scan every agent-case + cluster missing a skill → surface the estimate → confirm once → + replay the ``SkillClusterUpdated`` event each clustered case never + got handled for (see :func:`_scan_skill_source`) through a + dedicated engine that registers only ``extract_agent_skill`` → + wait for it to drain. + + Sync runs BEFORE :func:`_scan_skill_source` — not after the drain — + for the mid-Phase-3 Ctrl-C recovery path: ``extract_agent_skill`` + writes ``SKILL.md`` first and only reaches LanceDB via cascade + afterwards, so an interrupted Phase 3 can leave orphan ``SKILL.md`` + files on disk. Group F's on-disk idempotency probe + (:func:`_skill_md_exists_for_cluster`) skips clusters whose + ``SKILL.md`` already exists, so on a rerun ``_scan_skill_source`` + can legitimately return an empty list even though those md files + aren't yet indexed. Running sync unconditionally, before the scan, + guarantees those orphans get picked up regardless of whether the + scan finds fresh work. :func:`_sync_new_skill_files` is a no-op + when the cascade queue is empty, so the cost is one + ``scan_once`` + ``drain_until_empty`` round-trip. + """ + await _ensure_cluster_schema() + + # Preflight capability + OME lock BEFORE the (potentially large) + # cluster + membership scan. Phase 3 depends on Phase 2's + # clusters, which only exist if embed was available at OME + # startup — the feature label reflects what Phase 3 actually does + # (skill extraction) so the toml hint names the right context. + # Ordering mirrors :func:`_run_phase_clusters`. + capability = get_embedding_capability() + if not capability.available: + from everos.core.errors import ProviderNotConfiguredError + + err = ProviderNotConfiguredError( + provider="embedding", feature="skill_extraction_backfill" + ) + presenter.capability_missing( + provider="embedding", + feature="skill_extraction_backfill", + message=str(err), + ) + return _SkillPhaseResult(blocked_by_capability="embedding") + + if not await anyio.to_thread.run_sync(_probe_ome_lock_available): + presenter.server_running() + return _SkillPhaseResult(aborted=True, blocked_by_server=True) + + # Recover orphan SKILL.md files a prior interrupted Phase 3 may + # have left on disk (see docstring). Runs before the scan so the + # on-disk idempotency probe can't strand those md files unindexed. + await _sync_new_skill_files() + + source_rows = await _scan_skill_source() + if not source_rows: + presenter.nothing_to_backfill( + "Nothing to backfill. No agent skill clusters found." + ) + return _SkillPhaseResult() + + cluster_count = len({row.cluster_id for row in source_rows}) + presenter.estimate_skills(len(source_rows), cluster_count) + if not await presenter.confirm( + f"proceed with {len(source_rows):,} agent case(s) across " + f"{cluster_count:,} cluster(s)", + auto_yes=auto_yes, + ): + return _SkillPhaseResult(aborted=True) + + skills_before = await agent_skill_repo.count() + + engine = _build_skill_engine() + try: + await engine.start() + except EngineLockHeldError: + # Race window mirror of the Phase 2 handling in _run_phase_clusters. + logger.warning("cascade_backfill_skills_lock_race") + presenter.server_running() + return _SkillPhaseResult(aborted=True, blocked_by_server=True) + try: + emitted = 0 + for row in source_rows: + await engine.emit(_skill_source_to_event(row)) + emitted += 1 + _report_emit_progress(presenter, emitted, len(source_rows)) + drained = await engine.wait_idle(timeout=_SKILLS_WAIT_TIMEOUT_SECONDS) + if not drained: + logger.warning( + "cascade_backfill_skills_wait_timeout", + timeout=_SKILLS_WAIT_TIMEOUT_SECONDS, + ) + finally: + await engine.stop() + + # Sync of orphan SKILL.md files ran before the scan (see docstring); + # the new files this drain just wrote also need to land in LanceDB + # before ``skills_after`` is read, or the reported count would + # under-report until some later, unrelated cascade run happens. + await _sync_new_skill_files() + skills_after = await agent_skill_repo.count() + + presenter.phase_3_complete(skills_before, skills_after, emitted) + return _SkillPhaseResult( + events_emitted=emitted, + skills_before=skills_before, + skills_after=skills_after, + drained=drained, + ) + + +@dataclasses.dataclass +class _BackfillSummary: + """Accumulates each phase's result as the orchestrator runs them. + + A field stays ``None`` for a phase that never ran (single-``--phase`` + invocation, or an abort/interrupt before that phase started) — the + printed summary only shows a line for phases that actually executed, + rather than padding unrun phases with fake zeros. + """ + + vectors: _PhaseResult | None = None + clusters: _ClusterPhaseResult | None = None + skills: _SkillPhaseResult | None = None + + +def _count_failed_rows(summary: _BackfillSummary) -> int: + """Sum ``rows_failed`` across every phase that ran. + + Only Phase 1 exposes a row-level failure counter today — Phase 2 + (cluster formation) and Phase 3 (skill extraction) surface failures + through OME strategy retries and logging, not as row-level counters + on their result dataclasses. So only :class:`_PhaseResult` contributes. + Keeping this helper standalone leaves an obvious extension point once + Phase 2/3 grow their own ``rows_failed`` field. + """ + return summary.vectors.rows_failed if summary.vectors else 0 diff --git a/src/everos/memory/cascade/handlers/agent_case.py b/src/everos/memory/cascade/handlers/agent_case.py index 453c053d5..e9f4bcdd9 100644 --- a/src/everos/memory/cascade/handlers/agent_case.py +++ b/src/everos/memory/cascade/handlers/agent_case.py @@ -18,6 +18,8 @@ ``sections``: - ``TaskIntent``: short intent statement (embedded + BM25 primary). + Embedding is a soft dependency: when unavailable, ``vector`` is + written as ``None`` and the row stays BM25/scalar-searchable only. - ``Approach``: step-by-step approach (BM25 secondary field via ``approach_tokens``, **not** fed to the embedder — the retrieval anchor is task_intent). @@ -26,11 +28,15 @@ from __future__ import annotations +from everos.component.embedding import get_embedding_capability +from everos.core.observability.logging import get_logger from everos.infra.persistence.lancedb import AgentCase, ParentType, agent_case_repo from ._common import require_float, require_iso_timestamp from ._daily_log_base import BaseDailyLogHandler, ParsedEntry +logger = get_logger(__name__) + class AgentCaseHandler(BaseDailyLogHandler): """Cascade handler for ``agents//.cases/agent_case-*.md``.""" @@ -62,7 +68,14 @@ async def _build_row( key_insight = (s.sections.get("KeyInsight") or "").strip() or None intent_tokens = self._deps.tokenizer.tokenize(task_intent) approach_tokens = self._deps.tokenizer.tokenize(approach) - vector = await self._deps.embedder.embed(task_intent) + vector = await get_embedding_capability().embed_or_none(task_intent) + if vector is None: + logger.info( + "cascade_handler_embed_skipped", + kind=self.kind, + entry_id=entry.entry_id, + reason="embedding_capability_unavailable", + ) return AgentCase( id=f"{owner_id}_{entry.entry_id}", entry_id=entry.entry_id, diff --git a/src/everos/memory/cascade/handlers/agent_skill.py b/src/everos/memory/cascade/handlers/agent_skill.py index 12046d526..de0d38207 100644 --- a/src/everos/memory/cascade/handlers/agent_skill.py +++ b/src/everos/memory/cascade/handlers/agent_skill.py @@ -27,7 +27,9 @@ Embedding source: ``name + "\\n" + description`` (mirrors opensource AgentSkillExtractor — name belongs in the retrieval anchor too, -not just description). +not just description). Embedding is a soft dependency: when +unavailable, ``vector`` is written as ``None`` and the row stays +BM25/scalar-searchable only. """ from __future__ import annotations @@ -37,6 +39,8 @@ import anyio +from everos.component.embedding import get_embedding_capability +from everos.core.observability.logging import get_logger from everos.core.persistence import MarkdownReader from everos.infra.persistence.lancedb import AgentSkill, agent_skill_repo from everos.infra.persistence.markdown import AgentSkillFrontmatter @@ -46,6 +50,8 @@ from ._common import resolve_scope from .base import Handler +logger = get_logger(__name__) + class AgentSkillHandler(Handler): """Cascade handler for @@ -140,7 +146,14 @@ async def handle_added_or_modified(self, md_path: str) -> HandlerOutcome: content_tokens = " ".join(self._deps.tokenizer.tokenize(content)) # Embedding source: name + description joined (opensource parity). embed_text = "\n".join(s for s in [name, description] if s) - vector = await self._deps.embedder.embed(embed_text) + vector = await get_embedding_capability().embed_or_none(embed_text) + if vector is None: + logger.info( + "cascade_handler_embed_skipped", + kind=self.kind, + entry_id=skill_id, + reason="embedding_capability_unavailable", + ) row = AgentSkill( id=skill_id, diff --git a/src/everos/memory/cascade/handlers/atomic_fact.py b/src/everos/memory/cascade/handlers/atomic_fact.py index a33e94ede..3d1656fd6 100644 --- a/src/everos/memory/cascade/handlers/atomic_fact.py +++ b/src/everos/memory/cascade/handlers/atomic_fact.py @@ -14,16 +14,22 @@ ``sections``: - ``Fact``: the atomic-fact sentence — fed to the embedder and the - tokenizer (``fact_tokens`` BM25 field). + tokenizer (``fact_tokens`` BM25 field). Embedding is a soft + dependency: when unavailable, ``vector`` is written as ``None`` and + the row stays BM25/scalar-searchable only. """ from __future__ import annotations +from everos.component.embedding import get_embedding_capability +from everos.core.observability.logging import get_logger from everos.infra.persistence.lancedb import AtomicFact, ParentType, atomic_fact_repo from ._common import parse_inline_list, require_iso_timestamp from ._daily_log_base import BaseDailyLogHandler, ParsedEntry +logger = get_logger(__name__) + class AtomicFactHandler(BaseDailyLogHandler): """Cascade handler for ``users//.atomic_facts/atomic_fact-*.md``.""" @@ -47,7 +53,14 @@ async def _build_row( s = entry.structured text = s.sections.get("Fact", "").strip() tokens = self._deps.tokenizer.tokenize(text) - vector = await self._deps.embedder.embed(text) + vector = await get_embedding_capability().embed_or_none(text) + if vector is None: + logger.info( + "cascade_handler_embed_skipped", + kind=self.kind, + entry_id=entry.entry_id, + reason="embedding_capability_unavailable", + ) return AtomicFact( id=f"{owner_id}_{entry.entry_id}", entry_id=entry.entry_id, diff --git a/src/everos/memory/cascade/handlers/base.py b/src/everos/memory/cascade/handlers/base.py index d0c8bb477..58cb94c88 100644 --- a/src/everos/memory/cascade/handlers/base.py +++ b/src/everos/memory/cascade/handlers/base.py @@ -6,10 +6,13 @@ the per-kind row shape; everything around it (watcher / scanner / worker / orchestrator / CLI) is kind-agnostic. -Per-kind handlers share the same dependencies bundle — embedding, -tokenizer, memory-root path resolver — packaged as -:class:`HandlerDeps`. Construct it once at orchestrator startup, pass -to every handler factory, no per-row resolution churn. +Per-kind handlers share the same dependencies bundle — tokenizer, +memory-root path resolver — packaged as :class:`HandlerDeps`. +Construct it once at orchestrator startup, pass to every handler +factory, no per-row resolution churn. Embedding is a soft dependency: +handlers fetch it themselves via +:func:`everos.component.embedding.get_embedding_capability` rather +than receiving it here. """ from __future__ import annotations @@ -17,7 +20,6 @@ import abc import dataclasses -from everos.component.embedding import EmbeddingProvider from everos.component.tokenizer import Tokenizer from everos.core.persistence import MemoryRoot @@ -35,7 +37,6 @@ class HandlerDeps: """ memory_root: MemoryRoot - embedder: EmbeddingProvider tokenizer: Tokenizer diff --git a/src/everos/memory/cascade/handlers/episode.py b/src/everos/memory/cascade/handlers/episode.py index e28ac8c87..4c338d742 100644 --- a/src/everos/memory/cascade/handlers/episode.py +++ b/src/everos/memory/cascade/handlers/episode.py @@ -26,17 +26,25 @@ - ``Summary`` (optional): condensed narrative. - ``Content``: full episode narrative — fed to the embedder AND the tokenizer for the ``episode_tokens`` BM25 field. + +Embedding is a soft dependency: when unavailable, both ``vector`` and +``subject_vector`` are written as ``None`` and the row stays +BM25/scalar-searchable only. """ from __future__ import annotations import asyncio +from everos.component.embedding import get_embedding_capability +from everos.core.observability.logging import get_logger from everos.infra.persistence.lancedb import Episode, ParentType, episode_repo from ._common import parse_inline_list, require_iso_timestamp from ._daily_log_base import BaseDailyLogHandler, ParsedEntry +logger = get_logger(__name__) + class EpisodeHandler(BaseDailyLogHandler): """Cascade handler for ``users//episodes/episode-*.md``.""" @@ -72,14 +80,22 @@ async def _build_row( subject_text = s.sections.get("Subject", "").strip() # Embed content and subject concurrently; skip subject embed when absent. + capability = get_embedding_capability() if subject_text: vector, subject_vector = await asyncio.gather( - self._deps.embedder.embed(text), - self._deps.embedder.embed(subject_text), + capability.embed_or_none(text), + capability.embed_or_none(subject_text), ) else: - vector = await self._deps.embedder.embed(text) + vector = await capability.embed_or_none(text) subject_vector = None + if vector is None: + logger.info( + "cascade_handler_embed_skipped", + kind=self.kind, + entry_id=entry.entry_id, + reason="embedding_capability_unavailable", + ) # BM25 tokenization covers both body and subject keywords. tokenize_source = f"{text} {subject_text}" if subject_text else text diff --git a/src/everos/memory/cascade/handlers/foresight.py b/src/everos/memory/cascade/handlers/foresight.py index 50bae7728..e8eb9a948 100644 --- a/src/everos/memory/cascade/handlers/foresight.py +++ b/src/everos/memory/cascade/handlers/foresight.py @@ -21,11 +21,15 @@ ``sections``: - ``Foresight``: forward-looking inference text (embedded + BM25). + Embedding is a soft dependency: when unavailable, ``vector`` is + written as ``None`` and the row stays BM25/scalar-searchable only. - ``Evidence`` (optional): supporting excerpt (secondary BM25 only). """ from __future__ import annotations +from everos.component.embedding import get_embedding_capability +from everos.core.observability.logging import get_logger from everos.infra.persistence.lancedb import Foresight, ParentType, foresight_repo from ._common import ( @@ -36,6 +40,8 @@ ) from ._daily_log_base import BaseDailyLogHandler, ParsedEntry +logger = get_logger(__name__) + class ForesightHandler(BaseDailyLogHandler): """Cascade handler for ``users//.foresights/foresight-*.md``.""" @@ -68,7 +74,14 @@ async def _build_row( text = s.sections.get("Foresight", "").strip() evidence = (s.sections.get("Evidence") or "").strip() or None tokens = self._deps.tokenizer.tokenize(text) - vector = await self._deps.embedder.embed(text) + vector = await get_embedding_capability().embed_or_none(text) + if vector is None: + logger.info( + "cascade_handler_embed_skipped", + kind=self.kind, + entry_id=entry.entry_id, + reason="embedding_capability_unavailable", + ) evidence_tokens = ( " ".join(self._deps.tokenizer.tokenize(evidence)) if evidence else None ) diff --git a/src/everos/memory/cascade/handlers/knowledge_topic.py b/src/everos/memory/cascade/handlers/knowledge_topic.py index 80e2cdfd4..1b2cdd9ec 100644 --- a/src/everos/memory/cascade/handlers/knowledge_topic.py +++ b/src/everos/memory/cascade/handlers/knowledge_topic.py @@ -25,6 +25,8 @@ semantic drift. Embedding source: ``summary`` (mirrors the search recaller's anchor). +Embedding is a soft dependency: when unavailable, ``vector`` is +written as ``None`` and the row stays BM25/scalar-searchable only. """ from __future__ import annotations @@ -32,7 +34,9 @@ import json from typing import Any, ClassVar +from everos.component.embedding import get_embedding_capability from everos.component.utils.datetime import get_utc_now +from everos.core.observability.logging import get_logger from everos.core.persistence import MarkdownReader, ParsedMarkdown from everos.infra.persistence.lancedb import KnowledgeTopic, knowledge_topic_repo from everos.infra.persistence.sqlite import ( @@ -45,6 +49,8 @@ from ._common import resolve_scope from .base import Handler +logger = get_logger(__name__) + class KnowledgeTopicHandler(Handler): """Cascade handler for @@ -191,7 +197,14 @@ async def _build_lance_row( content_tokens = " ".join( self._deps.tokenizer.tokenize(fields["content"]), ) - vector = await self._deps.embedder.embed(fields["summary"]) + vector = await get_embedding_capability().embed_or_none(fields["summary"]) + if vector is None: + logger.info( + "cascade_handler_embed_skipped", + kind=self.kind, + entry_id=fields["node_id"], + reason="embedding_capability_unavailable", + ) now = get_utc_now() return KnowledgeTopic( id=fields["node_id"], diff --git a/src/everos/memory/cascade/orchestrator.py b/src/everos/memory/cascade/orchestrator.py index 0049d9aba..8a9972ce8 100644 --- a/src/everos/memory/cascade/orchestrator.py +++ b/src/everos/memory/cascade/orchestrator.py @@ -5,9 +5,11 @@ :meth:`stop` at shutdown. CLI ``cascade sync`` constructs its own instance but only invokes :meth:`drain_once` (no background tasks). -Construction is dependency-injected: the embedding / tokenizer -providers and the memory-root come in as constructor args so tests -can swap them without monkey-patching module-level singletons. +Construction is dependency-injected: the tokenizer provider and the +memory-root come in as constructor args so tests can swap them +without monkey-patching module-level singletons. Embedding is a soft +dependency handled directly by handlers via the capability accessor, +not threaded through here. """ from __future__ import annotations @@ -15,7 +17,6 @@ import asyncio import dataclasses -from everos.component.embedding import EmbeddingProvider from everos.component.tokenizer import Tokenizer from everos.core.observability.logging import get_logger from everos.core.persistence import MemoryRoot @@ -53,7 +54,6 @@ def __init__( self, *, memory_root: MemoryRoot, - embedder: EmbeddingProvider, tokenizer: Tokenizer, config: CascadeConfig | None = None, ) -> None: @@ -61,7 +61,6 @@ def __init__( self._config = config or CascadeConfig() deps = HandlerDeps( memory_root=memory_root, - embedder=embedder, tokenizer=tokenizer, ) self._handlers = build_handlers(deps) @@ -113,14 +112,31 @@ async def stop(self) -> None: self._started = False logger.info("cascade_orchestrator_stopped") - async def sync_once(self) -> int: - """One scan + drain cycle (used by CLI ``cascade sync``). + async def sync_once(self, *, kinds: set[str] | None = None) -> int: + """One scan + drain cycle (used by CLI ``cascade sync`` and + Phase-3 backfill's post-write skill-file sync). + + Args: + kinds: Optional restriction on which md kinds the scan + walks (e.g. ``{"agent_skill"}``). ``None`` — the default + for CLI ``cascade sync`` — scans every registered kind. + Phase 3 uses ``{"agent_skill"}`` so an unscoped sweep + doesn't tag unrelated kinds (notably ``knowledge_*``, + which the current process may not have handlers for) + as permanently failed. Returns the number of rows processed in this drain. The CLI loops on the returned count to know when to stop. + + The ``kinds`` filter is threaded through **both** the scanner + AND the worker drain: the scanner scopes what gets enqueued, + the worker scopes what gets claimed. Round-1 wired only the + scanner side, so a scoped Phase-3 sync could still drain a + knowledge md queued in a prior tick and mark it permanently + failed — round-2 closes that end. """ - await self._scanner.scan_once() - return await self._worker.drain_until_empty() + await self._scanner.scan_once(kinds=kinds) + return await self._worker.drain_until_empty(kinds=kinds) async def drain_once(self) -> int: """Drain the queue exactly once without scanning first.""" diff --git a/src/everos/memory/cascade/registry.py b/src/everos/memory/cascade/registry.py index 34de209e3..05279956b 100644 --- a/src/everos/memory/cascade/registry.py +++ b/src/everos/memory/cascade/registry.py @@ -173,5 +173,24 @@ def build_handlers(deps: HandlerDeps) -> dict[str, Handler]: Returns a ``{kind_name: Handler}`` map used by the worker for dispatch. Constructing once at orchestrator startup keeps the per-row hot path free of factory churn. + + Every handler registers unconditionally. Capability-dependent + steps live inside the handlers themselves: + + - :class:`KnowledgeDocumentHandler` is SQLite-only; no capability + dependency. + - :class:`KnowledgeTopicHandler.handle_added_or_modified` calls + :func:`embed_or_none`, which writes ``vector=None`` when the + embedding capability is unavailable — the column has been + nullable since the embedding-soft-dependency migration. + ``handle_deleted`` is pure SQL/LanceDB delete. + + Prior versions gated the two knowledge handlers off as an atomic + pair when embed OR rerank was unavailable — that gate broke the + delete path (worker marks the row failed with "no handler + registered", stranding SQLite / LanceDB rows after + ``shutil.rmtree`` cleared the md). Search-side gating is + enforced separately at the route level, so cascade does not need + to gate writes. """ return {spec.name: spec.handler_factory(deps) for spec in KIND_REGISTRY} diff --git a/src/everos/memory/cascade/scanner.py b/src/everos/memory/cascade/scanner.py index 87af7a0ce..9a5811365 100644 --- a/src/everos/memory/cascade/scanner.py +++ b/src/everos/memory/cascade/scanner.py @@ -72,17 +72,32 @@ async def stop(self) -> None: self._task = None logger.info("cascade_scanner_stopped") - async def scan_once(self) -> list[ReconcileDecision]: + async def scan_once( + self, *, kinds: set[str] | None = None + ) -> list[ReconcileDecision]: """One scan + reconcile pass; returns the decisions that were upserted into :class:`MdChangeState`. Exposed so the CLI ``cascade sync`` command can trigger a sweep without owning a long-lived scanner task. + + Args: + kinds: Restrict the walk to specific md kinds (e.g. + ``{"agent_skill"}``). ``None`` — the default — walks + every kind in :data:`KIND_REGISTRY`, matching the + background scanner loop's behaviour. """ scan_inputs = await asyncio.to_thread( - _collect_scan_inputs, self._memory_root.root + _collect_scan_inputs, self._memory_root.root, kinds ) state = await _load_state_snapshot() + if kinds is not None: + # reconcile() emits ``deleted`` for every state row missing + # from the scan input — with a kind-scoped scan, the state + # snapshot must be scoped too, or non-``kinds`` rows would + # get spuriously deleted just because we didn't look at + # them this sweep. + state = {p: s for p, s in state.items() if s.kind in kinds} decisions = reconcile(scan_inputs, state) for decision in decisions: await md_change_state_repo.upsert( @@ -113,9 +128,18 @@ async def _run_loop(self) -> None: continue -def _collect_scan_inputs(root: Path) -> list[ScanInput]: +def _collect_scan_inputs(root: Path, kinds: set[str] | None = None) -> list[ScanInput]: """Walk ``root`` once per registered kind, returning every match. + ``kinds`` — when non-``None`` — restricts the walk to specs whose + ``name`` appears in the set; other kinds' md files are neither + stat'd nor emitted. Used by kind-scoped sync callers (e.g. Phase-3 + backfill) so their sweep doesn't touch queue state for unrelated + kinds. The paired state-snapshot filter lives in :meth:`scan_once` + (reconcile itself is kind-agnostic) — both sides must be scoped or + reconcile would emit spurious ``deleted`` decisions for rows the + scan simply chose not to look at. + ``stat()`` failure mode discrimination is **load-bearing**: the reconciler treats "in state but not in scan" as a deletion signal, so if we silently drop a path here under a *transient* OS error, @@ -142,6 +166,8 @@ def _collect_scan_inputs(root: Path) -> list[ScanInput]: """ inputs: list[ScanInput] = [] for spec in KIND_REGISTRY: + if kinds is not None and spec.name not in kinds: + continue for absolute in root.glob(spec.path_glob()): try: mtime = absolute.stat().st_mtime diff --git a/src/everos/memory/cascade/worker.py b/src/everos/memory/cascade/worker.py index 6c7c71e99..eb74855b0 100644 --- a/src/everos/memory/cascade/worker.py +++ b/src/everos/memory/cascade/worker.py @@ -260,7 +260,7 @@ async def stop(self) -> None: self._rebuild_task = None logger.info("cascade_worker_stopped") - async def drain_once(self) -> int: + async def drain_once(self, *, kinds: set[str] | None = None) -> int: """Process one batch, return the number of rows handled. Used by CLI ``cascade sync`` and ``fix --apply`` to flush the @@ -276,8 +276,19 @@ async def drain_once(self) -> int: merged before returning (CLI ``cascade sync``) call :meth:`_flush_optimizers` — :meth:`drain_until_empty` does this on their behalf. + + Args: + kinds: Optional restriction on which ``kind`` values are + claimed. ``None`` (the default) claims across every + kind. Phase-3 backfill passes ``{"agent_skill"}`` so + the drain cannot flip an unrelated queued kind (e.g. a + queued ``knowledge_topic`` md whose handler this + process doesn't have registered) to + ``failed(retryable=False)``. """ - batch = await md_change_state_repo.claim_pending_batch(self._batch_size) + batch = await md_change_state_repo.claim_pending_batch( + self._batch_size, kinds=kinds + ) if not batch: return 0 results = await asyncio.gather(*(self._process_one(row) for row in batch)) @@ -286,7 +297,12 @@ async def drain_once(self) -> int: self._schedule_optimize(kind) return len(batch) - async def drain_until_empty(self, *, max_passes: int = 100) -> int: + async def drain_until_empty( + self, + *, + max_passes: int = 100, + kinds: set[str] | None = None, + ) -> int: """Drain repeatedly until the queue is empty (or ``max_passes``). Returns the total number of rows processed. Bounded passes @@ -298,10 +314,15 @@ async def drain_until_empty(self, *, max_passes: int = 100) -> int: (CLI ``cascade sync``) get a fully merged index — not for visibility (the data is already searchable) but so ``sync`` returns a deterministically optimized state. + + The ``kinds`` filter is forwarded to :meth:`drain_once` on every + pass so the scoping intent holds end-to-end for scoped syncs + (Phase-3 backfill's ``{"agent_skill"}`` sweep); ``None`` keeps + the CLI ``cascade sync`` path drainining every kind. """ total = 0 for _ in range(max_passes): - processed = await self.drain_once() + processed = await self.drain_once(kinds=kinds) if processed == 0: break total += processed diff --git a/src/everos/memory/reflection/orchestrator.py b/src/everos/memory/reflection/orchestrator.py index bb8fe129b..547adc643 100644 --- a/src/everos/memory/reflection/orchestrator.py +++ b/src/everos/memory/reflection/orchestrator.py @@ -114,6 +114,15 @@ async def run( Returns: List of successful ReflectionReport rows (typed as object because the table class lives in infra). + + Note: + No embedding-availability guard here — upstream callers are + responsible for skipping this run when the capability is + missing. In production the ``reflect_episodes`` strategy + body-guards on ``get_embedding_capability().available`` and + then constructs the orchestrator via ``.require()`` (which + raises before ``run()`` can be called if the provider is + missing), so an in-body check would be unreachable. """ candidates = await self._select_candidates( owner_id=owner_id, @@ -1013,7 +1022,7 @@ async def _patch_md_frontmatter( if is_deprecated and ep.md_path: path_to_entries[ep.md_path][ep.entry_id] = merged_entry_id - root = MemoryRoot.default().root + root = MemoryRoot.resolve().root for md_path, deprecated_map in path_to_entries.items(): await self._episode_writer.patch_frontmatter( root / md_path, diff --git a/src/everos/memory/search/manager.py b/src/everos/memory/search/manager.py index 14c1d27e8..99f849ac1 100644 --- a/src/everos/memory/search/manager.py +++ b/src/everos/memory/search/manager.py @@ -36,9 +36,12 @@ from everalgo.rank.fusion import rrf from everalgo.types import Candidate, RankInput +from everos.component.embedding import get_embedding_capability +from everos.component.rerank import get_rerank_capability from everos.component.utils.datetime import to_display_tz from everos.config import load_settings from everos.core.context import resolve_request_id +from everos.core.errors import ProviderNotConfiguredError from everos.core.observability.logging import get_logger from everos.core.observability.tracing import ( capture_input, @@ -754,51 +757,103 @@ def _apply_radius(cands: list[Candidate], radius: float | None) -> list[Candidat # ── Component guards ──────────────────────────────────────────── def _validate_components(self, req: SearchRequest) -> None: - """Fail fast when the chosen method needs components that are missing.""" + """Fail fast when the chosen method needs a provider that is missing. + + Every provider branch (embedding, rerank, LLM) raises the same + :class:`ProviderNotConfiguredError` -> HTTP 422 with a toml-pointing + message; vocabulary is uniform across the three. + + Embedding and rerank checks read BOTH the process-wide capability + singletons AND this manager's own injected providers. Either side + being "missing" is enough to fail the guard: accessor + says-unavailable means the tier never had it; ``self._embedding`` + is ``None`` means whoever built this manager didn't wire one in. + Because dispatch reads ``self._embedding`` directly, letting an + accessor-says-available-but-self-is-None case through would + produce an opaque ``AttributeError('NoneType')`` deep in recall + instead of a clean 422. Single source of truth = both must agree + the provider exists. + + Agent HYBRID has two rerank lanes selected by ``enable_llm_rerank``: + the LLM lane (``True``) reranks via the LLM and needs no rerank + provider; the cross-encoder lane (``False``, default) does. + + Design contract (PR #361 round-3 review #6): this manager is + instantiated by ``service.search`` — a single caller that always + sources its providers from the same capability accessors this + guard reads. On that path DI-injected provider and accessor + state cannot disagree, so the belt-and-suspenders check is a + near-no-op; it only fires on pathological drift (a subclass or + a test that constructs a manager with mismatched inputs). + Library consumers that embed :class:`SearchManager` directly + and hand it their OWN provider will hit 422 even when their + provider is valid, because the accessor reflects the + process-wide toml. This is an accepted limitation of the + current DI shape — the guard prefers false-positives + (over-refuse) to false-negatives (dispatch with a ``None`` + provider) because the latter surfaces as an opaque + ``AttributeError('NoneType')`` deep in recall. + """ method = req.method - needs_embedding = method != SearchMethod.KEYWORD - if needs_embedding and self._embedding is None: - raise RuntimeError( - f"method={method.value!r} requires an embedding provider; " - "configure [embedding] in settings" - ) - # LLM is only mandatory when the caller explicitly opts into - # Phase-5 rerank on HYBRID, or always for AGENTIC (sufficiency - # check + multi-query generation). - if ( - method == SearchMethod.HYBRID - and req.enable_llm_rerank - and self._llm is None + is_agent_hybrid = method == SearchMethod.HYBRID and req.owner_type == "agent" + + needs_embedding = method in ( + SearchMethod.VECTOR, + SearchMethod.HYBRID, + SearchMethod.AGENTIC, + ) + if needs_embedding and ( + not get_embedding_capability().available or self._embedding is None ): - raise RuntimeError( - "method='hybrid' with enable_llm_rerank=true needs an LLM; " - "configure [llm] in settings or drop the flag" + raise ProviderNotConfiguredError( + provider="embedding", + feature=_feature_name(method, req.owner_type), ) - # agent_skill HYBRID without LLM rerank reaches the cross-encoder - # lane; without the reranker it would AttributeError deep in the - # callback. Episode / agent_case HYBRID don't need it. + + # agent HYBRID cross-encoder lane (enable_llm_rerank=False, the + # default) reaches ``search_agent_skills_hybrid``, which needs a + # real rerank provider; the LLM lane reranks via the LLM instead + # and is exempt. if ( - method == SearchMethod.HYBRID - and req.owner_type == "agent" + is_agent_hybrid and not req.enable_llm_rerank - and self._reranker is None + and (not get_rerank_capability().available or self._reranker is None) ): - raise RuntimeError( - "owner_type='agent' with method='hybrid' requires a rerank " - "provider (skill cross-encoder lane); configure [rerank] in " - "settings, or set enable_llm_rerank=true to use the LLM lane" + raise ProviderNotConfiguredError( + provider="rerank", + feature="agent_hybrid", + alternative_hint=( + "Set enable_llm_rerank=true to route through the " + "LLM rerank lane instead of the cross-encoder " + "provider." + ), + ) + + if method == SearchMethod.AGENTIC and ( + not get_rerank_capability().available or self._reranker is None + ): + raise ProviderNotConfiguredError( + provider="rerank", feature="agentic_search" + ) + + # LLM has no Capability wrapper yet; guard on the injected client + # directly. Belt-and-suspenders with the embedding/rerank branches + # above -- same 422 vocabulary via ProviderNotConfiguredError. + is_hybrid_llm_lane = method == SearchMethod.HYBRID and req.enable_llm_rerank + if is_hybrid_llm_lane and self._llm is None: + raise ProviderNotConfiguredError( + provider="llm", + feature=_feature_name(method, req.owner_type), + alternative_hint=( + "drop the enable_llm_rerank flag to use the cross-encoder " + "rerank lane (needs a configured [rerank] provider)" + ), + ) + if method == SearchMethod.AGENTIC and self._llm is None: + raise ProviderNotConfiguredError( + provider="llm", + feature=_feature_name(method, req.owner_type), ) - if method == SearchMethod.AGENTIC: - if self._reranker is None: - raise RuntimeError( - "method='agentic' requires a rerank provider; " - "configure [rerank] in settings" - ) - if self._llm is None: - raise RuntimeError( - "method='agentic' requires an LLM client; " - "configure [llm] in settings" - ) def _scored_as_candidate(scored) -> Candidate: # type: ignore[no-untyped-def] @@ -815,6 +870,27 @@ def _scored_as_candidate(scored) -> Candidate: # type: ignore[no-untyped-def] ) +def _feature_name(method: SearchMethod, owner_type: str) -> str: + """Map a search method (+ owner partition) to its 422 ``feature`` tag. + + HYBRID splits by ``owner_type`` because the episode path (embed only) + and the agent path (embed, plus a rerank lane) have different + prerequisites and deserve distinct log/response tags. + + Note: this vocabulary (``"vector"``, ``"user_hybrid"``, ``"agent_hybrid"``, + ``"agentic_search"``) is separate from the capability-level vocabulary + in :mod:`everos.component.capabilities`. See that module's docstring + for the rationale and contract. + """ + if method == SearchMethod.VECTOR: + return "vector" + if method == SearchMethod.HYBRID: + return "agent_hybrid" if owner_type == "agent" else "user_hybrid" + if method == SearchMethod.AGENTIC: + return "agentic_search" + return method.value + + def _effective_llm_rerank(req: SearchRequest) -> bool: """LLM Phase-5 rerank only kicks in for ``HYBRID`` and only when the caller opts in. ``AGENTIC`` runs its own cross-encoder rerank loop diff --git a/src/everos/memory/strategies/extract_agent_case.py b/src/everos/memory/strategies/extract_agent_case.py index b6e09daee..53d99c560 100644 --- a/src/everos/memory/strategies/extract_agent_case.py +++ b/src/everos/memory/strategies/extract_agent_case.py @@ -44,7 +44,7 @@ def _get_writer() -> AgentCaseWriter: global _writer if _writer is None: - _writer = AgentCaseWriter(root=MemoryRoot.default()) + _writer = AgentCaseWriter(root=MemoryRoot.resolve()) return _writer diff --git a/src/everos/memory/strategies/extract_agent_skill.py b/src/everos/memory/strategies/extract_agent_skill.py index ceccf5dfb..7440e7390 100644 --- a/src/everos/memory/strategies/extract_agent_skill.py +++ b/src/everos/memory/strategies/extract_agent_skill.py @@ -39,11 +39,11 @@ from everalgo.types import AgentSkill as AlgoAgentSkill from everos.component.embedding import ( - EmbeddingNotConfiguredError, EmbeddingServiceError, - get_embedder, + get_embedding_capability, ) from everos.component.llm import get_llm_client +from everos.core.errors import ProviderNotConfiguredError from everos.core.observability.logging import get_logger from everos.core.persistence import MemoryRoot from everos.infra.ome.context import StrategyContext @@ -102,7 +102,7 @@ class _CaseNotYetIndexedError(RuntimeError): def _get_writer() -> AgentSkillWriter: global _writer if _writer is None: - _writer = AgentSkillWriter(root=MemoryRoot.default()) + _writer = AgentSkillWriter(root=MemoryRoot.resolve()) return _writer @@ -113,6 +113,25 @@ def _get_writer() -> AgentSkillWriter: max_retries=3, ) async def extract_agent_skill(event: SkillClusterUpdated, ctx: StrategyContext) -> None: + # Body-guard: capability is checked here for defensive degradation. + # Belt-and-suspenders even though the upstream + # trigger_skill_clustering already gates on the same capability — a + # direct emit of SkillClusterUpdated (tests, future features) should + # still degrade cleanly without an owner lock or OME retry pressure. + # Tier upgrades require a server restart; this guard is not a + # hot-reload mechanism. + # + # ``debug`` level (not ``info``) is intentional; see the body-guard + # in :func:`everos.memory.strategies.trigger_profile_clustering` for + # the shared rationale (PR #361 round-3 review #8). + if not get_embedding_capability().available: + logger.debug( + "strategy_gated_off_embedding_unavailable", + strategy_name="extract_agent_skill", + agent_id=event.agent_id, + ) + return + # Serialise on agent_id: SKILL.md is addressed by (agent_id, skill_name) # — concurrent runs across different clusters of the same agent can # both decide to add the same skill_name and clobber the file. Different @@ -269,10 +288,16 @@ async def _resolve_query_vector(target: LanceAgentCase) -> list[float]: return list(target.vector) if not target.task_intent: return [] + # ``.require()`` is defensive: the strategy body-guard checks + # ``.available`` before we reach this helper, so this branch will + # not raise ``ProviderNotConfiguredError`` in normal operation. The + # catch stays so a misconfiguration surfacing later (or a direct + # unit-test call to this helper without the guard) still degrades + # to ``[]`` instead of blowing up mid-strategy. try: - embedder = get_embedder() + embedder = get_embedding_capability().require() return list(await embedder.embed(target.task_intent)) - except (EmbeddingNotConfiguredError, EmbeddingServiceError) as exc: + except (ProviderNotConfiguredError, EmbeddingServiceError) as exc: logger.warning( "agent_skill_query_embed_failed", case_entry_id=target.entry_id, diff --git a/src/everos/memory/strategies/extract_atomic_facts.py b/src/everos/memory/strategies/extract_atomic_facts.py index b00ae44cf..0bc6a31e3 100644 --- a/src/everos/memory/strategies/extract_atomic_facts.py +++ b/src/everos/memory/strategies/extract_atomic_facts.py @@ -31,7 +31,7 @@ def _get_writer() -> AtomicFactWriter: """Return the lazily-initialised AtomicFactWriter singleton.""" global _writer if _writer is None: - _writer = AtomicFactWriter(root=MemoryRoot.default()) + _writer = AtomicFactWriter(root=MemoryRoot.resolve()) return _writer diff --git a/src/everos/memory/strategies/extract_foresight.py b/src/everos/memory/strategies/extract_foresight.py index 18ac5a531..b00df9e36 100644 --- a/src/everos/memory/strategies/extract_foresight.py +++ b/src/everos/memory/strategies/extract_foresight.py @@ -38,7 +38,7 @@ def _get_writer() -> ForesightWriter: global _writer if _writer is None: - _writer = ForesightWriter(root=MemoryRoot.default()) + _writer = ForesightWriter(root=MemoryRoot.resolve()) return _writer diff --git a/src/everos/memory/strategies/extract_user_profile.py b/src/everos/memory/strategies/extract_user_profile.py index c4c9a6e29..711eca096 100644 --- a/src/everos/memory/strategies/extract_user_profile.py +++ b/src/everos/memory/strategies/extract_user_profile.py @@ -1,15 +1,34 @@ """extract_user_profile strategy — synthesise the user's profile from clusters. -Listens to :class:`ProfileClusterUpdated` (fired after -``trigger_profile_clustering`` assigns a memcell to a cluster), pulls -the relevant memcells across all "fresh" clusters, and runs -:class:`everalgo.user_memory.ProfileExtractor` to INIT / UPDATE the -user's profile markdown. +Dual-trigger strategy: profile extraction must run whether or not embedding +is configured, so it listens on **two** events and gates itself so exactly +one path fires per memcell (see :func:`_profile_applies`): + +- **Cluster path** (Tier 2+, embedding available): fires on + :class:`ProfileClusterUpdated`, emitted by ``trigger_profile_clustering`` + after it assigns a memcell to a cluster. Pulls the relevant memcells + across all "fresh" clusters (:func:`_select_via_cluster`). +- **Direct path** (Tier 1, no embedding): fires on :class:`EpisodeExtracted` + directly. ``trigger_profile_clustering`` is always registered, but its + per-dispatch body-guard returns early when + :func:`get_embedding_capability` reports unavailable, so no + ``ProfileClusterUpdated`` is ever emitted in Tier 1 — this direct + path is the sole route to profile extraction there. Pulls memcells + via a scalar LanceDB timestamp filter, no embedding required + (:func:`_select_via_timestamp`). + +Both paths converge on the same LLM extraction + markdown persist tail of +``extract_user_profile`` — only memcell *selection* differs. Opensource parity (``mem_memorize.py`` Phase 2): -- **Throttle**: ``total_memcell_count % profile_extraction_interval == 0``; - default interval = 1 (every memcell triggers a re-extraction). +- **Throttle** (both paths): ``total_count % profile_extraction_interval + == 0``; default interval = 1 (every memcell triggers a re-extraction). + Applied at strategy entry so cluster and direct paths honor the same + gate. ``total_count`` is ``sum(c.count for c in user_clusters)`` on the + cluster path and :meth:`episode_repo.count_by_owner` on the direct + path — both express "cumulative units of source-memory for this owner" + and stay ~1:1 in the extract → memcell → episode → cluster pipeline. - **Target clusters**: every cluster whose ``last_ts`` is newer than the user's existing profile timestamp, plus the current cluster (so the freshly-arrived memcell is always counted even when its cluster's @@ -27,15 +46,18 @@ from __future__ import annotations +from everalgo.clustering import Cluster as AlgoCluster from everalgo.types import MemCell as AlgoMemCell from everalgo.types import Profile as AlgoProfile from everalgo.user_memory import ProfileExtractor +from everos.component.embedding import get_embedding_capability from everos.component.llm import get_llm_client from everos.core.observability.logging import get_logger from everos.core.persistence import MemoryRoot from everos.infra.ome.context import StrategyContext from everos.infra.ome.decorator import offline_strategy +from everos.infra.ome.events import BaseEvent from everos.infra.ome.triggers import Immediate from everos.infra.persistence.lancedb import episode_repo from everos.infra.persistence.markdown import ( @@ -45,7 +67,7 @@ ) from everos.infra.persistence.sqlite import cluster_repo, memcell_repo from everos.memory._partition_locks import get_partition_lock -from everos.memory.events import ProfileClusterUpdated +from everos.memory.events import EpisodeExtracted, ProfileClusterUpdated logger = get_logger(__name__) @@ -66,39 +88,185 @@ def _get_writer() -> ProfileWriter: global _writer if _writer is None: - _writer = ProfileWriter(root=MemoryRoot.default()) + _writer = ProfileWriter(root=MemoryRoot.resolve()) return _writer def _get_reader() -> ProfileReader: global _reader if _reader is None: - _reader = ProfileReader(root=MemoryRoot.default()) + _reader = ProfileReader(root=MemoryRoot.resolve()) return _reader +def _profile_applies(event: BaseEvent) -> bool: + """Route exactly one path per memcell to ``extract_user_profile``. + + - :class:`ProfileClusterUpdated` always applies: it is only ever + emitted by ``trigger_profile_clustering``, whose per-dispatch + body-guard short-circuits (returns before emit) whenever + :func:`get_embedding_capability` reports unavailable. Accepting + this event unconditionally is therefore safe — it can only arrive + when embedding is available, so it never overlaps with the direct + path below. + - :class:`EpisodeExtracted` applies only for pipeline-sourced + episodes (not Reflection's merged episodes) while embedding is + unavailable. Once embedding is available, ``trigger_profile_clustering`` + runs to completion and owns the same memcell via + ``ProfileClusterUpdated`` — so the direct path must stand down + here to avoid a double extraction. This check is a live capability + read (not a registration-time snapshot), so a Tier 3→1 downgrade + mid-process is picked up on the next event. + """ + if isinstance(event, ProfileClusterUpdated): + return True + if isinstance(event, EpisodeExtracted): + return event.source == "pipeline" and not get_embedding_capability().available + return False + + +async def _select_via_cluster( + event: ProfileClusterUpdated, + last_profile_ts: int, + user_clusters: list[AlgoCluster], +) -> list[str]: + """Cluster path (Tier 2+): resolve memcells via the user's cluster set. + + The caller (:func:`extract_user_profile`) already fetched + ``user_clusters`` for the strategy-entry throttle; we take it as a + parameter to avoid a redundant :meth:`cluster_repo.list_for_owner` + round-trip. Behavior of the selection step itself is unchanged from + the pre-dual-trigger implementation. + """ + # Pick clusters fresher than the existing profile (always include + # the one we just updated). + target_clusters = [ + c + for c in user_clusters + if c.last_ts > last_profile_ts or c.id == event.cluster_id + ] + if not target_clusters: + return [] + + # Resolve cluster members (episode entry_ids) → memcell_ids. + # Cluster members store episode entry_ids. To reach the memcell + # payloads we look up each episode's parent_id (= memcell_id) + # in LanceDB, skipping merged episodes (parent_type=cluster) + # whose source memcells were already processed before Reflection. + entry_ids = [m for c in target_clusters for m in c.members] + episodes = await episode_repo.find_by_owner_entries( + event.owner_id, + entry_ids, + app_id=event.app_id, + project_id=event.project_id, + ) + return [ + ep.parent_id for ep in episodes if ep.parent_type == "memcell" and ep.parent_id + ] + + +async def _select_via_timestamp( + event: EpisodeExtracted, last_profile_ts: int +) -> list[str]: + """Direct path (Tier 1): resolve memcells from the event + a LanceDB supplement. + + Always includes the current event's ``memcell_id`` — it is the + trigger for this run and its memcell is the one whose profile we + care about. The LanceDB scan is a best-effort supplement that adds + older memcells still eligible under ``last_profile_ts``; the cascade + daemon may not have indexed the freshly-arrived row yet (it runs on + its own schedule), so relying on LanceDB alone would miss the first + memory of a fresh Tier-1 install entirely. See + :class:`EpisodeExtracted` for the event-first contract that lets us + dodge this race — it carries ``memcell_id`` precisely so downstream + strategies do not need to poll LanceDB until the row appears. + + Mirrors the cluster path's ``c.id == event.cluster_id`` fallback + (see :func:`_select_via_cluster`): the currently-firing entity is + always in the candidate set even when its timestamp is not strictly + fresher than the last profile — this matters for bulk imports where + every incoming memcell may carry a historical timestamp. + + Order is unstable across runs (the return is a de-duplicated set + materialised to a list); downstream ``memcell_repo.find_by_ids`` + reorders back to the caller's list order, so this does not + destabilise later stages. + """ + memcell_ids: set[str] = {event.memcell_id} + # Column projection: this selector only needs `parent_id` to seed the + # memcell fetch — pulling full rows would drag the two 1024-D vector + # columns across the wire for every historical episode. Raw-dict + # return is contractual when `columns` is set. + supplement = await episode_repo.list_by_owner_after_ts( + owner_id=event.owner_id, + after_ts=last_profile_ts, + parent_type="memcell", + app_id=event.app_id, + project_id=event.project_id, + columns=["parent_id"], + ) + memcell_ids.update(row["parent_id"] for row in supplement if row.get("parent_id")) + return list(memcell_ids) + + @offline_strategy( name="extract_user_profile", - trigger=Immediate(on=[ProfileClusterUpdated]), + trigger=Immediate(on=[ProfileClusterUpdated, EpisodeExtracted]), + applies_to=_profile_applies, emits=[], max_retries=2, ) async def extract_user_profile( - event: ProfileClusterUpdated, ctx: StrategyContext + event: ProfileClusterUpdated | EpisodeExtracted, ctx: StrategyContext ) -> None: # Serialise on owner_id: user.md is a single per-user file and the # body is a read → LLM merge → overwrite sequence. Different users # run fully in parallel. partition = f"{event.app_id}:{event.project_id}:{event.owner_id}" async with get_partition_lock("extract_user_profile", partition): - # 1. Throttle: skip unless the Nth clustered memcell tick lands here. - user_clusters = await cluster_repo.list_for_owner( + existing = await _get_reader().read( event.owner_id, - "user_memory", + schema=UserProfileFrontmatter, app_id=event.app_id, project_id=event.project_id, ) - total_count = sum(c.count for c in user_clusters) + last_profile_ts = existing[0].profile_timestamp_ms if existing else 0 + + # Unified throttle: both paths gate on "cumulative units of + # source-memory for this owner". Cluster path sums cluster + # counts (pre-refactor semantics); direct path counts owner + # episode rows in LanceDB. Applied here so a bumped interval + # (e.g. 10x to cap LLM cost) throttles both Tier 1 and Tier 2+ + # equally instead of only the cluster path. + user_clusters: list[AlgoCluster] | None + if isinstance(event, ProfileClusterUpdated): + user_clusters = await cluster_repo.list_for_owner( + event.owner_id, + "user_memory", + app_id=event.app_id, + project_id=event.project_id, + ) + total_count = sum(c.count for c in user_clusters) + else: + user_clusters = None + # Scope the counter to parent_type='memcell' so it matches + # _select_via_timestamp's selector below — otherwise Reflection- + # merged rows (parent_type='cluster') inflate the throttle count + # without ever being selectable, firing the gate at the wrong cadence. + # TODO(profile-counter): reads LanceDB and therefore races the + # cascade daemon in the same way _select_via_timestamp used to + # (fresh Tier-1 install → count=0 until cascade catches up). + # The throttle only needs a monotonic per-owner integer, so a + # stale-but-monotonic value is acceptable for now; the followup + # is to source this from a sqlite ``memcell`` count-by-owner + # query — see PR #361 review finding M4. + total_count = await episode_repo.count_by_owner( + event.owner_id, + app_id=event.app_id, + project_id=event.project_id, + parent_type="memcell", + ) + if ( PROFILE_EXTRACTION_INTERVAL > 1 and total_count % PROFILE_EXTRACTION_INTERVAL != 0 @@ -108,43 +276,24 @@ async def extract_user_profile( owner_id=event.owner_id, total_count=total_count, interval=PROFILE_EXTRACTION_INTERVAL, + path="cluster" + if isinstance(event, ProfileClusterUpdated) + else "direct", ) return - # 2. Pick clusters fresher than the existing profile (always include - # the one we just updated). - existing = await _get_reader().read( - event.owner_id, - schema=UserProfileFrontmatter, - app_id=event.app_id, - project_id=event.project_id, - ) - last_profile_ts = existing[0].profile_timestamp_ms if existing else 0 - target_clusters = [ - c - for c in user_clusters - if c.last_ts > last_profile_ts or c.id == event.cluster_id - ] - if not target_clusters: - return + # Memcell selection is the only part that differs by trigger; the + # LLM extraction + persist tail below is shared by both paths. + # user_clusters is always populated on the cluster branch above, + # so the fallback to [] here is unreachable — it exists purely to + # satisfy the type checker without an assert. + if isinstance(event, ProfileClusterUpdated): + memcell_ids = await _select_via_cluster( + event, last_profile_ts, user_clusters or [] + ) + else: + memcell_ids = await _select_via_timestamp(event, last_profile_ts) - # 3. Resolve cluster members (episode entry_ids) → memcell_ids. - # Cluster members store episode entry_ids. To reach the memcell - # payloads we look up each episode's parent_id (= memcell_id) - # in LanceDB, skipping merged episodes (parent_type=cluster) - # whose source memcells were already processed before Reflection. - entry_ids = [m for c in target_clusters for m in c.members] - episodes = await episode_repo.find_by_owner_entries( - event.owner_id, - entry_ids, - app_id=event.app_id, - project_id=event.project_id, - ) - memcell_ids = [ - ep.parent_id - for ep in episodes - if ep.parent_type == "memcell" and ep.parent_id - ] if len(memcell_ids) < PROFILE_MIN_MEMCELLS: logger.info( "profile_extraction_below_min_memcells", @@ -154,7 +303,7 @@ async def extract_user_profile( ) return - # 4. Pull memcell payloads from SQLite, rehydrate to algo types. + # Pull memcell payloads from SQLite, rehydrate to algo types. memcell_rows = await memcell_repo.find_by_ids(memcell_ids) algo_memcells = sorted( (AlgoMemCell.model_validate_json(r.payload_json) for r in memcell_rows), @@ -163,14 +312,14 @@ async def extract_user_profile( if not algo_memcells: return - # 5. Run the LLM extractor — INIT (no prior) or UPDATE (existing). + # Run the LLM extractor — INIT (no prior) or UPDATE (existing). old_profile = _to_algo_profile(existing[0]) if existing else None extractor = ProfileExtractor(llm=get_llm_client()) new_profile = await extractor.aextract( algo_memcells, sender_id=event.owner_id, old_profile=old_profile ) - # 6. Write the fresh profile back to users//user.md. + # Write the fresh profile back to users//user.md. await _persist_profile( new_profile, owner_id=event.owner_id, @@ -180,8 +329,7 @@ async def extract_user_profile( logger.info( "user_profile_extracted", owner_id=event.owner_id, - cluster_count=len(target_clusters), - episode_count=len(episodes), + path="cluster" if isinstance(event, ProfileClusterUpdated) else "direct", memcell_count=len(algo_memcells), mode="UPDATE" if old_profile is not None else "INIT", ) diff --git a/src/everos/memory/strategies/reflect_episodes.py b/src/everos/memory/strategies/reflect_episodes.py index 8f72f2ac3..2d39b8795 100644 --- a/src/everos/memory/strategies/reflect_episodes.py +++ b/src/everos/memory/strategies/reflect_episodes.py @@ -14,7 +14,7 @@ import asyncio -from everos.component.embedding import get_embedder +from everos.component.embedding import get_embedding_capability from everos.component.llm import get_llm_client from everos.core.observability.logging import get_logger from everos.core.persistence import MemoryRoot @@ -43,7 +43,7 @@ def _get_episode_writer() -> EpisodeWriter: """Return the lazily-initialised EpisodeWriter singleton.""" global _episode_writer if _episode_writer is None: - _episode_writer = EpisodeWriter(root=MemoryRoot.default()) + _episode_writer = EpisodeWriter(root=MemoryRoot.resolve()) return _episode_writer @@ -61,6 +61,22 @@ async def reflect_episodes(event: CronTick, ctx: StrategyContext) -> None: event: Cron tick event (unused; triggers the scheduled run). ctx: OME strategy context for emit and logging. """ + # Body-guard: capability is checked here for defensive degradation. + # Reflection re-embeds merged narratives, so it cannot run without + # an embedder — silently no-op instead of raising deep inside the + # orchestrator. Tier upgrades require a server restart; this guard + # is not a hot-reload mechanism. + # + # ``debug`` level (not ``info``) is intentional; see the body-guard + # in :func:`everos.memory.strategies.trigger_profile_clustering` for + # the shared rationale (PR #361 round-3 review #8). + if not get_embedding_capability().available: + logger.debug( + "strategy_gated_off_embedding_unavailable", + strategy_name="reflect_episodes", + ) + return + # Deferred: avoid pulling LLM libs at module import time. from everalgo.user_memory import EpisodeReflector @@ -71,7 +87,7 @@ async def reflect_episodes(event: CronTick, ctx: StrategyContext) -> None: episode_writer=_get_episode_writer(), report_repo=reflection_report_repo, reflector=EpisodeReflector(llm=get_llm_client()), - embedder=get_embedder(), + embedder=get_embedding_capability().require(), ) owners = await cluster_repo.list_distinct_owners() diff --git a/src/everos/memory/strategies/trigger_profile_clustering.py b/src/everos/memory/strategies/trigger_profile_clustering.py index 83b8f0063..3bcc305fc 100644 --- a/src/everos/memory/strategies/trigger_profile_clustering.py +++ b/src/everos/memory/strategies/trigger_profile_clustering.py @@ -14,7 +14,7 @@ from everalgo.clustering import Cluster as AlgoCluster from everalgo.clustering import cluster_by_geometry -from everos.component.embedding import get_embedder +from everos.component.embedding import get_embedding_capability from everos.config import load_settings from everos.core.observability.logging import get_logger from everos.infra.ome.context import StrategyContext @@ -37,6 +37,36 @@ async def trigger_profile_clustering( event: EpisodeExtracted, ctx: StrategyContext ) -> None: + # Body-guard: capability is checked here for defensive degradation. + # When embedding is unavailable we cannot vectorise the episode, so + # the strategy silently no-ops — no work, no owner lock, no OME + # retry pressure. Tier upgrades require a server restart; this + # guard is not a hot-reload mechanism, it just keeps dispatch clean + # when capability was absent at engine start. + # + # Log level is intentionally ``debug`` here (and at the three sibling + # body-guards in ``trigger_skill_clustering``, ``extract_agent_skill``, + # and ``reflect_episodes``) rather than the ``info`` used by + # ``cascade.registry`` and the reflection orchestrator's + # ``cascade_registry_knowledge_gated_off`` / + # ``reflection_orchestrator_disabled`` events. Rationale (PR #361 + # round-3 review #8, accepted intentional asymmetry): the + # registry/orchestrator gate events fire ONCE per process at startup + # and are useful lifecycle signals worth ``info``. The per-dispatch + # body-guards fire on EVERY memorize/reflection dispatch under Tier 1 + # — a busy chat session emits hundreds. At ``info`` they would flood + # structured-log consumers even when the process-wide default level + # is ``WARNING`` (which suppresses stdout but not the sink). ``debug`` + # keeps this per-event signal cheap and opt-in via ``--verbose`` for + # diagnosing why a strategy isn't running. + if not get_embedding_capability().available: + logger.debug( + "strategy_gated_off_embedding_unavailable", + strategy_name="trigger_profile_clustering", + owner_id=event.owner_id, + ) + return + # Serialise on owner_id: the strategy reads the user's full cluster # set, picks merge target by geometry, then upserts — concurrent runs # on the same owner_id would race the read → decide → write cycle. @@ -46,7 +76,12 @@ async def trigger_profile_clustering( partition = f"{event.app_id}:{event.project_id}:{event.owner_id}" async with get_partition_lock("trigger_profile_clustering", partition): # 1. Embed the episode_text into a vector. - vector_list = await get_embedder().embed(event.episode_text) + # ``.require()`` is defensive: the body-guard above already + # returned when the capability was missing, so this cannot raise + # in the guarded path. Routing through the capability keeps a + # single shared provider (one client, one semaphore) per process. + embedder = get_embedding_capability().require() + vector_list = await embedder.embed(event.episode_text) vector = np.asarray(vector_list, dtype=np.float32) # 2. Load this user's existing user-memory clusters (scoped to space). diff --git a/src/everos/memory/strategies/trigger_skill_clustering.py b/src/everos/memory/strategies/trigger_skill_clustering.py index 151d43ccb..88213827c 100644 --- a/src/everos/memory/strategies/trigger_skill_clustering.py +++ b/src/everos/memory/strategies/trigger_skill_clustering.py @@ -23,7 +23,7 @@ from everalgo.clustering import Cluster as AlgoCluster from everalgo.clustering import cluster_by_llm -from everos.component.embedding import get_embedder +from everos.component.embedding import get_embedding_capability from everos.component.llm import get_llm_client from everos.core.observability.logging import get_logger from everos.infra.ome.context import StrategyContext @@ -49,6 +49,23 @@ async def trigger_skill_clustering( event: AgentCaseExtracted, ctx: StrategyContext ) -> None: + # Body-guard: capability is checked here for defensive degradation. + # When embedding is unavailable we cannot vectorise the case, so + # the strategy silently no-ops — no work, no agent lock, no OME + # retry pressure. Tier upgrades require a server restart; this + # guard is not a hot-reload mechanism. + # + # ``debug`` level (not ``info``) is intentional; see the body-guard + # in :func:`everos.memory.strategies.trigger_profile_clustering` for + # the shared rationale (PR #361 round-3 review #8). + if not get_embedding_capability().available: + logger.debug( + "strategy_gated_off_embedding_unavailable", + strategy_name="trigger_skill_clustering", + agent_id=event.agent_id, + ) + return + # Serialise on agent_id: the strategy reads the agent's full cluster # set, lets the LLM decide merge vs. mint, then upserts — concurrent # runs on the same agent_id would race the read → decide → write @@ -67,7 +84,12 @@ async def trigger_skill_clustering( return # 2. Embed the case's task_intent into a vector. - vector_list = await get_embedder().embed(event.task_intent) + # ``.require()`` is defensive: the body-guard above already + # returned when the capability was missing, so this cannot raise + # in the guarded path. Routing through the capability keeps a + # single shared provider (one client, one semaphore) per process. + embedder = get_embedding_capability().require() + vector_list = await embedder.embed(event.task_intent) vector = np.asarray(vector_list, dtype=np.float32) # 3. Load this agent's existing skill clusters (scoped to space). diff --git a/src/everos/service/knowledge.py b/src/everos/service/knowledge.py index a402bb95e..ac95f15aa 100644 --- a/src/everos/service/knowledge.py +++ b/src/everos/service/knowledge.py @@ -29,13 +29,15 @@ from everalgo.rank.fusion import rrf from everalgo.types import Candidate, CategorySpec, KnowledgeMemory, ParsedContent +from everos.component.embedding import get_embedding_capability +from everos.component.rerank import get_rerank_capability from everos.component.utils.datetime import get_utc_now from everos.core.errors import ( - ConfigurationError, DocumentNotFoundError, DuplicateDocumentError, ExtractionEmptyError, PathTraversalError, + ProviderNotConfiguredError, TopicNotFoundError, ) from everos.core.observability.logging import get_logger @@ -66,6 +68,7 @@ _MAX_MINT_RETRIES = 5 _SCOPE_ID_PATTERN = re.compile(r"^[a-zA-Z0-9_.\-@+]+$") _ORIGINAL_DIR_NAME = "_original" +_KNOWLEDGE_FEATURE = "knowledge" class KnowledgeExtractor(Protocol): @@ -519,7 +522,7 @@ async def delete_document( topic_count = await knowledge_topic_sqlite_repo.count_by_doc_id(doc_id) - memory_root = MemoryRoot.default() + memory_root = MemoryRoot.resolve() doc_dir = memory_root.root / Path(row.md_path).parent if await anyio.Path(doc_dir).is_dir(): await anyio.to_thread.run_sync(shutil.rmtree, doc_dir) @@ -605,7 +608,7 @@ async def _backup_doc_dir(doc_id: str) -> tuple[Path, Path] | None: without reverse-engineering the name, or ``None`` when no directory exists. """ - memory_root = MemoryRoot.default() + memory_root = MemoryRoot.resolve() row = await knowledge_document_repo.get_by_doc_id(doc_id) if row is None: return None @@ -844,7 +847,7 @@ async def patch_document( Raises: DocumentNotFoundError: When neither SQLite nor md files contain ``doc_id``. """ - memory_root = MemoryRoot.default() + memory_root = MemoryRoot.resolve() current = await _resolve_current_doc(doc_id, app_id, project_id, memory_root) new_title = title if title is not None else current.title @@ -883,7 +886,7 @@ async def list_categories(app_id: str, project_id: str) -> list[CategoryOverview Returns: List of CategoryOverview with document counts from SQLite. """ - knowledge_dir = MemoryRoot.default().knowledge_dir(app_id, project_id) + knowledge_dir = MemoryRoot.resolve().knowledge_dir(app_id, project_id) await ensure_taxonomy(knowledge_dir) specs = await parse_taxonomy(knowledge_dir / ".taxonomy.md") counts = await knowledge_document_repo.count_by_category(app_id, project_id) @@ -972,7 +975,7 @@ async def _resolve_original_file_path( safe_name = _safe_original_filename(source_name) except PathTraversalError: return None - memory_root = MemoryRoot.default() + memory_root = MemoryRoot.resolve() doc_dir = memory_root.root / Path(md_path).parent candidate = doc_dir / _ORIGINAL_DIR_NAME / safe_name if await anyio.Path(candidate).is_file(): @@ -1007,37 +1010,21 @@ async def _write_original_file( # ── Knowledge search ───────────────────────────────────────────────────────── -# Lazy singleton — mirrors the pattern in service/search.py. -_embedding: EmbeddingProvider | None = None -_embedding_resolved = False - def _get_embedding() -> EmbeddingProvider | None: - """Build the embedding client on first call. ``None`` when not configured.""" - global _embedding, _embedding_resolved - if _embedding_resolved: - return _embedding - - from everos.component.embedding import ( # Deferred: singleton - build_embedding_provider, - ) - from everos.config import load_settings # Deferred: singleton + """Return the process-wide embedding provider, or ``None`` when unset. - cfg = load_settings().embedding - if not cfg.model or cfg.api_key is None: - logger.warning( - "knowledge_embedding_not_configured", - hint="set [embedding] model / api_key to enable vector / hybrid search", - ) - _embedding = None - else: - _embedding = build_embedding_provider(cfg) - logger.info("knowledge_embedding_built", model=cfg.model) - _embedding_resolved = True - return _embedding + Delegates to :func:`get_embedding_capability` instead of keeping a + parallel module-level singleton — mirrors ``service/search.py``, so + this module answers the same "is embedding configured?" question as + the health endpoint and the cascade write path. + """ + return get_embedding_capability().provider -# Lazy singleton — mirrors the pattern for _embedding above. +# Lazy singleton — the recaller itself has no soft-dependency provider to +# delegate to (it wraps a tokenizer, not embedding/rerank), so it keeps its +# own module-level cache rather than a capability accessor. _recaller: KnowledgeTopicRecaller | None = None _recaller_resolved = False @@ -1061,30 +1048,12 @@ def _build_recaller() -> KnowledgeTopicRecaller: return _recaller -# Lazy singleton — mirrors the pattern for _embedding above. -_reranker: RerankProvider | None = None -_reranker_resolved = False - - def _get_reranker() -> RerankProvider | None: - """Build the rerank client on first call. ``None`` when not configured.""" - global _reranker, _reranker_resolved - if _reranker_resolved: - return _reranker - - from everos.component.rerank import ( # Deferred: singleton - build_rerank_provider, - ) - from everos.config import load_settings # Deferred: singleton + """Return the process-wide rerank provider, or ``None`` when unset. - cfg = load_settings().rerank - if not cfg.model or not cfg.base_url: - _reranker = None - else: - _reranker = build_rerank_provider(cfg) - logger.info("knowledge_reranker_built", model=cfg.model, provider=cfg.provider) - _reranker_resolved = True - return _reranker + Delegates to :func:`get_rerank_capability` — mirrors :func:`_get_embedding`. + """ + return get_rerank_capability().provider # ── Search result types ────────────────────────────────────────────────────── @@ -1261,23 +1230,25 @@ async def _to_search_hits( def _require_search_providers() -> tuple[EmbeddingProvider, RerankProvider]: """Return embedding + reranker providers, raising if not configured. + The HTTP router (``entrypoints.api.routes.knowledge``) already gates + both capabilities via ``Depends(_require_knowledge_capabilities)``, so + this check is normally redundant for HTTP callers -- it exists for + any non-HTTP caller of :func:`search_knowledge` that bypasses the + router dependency. + Raises: - ConfigurationError: When the embedding or rerank provider is not - configured (a required setting is missing). + ProviderNotConfiguredError: When the embedding or rerank provider + is not configured (-> HTTP 422 when it does surface via HTTP). """ - embedder = _get_embedding() - if embedder is None: - raise ConfigurationError( - "Embedding provider not configured. " - "Set EVEROS_EMBEDDING__MODEL and EVEROS_EMBEDDING__API_KEY." - ) - reranker = _get_reranker() - if reranker is None: - raise ConfigurationError( - "Rerank provider not configured. " - "Set EVEROS_RERANK__MODEL and EVEROS_RERANK__BASE_URL." + embed = _get_embedding() + if embed is None: + raise ProviderNotConfiguredError( + provider="embedding", feature=_KNOWLEDGE_FEATURE ) - return embedder, reranker + rerank = _get_reranker() + if rerank is None: + raise ProviderNotConfiguredError(provider="rerank", feature=_KNOWLEDGE_FEATURE) + return embed, rerank async def _run_category_pipeline( diff --git a/src/everos/service/memorize.py b/src/everos/service/memorize.py index 9fe806b55..3ce961390 100644 --- a/src/everos/service/memorize.py +++ b/src/everos/service/memorize.py @@ -31,7 +31,6 @@ from everos.component.llm import get_llm_client from everos.config import load_settings from everos.core.context import resolve_request_id -from everos.core.observability.logging import get_logger from everos.core.observability.tracing import memory_span from everos.core.persistence import MemoryRoot from everos.infra.ome.config import OMEConfig @@ -56,8 +55,6 @@ from everos.service._boundary import prepare_cells from everos.service._session_lock import get_session_lock -logger = get_logger(__name__) - class MemorizeResult(BaseModel): """What memorize returns to the caller (route serialises it).""" @@ -85,7 +82,7 @@ def _config_root() -> Path: def _get_episode_writer() -> EpisodeWriter: global _episode_writer if _episode_writer is None: - _episode_writer = EpisodeWriter(MemoryRoot.default()) + _episode_writer = EpisodeWriter(MemoryRoot.resolve()) return _episode_writer @@ -115,14 +112,55 @@ def _get_agent_pipeline() -> AgentMemoryPipeline: return _agent_pipeline +_STRATEGIES_ALWAYS = ( + extract_atomic_facts, + extract_foresight, + extract_agent_case, + extract_user_profile, +) +"""LLM-only strategies — no embed dependency, always registered.""" + +_STRATEGIES_REQUIRE_EMBED = ( + trigger_skill_clustering, + extract_agent_skill, + trigger_profile_clustering, + reflect_episodes, +) +"""Strategies whose body re-embeds or consumes a re-embedded cluster. + +All are registered unconditionally; each guards its own body via +:func:`everos.component.embedding.get_embedding_capability` at execution +time. When capability is unavailable the strategy no-ops (logged at +debug level) rather than raising deep inside its body. + +Why body-guard: the capability singleton at +:mod:`everos.component.embedding.accessor` is process-wide and never +invalidated at runtime — a tier upgrade (Tier 1 → Tier 2 by editing +everos.toml) is applied only after a server restart. Body-guard is +NOT a hot-reload mechanism; it exists for defensive degradation so +that direct-emit paths (unit tests, future features that bypass the +upstream clustering gate) still fail cleanly instead of crashing +inside recall. +""" + + def _get_engine() -> OfflineEngine: """Return the singleton OfflineEngine; constructed + registered on first call. + All strategies (both always-on and embed-requiring) are registered + unconditionally. Embed-requiring strategies guard their own bodies + with :func:`everos.component.embedding.get_embedding_capability`. + + Tier changes require a server restart to take effect: the accessor's + cached capability is not invalidated at runtime. Body-guard keeps + dispatches clean when capability was absent at start; it does not + substitute for restart on config changes. + Lifecycle (start/stop) is wired by ``OmeLifespanProvider``. """ global _ome_engine if _ome_engine is None: - root = MemoryRoot.default() + root = MemoryRoot.resolve() jobstore_path = root.ome_db jobstore_path.parent.mkdir(parents=True, exist_ok=True) engine = OfflineEngine( @@ -131,14 +169,8 @@ def _get_engine() -> OfflineEngine: config_path=root.ome_config, ) ) - engine.register(extract_atomic_facts) - engine.register(extract_foresight) - engine.register(extract_agent_case) - engine.register(trigger_skill_clustering) - engine.register(extract_agent_skill) - engine.register(trigger_profile_clustering) - engine.register(extract_user_profile) - engine.register(reflect_episodes) + for strategy in (*_STRATEGIES_ALWAYS, *_STRATEGIES_REQUIRE_EMBED): + engine.register(strategy) _ome_engine = engine return _ome_engine diff --git a/src/everos/service/search.py b/src/everos/service/search.py index cf9905ff3..8cb6ee85d 100644 --- a/src/everos/service/search.py +++ b/src/everos/service/search.py @@ -7,17 +7,24 @@ Component policy (matches :class:`SearchManager` guards): -* Embedding / rerank / LLM clients are **optional at boot**; they are - built lazily, and only the methods that need them fail (with a clear +* Embedding / rerank / LLM clients are **optional at boot**; the manager + is built lazily and only the methods that need them fail (with a clear message) when the corresponding section of settings is empty. * ``KEYWORD`` searches therefore work without any of the three clients, which makes the endpoint usable in a freshly-installed dev setup. +* All three providers are pulled from their process-wide capability / + singleton accessors (:func:`get_embedding_capability`, + :func:`get_rerank_capability`, :func:`get_llm_client`); this module + never keeps parallel singletons of its own. """ from __future__ import annotations from typing import TYPE_CHECKING +from everos.component.embedding import get_embedding_capability +from everos.component.llm import LLMNotConfiguredError, get_llm_client +from everos.component.rerank import get_rerank_capability from everos.component.tokenizer import build_tokenizer from everos.core.observability.logging import get_logger from everos.memory.search import SearchRequest, SearchResponse @@ -32,100 +39,36 @@ ) if TYPE_CHECKING: - from everos.component.embedding import EmbeddingProvider from everos.component.llm import LLMClient - from everos.component.rerank import RerankProvider logger = get_logger(__name__) -# Lazy singletons ──────────────────────────────────────────────────────── - +# Lazy singleton — the manager itself; every provider it needs comes +# from its own process-wide accessor (see module docstring). _manager: SearchManager | None = None -_embedding: EmbeddingProvider | None = None -_reranker: RerankProvider | None = None -_llm_client: LLMClient | None = None -_embedding_resolved = False -_rerank_resolved = False -_llm_resolved = False - - -def _get_embedding() -> EmbeddingProvider | None: - """Build the embedding client on first call. ``None`` when not configured.""" - global _embedding, _embedding_resolved - if _embedding_resolved: - return _embedding - - from everos.component.embedding import build_embedding_provider - from everos.config import load_settings - - cfg = load_settings().embedding - if not cfg.model or not cfg.api_key or not cfg.api_key.get_secret_value(): - logger.warning( - "embedding_not_configured", - hint="set [embedding] model / api_key to enable vector / hybrid search", - ) - _embedding = None - else: - _embedding = build_embedding_provider(cfg) - logger.info("search_embedding_built", model=cfg.model) - _embedding_resolved = True - return _embedding - - -def _get_reranker() -> RerankProvider | None: - """Build the rerank client on first call. ``None`` when not configured.""" - global _reranker, _rerank_resolved - if _rerank_resolved: - return _reranker - - from everos.component.rerank import build_rerank_provider - from everos.config import load_settings - - cfg = load_settings().rerank - has_key = cfg.api_key and cfg.api_key.get_secret_value() - if not cfg.model or not cfg.base_url or not has_key: - logger.warning( - "rerank_not_configured", - hint="set [rerank] model / api_key / base_url to enable agentic search", - ) - _reranker = None - else: - _reranker = build_rerank_provider(cfg) - logger.info("search_rerank_built", model=cfg.model, provider=cfg.provider) - _rerank_resolved = True - return _reranker def _get_llm_client() -> LLMClient | None: - """Lazily build the LLM client from settings (shared with memorize).""" - global _llm_client, _llm_resolved - if _llm_resolved: - return _llm_client - - from everos.component.llm import build_llm_provider - from everos.config import load_settings - - settings = load_settings() - cfg = settings.llm - if not cfg.api_key or not cfg.api_key.get_secret_value() or not cfg.base_url: + """Return the process-wide LLM client, or ``None`` when unset. + + Delegates to :func:`everos.component.llm.get_llm_client` (which + caches its own singleton, validates ``api_key`` / ``base_url``, and + already wraps the client with ``UsageRecordingClient`` when + observability is enabled — see ``component/llm/client.py``) instead + of maintaining a parallel module-level singleton here. LLM is + optional for search — ``KEYWORD`` works without it — so + :class:`LLMNotConfiguredError` is swallowed into ``None`` and the + manager surfaces a clear error only when a method that actually + needs the LLM is invoked. + """ + try: + return get_llm_client() + except LLMNotConfiguredError: logger.warning( "llm_not_configured", hint="set [llm] api_key / base_url to enable hybrid / agentic search", ) - _llm_client = None - else: - client = build_llm_provider(cfg) - # Record token usage for the hybrid/agentic path, mirroring - # get_llm_client() — otherwise the heaviest LLM spend (query - # decomposition + rerank judge) is invisible in Langfuse. - if settings.observability.enabled: - from everos.component.llm._usage_client import UsageRecordingClient - - client = UsageRecordingClient(client) - _llm_client = client - logger.info("search_llm_built", model=cfg.model) - _llm_resolved = True - return _llm_client + return None def _get_manager() -> SearchManager: @@ -138,8 +81,8 @@ def _get_manager() -> SearchManager: agent_case_recaller=AgentCaseRecaller(deps), agent_skill_recaller=AgentSkillRecaller(deps), profile_recaller=ProfileRecaller(), - embedding=_get_embedding(), - reranker=_get_reranker(), + embedding=get_embedding_capability().provider, + reranker=get_rerank_capability().provider, llm_client=_get_llm_client(), ) return _manager diff --git a/tests/conftest.py b/tests/conftest.py index c069c8eee..0fb8eda8c 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -48,6 +48,59 @@ def _reset_settings_cache() -> Iterator[None]: dt_module._display_tz.cache_clear() +@pytest.fixture(autouse=True) +def _reset_embedding_capability_singleton() -> Iterator[None]: + """Force embedding capability to ``available=False`` for every test. + + ``get_embedding_capability()`` lazily builds a process-wide singleton + on first call (see ``component.embedding.accessor``). Leaving the + cache as ``None`` between tests lets the accessor read ambient + settings — ``~/.everos/everos.toml`` on a developer machine, ``.env`` + when loaded — which turns test outcomes into a function of the host + environment. Pre-seeding a ``Capability(provider=None)`` keeps the + suite hermetic: any body-guard that reads ``.available`` sees + ``False`` unless the test explicitly opts in by re-assigning + ``acc._capability`` (or patching the strategy module's own + ``get_embedding_capability`` reference). + """ + import everos.component.embedding.accessor as acc + from everos.component.embedding import EmbeddingCapability + + acc._capability = EmbeddingCapability(provider=None) + yield + acc._capability = None + + +@pytest.fixture(autouse=True) +def _reset_rerank_capability_singleton() -> Iterator[None]: + """Force rerank capability to ``available=False`` for every test. + + See :func:`_reset_embedding_capability_singleton` for rationale — the + rerank accessor has the same lazy-singleton shape. + """ + import everos.component.rerank.accessor as acc + from everos.component.rerank import RerankCapability + + acc._capability = RerankCapability(provider=None) + yield + acc._capability = None + + +@pytest.fixture(autouse=True) +def _reset_multimodal_capability_singleton() -> Iterator[None]: + """Force multimodal capability to ``available=False`` for every test. + + See :func:`_reset_embedding_capability_singleton` for rationale — the + multimodal accessor has the same lazy-singleton shape. + """ + import everos.component.multimodal.accessor as acc + from everos.component.multimodal import MultimodalLLMCapability + + acc._capability = MultimodalLLMCapability(provider=None) + yield + acc._capability = None + + @pytest.fixture(scope="session") def long_conversation() -> dict: """LoCoMo conv_0 fixture (419 messages, 19 batches, one session).""" diff --git a/tests/e2e/conftest.py b/tests/e2e/conftest.py index 6958fe143..d3c3ea946 100644 --- a/tests/e2e/conftest.py +++ b/tests/e2e/conftest.py @@ -61,7 +61,7 @@ ) # OME strategy modules carry module-level lazy singletons (``_writer`` / -# ``_reader``) that capture ``MemoryRoot.default()`` at first call. They +# ``_reader``) that capture ``MemoryRoot.resolve()`` at first call. They # survive across tests, so the second test writes its output to the # **first test's** tmp_path. Reset all of them per-test. _STRATEGY_SINGLETONS: tuple[tuple[str, tuple[str, ...]], ...] = ( @@ -75,7 +75,7 @@ def _reset_strategy_singletons(monkeypatch: pytest.MonkeyPatch) -> None: """Null every strategy ``_writer`` / ``_reader`` so the next test - rebuilds against its own ``MemoryRoot.default()`` (driven by the + rebuilds against its own ``MemoryRoot.resolve()`` (driven by the fresh ``EVEROS_ROOT`` env var set by the calling fixture). """ for mod_name, attrs in _STRATEGY_SINGLETONS: diff --git a/tests/e2e/test_search_endpoint_e2e.py b/tests/e2e/test_search_endpoint_e2e.py index 5e74ac436..df1edfd51 100644 --- a/tests/e2e/test_search_endpoint_e2e.py +++ b/tests/e2e/test_search_endpoint_e2e.py @@ -40,7 +40,7 @@ from everalgo.clustering import Cluster as AlgoCluster from httpx import ASGITransport, AsyncClient -from everos.component.embedding import get_embedder +from everos.component.embedding import get_embedding_capability from everos.config import load_settings from everos.entrypoints.api.app import create_app from everos.infra.persistence.lancedb import ( @@ -96,21 +96,22 @@ async def client( async with _engine.begin() as _conn: await _conn.run_sync(_SQLModel.metadata.create_all) - # Search service: reset all lazy singletons so each test rebuilds + # Search service: reset the manager singleton so each test rebuilds # against the just-monkey-patched memory root + .env creds. - for attr in ( - "_manager", - "_embedding", - "_reranker", - "_llm_client", - ): - setattr(search_service_mod, attr, None) - for attr in ( - "_embedding_resolved", - "_rerank_resolved", - "_llm_resolved", - ): - setattr(search_service_mod, attr, False) + search_service_mod._manager = None + + # Embedding / rerank / LLM route through their process-wide + # capability / singleton accessors — reset those directly so each + # test rebuilds against the just-monkey-patched memory root + .env + # creds (also covered by the autouse fixtures in + # ``tests/conftest.py``; kept explicit here for locality with the + # rest of this fixture's singleton reset). + embedding_acc = import_module("everos.component.embedding.accessor") + rerank_acc = import_module("everos.component.rerank.accessor") + llm_client_mod = import_module("everos.component.llm.client") + embedding_acc._capability = None + rerank_acc._capability = None + llm_client_mod._llm_client = None app = create_app(lifespan_providers=[]) transport = ASGITransport(app=app) @@ -170,7 +171,7 @@ async def _seed_user_memory_cluster(eps: list[dict], *, owner_id: str) -> None: """ memcell_ids = list({ep["parent_id"] for ep in eps}) centroid_text = eps[0]["episode"] - centroid_vec = await get_embedder().embed(centroid_text) + centroid_vec = await get_embedding_capability().require().embed(centroid_text) await cluster_repo.upsert_with_members( AlgoCluster( id=mint_cluster_id(), @@ -428,14 +429,12 @@ async def _seed_one_agent_corpus(owner: str = "a1") -> None: can rank them (zero vectors are undefined under cosine distance — the dense path returns 0 hits for them). """ - from everos.service.search import _get_embedding - case_intent = "refactor authentication middleware" case_approach = "split provider lookup from session decode" skill_desc = "refactor authentication middleware reliably" skill_body = "step-by-step approach for auth refactors" - embedder = _get_embedding() + embedder = get_embedding_capability().provider if embedder is not None: case_vec, skill_vec = await embedder.embed_batch( [f"{case_intent}\n{case_approach}", f"{skill_desc}\n{skill_body}"] @@ -578,8 +577,6 @@ async def test_hybrid_rerank_bridges_skill_via_case_lineage( surface it on its own; the bridge is the path that does, and LLM rerank keeps it because the topic is genuinely relevant. """ - from everos.service.search import _get_embedding - case_id_with_owner = "a_bridge_ac_1" # mirrors AgentCase.id = "_" case_intent = "refactor authentication middleware" case_approach = "split provider lookup from session decode" @@ -595,7 +592,7 @@ async def test_hybrid_rerank_bridges_skill_via_case_lineage( "independently." ) - embedder = _get_embedding() + embedder = get_embedding_capability().provider assert embedder is not None, "live_llm test requires a real embedder" case_vec, skill_vec = await embedder.embed_batch( [f"{case_intent}\n{case_approach}", f"{skill_desc}\n{skill_body}"] diff --git a/tests/integration/search/conftest.py b/tests/integration/search/conftest.py index 5caffdce7..759599e60 100644 --- a/tests/integration/search/conftest.py +++ b/tests/integration/search/conftest.py @@ -224,15 +224,7 @@ async def search_client( # The search service has its own module-level singletons; reset # those too so re-attach is clean. search_svc = importlib.import_module("everos.service.search") - for attr in ( - "_manager", - "_embedding", - "_reranker", - "_llm_client", - "_embedding_resolved", - "_rerank_resolved", - "_llm_resolved", - ): + for attr in ("_manager", "_llm_client", "_llm_resolved"): if hasattr(search_svc, attr): monkeypatch.setattr( search_svc, @@ -241,6 +233,18 @@ async def search_client( raising=False, ) + # Embedding / rerank now route through the process-wide capability + # accessors (``everos.component.{embedding,rerank}.accessor``) instead + # of module-level singletons on ``everos.service.search`` — reset those + # singletons directly so re-attach is clean. (Also covered by the + # autouse fixtures in ``tests/conftest.py``; kept here for locality with + # the rest of this fixture's singleton reset.) + import everos.component.embedding.accessor as embedding_acc + import everos.component.rerank.accessor as rerank_acc + + monkeypatch.setattr(embedding_acc, "_capability", None, raising=False) + monkeypatch.setattr(rerank_acc, "_capability", None, raising=False) + from everos.entrypoints.api.app import create_app app = create_app() diff --git a/tests/integration/test_api/__init__.py b/tests/integration/test_api/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/integration/test_api/test_health_capabilities.py b/tests/integration/test_api/test_health_capabilities.py new file mode 100644 index 000000000..a822ec08a --- /dev/null +++ b/tests/integration/test_api/test_health_capabilities.py @@ -0,0 +1,327 @@ +"""Tests for GET /health endpoint capabilities and disabled_features response.""" + +from __future__ import annotations + +from pytest import mark, param + +from everos.component.capabilities import compute_disabled_features + + +class TestComputeDisabledFeatures: + """Test the compute_disabled_features helper function.""" + + def test_all_capabilities_available(self) -> None: + """When all capabilities are available, no features are disabled.""" + caps = { + "llm": True, + "embed": True, + "rerank": True, + "multimodal_llm": True, + "parser": True, + } + disabled = compute_disabled_features(caps) + assert disabled == [] + + def test_embedding_unavailable_disables_vector_search_features( + self, + ) -> None: + """When embedding unavailable, vector/hybrid/reflection/skill disabled.""" + caps = { + "llm": True, + "embed": False, + "rerank": True, + "multimodal_llm": True, + "parser": True, + } + disabled = compute_disabled_features(caps) + expected = { + "vector_search", + "hybrid_search", + "reflection", + "skill_extraction", + "knowledge", + } + assert set(disabled) == expected + + def test_rerank_unavailable_disables_agentic_search(self) -> None: + """When rerank is unavailable, agentic_search is disabled.""" + caps = { + "llm": True, + "embed": True, + "rerank": False, + "multimodal_llm": True, + "parser": True, + } + disabled = compute_disabled_features(caps) + assert "agentic_search" in disabled + + def test_both_embed_and_rerank_missing_disables_knowledge(self) -> None: + """When both embed and rerank are missing, knowledge is disabled.""" + caps = { + "llm": True, + "embed": False, + "rerank": False, + "multimodal_llm": True, + "parser": True, + } + disabled = compute_disabled_features(caps) + assert "knowledge" in disabled + + def test_multimodal_and_parser_missing_disables_multimodal_upload(self) -> None: + """When multimodal_llm is unavailable, multimodal_upload is disabled.""" + caps = { + "llm": True, + "embed": True, + "rerank": True, + "multimodal_llm": False, + "parser": True, + } + disabled = compute_disabled_features(caps) + assert "multimodal_upload" in disabled + + def test_parser_missing_disables_multimodal_upload(self) -> None: + """When parser is unavailable, multimodal_upload is disabled.""" + caps = { + "llm": True, + "embed": True, + "rerank": True, + "multimodal_llm": True, + "parser": False, + } + disabled = compute_disabled_features(caps) + assert "multimodal_upload" in disabled + + def test_tier1_only_llm(self) -> None: + """Tier 1 (LLM only): many features are disabled.""" + caps = { + "llm": True, + "embed": False, + "rerank": False, + "multimodal_llm": False, + "parser": False, + } + disabled = compute_disabled_features(caps) + # Tier 1 disables: vector/hybrid/agentic/reflection/skill/knowledge/multimodal + expected = { + "vector_search", + "hybrid_search", + "agentic_search", + "reflection", + "skill_extraction", + "knowledge", + "multimodal_upload", + } + assert set(disabled) == expected + + @mark.parametrize( + "caps,expected_disabled", + [ + param( + { + "llm": True, + "embed": True, + "rerank": True, + "multimodal_llm": True, + "parser": True, + }, + [], + id="all_available", + ), + param( + { + "llm": True, + "embed": False, + "rerank": True, + "multimodal_llm": True, + "parser": True, + }, + [ + "vector_search", + "hybrid_search", + "reflection", + "skill_extraction", + "knowledge", + ], + id="no_embed", + ), + param( + { + "llm": True, + "embed": True, + "rerank": False, + "multimodal_llm": True, + "parser": True, + }, + ["agentic_search", "knowledge"], + id="no_rerank", + ), + ], + ) + def test_combinations( + self, + caps: dict[str, bool], + expected_disabled: list[str], + ) -> None: + """Parametrized tests for various capability combinations.""" + disabled = compute_disabled_features(caps) + assert set(disabled) == set(expected_disabled) + + +@mark.asyncio +async def test_health_endpoint_includes_capabilities() -> None: + """GET /health returns capabilities dict with all 5 provider fields.""" + from httpx import ASGITransport, AsyncClient + + from everos.entrypoints.api.app import create_app + + app = create_app() + transport = ASGITransport(app=app) + async with AsyncClient(transport=transport) as client: + response = await client.get("http://testserver/health") + + assert response.status_code == 200 + data = response.json() + + assert "status" in data + assert "version" in data + assert "capabilities" in data + assert "disabled_features" in data + + caps = data["capabilities"] + assert isinstance(caps, dict) + assert "llm" in caps + assert "embed" in caps + assert "rerank" in caps + assert "multimodal_llm" in caps + assert "parser" in caps + + # All capability values should be boolean + for key, value in caps.items(): + msg = f"Capability {key} should be bool, got {type(value)}" + assert isinstance(value, bool), msg + + # disabled_features should be a list + disabled = data["disabled_features"] + assert isinstance(disabled, list) + for feature in disabled: + assert isinstance(feature, str) + + +@mark.asyncio +@mark.parametrize( + ("embed", "rerank", "multimodal_llm", "parser", "expected_disabled"), + [ + param( + False, + False, + False, + False, + { + "vector_search", + "hybrid_search", + "reflection", + "skill_extraction", + "agentic_search", + "knowledge", + "multimodal_upload", + }, + id="tier1_llm_only", + ), + param( + True, + False, + False, + False, + {"agentic_search", "knowledge", "multimodal_upload"}, + id="tier2_embed_only", + ), + param( + True, + True, + False, + False, + {"multimodal_upload"}, + id="tier3_embed_rerank_only", + ), + param( + True, + True, + True, + True, + set(), + id="full_stack_all_enabled", + ), + ], +) +async def test_health_endpoint_disabled_features_matches_hardcoded_expectation( + monkeypatch, + embed: bool, + rerank: bool, + multimodal_llm: bool, + parser: bool, + expected_disabled: set[str], +) -> None: + """GET /health disabled_features must match a hard-coded expected set. + + Pre-M12 this test asserted ``set(response['disabled_features']) == + set(compute_disabled_features(response['capabilities']))`` — a + tautology, because the endpoint uses the same + :func:`compute_disabled_features` to build the field. Any bug in + the helper would be present on both sides of the equation and go + undetected. Round-2 finding M12 replaces the tautology with a + per-tier hard-coded expected set, driven by parametrisation over + every soft-dependency capability toggle — so a regression in + :func:`compute_disabled_features` (e.g. dropping the + ``knowledge`` disable, mis-spelling a feature name) breaks the + test rather than staying invisible. + """ + from httpx import ASGITransport, AsyncClient + + from everos.component import embedding, multimodal + from everos.component import rerank as rerank_mod + from everos.component.parser import _core as parser_core + from everos.entrypoints.api.app import create_app + + class _StubCapability: + def __init__(self, available: bool) -> None: + self.available = available + + monkeypatch.setattr( + embedding, "get_embedding_capability", lambda: _StubCapability(embed) + ) + monkeypatch.setattr( + rerank_mod, "get_rerank_capability", lambda: _StubCapability(rerank) + ) + monkeypatch.setattr( + multimodal, + "get_multimodal_llm_capability", + lambda: _StubCapability(multimodal_llm), + ) + monkeypatch.setattr(parser_core, "parser_available", lambda: parser) + + # The health route imports the accessors at module load; patch there + # too so the values it actually reads honour the monkeypatch. + from everos.entrypoints.api.routes import health as health_route + + monkeypatch.setattr( + health_route, "get_embedding_capability", lambda: _StubCapability(embed) + ) + monkeypatch.setattr( + health_route, "get_rerank_capability", lambda: _StubCapability(rerank) + ) + monkeypatch.setattr( + health_route, + "get_multimodal_llm_capability", + lambda: _StubCapability(multimodal_llm), + ) + monkeypatch.setattr(health_route, "parser_available", lambda: parser) + + app = create_app() + transport = ASGITransport(app=app) + async with AsyncClient(transport=transport) as client: + response = await client.get("http://testserver/health") + + assert response.status_code == 200 + data = response.json() + + assert set(data["disabled_features"]) == expected_disabled diff --git a/tests/integration/test_api/test_knowledge_gates.py b/tests/integration/test_api/test_knowledge_gates.py new file mode 100644 index 000000000..7a7fc6ddb --- /dev/null +++ b/tests/integration/test_api/test_knowledge_gates.py @@ -0,0 +1,386 @@ +"""Verify per-endpoint scoping of the knowledge capability gate. + +Task 11 (``cascade.registry.build_handlers``) gates the knowledge +cascade handlers off as an atomic pair when embed + rerank aren't +both available — a doc queued for a gated-off kind gets marked +permanently failed by the worker instead of ever reaching LanceDB. +Before this router gate existed, the HTTP layer had no idea any of +that happened: a Tier 1 user's upload returned 200 (the md write +always succeeds) while the document silently never became +searchable. + +An earlier iteration attached the gate router-wide, which regressed +downgraded users: a client who dropped from Tier 3 → Tier 2/1 could +no longer even GET or DELETE their own docs, because the router-wide +dependency fired on every request. This module pins the corrected +scoping — writes and search stay gated (they can't succeed end-to- +end without the providers) while reads/deletes stay reachable so +users can inspect and clean up state they already have on disk. + +Uses ``httpx.AsyncClient`` + ``ASGITransport`` (not +``fastapi.testclient.TestClient``) to match the project's established +exception-handler test convention (see ``tests/integration/test_api/ +test_provider_error_mapping.py``). The knowledge router is mounted +standalone (no lifespan / full app) since the gate fires before any +body is even parsed — sub-dependency resolution runs ahead of +body/query/path validation in FastAPI's request handler, so a bare +request with no payload is enough to reach the gate on every route, +including the multipart upload endpoints. + +For endpoints without a gate we stub the service call so the test +proves the request reached the handler (i.e. no 422 from the +missing-provider check), independent of MemoryRoot / SQLite setup. +""" + +from __future__ import annotations + +from collections.abc import AsyncIterator +from typing import Any + +import pytest +from fastapi import FastAPI +from httpx import ASGITransport, AsyncClient + +import everos.component.embedding.accessor as embedding_accessor +import everos.component.rerank.accessor as rerank_accessor +from everos.component.embedding import EmbeddingCapability +from everos.component.rerank import RerankCapability +from everos.entrypoints.api.exception_handlers import register_handlers +from everos.entrypoints.api.routes import knowledge as knowledge_routes + +# Endpoints that MUST return 422 when embed or rerank is missing. +# These paths either write new md that cascade will need to embed +# (POST/PUT) or run a search that requires both providers +# (POST /search). PATCH is deliberately NOT here — see the ungated +# list below. +_GATED_ENDPOINTS: list[tuple[str, str]] = [ + ("POST", "/api/v1/knowledge/documents"), + ("PUT", "/api/v1/knowledge/documents/d_abcdef123456"), + ("POST", "/api/v1/knowledge/search"), +] + +# Endpoints that MUST stay reachable even when both capabilities are +# missing. All of these touch only md + SQLite state that already +# exists on disk — none of them need the embedding or rerank provider. +# PATCH is here because ``patch_document`` only rewrites md frontmatter +# and moves the doc directory when category changes; no embed or rerank +# code runs on that path. A user who downgrades from Tier 3 → Tier 1/2 +# must still be able to rename or recategorize the documents they +# created while providers were configured. +_UNGATED_ENDPOINTS: list[tuple[str, str]] = [ + ("DELETE", "/api/v1/knowledge/documents/d_abcdef123456"), + ("GET", "/api/v1/knowledge/documents"), + ("GET", "/api/v1/knowledge/documents/d_abcdef123456"), + ("GET", "/api/v1/knowledge/topics/d_abcdef123456_0"), + ("GET", "/api/v1/knowledge/categories"), +] + + +def _build_app() -> FastAPI: + app = FastAPI() + register_handlers(app) + # Match production mounting: create_app() mounts the router under + # /api/v1 (and /api/v2). The router's own prefix is /knowledge, so + # the effective paths become /api/v1/knowledge/... — same shape the + # _GATED / _UNGATED constants below use. + app.include_router(knowledge_routes.router, prefix="/api/v1") + return app + + +@pytest.fixture +async def client() -> AsyncIterator[AsyncClient]: + app = _build_app() + transport = ASGITransport(app=app, raise_app_exceptions=False) + async with AsyncClient(transport=transport, base_url="http://test") as c: + yield c + + +@pytest.fixture(autouse=True) +def _capabilities_unavailable_by_default(monkeypatch: pytest.MonkeyPatch) -> None: + """Default both singletons to unavailable; tests opt into availability + per-capability via the fixtures below. Mirrors ``tests/unit/test_memory + /test_search/test_validate_components.py``. + """ + monkeypatch.setattr( + embedding_accessor, "_capability", EmbeddingCapability(provider=None) + ) + monkeypatch.setattr(rerank_accessor, "_capability", RerankCapability(provider=None)) + + +@pytest.fixture +def embed_available(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + embedding_accessor, "_capability", EmbeddingCapability(provider=object()) + ) + + +@pytest.fixture +def rerank_available(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + rerank_accessor, "_capability", RerankCapability(provider=object()) + ) + + +@pytest.fixture +def _stub_read_services(monkeypatch: pytest.MonkeyPatch) -> None: + """Replace every read/delete/list service call with a benign stub. + + The tests in this module care only about whether the capability + gate fires. Ungated endpoints must reach their handler even when + providers are missing — but reaching the handler with no + MemoryRoot / SQLite setup would 500. Stubbing keeps the assertion + focused on the gate, not on end-to-end service behavior. + """ + from everos.service import knowledge as knowledge_service + + async def _fake_delete_document(*args: Any, **kwargs: Any) -> Any: + return knowledge_service.DeleteResult(doc_id="d_abcdef123456", deleted_topics=0) + + async def _fake_list_documents(*args: Any, **kwargs: Any) -> Any: + return knowledge_service.DocumentListResult( + documents=[], total=0, page=1, page_size=20 + ) + + async def _fake_get_document(*args: Any, **kwargs: Any) -> Any: + from everos.component.utils.datetime import get_utc_now + + now = get_utc_now() + return knowledge_service.DocumentDetail( + doc_id="d_abcdef123456", + category_id="General", + title="stub", + summary="stub", + source_name=None, + source_type=None, + original_file_path=None, + topics=[], + created_at=now, + updated_at=now, + ) + + async def _fake_get_topic(*args: Any, **kwargs: Any) -> Any: + from everos.component.utils.datetime import get_utc_now + + now = get_utc_now() + return knowledge_service.TopicDetail( + topic_id="d_abcdef123456_0", + doc_id="d_abcdef123456", + category_id="General", + topic_name="stub", + topic_path="stub", + depth=0, + summary="stub", + content="", + content_labels=[], + parent_topic_id=None, + children_topic_ids=[], + created_at=now, + updated_at=now, + ) + + async def _fake_list_categories(*args: Any, **kwargs: Any) -> list[Any]: + return [] + + monkeypatch.setattr(knowledge_routes, "delete_document", _fake_delete_document) + monkeypatch.setattr(knowledge_routes, "list_documents", _fake_list_documents) + monkeypatch.setattr(knowledge_routes, "get_document", _fake_get_document) + monkeypatch.setattr(knowledge_routes, "get_topic", _fake_get_topic) + monkeypatch.setattr(knowledge_routes, "list_categories", _fake_list_categories) + + +@pytest.fixture +def _stub_patch_document(monkeypatch: pytest.MonkeyPatch) -> None: + """Stub ``patch_document`` to echo the requested title back. + + Mirrors the real service contract: a title change lands in + ``updated_fields`` and the caller sees the mutation reflected in + the response envelope. Keeps the assertion focused on the gate + (or lack of it) instead of on md / SQLite plumbing. + """ + from everos.component.utils.datetime import get_utc_now + from everos.service import knowledge as knowledge_service + + captured: dict[str, Any] = {} + + async def _fake_patch_document( + doc_id: str, + app_id: str, + project_id: str, + *, + title: str | None = None, + category_id: str | None = None, + ) -> Any: + captured["title"] = title + captured["category_id"] = category_id + updated_fields: list[str] = [] + if title is not None: + updated_fields.append("title") + if category_id is not None: + updated_fields.append("category_id") + return knowledge_service.PatchResult( + doc_id=doc_id, + updated_fields=updated_fields, + updated_at=get_utc_now(), + ) + + monkeypatch.setattr(knowledge_routes, "patch_document", _fake_patch_document) + + +async def _call(client: AsyncClient, method: str, path: str) -> Any: + return await client.request(method, path) + + +# ── Gated endpoints: 422 when embed missing (rerank also missing) ──────── + + +@pytest.mark.parametrize("method,path", _GATED_ENDPOINTS) +async def test_gated_endpoint_422_when_embed_missing( + client: AsyncClient, method: str, path: str +) -> None: + """Write/search endpoints must 422 with an embedding-specific message + when embedding is missing (check-embedding-first order per Task 12/13). + """ + resp = await _call(client, method, path) + assert resp.status_code == 422 + body = resp.json() + assert body["error"]["code"] == "PROVIDER_NOT_CONFIGURED" + assert "embedding" in body["error"]["message"] + assert "knowledge" in body["error"]["message"] + + +# ── Gated endpoints: 422 when only rerank missing ─────────────────────── + + +@pytest.mark.parametrize("method,path", _GATED_ENDPOINTS) +async def test_gated_endpoint_422_when_rerank_missing( + client: AsyncClient, method: str, path: str, embed_available: None +) -> None: + """Write/search endpoints must 422 with a rerank-specific message + when only rerank is missing. + """ + resp = await _call(client, method, path) + assert resp.status_code == 422 + body = resp.json() + assert body["error"]["code"] == "PROVIDER_NOT_CONFIGURED" + assert "rerank" in body["error"]["message"] + assert "knowledge" in body["error"]["message"] + + +# ── Ungated endpoints: must NOT 422 even with both capabilities missing ─ + + +@pytest.mark.parametrize("method,path", _UNGATED_ENDPOINTS) +async def test_ungated_endpoint_not_422_without_any_capability( + client: AsyncClient, + method: str, + path: str, + _stub_read_services: None, +) -> None: + """Read/list/delete endpoints must stay reachable even when neither + embed nor rerank is configured — a Tier-3 → Tier-1 downgrade must + not lock users out of inspecting and cleaning up their own state. + """ + resp = await _call(client, method, path) + assert resp.status_code != 422, ( + f"{method} {path} returned 422 despite the gate being scoped away " + f"from read/delete endpoints; body={resp.text}" + ) + + +# ── Ungated endpoints: also fine with only embed (Tier-2 downgrade) ───── + + +@pytest.mark.parametrize("method,path", _UNGATED_ENDPOINTS) +async def test_ungated_endpoint_not_422_with_embed_only( + client: AsyncClient, + method: str, + path: str, + embed_available: None, + _stub_read_services: None, +) -> None: + """Tier-2 scenario (embed present, rerank missing): read/delete paths + must stay reachable — this is the concrete regression from the + router-wide-gate era that this fix targets. + """ + resp = await _call(client, method, path) + assert resp.status_code != 422, ( + f"{method} {path} returned 422 despite rerank not being needed; " + f"body={resp.text}" + ) + + +# ── Gated endpoints: pass the gate when both are available ────────────── + + +async def test_search_not_422_when_both_available( + client: AsyncClient, + monkeypatch: pytest.MonkeyPatch, + embed_available: None, + rerank_available: None, +) -> None: + """Both capabilities available: the gate must not fire on /search. + Stub the service call so the assertion is about the gate, not the + rest of the search pipeline (out of scope for this task). + """ + from everos.service import knowledge as knowledge_service + + stub_result = knowledge_service.SearchKnowledgeResult(hits=[], total=0, took_ms=0.0) + + async def _fake_search_knowledge(**kwargs: Any) -> Any: + return stub_result + + monkeypatch.setattr(knowledge_routes, "search_knowledge", _fake_search_knowledge) + resp = await client.post( + "/api/v1/knowledge/search", + json={"query": "hello"}, + ) + assert resp.status_code != 422 + + +# ── PATCH: metadata-only, must succeed regardless of provider config ──── + + +_PATCH_BODY: dict[str, str] = { + "app_id": "app1", + "project_id": "proj1", + "title": "renamed", +} + + +async def test_knowledge_patch_document_succeeds_without_rerank( + client: AsyncClient, + embed_available: None, + _stub_patch_document: None, +) -> None: + """Tier-2 scenario (embed configured, rerank missing): PATCH must + still land. Renaming an existing document does not touch rerank — + it only rewrites md frontmatter and upserts SQLite. + """ + resp = await client.patch( + "/api/v1/knowledge/documents/d_abcdef123456", + json=_PATCH_BODY, + ) + assert resp.status_code == 200, resp.text + body = resp.json() + assert body["data"]["doc_id"] == "d_abcdef123456" + assert "title" in body["data"]["updated_fields"] + + +async def test_knowledge_patch_document_succeeds_without_any_capability( + client: AsyncClient, + _stub_patch_document: None, +) -> None: + """Tier-1 scenario (no embed, no rerank): PATCH must still land. + Metadata-only edits are a cleanup verb — locking users out of + renaming their own docs after a full downgrade contradicts the + per-endpoint-gate contract documented on + ``_require_knowledge_capabilities``. + """ + resp = await client.patch( + "/api/v1/knowledge/documents/d_abcdef123456", + json=_PATCH_BODY, + ) + assert resp.status_code == 200, resp.text + body = resp.json() + assert body["data"]["doc_id"] == "d_abcdef123456" + assert "title" in body["data"]["updated_fields"] diff --git a/tests/integration/test_api/test_provider_error_mapping.py b/tests/integration/test_api/test_provider_error_mapping.py new file mode 100644 index 000000000..23a6a53bf --- /dev/null +++ b/tests/integration/test_api/test_provider_error_mapping.py @@ -0,0 +1,70 @@ +"""Verify ProviderNotConfiguredError maps to HTTP 422 via the FastAPI handler. + +Deviates from a naive flat `{"error_code": ..., "message": ...}` envelope: the +project already has a canonical error envelope (`ErrorResponse` / +`_error_response` in `exception_handlers.py`) shared by every domain +exception, so `ProviderNotConfiguredError` reuses it rather than inventing a +parallel shape. `ProviderNotConfiguredError` is a `ConfigurationError` +subclass; the assertions below also pin that the more-specific handler wins +over the parent `ConfigurationError` -> 500 mapping. + +Uses `httpx.AsyncClient` + `ASGITransport` (not `fastapi.testclient.TestClient`) +to match the project's existing exception-handler test convention -- the +installed starlette/httpx pair deprecates `TestClient`. +""" + +from __future__ import annotations + +from fastapi import FastAPI +from httpx import ASGITransport, AsyncClient + +from everos.core.errors import ProviderNotConfiguredError +from everos.entrypoints.api.exception_handlers import register_handlers + + +def _build_app_with_probes() -> FastAPI: + app = FastAPI() + register_handlers(app) + + @app.get("/probe_embed") + async def probe_embed() -> None: + raise ProviderNotConfiguredError(provider="embedding") + + @app.get("/probe_rerank_with_alt") + async def probe_rerank() -> None: + raise ProviderNotConfiguredError( + provider="rerank", + feature="agent_hybrid", + alternative_hint="Set enable_llm_rerank=true.", + ) + + return app + + +async def _get(path: str) -> tuple[int, dict]: + app = _build_app_with_probes() + transport = ASGITransport(app=app, raise_app_exceptions=False) + async with AsyncClient(transport=transport, base_url="http://test") as client: + resp = await client.get(path) + return resp.status_code, resp.json() + + +async def test_maps_to_422(): + status_code, _ = await _get("/probe_embed") + assert status_code == 422 + + +async def test_error_code_field(): + _, body = await _get("/probe_embed") + assert body["error"]["code"] == "PROVIDER_NOT_CONFIGURED" + + +async def test_message_field_carries_toml_path(): + _, body = await _get("/probe_embed") + assert "everos.toml" in body["error"]["message"] + + +async def test_alternative_hint_present_in_message(): + _, body = await _get("/probe_rerank_with_alt") + assert "Alternative:" in body["error"]["message"] + assert "enable_llm_rerank=true" in body["error"]["message"] diff --git a/tests/integration/test_api/test_startup_banner.py b/tests/integration/test_api/test_startup_banner.py new file mode 100644 index 000000000..283a73ef8 --- /dev/null +++ b/tests/integration/test_api/test_startup_banner.py @@ -0,0 +1,128 @@ +"""Test startup banner warning for unbackfilled memory rows. + +When LanceDB tables have rows with vector IS NULL (unbackfilled memories), +the startup banner should log a warning message pointing users to +`everos cascade backfill` command. +""" + +from __future__ import annotations + +import hashlib +from pathlib import Path + +import pytest +import structlog.testing + +from everos.component.utils.datetime import get_utc_now +from everos.core.observability.logging import get_logger +from everos.core.persistence import MemoryRoot +from everos.entrypoints.api.lifespans.lancedb import ( + LanceDBLifespanProvider, + _log_unbackfilled_hint, +) +from everos.infra.persistence.lancedb import Episode +from everos.infra.persistence.lancedb.repos import episode_repo + + +@pytest.fixture +def _isolated_memory_root(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Isolate each test with its own temporary MemoryRoot.""" + monkeypatch.setattr( + MemoryRoot, "resolve", classmethod(lambda cls: MemoryRoot(root=tmp_path)) + ) + (tmp_path / ".index" / "sqlite").mkdir(parents=True, exist_ok=True) + + +async def test_no_warning_when_empty(_isolated_memory_root: None) -> None: + """No warning is logged when the database is empty.""" + from fastapi import FastAPI + + app = FastAPI() + provider = LanceDBLifespanProvider() + await provider.startup(app) + + # When database is empty, no warning should be logged + with structlog.testing.capture_logs() as captured: + await _log_unbackfilled_hint() + + matching = [e for e in captured if e.get("event") == "unbackfilled_memory_rows"] + assert len(matching) == 0, "expected no warning when database is empty" + + await provider.shutdown(app) + + +async def test_warns_when_unbackfilled_rows_exist( + _isolated_memory_root: None, +) -> None: + """_log_unbackfilled_hint warns with correct count when vector=NULL rows exist.""" + from fastapi import FastAPI + + # Setup: initialize provider (connects to temp LanceDB) + app = FastAPI() + provider = LanceDBLifespanProvider() + await provider.startup(app) + + # Seed: insert one Episode row with vector=None + def _make_unvectorized_episode() -> Episode: + """Create an Episode row with vector=None.""" + return Episode( + id="test_owner_entry1", + entry_id="entry1", + owner_id="test_owner", + owner_type="user", + app_id="default", + project_id="default", + session_id="s_test", + timestamp=get_utc_now(), + parent_type="memcell", + parent_id="mc_test", + sender_ids=["test_owner"], + subject="test subject", + summary=None, + episode="test episode content", + episode_tokens="test episode", + md_path="users/test_owner/default/default/2026-07-25.md", + content_sha256=hashlib.sha256(b"test").hexdigest(), + deprecated_by=None, + vector=None, # explicitly unvectorized + ) + + await episode_repo.add([_make_unvectorized_episode()]) + + # Capture logs and call the hint function + with structlog.testing.capture_logs() as captured: + await _log_unbackfilled_hint() + + # Assert: the warning is logged with the correct count + matching = [e for e in captured if e.get("event") == "unbackfilled_memory_rows"] + assert len(matching) > 0, ( + "expected unbackfilled_memory_rows warning when vector=NULL rows exist" + ) + warning = matching[0] + assert warning.get("count") == 1, f"expected count=1, got {warning.get('count')}" + + await provider.shutdown(app) + + +async def test_log_unbackfilled_hint_runs_on_startup( + _isolated_memory_root: None, +) -> None: + """Startup should call _log_unbackfilled_hint after ensuring indexes.""" + from fastapi import FastAPI + + app = FastAPI() + provider = LanceDBLifespanProvider() + + # This startup should not raise - _log_unbackfilled_hint should be called + await provider.startup(app) + + await provider.shutdown(app) + + +async def test_startup_banner_logger_configuration() -> None: + """Verify the banner logger can be obtained.""" + logger = get_logger("everos.cli.server") + assert logger is not None + # Verify it has the expected logging methods + assert hasattr(logger, "warning") + assert hasattr(logger, "info") diff --git a/tests/integration/test_cascade_all_kinds_consistency.py b/tests/integration/test_cascade_all_kinds_consistency.py index 5c56859d7..c2310ecf3 100644 --- a/tests/integration/test_cascade_all_kinds_consistency.py +++ b/tests/integration/test_cascade_all_kinds_consistency.py @@ -87,6 +87,15 @@ async def cascade_runtime( monkeypatch.setenv("EVEROS_EMBEDDING__MODEL", "stub-model") monkeypatch.setenv("EVEROS_EMBEDDING__BASE_URL", "http://stub.invalid/v1") monkeypatch.setenv("EVEROS_EMBEDDING__API_KEY", "stub-key") + # Handlers fetch the embedder lazily via ``get_embedding_capability()`` + # rather than through ``CascadeOrchestrator`` — patch the process-wide + # singleton so cascade never hits the fake network target above. + import everos.component.embedding.accessor as acc + from everos.component.embedding import EmbeddingCapability + + monkeypatch.setattr( + acc, "_capability", EmbeddingCapability(provider=_StubEmbedder()) + ) await dispose_connection() await dispose_engine() engine = get_engine() @@ -94,7 +103,7 @@ async def cascade_runtime( await conn.run_sync(SQLModel.metadata.create_all) await ensure_business_indexes() (tmp_path / "ome.toml").write_text("# test\n") - yield MemoryRoot.default() + yield MemoryRoot.resolve() await dispose_connection() await dispose_engine() @@ -237,7 +246,6 @@ async def test_md_lance_strict_consistency_per_kind( memory_root = cascade_runtime orchestrator = CascadeOrchestrator( memory_root=memory_root, - embedder=_StubEmbedder(), tokenizer=build_tokenizer(), config=CascadeConfig( scan_interval_seconds=60.0, diff --git a/tests/integration/test_cascade_cli_integration.py b/tests/integration/test_cascade_cli_integration.py index b8307f82a..75b174725 100644 --- a/tests/integration/test_cascade_cli_integration.py +++ b/tests/integration/test_cascade_cli_integration.py @@ -4,7 +4,8 @@ tmp memory root. Validates the in-process orchestration that ``test_cascade_command`` (unit) cannot reach: ``_runtime()`` context, queue summary formatting, fix (no-rows path), and a full -``cascade sync `` round-trip with a stub embedder. +``cascade sync `` round-trip against an empty queue (no handler +ever runs, so embedding availability is irrelevant here). The CLI commands call ``asyncio.run(_run())`` internally, so this test is **synchronous** — pytest-asyncio's auto mode would otherwise wrap it @@ -22,23 +23,12 @@ import pytest from typer.testing import CliRunner -from everos.component.embedding import EmbeddingProvider from everos.config import load_settings from everos.entrypoints.cli.commands import cascade as cascade_mod from everos.infra.persistence.lancedb import dispose_connection from everos.infra.persistence.sqlite import dispose_engine -class _StubEmbedder(EmbeddingProvider): - dim = 1024 - - async def embed(self, text: str) -> list[float]: - return [0.0] * self.dim - - async def embed_batch(self, texts): # type: ignore[no-untyped-def] - return [[0.0] * self.dim for _ in texts] - - @pytest.fixture def cli_runtime(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Iterator[Path]: """Tmp memory root + clean singletons; CLI bootstraps the schema itself.""" @@ -101,11 +91,10 @@ def test_sync_on_empty_queue_with_stub_embedder( from everos.memory.cascade import CascadeOrchestrator def fake_build_orchestrator() -> CascadeOrchestrator: - root = MemoryRoot.default() + root = MemoryRoot.resolve() root.ensure() return CascadeOrchestrator( memory_root=root, - embedder=_StubEmbedder(), tokenizer=build_tokenizer(), ) @@ -148,8 +137,7 @@ def test_sync_with_unmatched_path( def fake_build_orchestrator() -> CascadeOrchestrator: return CascadeOrchestrator( - memory_root=MemoryRoot.default(), - embedder=_StubEmbedder(), + memory_root=MemoryRoot.resolve(), tokenizer=build_tokenizer(), ) diff --git a/tests/integration/test_cascade_fsevents_repro.py b/tests/integration/test_cascade_fsevents_repro.py index 7f95ec8b4..158a7ea64 100644 --- a/tests/integration/test_cascade_fsevents_repro.py +++ b/tests/integration/test_cascade_fsevents_repro.py @@ -72,6 +72,15 @@ async def cascade_runtime( monkeypatch.setenv("EVEROS_EMBEDDING__MODEL", "stub-model") monkeypatch.setenv("EVEROS_EMBEDDING__BASE_URL", "http://stub.invalid/v1") monkeypatch.setenv("EVEROS_EMBEDDING__API_KEY", "stub-key") + # Handlers fetch the embedder lazily via ``get_embedding_capability()`` + # rather than through ``CascadeOrchestrator`` — patch the process-wide + # singleton so cascade never hits the fake network target above. + import everos.component.embedding.accessor as acc + from everos.component.embedding import EmbeddingCapability + + monkeypatch.setattr( + acc, "_capability", EmbeddingCapability(provider=_StubEmbedder()) + ) await dispose_connection() await dispose_engine() @@ -82,7 +91,7 @@ async def cascade_runtime( await ensure_business_indexes() (tmp_path / "ome.toml").write_text("# test\n") - yield MemoryRoot.default() + yield MemoryRoot.resolve() await dispose_connection() await dispose_engine() @@ -127,7 +136,6 @@ async def test_high_freq_atomic_fact_append_no_loss( memory_root = cascade_runtime orchestrator = CascadeOrchestrator( memory_root=memory_root, - embedder=_StubEmbedder(), tokenizer=build_tokenizer(), config=CascadeConfig( scan_interval_seconds=60.0, diff --git a/tests/integration/test_cascade_integration.py b/tests/integration/test_cascade_integration.py index 5f25efb3e..d59e1d7bd 100644 --- a/tests/integration/test_cascade_integration.py +++ b/tests/integration/test_cascade_integration.py @@ -84,12 +84,25 @@ async def cascade_runtime( await ensure_business_indexes() (tmp_path / "ome.toml").write_text("# test\n") - yield MemoryRoot.default() + yield MemoryRoot.resolve() await dispose_connection() await dispose_engine() +def _patch_embedding_capability( + monkeypatch: pytest.MonkeyPatch, embedder: _StubEmbedder +) -> None: + """EpisodeHandler fetches the embedder lazily via + ``get_embedding_capability()`` — patch the process-wide singleton so + cascade never hits the fake network target set up by ``cascade_runtime``. + """ + import everos.component.embedding.accessor as acc + from everos.component.embedding import EmbeddingCapability + + monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=embedder)) + + async def _poll(condition, *, deadline_seconds: float = 10.0, interval: float = 0.05): # type: ignore[no-untyped-def] """Poll ``condition()`` (async) until truthy, or :class:`TimeoutError`. @@ -107,13 +120,14 @@ async def _poll(condition, *, deadline_seconds: float = 10.0, interval: float = async def test_append_to_md_propagates_to_lancedb( cascade_runtime: MemoryRoot, + monkeypatch: pytest.MonkeyPatch, ) -> None: """Happy path: writer append → watcher → state row → worker → LanceDB.""" memory_root = cascade_runtime embedder = _StubEmbedder() + _patch_embedding_capability(monkeypatch, embedder) orchestrator = CascadeOrchestrator( memory_root=memory_root, - embedder=embedder, tokenizer=build_tokenizer(), # Tight worker poll so the test wraps in seconds, not minutes. # Scanner interval kept long so the watcher path is the one @@ -188,12 +202,13 @@ async def _state_done(): # type: ignore[no-untyped-def] async def test_delete_md_wipes_lancedb_row( cascade_runtime: MemoryRoot, + monkeypatch: pytest.MonkeyPatch, ) -> None: """Append + drain, then ``unlink`` the md and watch the row evaporate.""" memory_root = cascade_runtime + _patch_embedding_capability(monkeypatch, _StubEmbedder()) orchestrator = CascadeOrchestrator( memory_root=memory_root, - embedder=_StubEmbedder(), tokenizer=build_tokenizer(), config=CascadeConfig( scan_interval_seconds=60.0, diff --git a/tests/integration/test_cascade_scenarios.py b/tests/integration/test_cascade_scenarios.py index a30fb47e4..d1ccbf0aa 100644 --- a/tests/integration/test_cascade_scenarios.py +++ b/tests/integration/test_cascade_scenarios.py @@ -83,6 +83,12 @@ async def cascade_runtime( monkeypatch.setenv("EVEROS_EMBEDDING__MODEL", "stub-model") monkeypatch.setenv("EVEROS_EMBEDDING__BASE_URL", "http://stub.invalid/v1") monkeypatch.setenv("EVEROS_EMBEDDING__API_KEY", "stub-key") + # Handlers fetch the embedder lazily via ``get_embedding_capability()`` + # rather than through ``CascadeOrchestrator`` — patch the process-wide + # singleton so cascade never hits the fake network target above. + # ``test_lap_append_during_handler_no_loss`` re-patches with a slow + # variant to force a handler-in-flight race. + _patch_embedding_capability(monkeypatch, _StubEmbedder()) await dispose_connection() await dispose_engine() @@ -93,18 +99,27 @@ async def cascade_runtime( await ensure_business_indexes() (tmp_path / "ome.toml").write_text("# test\n") - yield MemoryRoot.default() + yield MemoryRoot.resolve() await dispose_connection() await dispose_engine() +def _patch_embedding_capability( + monkeypatch: pytest.MonkeyPatch, embedder: EmbeddingProvider +) -> None: + """Patch the process-wide embedding capability singleton to *embedder*.""" + import everos.component.embedding.accessor as acc + from everos.component.embedding import EmbeddingCapability + + monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=embedder)) + + def _build_orchestrator( memory_root: MemoryRoot, *, scan_interval: float = 60.0 ) -> CascadeOrchestrator: return CascadeOrchestrator( memory_root=memory_root, - embedder=_StubEmbedder(), tokenizer=build_tokenizer(), config=CascadeConfig( scan_interval_seconds=scan_interval, @@ -486,6 +501,7 @@ async def test_concurrent_writes_different_owners_no_bleed( async def test_lap_append_during_handler_no_loss( cascade_runtime: MemoryRoot, + monkeypatch: pytest.MonkeyPatch, ) -> None: """Writer keeps appending while worker is mid-handler. @@ -500,9 +516,12 @@ async def embed(self, text: str) -> list[float]: await asyncio.sleep(0.05) # handler takes ~0.05*N entries return [0.0] * self.dim + # Override the fixture's default stub — this test needs latency to + # force a handler invocation to overlap later writer appends. + _patch_embedding_capability(monkeypatch, _SlowEmbedder()) + orchestrator = CascadeOrchestrator( memory_root=memory_root, - embedder=_SlowEmbedder(), tokenizer=build_tokenizer(), config=CascadeConfig( scan_interval_seconds=60.0, @@ -564,7 +583,6 @@ def _build_orchestrator_fast_scanner(memory_root: MemoryRoot) -> CascadeOrchestr don't wait 30s for the fallback path.""" return CascadeOrchestrator( memory_root=memory_root, - embedder=_StubEmbedder(), tokenizer=build_tokenizer(), config=CascadeConfig( scan_interval_seconds=2.0, diff --git a/tests/integration/test_cli/__init__.py b/tests/integration/test_cli/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/integration/test_cli/test_backfill_flags.py b/tests/integration/test_cli/test_backfill_flags.py new file mode 100644 index 000000000..4a0abdd7c --- /dev/null +++ b/tests/integration/test_cli/test_backfill_flags.py @@ -0,0 +1,315 @@ +"""Integration tests for Task 24's ``cascade backfill`` polish pass. + +Covers three behaviours layered on top of Task 20-23's scaffold + three +phase implementations: + +- **Summary block**: after ``run_backfill`` finishes (or is interrupted), + a consolidated "Backfill summary" section is printed, one line per + phase that actually ran, plus a final ``Exit: