Silent loss of uploaded collection files after hub teardown; mesh search returns zero results on deployed portal

[rosalie1984]

Silent loss of uploaded collection files + mesh search returning zero results

Environment: deployed portal memex.systemorph.com, platform 3.0.0-ci.866+build.639194691516252957.b367183f6923bdbbba994ede177e21aca7519ca1. Platform redeploy (migration stamp) at 2026-07-12 16:12 UTC. All observations below from the same day, via the MCP tool surface (get / search). Space/node names anonymized; exact paths available on request.

Symptom 1 — uploaded collection files vanish silently

Timeline (2026-07-12, UTC):

Time	Observation
18:25:18–19	7 files (docx/pdf) bulk-uploaded into a node's content collection via the portal Files view — all lastModified stamps within ~1 s
~18:30–18:45	get @{node}/content/ lists all 7 files; fetching an individual docx works (content conversion served correctly)
~2 h later	Same listing returns []; direct fetch returns File 'X' not found in collection 'content'

Additional facts:

• No delete was issued. The node's children are unchanged, no new/moved nodes in the space, and the parent node document itself was not modified (no version bump attributable to the loss).
• All 7 files disappeared as a unit, not individually.
• Node-document writes on the same partition the same day (creates + many edit_content rounds on sibling nodes) persisted fine — the loss is specific to the collection-file path.

Hypothesis

The upload wrote only into the owning hub's in-memory state (which served the successful reads for ~20+ min) and was never durably persisted. The next grain teardown — idle release, or process churn from an unrelated deployment/recompile in the same silo — erased it. Reads succeeding right after upload means users get no signal that their files are on borrowed time. This looks like the same silent-drop family as the 2026-07-02 virtual-data-source stream outage (RouteStreamMessage silent-drop / stream-cache idle-release suspects).

Suggested repro / verification

1. Upload files into a node's content collection; confirm the listing shows them.
2. Recycle the node's hub (DisposeRequest) — or wait for idle release.
3. List again. If the files are gone, that's a deterministic repro pinning the persistence defect.
4. In parallel: check the partition schema in Postgres (or the blob store, wherever collection files are supposed to land) immediately after step 1 — if no row/blob exists at that point, the write-side persistence never happened and the teardown is only the messenger.

Symptom 2 — search returns zero results for everything

On the same portal, search returns {count: 0} for queries that must match:

• path:{existing-node-path} select:name,version → 0 (the node is fetchable via get at that exact path)
• namespace:{space} scope:descendants → 0 (the space has 10+ children, all fetchable)
• free-text over content that demonstrably exists → 0

Reads via get work on all the same paths, so this is confined to the query/index path (catalog matview / vector index / PostgreSqlMeshQuery), not storage of the node documents themselves.

Possible common cause

Both symptoms are write-side/secondary-store failures while primary node reads work: collection-file persistence and index maintenance both broken on this deployment. Worth checking whether the 16:12 UTC migration/deploy left the index empty or re-pointed a store, and whether collection-file writes share a persistence path with index writes.

Impact

• Silent user data loss on file upload — the worst failure mode, since reads confirm success before the data evaporates.
• Search-dependent features (navigation search areas, agent retrieval, pickers) are blind on this deployment.

Per repo policy: looking for the root cause in the upload persistence / index write path — not a retry/watchdog mitigation.