Runtime-compile pipeline: self-update leaves stale NodeType assemblies, CompileWatcher wedges permanently, activation-fault loop surfaces raw Orleans rejection

[rosalie1984]

Runtime-compile pipeline: platform self-update leaves stale NodeType assemblies, CompileWatcher can wedge permanently, and activation-fault loops surface raw Orleans rejections to the GUI

Incident

memex portal (AKS memex namespace), 2026-07-13/14, UWDeepfield space (mesh-compiled plugin). After the continuous self-update rolled a main where the ContentCollections surface went reactive (GetFiles(path) / GetFolders(path) → pooled IObservable, one argument; GetContentAsync/DeleteFileAsync/DeleteFolderAsync/GetCollectionAsync removed in favour of GetContent/DeleteFile/DeleteFolder/GetCollection), every NodeType in the space stopped compiling — their shared compat source (UWDeepfield/Source/DeepfieldMeshCompat) still used the old two-argument await foreach shape.

User-visible symptom on every typed node (e.g. UWDeepfield/Attention, UWDeepfield/Clients):

Delivery to 'UWDeepfield/Clients' failed: Forwarding failed: tried to forward message
Request [S127.0.0.1:11111:… sys.client/hosted-…]→[… messagehub/UWDeepfield/Clients]
MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(…) #… [ForwardCount=2]
for 2 times after "DeactivateOnIdle was called." to invalid activation. Rejecting now.

plus an endless "Subscribing…" spinner on the resubscribing layout areas.

Causal chain: broken NodeType compile → instance grain cannot materialize a hub config → activation faults instantly → TryDeactivateOnIdle (retry-on-next-access) → every delivery (and every one of RoutingGrain.DeliverToGrainWithRetry's 6 transient retries) lands in a deactivation window → Orleans forwards twice, then rejects → the raw rejection text is NACKed to the GUI.

The plugin-side fix (rebridging the compat shim onto the new reactive surface) is applied on-mesh and verified: UwAttention, UwLossWatch, UwDecisionLog, TreatySubmission rebuilt green and UWDeepfield/Attention resolves again. The remaining NodeTypes cannot be rebuilt because of the framework defects below; only a pod restart clears them.

Defect 1 — no framework-version gating of cached assemblies (runtime MissingMethodException timebomb)

NodeType content carries compiledFrameworkVersion, but a compile request against a NodeType whose assembly was built on the previous platform returns the cached Ok without rebuilding:

• UWDeepfield/ReinsuranceClient → Ok, activity compile-2026071218 3635… (built 2026-07-12 18:36 — provably against the old API: its source only compiles pre-drift)
• UWDeepfield/UwPortfolio → Ok, activity compile-2026071213 0330…

Those assemblies reference the removed ContentCollection members and will throw MissingMethodException at runtime on the new binaries the moment their file-zone/ingest paths execute. Some types did get a fresh (failing) compile on first access after the update (UwAttention, stamped Error 2026-07-13 21:10), so the invalidation is at best inconsistent.

Expected: a platform-version bump invalidates (or at minimum flags) every cached NodeType assembly; compile never reports Ok for an assembly whose compiledFrameworkVersion ≠ current.

Defect 2 — CompileWatcher wedges permanently: stale source snapshot, then no build ever starts again

Sequence observed on UWDeepfield/ReinsuranceClientIndex (all timestamps 2026-07-13 UTC):

1. Source fix written 21:26:16; NodeType + source hub recycled; compile → fresh build 21:30:57 → Error against a stale source snapshot (expected CQRS lag, fine so far).
2. Second source fix written 21:38:04 (index confirmed caught up: v87, lastModified 21:38:04). NodeType hub recycled at ~21:37:5x — its re-init read the stale snapshot and parked: currentSourceVersions[DeepfieldMeshCompat] = ticks ⇒ 21:27:13 while the store had 21:38:04. The change-feed event for the write fired before/during hub re-init and was lost — the hub never converges.
3. From then on no compile request ever starts a build: lastCompileStartedAt stays frozen at 21:30:57 through all of: repeated compile calls (flip-to-Pending via remote-stream Update), recycling the NodeType hub again minutes later (its fresh init then tracked the correct source version 21:44:30 — and still no build), recycling the source node's hub, a version-bump touch-edit of the source, recycling the whole Space node, flipping compilationStatus via the patch route, and even writing a genuine Error → Pending transition (v421 → v422).
4. Adjacent oddity: after recycles, get_diagnostics / definition resolution intermittently answers NodeType '…' has no definition and stays there (observed on UWDeepfieldHome for >30 min).

Two sub-defects:

• (2a) Lost-on-reactivation of the source subscription: the hub's source-version tracking can initialize from a stale read and miss the concurrent change-feed emission — a re-subscribe/reconcile on init (compare stream state vs store after init) is missing.
• (2b) The build trigger is edge-triggered with no level check and can die silently: a watcher that misses the Pending transition (or whose subscription died with a mid-flight recycle) never rebuilds; a freshly re-initialized hub that finds compilationStatus == Pending (or Error with currentSourceVersions ≠ the versions the Error was computed from) does not start a build either. Only a process restart recovers. Expected: on hub init, Pending — or any mismatch between tracked source versions and the last build's inputs — starts a build (level-triggered reconciliation, not only edge).

Defect 3 — activation-fault loop surfaces the raw Orleans rejection instead of the real error

RoutingGrain.DeliverToGrainWithRetry correctly retries transient OrleansMessageRejectionExceptions, but when the target grain is in a persistent activation-fault loop (fault ≈ instantaneous → DeactivateOnIdle → the alive window is ~0), all 6 retries land in deactivation windows and the user receives the Orleans internals ("tried to forward message … after "DeactivateOnIdle was called." … Rejecting now") instead of the actual cause (Compilation failed for 'UWDeepfield/UwAttention': CS1501 …, which the grain logs on every activation).

Expected: the NACK for an exhausted retry against a repeatedly-faulting activation carries the activation error (the grain already resolves _hubReadyRaw.OnError(ex); the parked-caller path returns it — the rejection race path should too, e.g. by having the retry helper fall back to the last known activation failure for that grain key). The GUI's resubscribe loop then also stops spinning on a deterministic terminal error.

Repro sketch (deterministic, test-shaped)

1. Mesh with a runtime-compiled NodeType whose source set compiles green; let the grain activate.
2. Write a source change and recycle the NodeType hub concurrently (race the change-feed emission against hub re-init) → assert the hub's tracked source version converges to the store (fails today — 2a).
3. With compilationStatus already Pending, request a compile → assert a build starts (fails today — 2b, no-delta write emits no event; init does no level check).
4. Kill/replace platform binaries under a cached assembly (or fake compiledFrameworkVersion mismatch) → request compile → assert rebuild, not cached Ok (fails today — 1).
5. Grain whose activation always faults: deliver via RoutingGrain until retries exhaust → assert the NACK carries the activation exception, not the Orleans rejection text (fails today — 3).

Workaround used in production

On-mesh source fix + per-NodeType recycle+compile healed the types whose watchers were still alive; the wedged ones (ReinsuranceClientIndex, ReinsuranceClient, UWDeepfieldHome) require a portal pod restart (pending). Not a substitute for the fixes above — the next platform self-update that changes a public surface consumed by on-mesh code will reproduce all of this.