Fix idle-page crash: don't poison the read cache with transient grain-reactivation rejects by rbuergi · Pull Request #470 · Systemorph/MeshWeaver

rbuergi · 2026-07-14T06:40:49Z

The bug (two symptoms users hit on memex.meshweaver.cloud)

When Orleans idle-collects a per-node MessageHubGrain, navigating to that page intermittently crashes the area, and a manual browser reload "fixes" it. Two error shapes surface:

Forwarding reject — the next read's SubscribeRequest hits the mid-DeactivateOnIdle activation; Orleans forwards it up to MaxForwardCount (=2) times then transiently rejects:

Delivery to 'X' failed: Forwarding failed: tried to forward message
Request [...]->[messagehub/X] IMessageHubGrain.DeliverMessage(...) #...[ForwardCount=2]
for 2 times after "DeactivateOnIdle was called." to invalid activation. Rejecting now.

Activation timeout — if reactivation is slow, the 60 s hub-request timeout:

No response received in hub cache/... within 00:01:00 for request SubscribeRequest (id=...)
→ target X. The request may have been undeliverable or the target hub was not found.

Root cause

The retry infrastructure that already exists (RoutingGrain.DeliverToGrainWithRetry retries the transient Orleans reject; the grain reactivates) is sound and unchanged. The defect is one layer up, in MeshNodeStreamCache.

The read-path hydration bookkeeping recorded any upstream error into the storm-breaker negative cache, unconditionally:

var bookkeeping = inner.AsObservable().Subscribe(
    node => { if (node is not null) _negative.TryRemove(p, out _); },
    ex => RecordNegative(p, ex));   // ← records EVERY error, including a transient reject

So a transient grain-reactivation reject (OrleansMessageRejectionException → routing DeliveryFailure{ErrorType.Failed} "Forwarding failed … Rejecting now.", or the 60 s TimeoutException) was cached exactly like a genuinely-missing node. GetStreamRaw then replayed that raw Orleans reject to every reader for the whole backoff window (2 s, growing) and refused to re-probe the grain that had already reactivated. That is precisely why the page crashed and a manual reload "fixed" it — the reload merely outlasted the negative window so the re-probe landed on the fresh activation.

The write path already guarded this (only IsMissingNodeFailure records — "not RLS denial / transient, so a legitimately-existing node is never falsely suppressed"). The read path had no such guard — an asymmetry.

Orleans internals confirming the class of reject (Microsoft.Orleans.* 10.2.0):

TryForwardRequest rejects with RejectionTypes.Transient and the exact "Forwarding failed … for {ForwardCount} times … Rejecting now." message once MayForward (ForwardCount < MaxForwardCount, default 2) is false.
For a clean DeactivateOnIdle the DeactivationException is null, so the caller receives new OrleansMessageRejectionException(rejectionInfo) — a transient failure, retryable, never a missing node.

The fix

Gate the read-path RecordNegative with the same IsMissingNodeFailure guard the write path uses, and make that predicate transient-aware (transient takes precedence) via a new IsTransientOwnerFailure classifier that mirrors RoutingGrain.IsTransientFailure and AreaErrorClassifier.IsTransientHubFailure — so all three layers agree on what is worth a retry vs. a suppress.

ex => { if (IsMissingNodeFailure(ex)) RecordNegative(p, ex); });

A transient owner miss is forwarded to the current subscriber (whose caller / the area's transient classifier retries) but never recorded, so the very next read re-probes the reactivated grain immediately. Genuine missing nodes ("No node found") are still recorded — the storm-breaker keeps doing its real job.

Tests

MeshNodeStreamCacheNegativeClassifierTest (8, deterministic) — pins the exact transient-vs-missing decision using the verbatim prod message strings: the forwarding-reject and the 60 s timeout classify transient (NOT missing); a real "No node found" classifies missing; RLS denial is neither; the inner-exception chain is walked.
OrleansIdleReactivationNoWedgeTest (2) — end-to-end reactivation smoke test: a read right after the owning grain is deactivated transparently reactivates and returns the node, and an immediate re-read stays healthy (no poisoned window). ⚠️ A single in-memory test silo reactivates too cleanly to reliably hit the two-hop forwarding-reject window, so this is a smoke test for the reactivation path, not the before/after pin (that role belongs to the classifier test — verified: reverting the one-line read-path guard leaves the classifier RED and the Orleans smoke test green).
MissingSatelliteTest (existing) still green — genuine missing-node storm-breaker recording + fast-fail is preserved.

Local results (Release, `-warnaserror -p:CIRun=true` builds clean; 0 warnings)

MeshNodeStreamCacheNegativeClassifierTest:  Passed: 8,  Failed: 0
OrleansIdleReactivationNoWedgeTest + RoutingGrainDeliveryRetryTest:  Passed: 7,  Failed: 0
MissingSatelliteTest:  Passed: 2,  Failed: 0
MeshNodeStreamCache* (idle-release + classifier):  Passed: 15, Failed: 0

Why not the other candidate fixes

Config bump (CollectionAge / MaxForwardCount) — a band-aid. There is no misconfiguration: the grain reactivates correctly and the routing layer already retries the reject. The defect is the cache mis-classifying a transient reject as a permanent absence; raising a bound wouldn't fix that.
Retry at routing — already present (DeliverToGrainWithRetry) and correct; it re-activates the grain. The residual crash is the cache poisoning downstream of a successful reactivation.

Please review — this is prod concurrency code and I have not merged it. Please gut-check whether the IsTransientOwnerFailure substring set should also cover any other transient banner you've seen in the wild.

🤖 Generated with Claude Code

… rejects When Orleans idle-collects a per-node MessageHubGrain, the next read's SubscribeRequest can hit the mid-DeactivateOnIdle activation. Orleans forwards it up to MaxForwardCount (=2) times and then transiently rejects it: Delivery to 'X' failed: Forwarding failed: tried to forward message Request [...]->[messagehub/X] IMessageHubGrain.DeliverMessage(...) #...[ForwardCount=2] for 2 times after "DeactivateOnIdle was called." to invalid activation. Rejecting now. or, if reactivation is slow, the 60s hub-request timeout: No response received in hub cache/... within 00:01:00 for request SubscribeRequest (...) → target X. The request may have been undeliverable or the target hub was not found. Root cause: MeshNodeStreamCache's READ-path hydration bookkeeping recorded ANY upstream error into the storm-breaker negative cache (`ex => RecordNegative(p, ex)`), unconditionally. A transient reactivation reject was therefore cached exactly like a genuinely-missing node: GetStreamRaw then replayed that raw Orleans reject to every reader for the backoff window (2s, growing) AND refused to re-probe the grain that had already reactivated. That is precisely why navigating to a just-idle page intermittently crashed and a manual reload "fixed" it — the reload merely outlasted the negative window so the re-probe landed on the fresh activation. The WRITE path already guarded RecordNegative with IsMissingNodeFailure ("only missing-node failures record, not RLS denial / transient, so a legitimately- existing node is never falsely suppressed"). The READ path had no such guard — an asymmetry. Fix: gate the read-path RecordNegative with the same IsMissingNodeFailure guard, and make that predicate transient-aware (transient takes precedence) via a new IsTransientOwnerFailure classifier that mirrors RoutingGrain.IsTransientFailure and AreaErrorClassifier.IsTransientHubFailure. A transient owner miss (Orleans reject / "invalid activation" / "Rejecting now" / "Forwarding failed" / TimeoutException / "target hub was not found" / "undeliverable") is now forwarded to the current subscriber (whose caller / the area's transient classifier retries) but NEVER recorded — so the very next read re-probes the reactivated grain immediately. Genuine missing nodes ("No node found") are still recorded, so the storm-breaker keeps doing its real job. Tests: - MeshNodeStreamCacheNegativeClassifierTest (8, deterministic): pins the exact transient-vs-missing decision using the verbatim prod message strings — the forwarding-reject and the 60s timeout classify transient (NOT missing), a real "No node found" classifies missing, RLS denial is neither. - OrleansIdleReactivationNoWedgeTest (2): end-to-end reactivation smoke test — a read right after the owning grain is deactivated transparently reactivates and returns the node, and an immediate re-read stays healthy (no poisoned negative window). A single in-memory test silo reactivates too cleanly to reliably hit the two-hop forwarding-reject window, so this is a smoke test for the reactivation path, not the before/after pin (that role is the classifier's). - MissingSatelliteTest (existing) still green: genuine missing-node storm-breaker recording + fast-fail is preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-14T06:56:19Z

Test Results (shard 3)

12 files ±0 12 suites ±0 3m 56s ⏱️ -10s
873 tests ±0 873 ✅ ±0 0 💤 ±0 0 ❌ ±0
921 runs ±0 921 ✅ ±0 0 💤 ±0 0 ❌ ±0