Fix Layer0 e2e teardown SIGSEGVs: drain RPC scopes before their targets die, roll back started on failed start()

[ptesavol]

Problem

The streamr-dht end-to-end suite (streamr-dht-test-end-to-end, 7 tests) had a rare SIGSEGV when running the full suite (observed 2026-07-11 in Layer0MixedConnectionTypesTest.TwoNonServerPeersJoinFirst; passes in isolation and on most full-suite reruns). Runtime-evidence debugging against the two macOS crash reports from that day identified two independent teardown crash mechanisms.

Mechanism 1 — drain-order use-after-free (crash report 16:33)

The crash report captured it live on two threads: the main thread inside fixture destruction draining in-flight RPC coroutines (~DhtNode → ~RoutingRpcCommunicator → ~RpcCommunicatorClientApi → blockingWait(mScope.cancelAndJoinAsync())), while StreamrWorker1 resumed a drained Ping request that ran makeRpcRequest → outgoing callback → sendFn → ConnectionManager::send → WebsocketClientConnector::connect and crashed at 0x0 inserting into a freed map.

Root cause: the scope drains ran in the client/server API destructors, i.e. after the owning objects' members were already gone:

• DhtNode declared rpcCommunicator before ownedConnectionManager, so reverse-order destruction freed the ConnectionManager before the drain. A straggler send into a stopped ConnectionManager is a guarded no-op (deliberate design); into a destroyed one it is a use-after-free.
• The drain also ran after RoutingRpcCommunicator's own members (sendFn, ownServiceId) were destroyed.
• ConnectionManager's internal lock-RPC communicator drained after endpoints/endpointsMutex/state died, and sendIfStopped responses bypass the stopped-state guard in send().

Instrumented full-suite runs showed the unsafe window opened on every node teardown (350 ordering violations across 10 runs) with ~170 guarded post-stop straggler sends per run — the crash just needed one task still queued at fixture destruction, which is why it was rare and load-dependent.

Mechanism 2 — failed start() poisons stop() (crash report 07:21)

start() set started = true before creating the owned ConnectionManager. If the websocket server failed to bind, transportPtr stayed null while started stayed true; TearDown's stop() then called off() through the null pointer — SEGV at 0x1a0 (emitter mutex offset from null), byte-for-byte the 07:21 report. Reproduced deterministically by occupying the entry-point port (nc -l 10011) before Layer0Test.

Fix

• Idempotent drainAsyncTasks() (atomic once-flag; folly forbids a second scope join) on RpcCommunicatorClientApi, RpcCommunicatorServerApi, and RpcCommunicator; the API destructors keep their drains as a backstop.
• New ~RoutingRpcCommunicator calls drainAsyncTasks() so the drain runs while sendFn, ownServiceId, the owner, and its transport are all still alive (ListeningRpcCommunicator inherits this).
• ~ConnectionManager drains its own lock-RPC communicator in its body, while endpoints/endpointsMutex/state are still alive.
• DhtNode declares ownedConnectionManager before rpcCommunicator, so the stopped ConnectionManager outlives the drain and absorbs stragglers as guarded no-ops.
• started = true moves to the end of start() (no suspension points inside; nothing in start() reads it) — a failed start leaves the node safely stoppable, and the ConnectionManager destructor stops whatever came up.

Verification

Check	Before fix	After fix
Blocked-port repro (nc -l 10011 + Layer0Test)	SIGSEGV at 0x1a0	clean test failure (exit 1)
Drain-order violations (instrumented log analyzer)	350 / 10 runs (every teardown)	0 / 33 runs
Full e2e suite, 15 runs under CPU load	window open every run	15/15 pass, 0 findings
ASan loop (18 runs, load, --gtest_repeat=2)	0 heap reports, 3 tooling aborts	0 heap reports, 3 tooling aborts
Regressions	—	e2e 3×7/7 clean, streamr-dht unit 181/181, proto-rpc 24/24 + 4/4

The ASan aborts (same rate pre/post fix) are the known ASan-internal asan_thread.cpp:370 unwind CHECK artifact on arm64 (exception unwinding through WebrtcConnection::doClose → emit<Disconnected> on the AbortableTimers thread), not a heap error — consistent with the previously documented ASan/folly custom-stack false-positive family.

🤖 Generated with Claude Code

[ptesavol]

Update: rebased onto cbec392a and added a second commit fixing the linux-arm64 CI hang.

The failed leg was a single-test hang: DhtNodeExternalApiTest.ExternalStoreDataHappyPath hit its 1200 s ctest timeout twice, with the last log line inside Simulator::stop()'s drain-wait. Investigation (thread dumps from deterministic local reproductions with the shared worker pool capped at 1 thread) showed a pre-existing worker-pool deadlock, present on main (cbec392a) and this branch alike — both hang 2/2 at 1 pool thread with the identical stack:

sendResponse() (running on a pool thread inside an RPC server handler) destroyed its stack-local per-session ListeningRpcCommunicator; the destructor's scope join parked the pool thread waiting for the communicator's own queued fire-and-forget send task, which itself needs a pool thread. Once all workers park in such joins (plausible on the 3-core runner's 4-thread pool under concurrent recursive-operation responses), the queued sends can never run — permanent deadlock, which #80's new Simulator::stop() drain-wait then surfaces as this test's timeout. This PR's first commit did not introduce the mechanism (control experiment on cbec392a reproduces it), but it moves the scope joins one destructor level earlier, which is likely why this run tripped it.

The second commit fixes the root cause: per-session communicators are retired into a manager-owned list instead of being destroyed on the pool thread, destroyed opportunistically once their scopes report zero pending tasks (empty-scope joins need no pool thread; growth bounded by sessionRpcTimeout) and finally in stop() on the owner's thread. ListeningRpcCommunicator::destroy() now also detaches the previously-dangling Disconnected listener.

Verification: 1-thread pool 2/2 permanent hangs → 0 hangs (bounded 10 s failures, below the 4-thread design floor); CI-sized 4-thread pool 15/15 passes; full local regression green (e2e 3×7/7, dht unit 181/181, proto-rpc 24/24 + 4/4).

🤖 Generated with Claude Code