feat: N=1 cooperative fiber scheduler replacing per-thread host threads

[smmathews]

feat: N=1 cooperative fiber scheduler replacing per-thread host threads

What this replaces and why

Upstream runs each guest EE thread on its own host std::thread (g_hostThreads in Helpers/State.h, spawned from Thread.cpp/ps2_runtime.cpp). That model allows two guest threads to execute truly concurrently, which the real EE (a single core) never does — games' kernel-object usage (semaphores as pure ownership tokens, priority-based cooperative scheduling, RotateThreadReadyQueue round-robin) assumes exactly one running thread. Under the std::thread model this shows up as (DQ8 examples observed while debugging): audio-thread starvation, missed wakeups between WaitSema/SignalSema pairs, and lock-order deadlocks between the run token and guest mutexes (earlier attempt: #134).

This PR replaces that model with an N=1 cooperative fiber scheduler:

• Exactly one host OS thread (the guest executor) runs all guest fibers; one fiber runs at a time, highest priority first, FIFO within a priority (EE semantics; RotateThreadReadyQueue rotates the equal-priority group).
• Guest threads are fibers (ps2_fiber.{h,cpp}) with per-fiber R5900Context state; blocking syscalls park the fiber via an arm/publish/park protocol that cannot lose wakeups (see below).
• Host workers (IRQ/vsync/alarm/RPC) never run guest code directly; they either wake fibers through gated entry points or borrow the guest token (AsyncGuestScope) so handler code is serialized against fibers.
• Preemption points: recompiled code samples yield_point() every 128 back-edges (terminate/suspend/priority checks); blocking syscalls yield cooperatively.

Wakeup protocol (the core correctness argument)

A blocking syscall does: publish to the object wait-list under the object mutex → arm_park() (marks Blocked under the scheduler mutex) → release object mutex → block_current(). A waker that fires in the publish/arm window finds the fiber still running and records wake_pending instead of enqueueing (wake_locked); block_current() consumes it and returns WokenInWindow without parking; all wait sites re-check their predicate in a Mesa-style loop. Wakeups gate on suspendCount (a suspended waiter stays THS_WAITSUSPEND until ResumeThread). Stale wakeups after tid reuse are rejected by {generation, tid} tokens (enqueue_external_wakeup_validated). Host threads that call blocking syscalls (no fiber) use bounded-backoff Mesa loops on the same predicates.

Backends

Backend	Platform	Selection	Stack safety	Status
ucontext	Linux/macOS (default)	default	mmap + PROT_NONE guard page	tested: full suite N runs, ASan+LSan clean, deterministic
Win32 Fibers	Windows/MSVC	_WIN32	OS-native guard pages, FIBER_FLAG_FLOAT_SWITCH	compiles clean (mingw-w64 gcc+clang); validated by this PR's windows-msvc CI run
SceFiber	PS Vita	PLATFORM_VITA	sceKernelAllocMemBlock (no guard page — Vita has no per-subrange protection API; documented)	compiles clean against real VitaSDK (_sceFiberInitializeImpl, <psp2/fiber.h>); untested on hardware
pthread	Linux (testing only)	-DPS2X_FIBER_PTHREAD=ON	mmap guard + pthread_attr_setstack	exists so TSan can see the scheduler (TSan can't instrument swapcontext); full suite green

-DPS2X_SANITIZE=thread|address (top-level) wires sanitizers through every target; TSan requires the pthread backend (enforced at configure time).

Testing

• ~6k lines of scheduler tests: wakeup-window protocol (arm/park/wake races driven deterministically via atomics handshakes, not sleeps), suspend/resume gating, tid reuse + stale wakeup rejection (including a join ABA regression test), terminate-during-park-window, priority scheduling/rotation/join floors, shutdown/teardown ordering, borrowed-worker paths, EventFlag modes, alarm/vsync integration.
• Full suite: 351/351 on the ucontext and pthread backends (multiple consecutive runs, deterministic).
• ThreadSanitizer (pthread backend): 351/351; clean except two documented intentional reports (the guest-visible vsync flag/tick words are deliberately plain memory — memory-mapped-register emulation, as on hardware) and one pre-existing upstream report in updateGsCsrFieldForVSync (untouched by this PR; fixed separately in fix(runtime): make GS CSR atomic to fix vsync-worker/guest data race #145).
• AddressSanitizer + LeakSanitizer: 351/351, zero reports.
• The branch also went through a multi-agent adversarial review; bugs caught and fixed along the way: lost ReleaseWaitThread during Mesa retries, alarm-worker use-after-free vs CancelAlarm, non-fiber WaitEventFlag/WaitForNextVSyncTick returning without waiting, INTC/DMAC enable-mask races, a terminate/suspend wake dropped in the publish→park window, and a tid-recycling ABA in join_fiber.

Guest-visible semantics notes (for review against ps2sdk)

• Semaphore syscalls keep fix: semaphore syscalls return sid on success instead of KE_OK #136's return-value contract (sid on success).
• GetThreadId from the primary host thread returns 1 (upstream reserved tid 1 for main; the runtime claims it at construction).
• Suspended waiters: a SetEventFlag/SignalSema arriving while suspendCount > 0 leaves the thread THS_WAITSUSPEND; it completes the wait after ResumeThread. (Divergence from literal EE kernel timing, where the wake transitions WAITSUSPEND→SUSPEND immediately; the end state after resume is identical. Called out in the adapted kernel test.)
• Exit handlers may block: fibers have an Exiting state so a guest exit handler can make blocking syscalls without the scheduler freeing the live trampoline frame.

Vita status (honest labeling)

The SceFiber backend is compile-verified against the real VitaSDK toolchain (reproducible container harness; zero warnings) but has not run on hardware — I don't have a Vita to test with. The previous "vita is broken" state is resolved at the compile level: <psp2/fiber.h>, _sceFiberInitializeImpl (vita-headers exposes no sceFiberInitialize wrapper), and sceKernelAllocMemBlock stacks (VitaSDK has no <sys/mman.h>). If someone with hardware can boot-test, I'll support fixes.

Merge/rebase note

This branch is a single commit on top of current main (through #143), so it applies cleanly. If #140 lands first I'm happy to rebase over it.

[smmathews]

leaving this in draft while I give it a more thorough review, but overall I think it should be a more robust solution than #134

[smmathews]

still leaving in draft, I don't like how vita is broken and I can't test it. working on it.

[smmathews]

Updated and out of draft. Since the last revision:

• Merged upstream main through Feature/mpeg decoder #120 (conflict-free now; resolutions summarized in the merge commit — fix: semaphore syscalls return sid on success instead of KE_OK #136's sid-on-success semantics preserved throughout, including in the older scheduler tests).
• Windows: new Win32 Fibers backend; the windows-msvc CI leg builds and passes the full suite.
• Vita: the SceFiber backend now compiles against a real VitaSDK (<psp2/fiber.h>, _sceFiberInitializeImpl, sceKernelAllocMemBlock stacks). Untested on hardware — honestly labeled in the description.
• Test suite: 351/351, deterministic across repeated runs; TSan/ASan/LSan status documented in the description. Several real races found and fixed along the way (details in the description and commit messages).

Happy to re-merge over #140 if that lands first.

[smmathews]

Re-merged after #140/#141 landed — branch is again a single commit (93be65c) on current main, conflict-free, all CI green.

Notes on the resolution:

• feat: add SSE2 support and fix SSE4.1 paths for PADDSW and PSUBSW instructions #141 (SSE2/PADDSW/PSUBSW) has zero overlap with this branch.
• Refactor runtime for move speed and better code style #140's runtime-side changes merged cleanly with the scheduler. The new GuestBranchKind / MissingFunctionPolicy / dispatchGuestBranch machinery and the dense recompiled-function table are all kept as-is; the scheduler tests register through the new reset_ps2_test_function_table() BeforeEach hook.
• Refactor runtime for move speed and better code style #140 rewrote the guest-execution mutex machinery (enterGuestExecution et al.), which this branch deletes outright — under the N=1 scheduler only one fiber ever executes guest code, so the scope guards remain no-ops and the mutex block stays removed. Nothing else in the merged tree references it.
• Verified the two cooperative-yield surfaces survived the emitter split: recompiled back-edges still emit shouldPreemptGuestExecution() (control_flow_emitter.cpp), and the kernel-syscall ps2sched::maybe_yield() sites are untouched.

Local validation: 354/354 (incl. the new dispatchGuestBranch tests) on ucontext and pthread backends, TSan and ASan/LSan clean apart from the documented intentional MMIO-polling reports.

One observation from the merge, independent of this PR: #140 made lookupFunction exact-entry-only, so a mid-function PC now resolves to the missing-function stub instead of its owning function. That's inert for this branch (kernel PC loops step entry→entry and are hasFunction-guarded; interrupted fibers resume via context switch, not re-lookup), but flagging it in case other callers relied on mid-function resolution.