fix(serve): resource governor for multi-instance system freezes#142
Open
rajkumarsakthivel wants to merge 6 commits into
Open
fix(serve): resource governor for multi-instance system freezes#142rajkumarsakthivel wants to merge 6 commits into
rajkumarsakthivel wants to merge 6 commits into
Conversation
…eezes (#139) Multiple cce serve processes (one per project per AI session) exhaust system memory and create 10-20 minute desktop freezes. Root cause: no thread caps, no idle shutdown, no cross-process coordination, no memory pressure awareness. Adds resource_governor.py with four mechanisms: - ONNX Runtime thread caps (default 2 per process, via CCE_ORT_THREADS) - Per-project file lock so duplicate instances don't index simultaneously - Linux PSI memory-pressure detection to pause indexing under pressure - Idle auto-shutdown after 30 minutes of MCP inactivity New config keys: serve.idle_timeout_minutes, serve.max_ort_threads New env vars: CCE_ORT_THREADS, CCE_IDLE_TIMEOUT_MINUTES
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #139. Multiple
cce serveprocesses (one per project per AI session) exhaust system memory and cause 10–20 minute desktop freezes on Linux. The reported case had 68 CCE processes with 2,179 threads consuming 5.4 GiB RSS, triggering kernel OOM and PSI pressure spikes above 85%.This PR adds a resource governor (
resource_governor.py) with four mechanisms:ONNX Runtime thread caps: each
cce serveprocess defaults to 2 intra-op threads instead of cpu_count. Override viaCCE_ORT_THREADSenv var orserve.max_ort_threadsconfig key. Also setsTOKENIZERS_PARALLELISM=falseto prevent the tokenizers library from spawning additional threads.Per-project index lock: file-lock (
fcntl.flock) so when multiple AI sessions spawn duplicatecce serveprocesses for the same project, only one indexes at a time. Others skip and retry on the next file-change event. No-op on Windows (advisory lock not available).Linux PSI memory-pressure detection: reads
/proc/pressure/memoryand defers re-indexing whensome avg10exceeds 25%. This prevents CCE from contributing to the cascading page-cache thrashing described in the issue. No-op on non-Linux platforms.Idle auto-shutdown:
cce serveexits after 30 minutes of MCP inactivity (nocall_toolinvocations). This prevents zombie processes from accumulating when AI sessions disconnect or go idle. Configurable viaCCE_IDLE_TIMEOUT_MINUTESenv var,serve.idle_timeout_minutesconfig key, or set to 0 to disable.New config keys
Impact on the reported scenario
With the defaults from this PR, the 68-process scenario from the issue would see: