Skip to content

fix(serve): resource governor for multi-instance system freezes#142

Open
rajkumarsakthivel wants to merge 6 commits into
mainfrom
fix/resource-governor-139
Open

fix(serve): resource governor for multi-instance system freezes#142
rajkumarsakthivel wants to merge 6 commits into
mainfrom
fix/resource-governor-139

Conversation

@rajkumarsakthivel

Copy link
Copy Markdown
Member

Summary

Fixes #139. Multiple cce serve processes (one per project per AI session) exhaust system memory and cause 10–20 minute desktop freezes on Linux. The reported case had 68 CCE processes with 2,179 threads consuming 5.4 GiB RSS, triggering kernel OOM and PSI pressure spikes above 85%.

This PR adds a resource governor (resource_governor.py) with four mechanisms:

  • ONNX Runtime thread caps: each cce serve process defaults to 2 intra-op threads instead of cpu_count. Override via CCE_ORT_THREADS env var or serve.max_ort_threads config key. Also sets TOKENIZERS_PARALLELISM=false to prevent the tokenizers library from spawning additional threads.

  • Per-project index lock: file-lock (fcntl.flock) so when multiple AI sessions spawn duplicate cce serve processes for the same project, only one indexes at a time. Others skip and retry on the next file-change event. No-op on Windows (advisory lock not available).

  • Linux PSI memory-pressure detection: reads /proc/pressure/memory and defers re-indexing when some avg10 exceeds 25%. This prevents CCE from contributing to the cascading page-cache thrashing described in the issue. No-op on non-Linux platforms.

  • Idle auto-shutdown: cce serve exits after 30 minutes of MCP inactivity (no call_tool invocations). This prevents zombie processes from accumulating when AI sessions disconnect or go idle. Configurable via CCE_IDLE_TIMEOUT_MINUTES env var, serve.idle_timeout_minutes config key, or set to 0 to disable.

New config keys

serve:
  idle_timeout_minutes: 30   # 0 = disabled
  max_ort_threads: 2         # 0 = use ONNX default

Impact on the reported scenario

With the defaults from this PR, the 68-process scenario from the issue would see:

  • Thread count reduced from ~2,179 to ~340 (68 × ~5 threads each vs. 68 × ~32)
  • Idle processes auto-shutdown after 30 minutes, reducing the steady-state count
  • Concurrent indexing serialized per-project, preventing I/O amplification
  • Memory-pressure detection pausing heavy work before the system enters OOM territory

…eezes (#139)

Multiple cce serve processes (one per project per AI session) exhaust
system memory and create 10-20 minute desktop freezes. Root cause: no
thread caps, no idle shutdown, no cross-process coordination, no memory
pressure awareness.

Adds resource_governor.py with four mechanisms:
- ONNX Runtime thread caps (default 2 per process, via CCE_ORT_THREADS)
- Per-project file lock so duplicate instances don't index simultaneously
- Linux PSI memory-pressure detection to pause indexing under pressure
- Idle auto-shutdown after 30 minutes of MCP inactivity

New config keys: serve.idle_timeout_minutes, serve.max_ort_threads
New env vars: CCE_ORT_THREADS, CCE_IDLE_TIMEOUT_MINUTES
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extreme (300x) load / system-wide freeze with multiple cce serve instances

1 participant