Skip to content

feat(win): fix silent CPU fallback, launcher suite, DirectStorage expert loads - #670

Open
khalilswdp wants to merge 4 commits into
JustVugg:devfrom
khalilswdp:feat/windows-gpu-suite
Open

feat(win): fix silent CPU fallback, launcher suite, DirectStorage expert loads#670
khalilswdp wants to merge 4 commits into
JustVugg:devfrom
khalilswdp:feat/windows-gpu-suite

Conversation

@khalilswdp

Copy link
Copy Markdown
Contributor

**Stacked on #667
#667 ** (model-architecture seam) — its commit shows here until
it merges. This PR's own work is three commits, reviewable independently:

  1. fix(win): static cudart + launcher/cap-warning fixes
  2. feat(win): PowerShell launcher suite
  3. feat(win): DirectStorage expert-load transport (opt-in)

Summary

On Windows every run silently fell back to CPU: coli_cuda.dll linked dynamic
cudart, but CUDA 13.x keeps cudart64_*.dll in bin\x64 (not on PATH), so
LoadLibrary failed quietly. Fix: -cudart static (the DLL then imports only
KERNEL32). The rest makes the fix visible and usable:

  • coli launcher: banner prints the real backend instead of a hardcoded
    "streaming CPU"; numeric env vars parse tolerantly (CUDA_EXPERT_GB=auto no
    longer crashes); --ram accepts fractional budgets.
  • cap=1 warning: quotes the actual RAM shortfall and what --ram would
    buy a second cache slot, instead of unconditionally blaming PIN_GB.
  • windows/*.ps1 suite: Build (toolchain detect + doctor), Run (wizard,
    attaches to a running serve instead of starting a second engine), Chat, Web,
    Warmup. Saved with a UTF-8 BOM — PowerShell 5.1 reads BOM-less files as
    ANSI, which breaks the multilingual warmup prompts.
  • warmup.ps1: -Backend auto|gpu|cpu (the .coli_usage pin is
    backend-flavoured), -PromptFile, +40 multilingual/code prompts.
  • DirectStorage expert loads (opt-in COLI_DSTORAGE=1, default off): the
    miss path's coalesced ~19 MB slab pread can be served by a DirectStorage
    host-destination DMA instead. Pure transport swap — bytes land in the slab
    the expert cache already owns; cache/eviction/layout untouched; primary
    replica only; any failure (no SDK at build, no dstorage.dll, no adapter,
    request error) falls back to pread. The depot PR reuses this transport for
    disk→VRAM. No performance claims yet — correctness-verified only.

Validation

  • make -C c check full pass at this tip; zero new warnings
  • make CUDA_DLL=1 colibri.exe clean; DLL builds with and without
    DSTORAGE_HOME
  • DirectStorage A/B on a real int4 container: COLI_DSTORAGE=0 vs 1
    byte-identical output; served loads reported at exit
  • Token-exact oracle with the flag off (default): 32/32 TF, 20/20 greedy
  • All six .ps1 scripts parse under Windows PowerShell 5.1 and PS 7
  • Static-cudart DLL: dumpbin shows KERNEL32 imports only; GPU engages on a
    box where the dynamic link failed silently

Compatibility

  • Default CPU build unchanged and dependency-free — DirectStorage is
    opt-in at build time (DSTORAGE_HOME), optional in the loader;
    Linux/macOS unchanged
  • No model files or binaries included
  • Defaults changed: none in the engine.

@JustVugg

Copy link
Copy Markdown
Owner

Holding this behind #667 (its arch-seam base). Please rebase #667 first onto latest dev, then this one on top — CI green on both, just needs the merges resolved.

@JustVugg JustVugg added the enhancement New feature or request label Jul 28, 2026
@khalilswdp
khalilswdp force-pushed the feat/windows-gpu-suite branch from 1e78fec to 58a241a Compare July 28, 2026 21:27
@khalilswdp
khalilswdp force-pushed the feat/windows-gpu-suite branch from 58a241a to d74d087 Compare July 28, 2026 23:03
@JustVugg

Copy link
Copy Markdown
Owner

dev has moved a fair amount (v1.3.0: Kimi K3 and Inkling engines, the ragged-attention grouped-scale fix, CUDA_RELEASE_HOST default, the Inkling expert-cache fix). This PR now conflicts — could you rebase onto latest dev? Nothing is wrong with the change itself; CI was green where it ran. Ping me once it's clean and I'll review/merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request needs-rebase Confligge, serve rebase dell'autore

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants