Systems engineer focused on LLM inference infrastructure, GPU memory systems, and serving optimization.
I build simulators and tools to study the latency, memory, throughput, routing, and quantization tradeoffs inside LLM serving systems — and validate them against real GPU measurements.
| Project | Focus | Key Finding |
|---|---|---|
| kv-cache-compaction-lab | KV-cache page compaction | ThresholdCompaction dominates; 11 free-compaction points |
| prefix-cache-sim | Prefix sharing with RadixTree | LFU dominates under Zipf; multi-turn hit rate 60%+ |
| llm-inference-scheduler | Continuous batching scheduler | ChunkedPrefill eliminates starvation; FCFS collapses under load |
| tensor-memory-allocator | GPU tensor memory allocation | Free-list beats buddy/slab for continuous size distributions |
| llm-serving-sim | End-to-end LLM serving | ChunkedPrefill + LFU: 41% lower TTFT p95, 94% prefix hit rate |
| speculative-decoding-sim | Speculative decoding | 6.06x max speedup; breakeven at cost_ratio = 0.25 |
| moe-router-sim | MoE routing and load balancing | ExpertChoice best balance; NoisyTopK best practical tradeoff |
| admission-control-sim | Admission control under overload | Tight token budget maximizes goodput; proactive beats reactive |
| kv-cache-disaggregation-sim | Prefill/decode disaggregation | Disagg wins at arrival>=20 AND prompt>=1024; 29% TTFT gain |
| speculative-decoding-validation | Real GPU validation | Median 1.14x speedup with KV cache; simulation confirmed |
| quantization-impact-analyzer | Weight quantization sensitivity | INT8-g32: 1.8x compression, +0.13 PPL; group-wise reduces INT4 error 99% |
| latency-breakdown-simulator | Where each millisecond goes | Compute = 99.8%; prefix cache saves 17% TTFT; disagg adds 8-20% overhead |
| request-lifecycle-tracker | Per-request event tracing | 24 event types; full lifecycle from arrival to memory release |
| real-model-profiler | Real GPU cost measurement | Prefill: 30-70 us/tok; decode memory-bound at 5300-10800 us/tok |
| attention-kernel-profiler | Attention kernel profiling | sdpa 7.64x faster than naive; memory O(n^0.25) vs O(n^2) at seq=2048 |
| continuous-batching-profiler | Real continuous batching on GPU | ChunkedPrefill is a fairness mechanism; EagerContBatch wins on mean TTFT |
| kv-cache-profiler-real | Real KV cache measurements | 36 KB/tok validated; compaction+decode 30% faster than compaction alone (CUDA stream overlap) |
All projects: C++20 or Python, quantitative results, open source.
Optimizing one component in isolation is not enough.
- The scheduler that minimizes TTFT hits OOM first under memory pressure
- The prefix cache that saves compute also consumes memory
- The allocator that reduces fragmentation can increase lookup cost
- Speculative decoding can hurt throughput if the draft model is too expensive
- The MoE router that achieves perfect balance sacrifices expert specialization
- Admission control that accepts everything destroys goodput under overload
- Disaggregation that eliminates interference pays KV transfer cost instead
- Quantization that maximizes compression destroys model quality
- Kernel choice matters more than model architecture at long sequences
- ChunkedPrefill does not maximize mean TTFT -- it protects decode tail latency
- Compaction during decode is 30% faster than alone (CUDA overlaps copy+compute streams)
End-to-end systems thinking matters more than any single optimization.
- C++20 -- allocators, schedulers, caches, routers, simulators, tracers
- Python + PyTorch -- real model profiling, attention kernels, quantization, serving
- CMake + Ninja -- build system
- RTX 2070 (8GB, sm75) -- GPU for real measurements
- Real batched decode with paged attention
- Per-token streaming latency under realistic load
- Comparing simulation predictions with vLLM production metrics