Awesome-LLM-KV-Cache: A curated list of 📙Awesome LLM KV Cache Papers with Codes.
-
Updated
Jun 17, 2026
#
kv-cache-quantization
Here are 14 public repositories matching this topic...
Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).
raspberry-pi privacy offline-first distributed-computing self-hosted orange-pi ai-agents data-sovereignty ai-platform local-first agent-framework apple-silicon llm vllm local-llm llm-inference kv-cache-quantization rockchip-npu turboquant
-
Updated
Jul 10, 2026 - Python
[ICLR 2025] Palu: Compressing KV-Cache with Low-Rank Projection
-
Updated
Feb 20, 2025 - Python
SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.
memory cuda self-hosted quantization gemma rag awq llm-serving pgvector vllm local-llm llm-inference qwen speculative-decoding kv-cache-quantization tool-calling consumer-gpu rtx-4090 rtx-3090 spec-decode
-
Updated
Jul 8, 2026 - Python
Empirical study of KV-cache quantization in self-forcing video generation
-
Updated
Mar 14, 2026 - Python
W4A4 and INT8 KV-cache quantization for Infinity VAR models. Optimized for high-fidelity generative AI deployment on edge GPUs (e.g. NVIDIA Jetson).
computer-vision pytorch gpu-acceleration quantization model-compression nvidia-jetson inference-optimization edge-ai on-device-ml weight-quantization post-training-quantization autoregressive-models generative-ai kv-cache-quantization activation-quantization visual-autoregressive-model svdquant infinity-var
-
Updated
Jun 11, 2026 - Python
Measure MLX quantization quality loss — KL divergence, perplexity, top-token agreement for KV cache and weights
python machine-learning metal diagnostics quantization mlx kl-divergence perplexity model-quantization kv-cache apple-silicon llm llm-inference llm-eval kv-cache-quantization mlx-lm quantization-quality
-
Updated
Jul 4, 2026 - Python
Evaluation harness and norm-direction method for KV cache compression. Cross-model worst-case quality metrics.
-
Updated
May 16, 2026 - Python
Reproduction of TurboQuant.
transformers quantization kv-cache large-language-models llm-inference kv-cache-quantization polarquant
-
Updated
Jun 23, 2026 - Python
MLX-native port of KVarN — variance-normalized KV-cache quantization for Apple Silicon. 3.3× compression at 71% FP16 speed, matches FP16 on GSM8K within ~2%.
mac metal transformer quantization mlx kv-cache apple-silicon llm llm-inference kv-cache-quantization apple-ml
-
Updated
Jun 12, 2026 - Python
Attention-aware KV cache quantization for LLM inference
ai attention quantization ai-agents inference-optimization kv-cache llm long-context long-context-attention kv-cache-quantization
-
Updated
Jun 30, 2026 - Python
Deploy Nemotron 3 Nano 30B with 1M context window on NVIDIA DGX Spark using llama.cpp (Blackwell sm_121, Q4_0 KV cache quantization)
cuda inference aarch64 mamba mixture-of-experts blackwell long-context llama-cpp local-llm gguf kv-cache-quantization nemotron nvidia-dgx-spark 1m-context-window
-
Updated
Mar 22, 2026 - Shell
Neutral tensor-level KV-cache quantization benchmark: 11 methods as first-class peers, per-metric leaderboards, matched-budget Pareto, multi-seed CI, and per-method fidelity tags.
benchmark machine-learning pytorch quantization inference-optimization kv-cache llm llm-inference kv-cache-quantization
-
Updated
Jun 8, 2026 - Python
KV Cache is one of the most critical optimization techniques in modern Large Language Models. However, it also creates one of the biggest memory bottlenecks in AI inference.
-
Updated
Jul 1, 2026 - Jupyter Notebook
Improve this page
Add a description, image, and links to the kv-cache-quantization topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the kv-cache-quantization topic, visit your repo's landing page and select "manage topics."