LMCache: Open-Source KV Cache Management for Faster and More Efficient LLM Inference

14x faster and 90% cheaper LLM inference. (100% open-source, KV cache management) LMCache is an open-source KV cache management layer that plugs into vLLM, SGLang, and TensorRT-LLM. Here's how it works: LLMs recompute their understanding of the same content on every request. The same system prompts, the same documents, processed from scratch every time, and a single GPU throws away roughly 15 TB of this reusable cache per day. LMCache stores that cache and serves it back on repeat requests, running as a separate process completely outside the inference engine. The engine just asks for the cache blocks it needs. LMCache handles all the heavy data movement across GPU, CPU, disk, and remote storage in parallel, so cache work never steals compute from inference. It also reuses cache beyond exact prefixes. Their CacheBlend technique (EuroSys 2025 Best Paper) keeps RAG documents cached no matter what order they appear in. On H200s with a 235B model, that adds up to 14x faster time-to-first-token and 4x faster decoding. And since reuse skips the compute entirely (the same reason providers discount cached tokens by 90%), the cost savings follow directly. The repo has the full architecture breakdown, benchmarks, and a Kubernetes operator for production use. Link in the first comment. ____ Share this with your network if you found this insightful ♻️ Follow me (Akshay Pachaar) for more insights and tutorials on AI and Machine Learning!

This post is based on a real open-source project and accurately describes many of its capabilities. However, some performance and cost claims are benchmark-specific and should not be interpreted as universal results.

Confirmed Accurate

• LMCache is a real open-source KV cache management layer for LLM inference.
• It integrates with inference frameworks such as vLLM, SGLang, and TensorRT-LLM.
• LMCache stores and reuses KV cache across requests, reducing redundant computation for repeated prompts and documents.
• It manages cache across GPU memory, CPU memory, disk, and remote storage independently of the inference engine.
• CacheBlend is a real technique developed by the LMCache team for improving cache reuse across reordered RAG documents, and the associated paper received the EuroSys 2025 Best Paper Award.
• The project provides production tooling, including a Kubernetes operator.

Not Fully Accurate / Benchmark-Specific

• The claim of 14× faster inference is based on specific benchmark conditions (hardware, model size, workload, and cache hit rate). It should not be interpreted as typical performance for all deployments.
• The statement 90% cheaper LLM inference is an oversimplification. Reusing cached KV states can significantly reduce inference costs, but actual savings depend on workload characteristics, cache hit rates, infrastructure, and model provider pricing.
• The claim that a GPU discards 15 TB of reusable cache per day is an illustrative estimate rather than a universally applicable measurement.