kvcached: Virtual Memory for GPU KV Cache, and Why It Matters for Multi-LLM Serving

kvcached is a KV-cache memory manager for LLM serving that brings an OS-style virtual-memory idea to GPU memory => instead of forcing engines to reserve a fixed amount of physical GPU memory for KV cache at startup, it separates the cache’s virtual address space from the physical memory backing it, so memory can be allocated and reclaimed as demand changes. This lets multiple LLMs share the same GPU more elastically, supports workloads where models sleep and wake on demand, and makes it easier to colocate inference with training, fine-tuning, or vision jobs. It integrates with vLLM and SGLang and also supports prefix caching.
treat KV-cache memory as a dynamic shared resource instead of permanently carving up GPU memory per model.
GitHub Repo: https://lnkd.in/gbzMaNMq

This is an accurate, well-paraphrased summary of a real project — nothing invented, nothing exaggerated.

kvcached is real, hosted on GitHub as ovg-project/kvcached, described in its own README as: kvcached (KV cache daemon) is a KV cache library for LLM serving/training on shared GPUs. By bringing OS-style virtual memory abstraction to LLM systems, it enables elastic and demand-driven KV cache allocation, improving GPU utilization under dynamic workloads.

The virtual/physical memory decoupling claim is confirmed precisely. The project achieves this by decoupling GPU virtual addressing from physical memory allocation for KV caches, allowing serving engines to initially reserve virtual memory only and back it with physical GPU memory only when the cache is actively used — this is exactly the "separates virtual address space from physical memory backing" framing in your post.

"Fixed reservation at startup" contrast is accurate. The project's own comparison states that current serving engines need to statically reserve GPU memory at startup, which is the exact problem kvcached is positioned against.

Multi-LLM elastic sharing — confirmed. kvcached allows multiple LLMs to share a GPU's memory elastically, enabling concurrent deployment without the rigid memory partitioning used today.

Sleep/wake and serverless claim — confirmed. The project has a frontend router and sleep mode that puts models to sleep when idle, and a serverless use case where models can spin up and down on demand by allocating KV cache only when needed.

Colocation with training/fine-tuning/vision — confirmed nearly verbatim as a named use case: GPU workload colocation — kvcached allows LLM inference to coexist with other GPU workloads such as training jobs, fine-tuning, or vision models.

vLLM/SGLang integration and prefix caching — both confirmed. It integrates with SGLang and vLLM, and as of an April 2026 update, supports automatic prefix caching (APC) for vLLM and RadixCache for SGLang.

One thing your post leaves out that would strengthen it: the project reports concrete numbers — kvcached achieves 2-28x TTFT reduction compared to current serving engines in a benchmark serving three Llama-3.1-8B models on an A100-80GB GPU under intermittent-peak workloads. Worth adding if you want a harder proof point instead of just the architectural description.