7 Techniques That Actually Make LLM Inference Cheaper (And Why They Work)

𝗟𝗟𝗠 𝗦𝗲𝗿𝘃𝗶𝗻𝗴: 𝟳 𝗪𝗮𝘆𝘀 𝘁𝗼 𝗠𝗮𝗸𝗲 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗖𝗵𝗲𝗮𝗽𝗲𝗿
1. Continuous batching: Instead of waiting for every sequence in a batch to finish, evict a sequence the moment it emits its stop token and admit a waiting one into the free slot. The GPU stops idling on the longest generation in the group.
 - Used when traffic is concurrent and output lengths vary widely.
2. Paged attention: Store the KV cache in fixed-size blocks that need not be contiguous, the way an operating system pages memory. Removes the need to reserve worst-case contiguous space per sequence.
 - Used when memory fragmentation, not compute, is what caps concurrency.
3. Prefix caching: Reuse the KV cache for a prompt prefix that has already been computed, so a shared system prompt or a long document is encoded once rather than once per request.
 - Used when many requests begin with identical tokens.
4. Speculative decoding: A small draft model proposes several tokens ahead, and the large model verifies them in one forward pass, accepting the longest correct run. Rejected guesses cost a pass that would have happened anyway.
 - Used when decoding is memory-bandwidth-bound and spare compute is available.
5. Quantization: Serve weights, and sometimes activations or the KV cache, at lower precision than training used. Fewer bytes moved per token is directly fewer milliseconds per token.
 - Used when the model barely fits, or fits with no room for concurrency.
6. Prefill-decode disaggregation: Run the compute-bound prefill phase and the bandwidth-bound decode phase on separate pools of hardware, so neither one queues behind the other.
 - Used when time-to-first-token and inter-token latency have separate targets.
7. Multi-query and grouped-query attention: Share key and value heads across query heads so the KV cache holds a fraction of the entries per token.
 - Used when long contexts or high concurrency make the cache the binding constraint.
Three of these buy throughput by keeping the GPU busy (1, 4, 6) and four buy it by moving fewer bytes (2, 3, 5, 7). That split is the whole subject: decoding is limited by memory bandwidth far more often than by arithmetic, which is why shrinking the KV cache tends to pay more than any scheduler change.

Unlike a product-announcement post, this is a technical explainer with checkable claims — each technique is a documented, real systems optimization, and the descriptions are accurate.

All seven techniques are correctly named and correctly described:

• Continuous batching — accurate. This is the Orca/vLLM scheduling approach: evicting a finished sequence and admitting a queued one into the freed slot rather than waiting for the whole static batch to finish, avoiding idle GPU time on the longest generation.
• PagedAttention — accurate description of the vLLM mechanism: KV cache stored in non-contiguous fixed-size blocks, OS-paging-style, removing the need to reserve worst-case contiguous memory per sequence.
• Prefix caching — accurate; this is standard practice (vLLM automatic prefix caching, SGLang RadixAttention) for reusing KV cache on shared prompt prefixes.
• Speculative decoding — accurate; draft-then-verify with acceptance of the longest correct run is the standard description from the original speculative decoding papers.
• Quantization — accurate; weight/activation/KV precision reduction directly maps to less HBM traffic per token.
• Prefill-decode disaggregation — accurate and matches the DistServe framing precisely: prefill is compute-bound (large matmuls over the full prompt), decode is memory-bandwidth-bound (loading weights + growing KV cache one token at a time), and colocating them causes prefill bursts to stall active decode requests — which is exactly why separate hardware pools with separate TTFT/ITL targets make sense.
• Multi-query / grouped-query attention — accurate; sharing KV heads across query heads directly shrinks per-token KV cache size, which is the correct mechanism (not an approximation of it).

The closing claim — "decoding is memory-bandwidth-bound far more often than compute-bound, so shrinking the KV cache pays more than scheduler changes" — is the correct, well-established framing from inference systems research (this is precisely the premise DistServe and PagedAttention are both built on).

One nuance worth flagging on the 3-vs-4 categorization: grouping technique #2 (PagedAttention) under "moving fewer bytes" is a slight simplification. PagedAttention's actual mechanism is reducing fragmentation and reserved-but-unused memory (allowing more concurrent sequences to fit), not reducing the number of bytes moved per token during a decode step the way quantization, prefix caching, or GQA/MQA literally do. It increases throughput via better memory utilization, which is a related but distinct lever from the other three in that bucket. Not wrong, just a slightly looser fit than the other three.

Nothing here is fabricated or outdated — this reads like someone who's actually read the vLLM/DistServe/GQA papers, not someone summarizing marketing blogs.