Someone Finally Organized LLM Inference Knowledge — Here's What's Inside

𝗦𝗼𝗺𝗲𝗼𝗻𝗲 𝗳𝗶𝗻𝗮𝗹𝗹𝘆 𝗽𝘂𝘁 𝗮𝗹𝗹 𝗼𝗳 𝗟𝗟𝗠 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗶𝗻 𝗼𝗻𝗲 𝗽𝗹𝗮𝗰𝗲. ━━━━━━━━━━━━━━━ Inference knowledge is scattered. Half a paper here, a vendor blog there, the real answer buried in a GitHub issue from eight months ago. Modular open-sourced an LLM Inference Handbook that pulls it together. Free, no signup. ━━━━━━━━━━━━━━━ The calculators are the part I'd actually use: → GPU memory calculator — VRAM for a given model and config → KV cache calculator — the number that decides your batch size → Batching simulator — static vs dynamic vs continuous → Chunked prefill scheduler — see a long prefill stall your decodes → Quantization memory visualizer → GPU ↔ model matching table, NVIDIA and AMD Plus the written coverage: TTFT, TPOT, goodput, prefix caching, prefill-decode disaggregation, GPU architecture. ━━━━━━━━━━━━━━━ Two notes: → Append .md to any URL for markdown. There's an llms.txt index — feed the whole thing to your coding agent. → It's Modular's, so deployment framing leans their way. Fundamentals don't. Bookmark it. You'll come back to the KV cache calculator. 🔗 Link in the comments 👇 ♻️ Repost so it reaches someone sizing a GPU cluster this week.

Confirmed accurate:

• The LLM Inference Handbook is real and free, hosted at handbook.modular.com, no signup required.
• The .md URL append trick and the llms.txt index are both explicitly confirmed — the handbook's own pages state: "For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL." This matches the post's tip almost verbatim.
• Several named calculators/visualizers are directly confirmed: GPU Memory Calculator, KV Cache Calculator, GPU Comparison Table (matching models to NVIDIA/AMD GPUs), and a Latency Metrics Playground covering TTFT, TPOT/ITL, E2EL, and SLO-based goodput — all match the post's description.
• Topics like KV cache, prefix caching, quantization, and GPU memory sizing are all substantively covered in the handbook's content.

One important nuance the post glosses over:

• The post frames this as "Modular open-sourced" the handbook. The actual source repository is currently hosted under bentoml/llm-inference-handbook on GitHub (BentoML's org) — and its own README states it is "styled with the Modular design language system" and that its "future canonical source home is modular/llm-inference-handbook." So this looks like either a joint effort or a handoff-in-progress between BentoML and Modular, rather than a handbook purely authored and released by Modular. The docs are licensed CC BY 4.0. This is a meaningful attribution wrinkle — not wrong that Modular hosts/promotes it, but "Modular open-sourced it" oversimplifies who actually built it.
• The post's own caveat — "It's Modular's, so deployment framing leans their way" — is a fair and honest disclosure regardless of the authorship nuance above.

Not independently confirmed (but plausible given the site's scope): the specific batching simulator, chunked prefill scheduler, and quantization memory visualizer tools — I found strong evidence of the handbook's calculator-heavy format and overlapping content, but didn't turn up those three tools by name in what I could access. Given the confirmed presence of several very similar interactive tools, their existence is credible but unverified from my end.

Verdict: Overwhelmingly accurate description of a real, useful resource — the one correction worth making is that the handbook appears to originate from (or be co-developed with) BentoML, not solely "Modular," even though Modular now hosts and brands it.