5 Ways to Shrink a 70B Model Onto One GPU: A Quantization Primer

LLM quantization techniques I'd learn if I had to fit a 70B model on a single GPU: (bookmark this) A 70B model in FP16 needs 140GB for weights alone. At 4-bit, that drops to 35GB, which fits on one card. But naive rounding fails on large models. Roughly 0.1% of hidden dimensions carry values up to 20x larger than anything else in the tensor, and they wreck the quantization grid for everything else. Each of these 5 methods handles those outliers at a different point: 1. RTN: ignores them. Rounds every weight to the nearest grid level with no calibration data. Cheapest option, weakest at low bit widths. 2. GPTQ: repairs after rounding. Quantizes a layer column by column and adjusts the remaining weights to absorb the error before moving on. 3. AWQ: protects before rounding. Finds the ~1% of weight channels that matter most and scales them up so they survive quantization. Everything still ends up in plain INT4. 4. LLM. int8(): isolates at inference. Outlier dimensions run in FP16, the other 99.9% run in INT8, and the results are merged. 5. QAT: solves it during training. The model is fine-tuned with rounding baked into every forward pass, so it adapts to the damage before quantization is actually applied. All five produce the same artifact, a model at a fraction of its trained precision. They differ only in where the outlier problem gets addressed. The visual below nicely summarise these techniques. There's a really good paper that provides a comprehensive study of LLM quantization techniques. I have shared the link to the paper in the first comment. ____ Share this with your network if you found this insightful ♻️ Follow me (Akshay Pachaar) for more insights and tutorials on AI and Machine Learning!

This is a solid, largely accurate technical explainer. I verified the core numbers and technique descriptions against primary sources.

Confirmed accurate:

• Memory math checks out: A 70B model in FP16 (2 bytes/parameter) needs ~140GB for weights (70B × 2 bytes = 140GB). At 4-bit (0.5 bytes/parameter), that's ~35GB — both figures are simple, correct arithmetic.
• The outlier phenomenon is real and well-documented — this comes from the original LLM.int8() paper (Dettmers et al., 2022). The paper found that beyond roughly 6.7B parameters, transformers develop "emergent outlier features" — specific hidden dimensions with values dramatically larger than the rest, appearing consistently across layers. The paper documents these outliers as ~0.1% of feature dimensions with magnitudes 20x–100x larger than typical values. The post's "20x larger" and "0.1% of hidden dimensions" figures are accurate, if on the conservative end of the paper's reported 20–100x range.
• RTN (round-to-nearest) — correctly described as calibration-free, weakest at low bit-widths.
• GPTQ — correctly described as post-hoc, layer-by-layer (column-by-column) error compensation; this matches the actual GPTQ algorithm, which quantizes weights sequentially and adjusts remaining unquantized weights to compensate for introduced error.
• AWQ (Activation-aware Weight Quantization) — correctly described; AWQ does identify a small fraction (~1%) of salient weight channels based on activation magnitude and scales them before quantization, keeping everything in plain INT4 rather than mixed precision.
• LLM.int8() — correctly described; this is exactly the paper's mixed-precision decomposition approach: outlier dimensions computed in FP16, the remaining ~99.9% in INT8, with results merged — directly confirmed in the original paper and Dettmers' own explanation of the method.
• QAT (Quantization-Aware Training) — correctly described as baking quantization effects into training/fine-tuning so the model adapts to rounding error before deployment; this is the standard, accurate characterization of QAT.

Minor notes:

• The post says outliers are "up to 20x larger" — the actual paper documents this range as 3x–100x, with 20x cited in some illustrative examples elsewhere. Not wrong, but slightly narrows the actual reported range.
• The framing "all five produce the same artifact... they differ only in where the outlier problem is addressed" is a reasonable, clean pedagogical simplification, though QAT in particular is a meaningfully different category (a training-time approach vs. the other four being post-training/inference-time methods) — a nuance the post doesn't flag.

Verdict: This is an accurate, well-grounded technical summary that matches the peer-reviewed literature closely (LLM.int8() paper, GPTQ, AWQ papers). No fabricated numbers or false claims found.