Can AirLLM run a 70B LLM on 4GB VRAM and Llama 3.1 405B on 8GB without quantization?

Do not buy a new GPU this year.
Seriously. Close the Nvidia tab and read this first.
A free Python library just ran a 70B model on a 4GB graphics card.
Then it ran Llama 3.1 405B on about 8GB.
No quantization needed. No pruning.
It's called AirLLM, and the trick behind it is smarter than it sounds.
A large model is a stack of layers. Llama 70B has 80 of them. Your prompt goes into layer 1, comes out, goes into layer 2, and keeps climbing until the answer pops out the top.
Normal setups load all 80 layers into your GPU at once. That's why a 70B model wants around 140GB of VRAM at full precision.
AirLLM does something different.
It loads one layer. Runs it. Flushes it. Loads the next one.
Your GPU only ever holds a single layer at a time. So the VRAM you need depends on the size of one layer, not the size of the whole model.
That one idea is why these numbers look fake:
→ Llama 3.x 70B: ~4GB → Qwen3-235B: ~3GB → Llama 3.1 405B: ~8GB → DeepSeek-V3 (671B): ~12GB
Now the part nobody tells you.
It's slow. Every new word means streaming the model through your GPU layer by layer. Your disk becomes the bottleneck, not your graphics card. A fast NVMe SSD matters more than anything else here.
It eats disk space. Before the first run, AirLLM splits the model into separate layer files. Set delete_original=True and it keeps only the split copy, which cuts disk use roughly in half.
Want more speed? Pass compression='4bit' or '8bit'. The README claims up to 3x faster inference with almost no accuracy loss, because it only shrinks the weights being loaded from disk.
On a Mac? Apple silicon only, and you need mlx installed.
Getting started takes two steps:
pip install airllm
Then pass any Hugging Face model ID into AutoModel.from_pretrained(). Llama, Qwen, DeepSeek, Mistral, Gemma, and Phi all work through that same line.
Where this actually shines:
→ Testing a giant model before paying for cloud GPUs → Overnight batch jobs where speed doesn't matter → Running models fully offline on hardware you already own → Finally seeing how a 405B model thinks without a server rack
Where it doesn't: anyone who wants ChatGPT-speed replies.
Your old gaming laptop just got a second career.
35.4K stars. Apache 2.0. 100% Opensource.

AirLLM is an open-source Python library that enables large language models to run with limited GPU memory by loading model layers sequentially instead of loading the entire model into GPU memory at once. Its official documentation reports that Llama 3.x 70B can run with approximately 4 GB of GPU VRAM and Llama 3.1 405B with approximately 8 GB of GPU VRAM.

However, low GPU memory requirements do not mean that these models will run quickly or smoothly on ordinary computers. The complete model still requires substantial storage space, sufficient system memory, and repeated data transfers between storage and the GPU. These transfers can significantly slow down inference. Actual requirements and performance depend on the model format, sequence length, hardware, and system configuration.

Therefore, the core claim about reducing GPU memory requirements is supported by the project's documentation, but the figures should not be interpreted as a guarantee of practical performance on every computer. AirLLM is useful for experimenting with large models on memory-constrained hardware, although its performance may be unsuitable for interactive applications or production workloads.