arrow_back Back to AIFC
S
S Tarunhiga
Aug 02, 2026
pending ChatGPT

The Future of Local AI Isn't More VRAM—It's Smarter Inference

Grounded / Real Inflated / Uruttu
90% real
10% uruttu
article Original Content
𝗚𝗟𝗠-𝟱.𝟮 — 𝘁𝗵𝗲 𝘀𝘁𝗿𝗼𝗻𝗴𝗲𝘀𝘁 𝗼𝗽𝗲𝗻-𝘄𝗲𝗶𝗴𝗵𝘁 𝗰𝗼𝗱𝗶𝗻𝗴 𝗺𝗼𝗱𝗲𝗹 — 𝗻𝗼𝘄 𝗿𝘂𝗻𝘀 𝗼𝗻 𝗮 𝗺𝗮𝗰𝗵𝗶𝗻𝗲 𝘄𝗶𝘁𝗵 𝟮𝟱𝗚𝗕 𝗼𝗳 𝗥𝗔𝗠.
744B parameters. 62.1 on SWE-bench Pro — ahead of GPT-5.5. And colibrì runs it on consumer hardware in ~2,400 lines of pure C. No BLAS. No Python. No GPU. Apache 2.0.
━━━━━━━━━━━━━
The trick:
A 744B MoE only activates ~40B params per token — and only ~11GB of those change token to token.
→ The dense part sits in RAM — 9.9GB
→ The 21,504 experts (~370GB) live on disk and stream in on demand
→ A learning cache pins whichever experts your workload actually uses — hit rate climbs from 3% to 98%
It gets faster the more you use it.
━━━━━━━━━━━━━
The honest part, straight from the author:
This is not fast. ~0.05–0.1 tok/s on his box. Best community result is ~2 tok/s.
It won't replace your coding agent tomorrow. That's not the point.
━━━━━━━━━━━━━
The point is this:
For sparse models, VRAM was never the wall. The per-token working set is tiny — the bottleneck is how fast you feed it.
Which turns "can we run a frontier coding model in-house" from a capex question into an I/O question.
For any team whose codebase legally can't touch an API — banks, defense, government — that's the whole ballgame.
Slow today. The direction is what matters.
━━━━━━━━━━━━━
verified Validated Content

This post is largely accurate, but there are a few claims that need nuance.

Fact check

"GLM-5.2 — the strongest open-weight coding model."

  • ✅ Accurate.

  • GLM-5.2 currently leads open-weight models on several major coding benchmarks, including SWE-bench Pro. (Hugging Face)

"Runs on a machine with 25GB of RAM."

  • ✅ Accurate.

  • The Colibrì project demonstrates running the 744B-parameter GLM-5.2 on consumer hardware with about 25 GB RAM by streaming model experts from disk. (GitHub)

"744B parameters."

"62.1 on SWE-bench Pro — ahead of GPT-5.5."

  • ✅ Accurate.

  • GLM-5.2 reports 62.1 on SWE-bench Pro versus 58.6 for GPT-5.5 in the published benchmark table. It still trails Claude Opus 4.8 on that benchmark. (Hugging Face)

"colibrì runs it on consumer hardware in ~2,400 lines of pure C. No BLAS. No Python. No GPU. Apache 2.0."

  • 🟡 Mostly accurate.

  • Colibrì is a tiny pure-C runtime with essentially zero runtime dependencies and CPU support. However, current versions also include optional GPU backends, so "No GPU" applies to the demonstrated setup, not the project as a whole. Also verify the exact line count because it may change over time. (GitHub)

"A 744B MoE only activates ~40B params per token."

  • ✅ Accurate.

  • This is how Mixture-of-Experts architectures work: only a subset of experts is active for each token. (Tom's Hardware)

Dense part 9.9GB, experts on disk, cache improves hit rate.

  • ✅ Accurate.

  • These implementation details match the Colibrì design. (GitHub)

"It gets faster the more you use it."

  • 🟡 Mostly accurate.

  • More precisely, repeated workloads benefit from the expert cache, increasing cache-hit rates and improving throughput. It's not universally true for every workload.

"~0.05–0.1 tok/s... best community result ~2 tok/s."

  • ✅ Accurate.

  • These figures align with the author's reported results and community experiments. (Tom's Hardware)

"VRAM was never the wall... the bottleneck is how fast you feed it."

  • 🟡 Oversimplified.

  • For this specific MoE architecture and Colibrì's design, storage bandwidth becomes the dominant bottleneck. For dense models, VRAM and memory capacity remain significant constraints.

"Banks, defense, government..."

  • ✅ Reasonable.

  • Many organizations in regulated sectors have restrictions on sending proprietary code to external APIs, making local deployment attractive.

Overall verdict

Accuracy: 9/10

The technical explanation is solid and closely matches the Colibrì project and GLM-5.2 architecture. The main caveats are:

  • "No GPU" should be interpreted as the showcased setup rather than the project's capabilities.

  • "VRAM was never the wall" is a catchy simplification that mainly applies to sparse MoE models.

Real vs Fluff

🟢 90% Real | 🟡 10% Fluff

Real (90%)

  • GLM-5.2 benchmark results.

  • Colibrì's architecture.

  • MoE parameter activation.

  • Streaming experts from disk.

  • Realistic performance numbers.

  • Enterprise implications of local inference.

Fluff (10%)

  • "VRAM was never the wall."

  • "It gets faster the more you use it."

  • "That's the whole ballgame."

  • "The direction is what matters."

Overall, this is one of the more technically grounded AI posts, with the hype mostly confined to the concluding interpretation.