Spotify's 90% Token Cut Wasn't a Prompting Trick — It Was a Routing Decision
The architecture is a model cascade, which anyone who has served ML in production will recognize on sight. Most of what a coding agent does is not reasoning, it's moving text. Reading a 2,000 line file into a frontier model's context bills at the top rate, then re-bills on every subsequent turn. So they set a threshold at 350 lines. Above it, the read gets intercepted, handed to a cheap worker at temperature 0.2, and only structured bullets return. The file never enters the expensive context, which makes follow-up questions free.
From what I have seen, every agentic workload has a high-volume, low-judgment tier that should never touch your most expensive model. The work is drawing that boundary honestly, then enforcing it somewhere the model can't talk its way past.
#GenAI #MLOps #LLMOps #InferenceCost
Spotify engineering published results showing they cut Claude Code token usage by roughly 90% on bulk file reads, using a plugin called shunt, built on top of their internal developer platform, Portal. The number itself is the least interesting part. What's notable is that they solved it as a routing problem, not a prompting problem.
The architecture is a model cascade — a pattern familiar to anyone who has run ML systems in production. Most of what a coding agent does isn't reasoning, it's moving text. Reading a 2,000-line file into a frontier model's context burns tokens at the top rate, then keeps burning on every subsequent turn.
Spotify's first attempt was pure prompting: instructions in CLAUDE.md telling Claude to delegate large reads itself. It only sort of worked — the rules were advisory, and Claude could ignore them deep into a session. The fix was enforcement: a PreToolUse hook that physically intercepts any file read over 350 lines and reroutes it to a cheap worker model (Gemini 2.5 Flash, temperature 0.2) running in an ephemeral "AiKA Mode." Only structured bullet-point output comes back — no prose, no greetings. The full file never enters the expensive model's context.
Worth flagging: this is a self-reported result from one engineer's testing across four scenarios on a single Java monorepo — not an audited benchmark. Spotify itself is explicit that the cheap-model tier is unsuitable for debugging, architectural decisions, or safety-critical reasoning — in testing, the worker model missed a subtle thread-safety bug that Claude caught immediately once given full context. There's also a real latency cost: each delegation adds 10–30 seconds, which can erase savings on small files.
The generalizable lesson: most agentic workloads likely have a high-volume, low-judgment tier that shouldn't touch your most expensive model — but the enforcement needs to sit at the tool boundary (a hook), not in the prompt, or the model will eventually talk its way past it.