Pay-Per-Token vs. Serverless GPU: The Real Economics of LLM Inference

Choosing an LLM provider isn't just about choosing the best model. You're also choosing: • What you pay for • Who serves it • How it scales • How much infrastructure you control I noticed this while adding 3 different providers to Decode (the coding agent I'm building from scratch). Specifically, I picked providers that represent 3 different ways of running inference: 𝟭/ 𝗚𝗲𝗺𝗶𝗻𝗶 → 𝗕𝘂𝘆 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹 This is the traditional proprietary API approach. Google owns the model and infrastructure. You send requests and pay for usage. I added Gemini because its generous free tier makes it an easy way to experiment with Decode without paying anything. 𝟮/ 𝗢𝗽𝗲𝗻𝗥𝗼𝘂𝘁𝗲𝗿 → 𝗕𝘂𝘆 𝘁𝗵𝗲 𝘀𝗲𝗿𝘃𝗶𝗻𝗴 OpenRouter gives you one interface to a huge pool of models. So you can experiment with different models without worrying about the underlying infrastructure. 3/ Modal → 𝗦𝗲𝗿𝘃𝗲 𝗼𝗽𝗲𝗻-𝘄𝗲𝗶𝗴𝗵𝘁 𝗺𝗼𝗱𝗲𝗹𝘀 𝘆𝗼𝘂𝗿𝘀𝗲𝗹𝗳 This gives you more control over the inference layer. With Modal Endpoints, I can serve Qwen3.6 35B, GLM-5.2, or even swap in my own fine-tuned model. Without changing anything in the agent harness. But the most interesting difference is the economics... Instead of paying per token, you can pay for GPU compute while the workload runs. This is attractive for large, bursty workloads. For example, imagine processing 1,000 documents containing roughly 30M input tokens. Using Sonnet, my napkin math puts that at roughly $97. Process the same workload batched at ~3,000 tokens/sec on a single H200 through Modal, and you're looking at under 3 hours. At $4.54/hour, that's roughly $13 of GPU compute. Of course, serverless GPU inference isn't automatically cheaper. If Decode asks me for confirmation and I leave an H200 sitting idle overnight, I could burn another ~$45 doing absolutely nothing 😂 That's the point… There isn't one "best" way to run inference. • Interactive workloads might favor pay-per-token APIs. • Large asynchronous workloads might favor serverless GPUs. • Fine-tuned models might push you toward serving open weights yourself. Which is why the most important architectural decision I made was keeping all of this outside the agent loop. The loop doesn't know whether it's talking to Gemini, OpenRouter, Modal, or whatever provider I add next. It simply requests the next completion. Agent loop → Model interface → Provider Keep that boundary clean and you can change the infrastructure underneath as your workload changes. I break down exactly how I implemented this inside Decode in Lesson 2 of my open-source Building a Coding Agent From Scratch series. Read it here: https://lnkd.in/d-3NGttv

Choosing an LLM provider involves more than picking the best model — it also determines cost structure, infrastructure control, and scalability. While building a coding agent called Decode, three different inference approaches were integrated: Gemini, representing the traditional proprietary API model where the provider owns both the model and infrastructure and offers a generous free tier for experimentation; OpenRouter, which provides a unified interface across many models without requiring management of underlying infrastructure; and Modal, which allows self-serving open-weight models with full control over the inference layer, including the ability to swap in custom fine-tuned models without changing the agent's core logic. The key economic difference is that Modal-based serving charges for GPU compute time rather than per token, which can favor large, bursty batch workloads — illustrated by an estimate that processing roughly 30 million input tokens across 1,000 documents might cost approximately $97 via Sonnet's per-token pricing, versus roughly $13 in GPU compute using a single H200 on Modal at its base rate of $4.54/hour for about three hours of processing. However, serverless GPU inference isn't inherently cheaper, since idle GPU time — such as waiting on a confirmation prompt overnight — still accrues cost. The architectural takeaway is that keeping the inference provider decoupled from the agent loop, so the loop simply requests completions without knowing which provider is being used, allows the underlying infrastructure to be changed as workload needs evolve.