How Does an LLM Choose Its Next Token? Temperature, Top-p, and Top-k Explained

An AI engineering interview question most people get wrong: How does an LLM choose its next token? The model writes one token at a time. → At each step, it scores every token in its vocabulary. For instance, GPT-4o holds 200,019 tokens. → Softmax turns those scores into probabilities: a distribution to pick from. → The next token is sampled from this distribution. This is 𝘁𝗼𝗸𝗲𝗻 𝘀𝗮𝗺𝗽𝗹𝗶𝗻𝗴. Here are three runtime settings that can change how that pick happens. 𝗧𝗲𝗺𝗽𝗲𝗿𝗮𝘁𝘂𝗿𝗲 Scales the scores before softmax. It reshapes the probability distribution. → T = 1: no scaling. The distribution stays exactly as the model produced it. → Higher T flattens it. Long-shot tokens get a chance, so answers are more varied and creative. → Lower T sharpens it. High-probability tokens dominate, so answers are more focused and predictable. As T approaches 0, sampling approaches argmax. The highest-probability token dominates. ↳ Even temperature 0 may not produce identical outputs every time. ↳ Floating-point operations, batching, and hardware differences can introduce variation. 𝗧𝗼𝗽-𝗽 (𝗻𝘂𝗰𝗹𝗲𝘂𝘀) Adds the highest-probability tokens until their cumulative probability reaches p. The pool grows or shrinks with the model's confidence. → p = 1.0: every token stays in play. → p = 0.9: only the tokens covering 90% of the probability. → p = 0.1: very restricted, close to greedy. 𝗧𝗼𝗽-𝗸 Keeps only the k highest-probability tokens. The pool size is fixed. → k = 50 and the model is confident: you still sample from 50 when only 3 matter. → k = 50 and the model is uncertain: 50 tokens may not be enough. 𝗢𝗻𝗲 𝗿𝘂𝗹𝗲 𝗼𝗳 𝘁𝗵𝘂𝗺𝗯 Temperature and top-p both act on the same distribution. → Change one at a time. → Move both and you stack two diversity controls. The effect gets harder to reason about. Temperature reshapes the curve. Top-p and top-k filter the pool. 𝗧𝗼𝗸𝗲𝗻 𝗦𝗲𝗹𝗲𝗰𝘁𝗶𝗼𝗻 One token is selected from what's left. → After filtering, the remaining probabilities are rescaled to sum to 1. → Higher-probability tokens have a higher chance of being picked. → A weighted random draw picks one token based on those probabilities. The selected token is added to the sequence. Then the process repeats. That is how an LLM chooses its next token. And because the draw can land on a different token each time, the same prompt can produce a different answer. 💾 Save this for your next AI interview ➕ Follow for more AI engineering insights ♻ Repost to help someone in your network

This is a technical explainer rather than a product/news claim, so the "facts" to check are the mechanics themselves and the one concrete number cited.

Confirmed accurate:

• The vocabulary size claim checks out precisely: OpenAI's o200k_base tokenizer, used in GPT-4o, has a vocab size of exactly 200,019 tokens. This is an oddly specific number to get right, and the post nails it exactly rather than rounding to "~200k." arxiv
• The core sampling pipeline described — score every token, apply softmax to get a probability distribution, then sample from it — is the standard, correct description of autoregressive next-token generation. Nothing to dispute here; this is well-established mechanics, not a claim needing external verification.
• Temperature: scaling logits before softmax, T=1 as no-op, higher T flattening the distribution, lower T sharpening it, and T→0 approaching argmax are all standard, correct descriptions of temperature scaling as used in virtually every LLM API (OpenAI, Anthropic, open-source inference stacks).
• The note that temperature 0 doesn't guarantee identical outputs is a real, well-documented phenomenon — floating-point non-associativity, GPU batching effects, and kernel non-determinism are commonly cited explanations for why "deterministic" temperature-0 sampling can still vary run to run in production systems.
• Top-p (nucleus sampling) and top-k are both described correctly: top-p as a cumulative-probability cutoff with a variable pool size, top-k as a fixed-size pool of the k highest-probability tokens. The k=50 examples (too many when the model is confident, potentially too few when it's uncertain) illustrate a real, commonly discussed limitation of fixed-k sampling versus the adaptive pool size of top-p.
• The advice to avoid changing temperature and top-p simultaneously mirrors guidance that appears in API documentation from major LLM providers, which generally recommend adjusting one or the other rather than both at once, since their combined effect is harder to predict.

No fabricated numbers, no incorrect mechanics, and the one specific figure included is exactly right — a good sign this was written by someone who checked the number rather than approximating it.