Autoregressive Generation: Training vs Inference

Given a sequence of tokens, the model computes a probability distribution over every possible next token, conditioned on everything that came before it. Generation samples one token from that distribution, appends it to the sequence, and runs the whole forward pass again on the new, longer sequence. Because each step depends on the result of the previous one, generation is inherently sequential: you cannot parallelize across output tokens the way you can parallelize across a batch of separate requests.

That single structural fact is why latency scales with the number of output tokens rather than the length of the prompt, and why the model is effectively re-reading the entire conversation on every single call rather than just the newest message.

Training and inference are the same forward pass used for opposite purposes. During training, the model sees real text, predicts the next token, and the error against what the text actually says updates the model's weights; this is the only phase where anything is learned. During inference, the trained distribution is sampled to produce new text, the weights are frozen, and nothing is learned from the interaction no matter how many times you run it.

Two consequences follow in practice. The first is the KV-cache: at inference time the model caches the key and value activations for every previous token, so each new token only has to attend to itself against that cache rather than recomputing attention over the whole sequence from scratch, which is the main reason inference is fast enough to ship at all. The second is that identical prompts can produce different outputs, because sampling is stochastic by design; getting the same output twice generally requires temperature zero and a fixed seed, and even then is not guaranteed across different providers or model versions.

Token n+1 is conditioned on token n, so the two cannot be computed independently of each other, and that one fact explains most of what surprises engineers the first time they build on top of an LLM API. Latency is dominated by output length rather than prompt length, because the prompt is processed once up front and then one additional forward pass runs per output token. Streaming exists specifically because each token becomes available the moment it is produced rather than only once the entire completion is finished. And the context window is shared between the prompt and the completion, so every output token the model generates also gets added back into context and consumes budget that could otherwise go toward a longer prompt or a longer conversation history.