How greedy decoding works
At each generation step, the model produces a probability distribution over its vocabulary. In greedy decoding, the token with the highest probability is selected unconditionally - no sampling, no randomness. The next step then conditions on that token, produces a new distribution, and again takes the maximum. This continues until the model emits a stop token or hits the output limit.
You trigger greedy decoding by setting [[temperature]] to 0, which collapses the distribution to a spike on the top token, or by setting [[top-k]] to 1, which filters to a single candidate. Both have the same effect: the random number generator is never consulted, and the same input always produces the same output - assuming identical numerical conditions on the hardware.
The repetition loop problem
Greedy decoding's determinism is its appeal and its failure mode. Because the model always takes the most probable path, it can get stuck. If a sequence of tokens leads into a pattern where repeating or paraphrasing is the most probable continuation, greedy decoding follows that path forever. Sampling would occasionally pick a less probable token and break out; greedy decoding cannot.
We have observed this directly on reasoning models: at temperature 0, a model solved a logic puzzle correctly, then entered an infinite synonym cycle - restating its conclusion in different words until it hit the token limit. At the publisher's recommended settings (temperature 1.0 with [[top-p]] and top-k), the same prompts completed normally. The loop exists as a fixed point in the model's probability distribution; sampling can escape it, greedy decoding cannot.
When to use it - and when not to
Greedy decoding is appropriate for tasks where consistency matters more than diversity and the output is short or highly constrained - classification labels, structured extraction, or JSON output via constrained decoding. For open-ended generation, reasoning, or any task where the model might enter a self-reinforcing pattern, use the publisher's recommended sampling parameters instead. On Infercom, if you omit temperature, most models default to greedy decoding - which is why our documentation warns to always set it explicitly.
Sources
Related terms
Temperature (Sampling)
The parameter controlling randomness in token selection - where 1.0 is the baseline and 0 forces greedy decoding.
Top-P (Nucleus Sampling)
A sampling method that keeps only enough high-probability tokens to cover a cumulative probability p - adapting the candidate pool to the model's confidence.
Top-K Sampling
A sampling method that limits selection to the k highest-probability tokens - a hard cap on the candidate pool.
Inference
Running a trained AI model to produce outputs - the production workload of AI, and the one whose cost and speed compound with usage.
Learn how SambaNova's dataflow architecture changes the economics of inference - and why we built on it.