How top-k works
After [[temperature]] adjusts the probability distribution, top-k filters by rank. The k tokens with the highest probabilities are kept; the rest get zeroed out. The surviving probabilities are renormalized to sum to 1, and sampling draws from this set. k is typically set between 10 and 100 depending on the model and use case.
At the extreme, top_k=1 means only the single highest-probability token survives - which is functionally equivalent to [[greedy-decoding]]. At the other extreme, top_k set to the full vocabulary size means no filtering at all. Between those, top-k acts as a hard ceiling on how far into the probability tail the model can reach.
Why top-k is often paired with top-p
Top-k's fixed pool size is both its strength and its weakness. It guarantees the model cannot sample from extremely unlikely tokens, which prevents some forms of incoherent output. But it does not adapt: when the model is very confident (one token at 95%), k=50 still keeps 50 candidates even though 49 of them are noise. When the model is uncertain (many tokens at similar probabilities), the same k might cut off options that were genuinely plausible.
[[Top-p]] addresses this by adapting the pool size to the distribution shape. In practice, model publishers often recommend using both: top-p as the adaptive filter (keep tokens summing to 95% probability) and top-k as a safety cap (never more than 40-100 candidates). The two filters apply in sequence after temperature, giving a pipeline of: reshape, trim by mass, cap by count.
Recommended values
On Infercom's EU-sovereign models, publisher-recommended top-k values are: MiniMax-M2.7 top_k 40, gemma-4-31B-it top_k 64. gpt-oss-120b and DeepSeek-V3.2 do not specify a top-k, relying on temperature and top-p alone. The Infercom API accepts top_k from 1 to 100. If you are unsure, start with the publisher's recommendation or omit it and let top-p do the filtering.
Sources
Related terms
Temperature (Sampling)
The parameter controlling randomness in token selection - where 1.0 is the baseline and 0 forces greedy decoding.
Top-P (Nucleus Sampling)
A sampling method that keeps only enough high-probability tokens to cover a cumulative probability p - adapting the candidate pool to the model's confidence.
Greedy Decoding
The decoding strategy that always picks the highest-probability token - deterministic, but prone to repetition loops.
Inference
Running a trained AI model to produce outputs - the production workload of AI, and the one whose cost and speed compound with usage.
Learn how SambaNova's dataflow architecture changes the economics of inference - and why we built on it.