How top-p works
After [[temperature]] reshapes the probability distribution, top-p filters it. Tokens are sorted by probability; the filter keeps adding from the top until the cumulative probability reaches p. Everything below that line is removed and the remaining probabilities are renormalized. Sampling then draws from the reduced set.
For example, with top_p=0.9: if the top token has 92% probability, only that token survives - the model is confident and the nucleus is tight. If instead the top token has 40%, the next 30%, the next 15%, and so on, the filter keeps adding tokens until the cumulative crosses 90%, yielding a wider pool. This is why top-p adapts to confidence where [[top-k]] does not - the same threshold produces different pool sizes depending on how spread out the distribution is.
Top-p vs. top-k
Top-k fixes the pool size: keep exactly k tokens, regardless of how the probabilities are distributed. Top-p fixes the cumulative probability mass: keep however many tokens it takes to cover p of the total. When the model is confident, top-k may include far more candidates than necessary; when the model is uncertain, it may cut off too aggressively. Top-p handles both cases with a single parameter.
In practice, many model publishers recommend using both: top-p as the primary filter (0.9-0.95 is common) with a top-k as a hard ceiling (40-100) to catch edge cases where even the nucleus is very large. Temperature, top-p, and top-k together form a three-stage pipeline: reshape the distribution, trim by cumulative probability, then cap the count.
Recommended values
On Infercom's EU-sovereign models, publisher-recommended values are: MiniMax-M2.7 top_p 0.95, gpt-oss-120b top_p 1.0 (no filtering), gemma-4-31B-it top_p 0.95, DeepSeek-V3.2 top_p 0.95. A top_p of 1.0 means no filtering at this stage - the full distribution passes through. Lower values tighten the nucleus; values below 0.5 are unusual and may over-constrain the output. See our API documentation for the full parameter reference.
Sources
Related terms
Temperature (Sampling)
The parameter controlling randomness in token selection - where 1.0 is the baseline and 0 forces greedy decoding.
Top-K Sampling
A sampling method that limits selection to the k highest-probability tokens - a hard cap on the candidate pool.
Greedy Decoding
The decoding strategy that always picks the highest-probability token - deterministic, but prone to repetition loops.
Inference
Running a trained AI model to produce outputs - the production workload of AI, and the one whose cost and speed compound with usage.
Learn how SambaNova's dataflow architecture changes the economics of inference - and why we built on it.