What temperature actually does
When an LLM generates a token, it first produces a raw score (logit) for every token in its vocabulary, then converts those scores into probabilities via the softmax function. Temperature is applied inside this conversion: each logit is divided by T before softmax. This rescales the probability distribution - T less than 1 sharpens the peaks, T greater than 1 flattens it.
A common misconception: temperature 1.0 is not 'maximum randomness.' It is the baseline - the distribution the model was trained on. Think of it like contrast on a photograph: T=1 is the original; anything else is an adjustment. Model publishers typically recommend T=1.0 for their models, paired with [[top-p]] and [[top-k]] to cut off the long tail of unlikely tokens.
Why T=0 is risky for reasoning models
At temperature 0, there is no sampling - the model deterministically picks the highest-probability token every time. This sounds like consistency, but it introduces a failure mode: if the model enters a self-reinforcing pattern, it cannot escape. Sampling provides an exit route; greedy decoding does not. We have observed reasoning models at T=0 enter infinite synonym cycles - correctly solving a problem, then cycling through rephrased conclusions until they hit the token limit.
This is why our documentation warns: always set temperature explicitly, and use the value the model publisher recommends. Omitting it causes most models to default to greedy decoding (T=0), which risks non-terminating loops on reasoning workloads. The Infercom API has no service-wide default precisely to make this choice visible.
Practical guidance
For EU-sovereign models on Infercom, use the publisher's recommended values: MiniMax-M2.7 specifies temperature 1.0, top_p 0.95, top_k 40; gpt-oss-120b specifies temperature 1.0, top_p 1.0. If you want more focused output, lower temperature to 0.3-0.7 rather than zero - you narrow the variance without triggering greedy decoding's failure modes. And remember: even at T=0, byte-identical outputs are not guaranteed across distributed [[inference]] - floating-point variance in parallel computation can flip close calls.
Sources
Related terms
Top-P (Nucleus Sampling)
A sampling method that keeps only enough high-probability tokens to cover a cumulative probability p - adapting the candidate pool to the model's confidence.
Top-K Sampling
A sampling method that limits selection to the k highest-probability tokens - a hard cap on the candidate pool.
Greedy Decoding
The decoding strategy that always picks the highest-probability token - deterministic, but prone to repetition loops.
Inference
Running a trained AI model to produce outputs - the production workload of AI, and the one whose cost and speed compound with usage.
Learn how SambaNova's dataflow architecture changes the economics of inference - and why we built on it.