Stop Chasing Determinism - A Practical Guide to LLM Consistency

Why temperature=0 doesn't mean what you think, and how to actually get reliable LLM outputs.

A reasoning model we were testing got stuck. Same prompt, same parameters, temperature set to zero for "consistency." It solved the puzzle correctly, then kept going - cycling through synonyms for "boat crossing" until it hit the token limit. Every single time.

We raised temperature to 1.0 and added the publisher's recommended top_p and top_k. Thirty-six runs, no loop.

If those names mean nothing to you yet, here is the whole idea in one paragraph. A language model does not look up an answer and hand it to you. It builds the answer one word at a time, and at every single step it holds a ranked list of candidates for what comes next. Temperature, top_p and top_k are the three dials that decide how it picks from that list: whether it always takes the front-runner, or sometimes reaches a little further down. That is all they do. They exist in every LLM API, their names explain nothing, their defaults differ from one provider to the next, and most people set them by copying a value from a blog post and never touching them again.

That opening result is backwards from how most people think these dials work. See whether any of this sounds familiar:

• You set temperature: 0 so the output would be reproducible.
• You left temperature out of the request entirely, assuming the platform picks something sensible.
• Your evaluation suite passes on Monday and fails on Thursday, with nothing changed on your side.
• A model started repeating itself, so you reached for frequency_penalty.
• You have read three explanations of top_p and top_k and still could not say when to use which.

None of those instincts is unreasonable. Most of them do not do what you think, and the first one is the most reliable way there is to make a reasoning model fail.

Temperature=0, the setting developers reach for when they want consistency, is the setting most likely to break things. And the consistency it was supposed to buy does not work the way most people think either. Send the same request twice on a busy endpoint and the answer can change, for a reason that is not in your request, not in your parameters, and not something a seed can fix.

Every measurement labelled as ours is a live run against our API, and you can repeat all of them. Where a number comes from someone else's work, it says so.

TL;DR

Language models give different answers to the same question for two reasons: choosing the next word involves a deliberate element of chance, and a busy server quietly changes the arithmetic underneath. Neither is a bug. Both are manageable once you know which is which.

If you are building against an LLM API, these five lines are the whole article:

• Set temperature on every request. Leaving it out is not a neutral choice. On most models here it means greedy decoding, which is the riskiest setting available.
• Use the value the model's publisher recommends. It is usually 1.0, not 0. The per-model table is further down.
• temperature: 0 is how you get repetition loops. A reasoning model can get stuck restating the same point until it runs out of tokens. Raising max_tokens makes it worse, not better.
• frequency_penalty and presence_penalty do nothing on this API. repetition_penalty is the control that works. Send it in extra_body and keep it at or below 1.2.
• Never write a test that asserts on exact output. It holds today and breaks at the next model upgrade, and on a busy endpoint it may not hold today either. Assert on properties: valid JSON, the right schema, a number inside a range.

How token selection works

When an LLM generates text, it does not retrieve pre-written responses. For each token it outputs, it runs three steps:

1. Produce logits - raw scores for every token in the vocabulary
2. Apply softmax - convert scores to probabilities
3. Sample - pick one token based on those probabilities

Temperature, top_p, and top_k each act on a different one of those steps.

Step 1: the model scores every token it knows

The model's final layer outputs one number per token in its vocabulary - typically 100,000+ scores. These are called logits.

A logit is not a probability. It is an unbounded score: positive, negative, -15.3, +8.7, whatever. The term comes from statistics - short for "logistic unit" - where it refers to log-odds. In neural networks, it means "the raw output before the final activation function."

Here's what logits might look like for a single generation step:

A logit of +5.2 does not mean 520% chance. It means "relatively more likely than tokens with lower scores." The scores only become probabilities after the next step.

Step 2: the scores become probabilities

Raw scores are not much use as they stand. Some are negative, they add up to nothing in particular, and "+5.2" does not tell you how much more likely "the" is than "a". What the model needs is a proper probability distribution: every value positive, and the whole set summing to 1.

The function that does this is called softmax, and it works in two moves. First it raises e to the power of each score, which makes every number positive and pushes the high scores far away from the low ones. Then it divides each result by the total, which brings the set to 1.

Our four candidates come out like this:

The exponential widens the gap more than you would guess. On the raw scores, "the" leads "a" by 2.1, which does not sound decisive. After the exponential it is an eight-to-one lead. Small differences in score become large differences in probability, and that is the lever temperature pulls on.

Then there is the bottom row. "quantum" scored -4.3, it is a nonsense continuation, and it still comes out with a probability above zero. Nothing is ever fully ruled out. Every token in the vocabulary keeps a sliver of a chance, which is both why a model can occasionally say something strange and why filtering the tail is a parameter you are given.

Step 3: the model picks one

You now have probabilities. Sampling means picking one token randomly, weighted by those probabilities.

Lay the probabilities end to end on a line from 0 to 1, each token taking the width of its own share:

Draw a random number between 0 and 1 and see where it lands. 0.372 falls in "the". 0.891 falls in "a". Over many draws, "the" wins about 82% of the time, and the long tail still gets its turn.

This is where randomness enters. Same probabilities, different draw, different token.

Where temperature fits in

Temperature acts inside that exponential step. Before the scores are raised to a power, every one of them is divided by the temperature value. That single division is the entire mechanism.

Divide by something below 1 and the scores get bigger and further apart, so the front-runner pulls away and the distribution sharpens. Divide by something above 1 and they squash together, so the also-rans get a real chance. Divide by exactly 1 and nothing changes at all, which is why 1.0 is the baseline rather than the maximum.

Temperature=1.0 is not "maximum randomness." It is the baseline - the distribution the model was trained on. Think of it like contrast on a photo. T=1 is the original; anything else is an adjustment.

At T=0, the distribution collapses to a spike. One token gets ~100%, everything else gets ~0%. This is called greedy decoding - the model always picks the most probable token, no randomness involved.

Where top_p and top_k fit in

Both parameters filter candidates after softmax, before sampling:

Top-k keeps only the k highest-probability tokens:

Top-p (nucleus sampling) sorts tokens by probability and keeps the smallest set whose total reaches p:

"the" alone is 82.4%, short of 90%. Adding "a" reaches 92.5%, which crosses the threshold, so the set closes there. Everything below is discarded, and the survivors are renormalized.

Top-p adapts to confidence: if one token has 95% probability and p=0.9, only that token survives. If probabilities are spread out, more candidates remain.

An obvious question: isn't T=0 the same as top_k=1?

Functionally, yes - both result in always picking the highest-probability token. In theory they act at different stages: T=0 reshapes the distribution into a spike, top_k=1 filters to one candidate after the probabilities are computed.

In practice, dividing a logit by zero is undefined, so every implementation special-cases it. On the Infercom API the two are one mechanism, with a consequence:

top_k does nothing unless temperature is 0.001 or above. Below that threshold - including when you omit temperature entirely - your top_k value is discarded, top_k is forced to 1, and decoding is greedy. Nothing in the response says your value was dropped.

So {"top_k": 100} on its own does not widen anything. It returns argmax.

It is a cheap mistake to make: a reasonable-looking request, accepted in full, with one of its two parameters doing nothing and the other sitting at the riskiest value on the dial.

Two more things about top_k on this API. It is not part of the OpenAI dialect, so the OpenAI SDKs reject it as a top-level argument. Send it through extra_body. And send a value of 1 or greater: unlike some providers, -1 and 0 are not accepted as a way to switch it off. Omit it instead.

See it in action

Every block below is a live run against gpt-oss-120b on the Infercom API, measured on 18 September 2026. Prompt: "Write a creative one-word name for a cat. Reply with the name only." Five runs per arm, because three is not enough to tell a pinned output from a lucky one.

Temperature: consistency vs. variety

At T=0 the model returns the same name every time. At T=1.0 it explores.

The omitted-temperature trap

Now the same prompt with no temperature field at all, and with top_k: 100 - a request that asks, in plain reading, for a wide sampling pool:

Both are greedy. Both return the T=0 answer. The Infercom API applies no service-wide temperature default, so leaving it out is not "use the model's default" - on most models it is temperature 0, and it silently voids your top_k.

For completeness, top_k=1 with temperature=1.0 returns Quasar five times out of five as well. Filtering to one candidate is greedy decoding by another route.

Set temperature and top_k together, and the pool opens up:

Seed: it pins the output, within one deployment

Same prompt, temperature=1.0:

A seed holds the output. A different seed gives a different output. No seed varies. That is what a working seed looks like, and it is the tool to reach for when you want a repeatable evaluation run. It comes with three limits, which get their own section later.

Look again at what just happened. Three of those six blocks returned Quasar five times out of five, and in two of them we had asked, in plain reading, for variety. Greedy decoding is much easier to fall into than to fall out of.

Which brings us back to the model that would not stop talking about boats.

The real risk of T=0: repetition loops

Greedy decoding has a failure mode of its own, and it is worse than an inconsistent answer.

LLMs are autoregressive: each token depends on all previous tokens, including the model's own output. If the model enters a pattern that reinforces itself, it can get stuck.

At temperature > 0, sampling provides an escape route. The model might pick a slightly less probable token that leads elsewhere. At temperature = 0, there is no escape. The same path is followed every time, and if it leads into a loop, it stays there forever.

We have observed this directly with reasoning models. At temperature=0, a model entered an infinite synonym cycle:

... boat crossings. / moves. / steps. / transitions. / state changes.
... boat trips. / crossings. / voyages. / journeys.

It solved the problem correctly, then never emitted a stop token - cycling through near-synonyms until it hit the token limit. At temperature 0 it looped on every one of three runs. At the publisher's recommended settings (temperature=1.0, top_p=0.95, top_k=40), thirty-six runs completed normally.

The loop is a fixed point in the model's probability distribution. At each step, the highest-probability next token leads back into the cycle. Greedy decoding follows that path forever. With sampling, the model occasionally picks a slightly less probable token, and that is enough to break out.

Publishers recommend temperature > 0 for their reasoning models, and this is a large part of why. It is not about creativity. It is about not getting stuck.

Raising max_tokens does not fix it. A loop with more room to run is still a loop, and you pay for every token of it.

It is not a laboratory curiosity. We first hit it in a test. Since then we have watched three customers hit it in production, and every one of them arrived by the same road: they lowered temperature on purpose, for a sensible reason. One wanted repeatable benchmark numbers. One wanted a voice agent to sound predictable. The instinct that makes you reach for a low temperature is exactly the instinct this failure mode punishes.

The repetition parameter that works, and the two that don't

Say you have set a sensible temperature and some repetition survives. The reflex, for anyone arriving from the OpenAI API, is frequency_penalty or presence_penalty.

On the Infercom API, neither does anything. They are accepted for API compatibility and never applied, on any model. You get an HTTP 200, a normal-looking response, and no signal at all that the value was dropped. Here is the same prompt at temperature 0, measured on 18 September 2026:

Byte-identical, down to the token count, with a value far outside the documented range.

This is the expensive kind of surprise, because a parameter your provider ignores looks exactly like a parameter that isn't helping. Both readings fit the evidence on your screen. One sends you to the docs. The other sends you to tune it harder. A customer of ours spent a week on the second road.

The general rule behind it is not specific to us: OpenAI-compatible does not mean OpenAI-equivalent. Every provider implements a subset of that API surface, the subset is different for each one, it changes between releases, and the response body never tells you which parts you got. Before you tune any sampling parameter, on any platform, check that the platform reads it.

If your requests currently carry either parameter, remove them or set them to 0. They do nothing today, but an upcoming platform release will reject any other value with an error. Requests without them, or with 0, are unaffected. Better to take that out now than to find it in a deploy.

The control that does work here is repetition_penalty:

Different generation, same answer. Five rules for using it:

1. Send it through extra_body. It is not part of the OpenAI dialect, so the SDKs reject it as a top-level argument.
2. The accepted range is 1 to 2, and 1.0 means no penalty. Anything outside the range returns a 400, which is the behaviour you want - unlike the two penalties above, this parameter tells you when it is unhappy.
3. Stay at or below 1.2. Higher values make a reasoning model think longer before it answers. On gpt-oss-120b, a one-sentence answer used 104 completion tokens at 1.0 and 244 at 1.3, so a tight max_tokens starts truncating. At 1.8 the model returns no content at all.
4. It is not a drop-in replacement for frequency_penalty. repetition_penalty also penalises tokens that appear in your prompt, which the OpenAI penalties never do. With a long system prompt - a voice agent, say - that can suppress the exact vocabulary the prompt depends on.
5. It applies on /v1/chat/completions. /v1/responses and /v1/messages accept the value and discard it. Worth remembering if you move an integration between endpoints: the parameter did not stop working, it stopped being read.

And set temperature first. Repetition that comes from greedy decoding is a temperature problem, and no penalty is a substitute for fixing it.

Why your neighbours change your output

Set a real temperature. Add a seed. Now you have repeatable output, so the obvious next move is to write a test that asserts on the exact string.

Do not. That test can go red without a single thing changing on your side.

Most of the time, repeating a request with the same parameters does give you the same bytes back. That is what makes this failure mode awkward. It looks stable right up until it isn't, and the thing that changed is not in your code, not in your request, and not on your side at all.

The explanation everybody gives, and why it is wrong

Ask why LLM inference is not reproducible and you get a stock answer: floating-point addition is not associative, (a + b) + c does not always equal a + (b + c), GPUs run thousands of threads that finish in a different order every time, so the sums land differently and the logits wobble.

The first half is true. The second half is not.

Horace He and Thinking Machines Lab took this apart in Defeating Nondeterminism in LLM Inference (September 2025), and the finding is that individual GPU kernels are already run-to-run deterministic. Run the same matrix multiplication on the same data a thousand times and you get bitwise identical results every time. The LLM forward pass does not contain a single atomic add, which is the operation the folk explanation depends on. There is enough parallelism along the batch dimension that nobody needs to race threads along the reduction dimension.

So the kernels are not the problem. Your request is.

Batch invariance, or: who else was talking

A production endpoint does not run your request alone. It batches yours with whatever else arrived in the same window, and the batch size moves with load.

The kernels that matter here - RMSNorm, matrix multiplication, attention - are not batch-invariant. Change the batch size and the reduction is split differently, the rounding differs, and the numbers coming out for your row change, even though your input did not.

Your output depends on how many strangers were hitting the same endpoint at the same moment. That is not a knob you can set. It is not in your request. It is not something a seed can pin.

Most of the time this changes nothing, because the leading token is winning comfortably and a rounding difference in the sixth decimal place does not reach it. The trouble is the near-ties. Picture two candidates separated by a millionth of a point, plus at 3.700001 against + at 3.699999. In a batch of four, plus takes it. In a batch of thirty-two the rounding falls the other way, and + takes it by the same invisible margin.

At temperature 0 there is no recovery from that. Greedy decoding commits and follows the consequences to the end of the generation, so one flipped token can rewrite everything after it.

On Qwen3-235B at temperature 0, a thousand identical requests produced eighty different completions. The most common one appeared 78 times. The first divergence was at token 103, where 992 runs said "Queens, New York" and 8 said "New York City" - and from there the answers separated.

It is fixable, and it costs you

Thinking Machines wrote batch-invariant versions of the three kernels and published them. With those in place the same experiment returned a thousand identical completions out of a thousand. SGLang shipped deterministic inference on the same principle two weeks later.

The bill comes as latency. On their benchmark, vLLM's default path ran 1,000 sequences in 26 seconds; the batch-invariant path took 42 seconds with an improved attention kernel, and 55 seconds before that optimisation. Call it 1.6x, on kernels nobody has finished tuning.

That is the trade in one line: bit-exact output is purchasable, and the currency is throughput. No major inference API turns it on by default, because most workloads would rather have the tokens.

What this means for anything you build

The mechanism belongs to batched inference, not to any one vendor or any one kind of chip. Any engine that groups requests together and reduces across a variable batch dimension is exposed to it, and grouping requests is how every production inference stack gets its throughput. Thinking Machines demonstrated it on GPUs against vLLM because that is the stack they had in front of them, not because it is a GPU problem.

There is a second way to lose byte-identity, unrelated to load, and most teams meet this one first: two differently compiled deployments of the same model return two different answers to the same prompt, each of them perfectly stable. Same model name, same parameters, different build. A model upgrade does exactly this, on purpose, every time.

So, the rule about scope:

Which means: do not write a test that asserts on an exact string. Assert on the properties you need. Valid JSON. The right schema. A number inside a range. A claim the source supports. Those survive a busy Tuesday and a model upgrade.

What about seed?

Within one deployment, though, repeatability is available on purpose rather than by accident, and seed is how you ask for it.

It controls the random number generator in the third step, the one that picks a token. Set seed=42, and the RNG produces the same sequence of draws for the same request. Same draws applied to the same probabilities give the same tokens, which is why Nebulyn came back five times out of five.

It only works where sampling happens. At temperature 0 there is no sampling, so the RNG never runs and the seed does nothing. The identical output you get at temperature 0 is not a pinned seed. It is argmax, and argmax moves when the numbers underneath it move.

It is not honoured on every model. Some models in the catalog accept a seed and ignore it, and nothing in the response tells you which. The OpenAI compatibility page lists where it is honoured. Check it before you build an evaluation harness on top of a seed.

It fixes the draws, not the distribution. A seed pins the random numbers. It does not pin the probabilities those numbers are applied to, so it cannot protect you from a batch-size shift or a model upgrade. Use a seed for a debugging session or a review cycle, not as a long-term contract.

What consistency you CAN get

Not a string your tests can assert on for the next two years. But the consistency your application actually needs, yes.

Start with the publisher's recommended parameters. They are not arbitrary:

A dash means the publisher does not publish a value. Notice they all use T=1.0 - the baseline the model was trained on.

Always set temperature explicitly. The Infercom API applies no service-wide default. Omit it, and gpt-oss-120b, gemma-4-31B-it, DeepSeek-V3.2 and Meta-Llama-3.3-70B-Instruct all decode greedily. MiniMax-M2.7 is the exception and samples. A client that never sends temperature is running at T=0 on four of five models without knowing it, which is the highest-risk setting for the loop above.

This applies to /v1/responses and /v1/messages too. If you are porting code from the Anthropic SDK, note that the Anthropic API documents a temperature default of 1.0 and we apply none. Your ported code switches to greedy decoding unless you set it.

Use structured output for format consistency. JSON mode or function calling constrains the shape without constraining the content. If you need a specific format, this is more reliable than temperature tuning. Set max_tokens generously when your schema is large.

Use a seed for repeatable runs. A seed pins the output on the models that honour it, which makes evaluation runs and bug reports reproducible. Check the compatibility table for your model, and re-baseline after a model upgrade.

Lower temperature for narrower variance - not zero. Want more focused outputs? Try T=0.3 to 0.7. You get less variety without the failure modes of greedy decoding.

Assert on properties, not on strings. The one change that makes a test suite stop flaking.

The tradeoff

Two different things have been calling themselves variance here, and separating them is the whole lesson.

One is sampling. You asked for it, it is what lets the model try a different phrasing or climb out of a loop, and switching it off is what got a reasoning model stuck on a boat puzzle for a thousand tokens. Keep it.

The other is numerical noise from a batch you did not choose. Nobody wants that one. It is buyable - the kernels exist, they work, a thousand runs came back a thousand times identical - and the price is roughly 1.6x on throughput, which is why no major API bills you for it by default. If your workload genuinely needs bit-exact replay, that price is worth knowing about. Most workloads need something cheaper and more honest: output that is correct every time rather than identical every time.

So the question was never "how do I eliminate variance?" It is "which variance am I looking at, and is this one worth paying to remove?"

The model that got stuck on the boat puzzle had already solved it. The answer was sitting there in its reasoning, three lines above the point where it started looping. What it could not do was stop. At temperature 0, the token that ends the sentence has to beat the token that starts one more synonym on every single step, and the first time it lost that contest it had lost it for good. One notch of randomness was the entire fix.

Use the recommended parameters. Set temperature explicitly. Constrain outputs through structure and through a seed, not by forcing the sampling to zero. And before you spend a week tuning a parameter, spend a minute proving it does something. That is it.

For model-specific defaults, see Recommended sampling parameters. For which parameters each model honours, see OpenAI compatibility.

Sources

• Horace He and Thinking Machines Lab, Defeating Nondeterminism in LLM Inference, 10 September 2025 - the batch-invariance result, the Qwen3-235B experiment and the latency figures quoted above
• thinking-machines-lab/batch_invariant_ops - their batch-invariant kernels, open source
• LMSYS, Towards Deterministic Inference in SGLang, 22 September 2025