Decoding parameters: temperature, top-p, top-k

• Token — the unit a model reads and writes: a word, word fragment or punctuation chunk (see the tokenization pages for how text is split).
• Vocabulary (V) — the fixed list of all tokens the model can emit; typically tens of thousands to a few hundred thousand entries.
• Logit (z) — the raw, unnormalised score the model’s final layer gives each vocabulary token for “what comes next”. Can be negative or greater than 1; larger means more likely. One logit vector has one entry per token, so shape (|V|,).
• Softmax — the function that turns a logit vector into probabilities that are positive and sum to 1.
• Probability distribution (p) — the vector of |V| probabilities over the next token after softmax.
• Sampling — drawing one token at random, with chance proportional to its probability.
• Greedy decoding — always taking the single most probable token (argmax); no randomness.
• Argmax — the index of the largest value in a vector.
• Temperature (T) — a positive number that divides the logits before softmax; controls how peaked or flat the distribution is.
• Top-k — truncation rule: keep only the k most probable tokens.
• Top-p / nucleus sampling — truncation rule: keep the smallest set of top tokens whose probabilities add up to at least p.
• Min-p — truncation rule: keep tokens whose probability is at least min_p times the top token’s probability.
• Renormalisation — dividing the surviving probabilities by their sum so they total 1 again.
• Frequency / presence / repetition penalty — logit adjustments that discourage tokens already generated.
• logit_bias — a per-token additive logit offset supplied by the caller (for example to ban a token).
• Multinomial draw — the actual random pick from a discrete distribution.
• Seed — the initial value of the pseudo-random generator; the same seed gives the same random numbers.
• Non-associativity (floating point) — (a + b) + c can differ from a + (b + c) in the last bits when numbers are rounded, so summation order matters.
• Kernel (GPU kernel) — a small program that runs on the GPU, e.g. one matrix multiply; the library picks which kernel (and so which addition order) to use.
• All-reduce — the operation that sums partial results held by several GPUs so each ends with the total.
• Float32 / bf16 / fp16 — 32- or 16-bit number formats; fewer bits means coarser rounding (see Quantization and inference efficiency).
• Tensor parallelism — splitting each weight matrix across several GPUs; partial results are summed across GPUs.
• MoE (Mixture of Experts) — a model where each token is routed to a few expert sub-networks; see MoE and architecture families.

What it is and why it exists

A language model never outputs “a word”. For every position it outputs one score per vocabulary token (the logits). Turning that vector into a single chosen token is a separate, configurable step called decoding. Every API parameter in this page (temperature, top_p, top_k, min_p, penalties, seed) is a knob on that step. The model weights are untouched; only the choice rule changes. This is why the same model can be a precise extractor at one setting and a brainstorming partner at another.

The pipeline, step by step

logits z  (shape |V|)
   |
   v  1. penalties:   z_i <- z_i - alpha*count_i - beta*[count_i>0]   (+ logit_bias)
   v  2. temperature: z_i <- z_i / T
   v  3. softmax:     p_i = exp(z_i) / sum_j exp(z_j)
   v  4. truncation:  top_k and/or top_p and/or min_p  -> zero out the tail
   v  5. renormalise: p'_i = p_i / sum of survivors
   v  6. draw one token  (multinomial)   [or argmax if greedy]
   |
   v  token appended to the text; the whole model runs again for the next position

Different libraries order steps 1-4 slightly differently (some apply top-p before temperature, for example), so treat the order above as the common convention and check your serving stack’s docs when exact behaviour matters. What is universal: the only random step is the last one; everything before it is deterministic arithmetic on the logits.

Running example used on this page

Five candidate tokens for the continuation of “The capital of France is”. The logits are illustrative numbers chosen for clarity, not taken from a real model:

index	token	logit z
0	Paris	3.0
1	Lyon	2.0
2	the	1.0
3	a	0.0
4	banana	-1.0

Softmax at T=1 gives [0.6364, 0.2341, 0.0861, 0.0317, 0.0117]. Every later section transforms this exact vector, so you can compare the effect of each knob.

Where the parameters come from

All of these are chosen by you at request time (or defaulted by the provider). None is learned. The model was trained to be sensible at T = 1 with no truncation, which is why that is the usual default. Some APIs expose only a subset (many hosted APIs have temperature and top_p, but not min_p; local runtimes such as llama.cpp or vLLM expose more).

The equation in words

Temperature divides every logit by the same positive number T before softmax. Softmax then turns the result into probabilities. Because softmax only cares about differences between logits, dividing by T stretches the gaps (T < 1) or squeezes them (T > 1).

p_i = exp(z_i / T) / sum_j exp(z_j / T)

Every symbol

symbol	meaning	shape / range	where it comes from
z_i	logit of vocabulary token i	real number, one per token	computed by the model’s final linear layer from the context
T	temperature	T > 0 (0 is special-cased as greedy)	chosen by the caller
exp	natural exponential e^x; makes every term positive and turns additive gaps into multiplicative ratios		fixed maths
sum_j	sum over all `	V	` tokens
p_i	probability of token i	0 < p_i < 1, all sum to 1	output

Why it works: p_i / p_j = exp((z_i - z_j) / T). The odds between two tokens are the exponential of their logit gap divided by T. Halve T and every gap doubles, so the exponent doubles, so odds are squared.

Worked example: four tokens

Logits z = [4, 3, 1, 0] (computed with Python):

T	p (4 tokens)	reading
0.5	[0.8786, 0.1189, 0.0022, 0.0003]	sharp: top token ~88%
1	[0.6964, 0.2562, 0.0347, 0.0128]	the model’s native distribution
2	[0.5089, 0.3087, 0.1136, 0.0689]	flatter: tail tokens gain mass
100	[0.2550, 0.2525, 0.2475, 0.2450]	almost uniform
0.01	[1.0, 0.0, 0.0, 0.0]	effectively one-hot

Check one cell by hand at T=1: exp(4)=54.60, exp(3)=20.09, exp(1)=2.718, exp(0)=1, sum =78.40, so p_0 = 54.60/78.40 = 0.6964.

Limiting behaviour

• T -> 0+: the largest logit dominates completely; p becomes one-hot on argmax(z). This is greedy decoding. Implementations special-case T = 0 to avoid dividing by zero.
• T = 1: the trained distribution, unchanged.
• 0 < T < 1: sharpens; high-probability tokens gain, the tail is suppressed.
• T > 1: flattens; more diverse, more risk of incoherence.
• T -> infinity: z_i / T -> 0 for all i, so p approaches uniform over the whole vocabulary: noise.

Code and gotchas

import math

def softmax(z, T=1.0):
    m = max(z)                                  # subtract max: avoids overflow in exp
    e = [math.exp((x - m) / T) for x in z]
    s = sum(e)
    return [x / s for x in e]

z = [4.0, 3.0, 1.0, 0.0]
for T in (0.5, 1, 2):
    print(T, [round(p, 4) for p in softmax(z, T)])
# 0.5 [0.8786, 0.1189, 0.0022, 0.0003]
# 1   [0.6964, 0.2562, 0.0347, 0.0128]
# 2   [0.5089, 0.3087, 0.1136, 0.0689]

Subtracting max(z) does not change the result (it cancels in the ratio) but prevents exp from overflowing when logits are large. Gotchas: temperature changes calibration of randomness, not knowledge; a model that is wrong at T=0 is not fixed by a different T. And T is not a universal unit: the same value feels different on models whose logits have different spread.

Why truncate at all

Even at T = 1, a vocabulary of 100,000 tokens has a long tail of nonsense tokens, each with a tiny probability (say 0.00001). Together the tail can hold a few percent of the mass, so over a 500-token answer you will sample some tail token surprisingly often. Truncation removes the tail before the draw.

Top-k

Keep only the k tokens with the highest probability, zero the rest, renormalise.

V_k  = the k indices with largest p_i
p'_i = p_i / sum_{j in V_k} p_j    for i in V_k,   else 0

Symbols: k is an integer chosen by the caller (for example 40); V_k is the kept set; p_i the pre-truncation probability; the denominator is the total mass of the kept set, which rescales survivors to sum to 1.

Worked example (running vocabulary, k = 2): keep Paris 0.6364 and Lyon 0.2341, sum 0.8705. New probabilities: 0.6364/0.8705 = 0.7311, 0.2341/0.8705 = 0.2689.

Top-p (nucleus)

Sort tokens by probability descending, walk down accumulating mass, and stop at the first prefix whose cumulative probability reaches p. Keep exactly that set, renormalise.

sort so that p_(1) >= p_(2) >= ...
V_p = { (1), ..., (m) }   where   m = min { m' : sum_{i=1..m'} p_(i) >= p }

Symbols: p (the parameter top_p, in (0, 1]) is chosen by the caller and is not the same thing as p_i, the token probabilities; (i) means “the i-th most probable token”; m is the resulting nucleus size, which varies per step.

Worked example (running vocabulary, top_p = 0.9): cumulative sums are 0.6364, 0.8705, 0.9567, 0.9883, 1.0. The first prefix at or above 0.9 is 3 tokens (Paris, Lyon, the; mass 0.9567). Renormalised: [0.6652, 0.2447, 0.0900, 0, 0]. “a” and “banana” can no longer be drawn.

Why top-p usually wins

The number of reasonable continuations differs enormously by context.

• After “The capital of France is” the distribution is nearly one-hot. top_k = 40 admits 39 junk tokens, though they are low-weighted.
• In the middle of open prose hundreds of tokens are plausible; top_k = 40 cuts good ones.

Top-p adapts: it keeps 1-2 tokens when the model is confident and many when it is not, because it thresholds on cumulative mass rather than a fixed count. They can be combined (apply both, keep the intersection). Use top-k when you want a hard cap on work or when your runtime only offers it.

Code

def top_k_filter(p, k):
    keep = set(sorted(range(len(p)), key=lambda i: -p[i])[:k])
    return [x if i in keep else 0.0 for i, x in enumerate(p)]

def top_p_filter(p, top_p):
    order = sorted(range(len(p)), key=lambda i: -p[i])
    keep, cum = set(), 0.0
    for i in order:
        keep.add(i)              # add first, then test: the token that crosses the line stays
        cum += p[i]
        if cum >= top_p:
            break
    return [x if i in keep else 0.0 for i, x in enumerate(p)]

def renorm(p):
    s = sum(p)
    return [x / s for x in p]

Walk-through: top_p_filter sorts indices by probability, adds tokens to the keep set one by one, and stops after the token that makes the running total reach top_p. Adding before testing matters: otherwise a very confident model (top token already above top_p) would keep zero tokens.

The definition

Keep every token whose probability is at least min_p times the probability of the single most likely token:

V = { i : p_i >= min_p * max_j p_j }

Symbols: min_p is a chosen fraction (commonly 0.05-0.1); max_j p_j is the probability of the top token at this step; so the threshold slides with the model’s confidence. Then renormalise as usual.

Worked example

Running vocabulary at T = 1, min_p = 0.1: threshold = 0.1 * 0.6364 = 0.0636. Tokens with p >= 0.0636: Paris 0.6364, Lyon 0.2341, the 0.0861. “a” (0.0317) and “banana” (0.0117) are dropped. Same three survivors as top_p = 0.9 here, but the logic differs.

Now the case min-p was designed for, high temperature. At T = 2 the distribution becomes [0.4287, 0.2600, 0.1577, 0.0956, 0.0580].

• top_p = 0.9: cumulative 0.4287, 0.6886, 0.8463, 0.9420 so 4 tokens survive, including a 0.0956 token.
• min_p = 0.15: threshold 0.15 * 0.4287 = 0.0643, so 4 tokens survive (0.0580 drops).

In this tiny vocabulary they look similar. With a real 100k vocabulary the difference appears: flattening spreads mass over thousands of weak tokens, so the cumulative mass needed for 0.9 reaches deep into the tail, while the min-p threshold stays pinned to the peak and keeps cutting those weak tokens.

Why it exists

Top-p’s cumulative threshold behaves badly when T flattens the distribution: no individual tail token has much mass, but together they cross the line. Min-p is relative to the peak: if the top token is at 0.4, min_p = 0.1 admits only tokens above 0.04. That scales with confidence and is more robust to changing temperature, so you can raise temperature for diversity without the incoherence top-p would let through.

Code and caveat

def min_p_filter(p, min_p):
    thr = min_p * max(p)
    return [x if x >= thr else 0.0 for x in p]

The top token always survives (it is at least min_p times itself for any min_p <= 1), so the filter can never leave an empty set. Caveat: min-p is not offered by every hosted API; where it is not, use a moderate top-p instead.

This puts the pipeline together so you can see each knob act. It uses only the standard library.

import math, random
from collections import Counter

def softmax(z, T=1.0):
    m = max(z)
    e = [math.exp((x - m) / T) for x in z]
    s = sum(e)
    return [x / s for x in e]

def apply_penalties(z, counts, freq=0.0, pres=0.0):
    return [x - freq * c - pres * (c > 0) for x, c in zip(z, counts)]

def top_k_filter(p, k):
    keep = set(sorted(range(len(p)), key=lambda i: -p[i])[:k])
    return [x if i in keep else 0.0 for i, x in enumerate(p)]

def top_p_filter(p, top_p):
    keep, cum = set(), 0.0
    for i in sorted(range(len(p)), key=lambda i: -p[i]):
        keep.add(i); cum += p[i]
        if cum >= top_p:
            break
    return [x if i in keep else 0.0 for i, x in enumerate(p)]

def min_p_filter(p, min_p):
    thr = min_p * max(p)
    return [x if x >= thr else 0.0 for x in p]

def renorm(p):
    s = sum(p)
    return [x / s for x in p]

def sample(z, counts=None, T=1.0, top_k=None, top_p=None, min_p=None,
           freq=0.0, pres=0.0, rng=random):
    counts = counts or [0] * len(z)
    z = apply_penalties(z, counts, freq, pres)
    if T == 0:                                    # greedy
        return max(range(len(z)), key=lambda i: z[i])
    p = softmax(z, T)
    if top_k: p = top_k_filter(p, top_k)
    if top_p: p = top_p_filter(p, top_p)
    if min_p: p = min_p_filter(p, min_p)
    p = renorm(p)
    return rng.choices(range(len(p)), weights=p)[0]

vocab = ["Paris", "Lyon", "the", "a", "banana"]
z = [3.0, 2.0, 1.0, 0.0, -1.0]
rng = random.Random(0)
for kw in [dict(T=1.0), dict(T=1.0, top_p=0.9), dict(T=2.0, min_p=0.15), dict(T=0)]:
    c = Counter(vocab[sample(z, rng=rng, **kw)] for _ in range(10000))
    print(kw, dict(c.most_common()))
# {'T': 1.0}                 {'Paris': 6351, 'Lyon': 2395, 'the': 855, 'a': 290, 'banana': 109}
# {'T': 1.0, 'top_p': 0.9}   {'Paris': 6633, 'Lyon': 2439, 'the': 928}
# {'T': 2.0, 'min_p': 0.15}  {'Paris': 4547, 'Lyon': 2758, 'the': 1661, 'a': 1034}
# {'T': 0}                   {'Paris': 10000}

Walk-through of the output: with plain T=1, all five tokens appear, in roughly the probabilities we computed (0.6364 of 10,000 is about 6,364; got 6,351). Adding top_p=0.9 removes “a” and “banana” entirely and shifts their mass to the survivors. At T=2 with min_p=0.15 the distribution is flatter (“a” appears 10%), but “banana” is still cut. T=0 is deterministic. Exact counts depend on the seed; the proportions are what matter.

Note that this is one step. A real generator calls this once per output token, feeding the chosen token back in as context.

What problem they solve

Models, especially at low temperature, can fall into loops (“the the the” or repeating a sentence) because a token that was just emitted makes itself more likely again. Penalties push the logits of already-generated tokens down before softmax.

The three variants

frequency:  z_i <- z_i - alpha * count_i
presence:   z_i <- z_i - beta  * [count_i > 0]
repetition: z_i <- z_i / gamma   (if z_i > 0 and token seen)
            z_i <- z_i * gamma   (if z_i < 0 and token seen)     (HF-style)

symbol	meaning	chosen by
count_i	how many times token i already appears in the generated (some APIs: prompt + generated) text	measured from the sequence
alpha	frequency penalty strength, >= 0	caller
beta	presence penalty strength, >= 0	caller
gamma	repetition penalty, > 1 discourages	caller
[count_i > 0]	1 if the token has appeared at all, else 0	measured

• Frequency scales with how many times the token has appeared: heavy repetition is punished heavily.
• Presence is a one-off flat fee for having appeared at all, nudging the model towards new topics.
• Repetition (older, multiplicative style) divides positive logits and multiplies negative ones so that the logit always moves down. A naive z / gamma would push a negative logit up towards zero, which is why implementations branch on sign.

Worked example

Logits z = [3, 2, 1, 0] for tokens A, B, C, D. Counts so far: A seen 3 times, B once, C and D never.

• Frequency, alpha = 0.5: [3-1.5, 2-0.5, 1, 0] = [1.5, 1.5, 1.0, 0.0]. A has fallen from first to a tie. Softmax of that is [0.3534, 0.3534, 0.2143, 0.0789], versus about [0.64, 0.24, 0.09, 0.03] before.
• Presence, beta = 0.6: [2.4, 1.4, 1.0, 0.0]. A and B both lose 0.6 regardless of count, so A’s lead over B is unchanged (1.0), while both fall relative to unseen C and D.
• Repetition, gamma = 1.3: [3/1.3, 2/1.3, 1, 0] = [2.31, 1.54, 1.0, 0.0]. Note this is relative: a logit of 3 loses 0.69 while a logit of 2 loses 0.46. A seen token with logit -2 would go to -2 * 1.3 = -2.6.

Code

def apply_penalties(z, counts, freq=0.0, pres=0.0):
    return [x - freq * c - pres * (c > 0) for x, c in zip(z, counts)]

print(apply_penalties([3, 2, 1, 0], [3, 1, 0, 0], freq=0.5))   # [1.5, 1.5, 1.0, 0.0]
print(apply_penalties([3, 2, 1, 0], [3, 1, 0, 0], pres=0.6))   # [2.4, 1.4, 1.0, 0.0]

Gotchas

• Penalties are blunt: they also punish necessary repeats (articles, a variable name in code, the correct answer, JSON keys). Small values (roughly 0 to 0.5 for the additive ones) are usually right. For code and structured output prefer none.
• Counting scope differs between implementations (generated only vs prompt + generated); read your provider’s definition.
• logit_bias is the manual sibling: you add a fixed number to a chosen token’s logit, with a large negative value effectively banning it. It acts at step 1 of the pipeline.
• For hard structural rules (never emit anything outside a schema) use masking, not penalties; see Structured output and constrained decoding.

The common belief is “I set temperature = 0, so the same prompt always returns the same text.” It feels true because temperature controls randomness, and zero randomness sounds like determinism; in a single-threaded program with a fixed seed it would be right. But temperature = 0 (greedy decoding) removes only the sampling randomness (the last step of the pipeline becomes an argmax); it does not guarantee bit-identical output. The logits themselves can differ slightly between two runs of the same prompt, and an argmax over near-tied logits is extremely sensitive to tiny differences. This concept builds the argument from the bottom: how computers add numbers, why order matters, how that reaches the logits, and what you can and cannot do about it.

Step 1 - floating-point addition is not associative

Computers store real numbers with a fixed number of bits (see Quantization and inference efficiency for the bit layouts). Every addition rounds the exact result to the nearest representable number. Rounding makes grouping matter: (a + b) + c and a + (b + c) can differ.

import numpy as np

print(0.1 + 0.2)                      # 0.30000000000000004  (0.3 is not exactly representable)
print((0.1 + 0.2) + 0.3)              # 0.6000000000000001
print(0.1 + (0.2 + 0.3))              # 0.6                  same three numbers, different grouping
print((0.1 + 0.2) + 0.3 == 0.1 + (0.2 + 0.3))   # False

a, b = np.float32(1e8), np.float32(1.0)
print((a + b) - a)                    # 0.0   the 1.0 was absorbed: float32 spacing near 1e8 is 8
print((a - a) + b)                    # 1.0   same numbers, different order, different answer

import random
r = random.Random(1)
xs = [r.uniform(-1, 1) * 10 ** r.randint(0, 8) for _ in range(10000)]
def acc(v):
    s = 0.0
    for x in v:
        s += x
    return s
print(acc(xs) == acc(reversed(xs)), acc(xs) - acc(reversed(xs)))
# False -8.58306884765625e-06    the same 10,000 numbers summed in reverse order give a different total

Walk-through: the first block uses 64-bit doubles. The result of 0.1 + 0.2 is rounded, and the second rounding happens at a different magnitude depending on grouping. The second block shows the extreme case: near 100,000,000 the gap between adjacent float32 values is 8, so adding 1.0 changes nothing (“absorption”). Subtracting first keeps the 1.0. The third block scales this up: summing the same 10,000 numbers of mixed magnitudes forwards and backwards gives totals that differ in the last bits. Neither answer is “wrong”; they are different roundings of the same exact sum.

Step 2 - where sums happen inside a model

Every logit is a long sum. The final layer computes z_i = h . u_i, a dot product of the final hidden state h (length d_model, e.g. 4096) with token i’s unembedding row u_i: 4096 multiplications and a sum of 4096 numbers. The same pattern appears in every matrix multiply, in the softmax denominator inside attention, in normalisation (mean(x^2)), and in the all-reduce that adds partial results from several GPUs.

A GPU does not add 4096 numbers left to right. It splits them across many threads, each adds its share, and the partial sums are combined in a tree. The split depends on the kernel the library chooses, which depends on matrix shapes, the batch size, the GPU model, and the library version. A different split is a different grouping, hence a (slightly) different rounded result.

import numpy as np

rng = np.random.default_rng(7)
d = 4096
h = rng.standard_normal(d).astype(np.float32)      # a pretend final hidden state
wa = rng.standard_normal(d).astype(np.float32)     # unembedding row of token A
wb = wa.copy()
wb[0] -= np.float32(0.000003) / h[0]               # token B is truly smaller than A by 3e-6

def logit(w, h, chunks):
    """Dot product in float32, accumulated as `chunks` interleaved partial sums
    (a stand-in for how different GPU kernels split the work)."""
    prod = w * h
    parts = [np.float32(0)] * chunks
    for i, p in enumerate(prod):
        parts[i % chunks] = np.float32(parts[i % chunks] + p)
    total = np.float32(0)
    for p in parts:
        total = np.float32(total + p)
    return total

for chunks in (1, 2, 4, 8, 16, 64):
    a, b = logit(wa, h, chunks), logit(wb, h, chunks)
    print(chunks, a, b, "winner", "A" if a > b else "B")
# 1   -30.902002 -30.902002 winner B     (tie, so the "A > B" test fails)
# 2   -30.902128 -30.902128 winner B
# 4   -30.902092 -30.902092 winner B
# 8   -30.902119 -30.902122 winner A
# 16  -30.902075 -30.902082 winner A
# 64  -30.9021   -30.9021   winner B

Walk-through: A’s exact logit is larger than B’s by 3e-6, a gap smaller than the float32 spacing near 31 (about 2e-6 per step, so the gap is only one or two representable steps). Each kernel layout (chunks) lands on a slightly different rounded total, so sometimes the two logits tie, sometimes A wins, and the “winner” changes with the layout while the inputs are identical. Real logit noise from this effect is larger in bf16/fp16 models (bf16 keeps only about 3 significant decimal digits). Whenever the top-two gap is smaller than that noise, the argmax can flip.

Step 3 - why one flipped token changes everything

Generation is autoregressive: the token chosen at step t becomes input for step t + 1. After a flip, the two runs condition on different histories, and the divergence compounds:

run 1: ... "the answer is" -> " clear"   -> ", because ..."
run 2: ... "the answer is" -> " obvious" -> ", since ..."       (a 1e-4 logit difference decided this)

Most positions have a clear winner (top-two gap far above the noise), so most tokens agree; but over a 500-token answer there are many chances for a near-tie, so full-text identity across runs becomes unlikely even though each individual token is almost always stable. A rough model: if each token independently has probability q of being unaffected, a 500-token answer is identical with probability q^500. With q = 0.999 that is about 61%; with q = 0.99 about 0.7% (illustrative values; computed as 0.999**500 and 0.99**500).

Causes at the serving layer

cause	why the numerics change
Batching	the kernel and its reduction order can depend on how many requests share the batch; your result can depend on other users’ concurrent traffic
Padding / sequence packing	different padded shapes select different kernels
Tensor parallelism	the order in which GPUs’ partial sums are added in the all-reduce is not always fixed
Hardware / driver / library upgrades	new kernels, new reduction trees
MoE routing	tiny hidden-state differences can send a token to a different expert (see Mixture of experts and architecture families)
Different quantised or speculative-decoding paths	different arithmetic for the same model
Provider-side model updates	a “same name” model alias can point to new weights

The seed parameter (where offered) fixes the pseudo-random generator for temperature > 0, but it rides on the same numeric substrate: same seed plus different logits can still give different tokens. It is best effort, not a guarantee.

What to do

• Pin the exact model version (not a floating alias), keep the prompt byte-identical, set temperature = 0 and a seed if offered.
• Evaluate on agreement rate: run each case several times and measure how often outputs match. Treat 100% as a bonus, not an assumption.
• Design downstream code so equivalent answers are accepted: compare parsed fields, not raw strings.
• Locally you can get stronger determinism (batch size 1, deterministic kernel flags such as torch.use_deterministic_algorithms(True), the same GPU model), at a speed cost. Some inference engines offer batch-invariant kernels that make results independent of batch composition; check whether your stack has one and what it costs.
• If you need strict reproducibility (a replayable audit trail, a regression snapshot), cache the output keyed by the exact prompt and parameters and serve that, rather than regenerating and hoping for the same text.
• Reproducibility is a property of the whole serving stack, not of a decoding flag.

They compose but are partly redundant, and setting both aggressively compounds. Temperature reshapes the whole distribution’s sharpness; top-p then truncates the tail of whatever temperature produced.

Worked interaction

Running vocabulary with top_p = 0.9:

T	distribution	nucleus kept
0.5	[0.8647, 0.1170, 0.0158, 0.0021, 0.0003]	2 tokens (cumulative 0.8647, then 0.9817)
1	[0.6364, 0.2341, 0.0861, 0.0317, 0.0117]	3 tokens
2	[0.4287, 0.2600, 0.1577, 0.0956, 0.0580]	4 tokens

Raising temperature widens the nucleus; a tight top_p then claws some of the diversity back. Lowering temperature first makes top_p nearly irrelevant, because the distribution is already peaked.

Practical guidance

• Change one at a time. Tune temperature and leave top_p at a safe default (about 0.9-1.0), or the reverse. Some providers recommend altering only one of them.
• Factual, extraction, code: low temperature (0 to 0.3), or greedy. Top-p barely matters.
• Creative, brainstorming: temperature roughly 0.7-1.0, and prefer min-p or a moderate top-p to catch incoherent tail tokens.
• Avoid T > 1.2 with top_p = 1.0: that is where word salad appears.
• Defaults (often T = 1.0, top_p = 1.0, meaning the native distribution) are a sound starting point because the model was aligned to behave well at T = 1.

Decision table

goal	temperature	truncation	penalties
deterministic extraction / classification	0	none needed	none
code generation	0 to 0.3	none or top-p 0.95	none
chat assistant	0.5-1.0	top-p 0.9-1.0	light, optional
creative writing	0.8-1.1	min-p 0.05-0.1 or top-p 0.9	light presence

These ranges are common rules of thumb, not guarantees; validate on your task.

“Too random” and “contradicts itself” are two different symptoms that people often blame on the same knob. Only some of the causes are decoding. This concept gives a procedure: first rule out causes outside decoding, then measure, then change one setting at a time.

Step 1 - rule out non-decoding causes

Decoding knobs cannot fix a bad context. Check these before touching temperature:

• Is the prompt underspecified or itself contradictory (one paragraph says “be brief”, another says “explain in detail”)? See Messages, roles, and prompt structure.
• Is retrieved context inconsistent (two documents that disagree, stale and fresh versions of the same fact)? The model will faithfully reflect the disagreement.
• Does the conversation history carry stale claims from earlier turns?
• Is the answer long enough that contradictions are natural (a model drifting over 2,000 tokens, with the early claims far away in context)?

A quick test: run the same input 5 times at temperature = 0. If the answer is already wrong or already self-contradictory every time, the cause is the prompt, context or model capability, not sampling randomness.

Step 2 - measure the randomness instead of guessing

Randomness shows up as disagreement between runs. Measure it:

from collections import Counter

def agreement(generate, prompt, n=10, **params):
    """Run the same prompt n times; report how concentrated the answers are.
    generate(prompt, **params) -> str  is a placeholder for your API call."""
    answers = [generate(prompt, **params).strip() for _ in range(n)]
    counts = Counter(answers)
    top, top_n = counts.most_common(1)[0]
    return {"distinct": len(counts), "top_share": top_n / n, "top": top[:60]}

# Example output for a factual question (illustrative):
# agreement(gen, "Capital of Australia?", temperature=1.0) -> {'distinct': 4, 'top_share': 0.6, ...}
# agreement(gen, "Capital of Australia?", temperature=0.2) -> {'distinct': 1, 'top_share': 1.0, ...}

Read the result: distinct = 1 means sampling is not your problem. Many distinct answers to a question that has one right answer means the distribution is too flat (temperature) or the tail is too heavy (truncation). Compare top_share before and after each change.

Step 3 - why a loose tail produces “weird” tokens (worked example)

With a 50,000-token vocabulary, individual junk tokens have tiny probability, but there are tens of thousands of them. The mass of the whole tail depends strongly on temperature. The next simulation uses synthetic, illustrative logits (three plausible tokens at 9.0, 7.5, 6.5 and 49,997 junk tokens drawn from a normal distribution with mean -6); real models differ, but the shape of the effect is the point:

temperature	total probability on the 3 plausible tokens	total probability on the 49,997 junk tokens
0.5	100.0%	about 0%
1.0	96.5%	3.5%
1.5	29.3%	70.7%
2.0	4.6%	95.4%

At T = 1.0 a 3.5% junk chance per token means that over a 300-token answer you expect about 10 junk draws (300 * 0.035). Truncation (top_p = 0.9 or min_p = 0.05) deletes the junk tokens before the draw, which is why it fixes “occasional bizarre word” symptoms without lowering diversity among the plausible tokens. Past T = 1.5 most of the mass is junk, and nothing short of lowering T helps: this is the “word salad” regime.

Step 4 - symptom to knob table

symptom	likely knob	fix
different answer every run to a factual question	T too high	go to 0-0.3
occasional bizarre word or wrong-language token in fluent text	truncation too loose	add top_p 0.9-0.95 or min_p 0.05
gibberish after a few sentences	T far above 1 with no truncation	lower T, add min-p
model avoids a key term it must repeat; odd synonyms	penalties too high	reduce to zero, retest
fine early, drifts and contradicts itself late in a long answer	not sampling	shorten the output, ask for structure, split into steps
loops of the same phrase	T too low plus no penalty	raise T slightly, add a small frequency penalty
contradictions between runs but each run is internally consistent	randomness, not a bug	lower T, or accept and vote across runs

Worked example of the penalty trap

Asking for a summary of a contract in which “the Supplier” must appear 20 times. With a frequency penalty alpha = 1.0, the logit of “Supplier” drops by alpha * count. After 5 uses it is 5 logits down, which is a factor of e^5 (about 148) less likely. After 10 uses it is e^10, about 22,000 times less likely. The model starts writing “the vendor”, “the other party”, “the provider”, and the summary becomes inconsistent about who is who. The fix is not a better prompt but alpha = 0.

import math
for uses in (0, 1, 3, 5, 10):
    print(uses, "uses -> relative likelihood", f"{math.exp(-1.0 * uses):.5f}")
# 0 uses -> relative likelihood 1.00000
# 1 uses -> relative likelihood 0.36788
# 3 uses -> relative likelihood 0.04979
# 5 uses -> relative likelihood 0.00674
# 10 uses -> relative likelihood 0.00005

Procedure summary

1. Confirm the context and prompt are not the cause (rerun at T = 0).
2. Measure run-to-run agreement on 5-10 runs.
3. Change one knob at a time, from the table above, and re-measure on the same inputs.
4. A single sample tells you almost nothing about a stochastic process; always compare distributions of outputs, not individual outputs.