Reasoning models and test-time compute

• Token — a chunk of text (a word piece); the unit the model reads, writes and is billed for.
• Forward pass — one run of the network that produces the next token. A fixed amount of computation per generated token.
• Inference / test time — using a trained model to answer a request (as opposed to training it).
• Test-time compute — extra computation spent at inference (more tokens, more samples, search) to get a better answer.
• Chain-of-thought (CoT) — intermediate reasoning written out as tokens before the final answer.
• Reasoning / thinking tokens — the CoT tokens a reasoning model generates; usually billed as output tokens and counted in the context.
• RL (Reinforcement Learning) — training by trial and reward: generate, score, make high-scoring behaviour more likely.
• RLVR (RL with Verifiable Rewards) — RL where the reward comes from an automatic check (a unit test, a numeric answer match) rather than a human or learned judge.
• Verifier — code or a model that checks whether an answer is correct.
• Policy — the model being trained, viewed as a distribution over what it will generate.
• Rollout / sample — one complete generated response, including its reasoning.
• Advantage — how much better a particular sample’s reward is than the typical reward for that prompt.
• GRPO (Group Relative Policy Optimization) — an RL method that compares several samples of the same prompt with each other instead of using a learned value network.
• ORM (Outcome Reward Model) — scores only the final answer.
• PRM (Process Reward Model) — scores each intermediate reasoning step.
• Self-consistency — sample several reasoning paths and take the most common final answer (majority vote).
• Best-of-N — sample N candidates and keep the one a verifier or judge scores highest.
• pass@k — the probability that at least one of k samples is correct.
• Temperature — sampling randomness; above 0 gives different samples on repeated runs.
• Reasoning effort / thinking budget — a setting limiting or steering how many thinking tokens the model may spend.
• Faithful (trace) — a reasoning trace is faithful if the reasons it states are the reasons that actually produced the answer.
• PII (personally identifiable information) — data that identifies a person (emails, card numbers); a reason to restrict and redact stored traces.
• KV cache — stored attention keys/values for already-processed tokens; grows with every token, thinking tokens included (see Self-attention and positional structure).

Test-time compute is compute spent at inference, beyond a single forward pass per output token, to improve the answer — generating an extended reasoning trace before the final answer, sampling multiple candidate solutions and picking the best, or running a search process. It helps on tasks where the correct answer is hard to produce directly but easier to verify or refine — multi- step math, algorithmic problems, proofs — because it lets the model use its own intermediate output as working memory and self-correct, rather than being forced to get a multi-step derivation exactly right in one forward pass with no scratch space. It is the inference-side analogue of scaling training compute: for a fixed model, there is a real accuracy/cost curve as you spend more tokens “thinking” before answering, and modern reasoning models are trained specifically to use that budget well rather than pad it with irrelevant tokens.

Why tokens are “compute”

A transformer does a fixed amount of work per generated token: one forward pass through a fixed number of layers. Whatever it can compute “in its head” while emitting one token is bounded by that depth. A problem needing more sequential steps than the network has layers cannot be solved in a single token no matter how well trained it is. Generating intermediate tokens fixes this: each token’s output is fed back as input, so the model can chain many forward passes, writing partial results into the context and reading them back via attention.

Question: What is 17 * 24 + 9 ?

No thinking:    model must emit the answer's first digit immediately -> must "know" 417 in one pass
Thinking:       "17 * 24 = 17 * 20 + 17 * 4 = 340 + 68 = 408.  408 + 9 = 417."  -> answer: 417
                 each partial result is now in the context, so each step is easy

Thinking tokens are therefore working memory plus extra serial depth. The cost is linear in the number of tokens generated (and more, since each token attends to a longer context).

Three ways to spend test-time compute

Method	What it does	Best when	Cost
Long reasoning trace	one sample with a long chain-of-thought	multi-step problems; model trained for it	one long generation
Parallel sampling (self-consistency, best-of-N)	N independent attempts, vote or verify	answer is checkable or has few distinct values	N x generation, but parallelisable
Search (tree / beam over steps with a step scorer)	explore alternatives, prune bad branches	a reliable step-level signal exists (see PRMs below)	highest, most engineering

These combine: a reasoning model producing long traces, sampled several times, with a verifier choosing.

When it helps and when it does not

Helps: tasks with intermediate state or a verifiable answer (arithmetic, logic puzzles, code with tests, planning). Does little: recalling a fact the model does not know (more thinking cannot add knowledge), simple formatting, short-latency classification.

Thinking is not a hidden module or a separate “reasoning engine”. A reasoning model is the same kind of next-token predictor as any other: it generates a long run of ordinary tokens (the “thinking”) before the tokens of the final answer. The only differences are that training taught it to use that run well (next concept) and that the API usually separates the two parts.

The layout of one response

[prompt tokens]  ->  [thinking tokens ........................]  ->  [answer tokens]
 you pay input        generated one by one, billed as output,        what the user sees
                      occupy context window and KV cache

Each thinking token is produced by the same forward pass as every other token: the model reads everything so far (prompt plus the thinking written so far), outputs a distribution over the vocabulary, one token is sampled, and it is appended. When the model emits a special end-of-thinking marker (or the budget runs out), it starts writing the answer, now conditioned on the whole trace.

What a trace looks like (illustrative)

Question: “A train leaves at 14:20 and the trip takes 2 h 55 min. When does it arrive, and is that before 17:30?” A trace might read:

Start 14:20. Add 2 h -> 16:20. Add 55 min -> 16:75, which is 17:15.
Check: 14:20 + 3 h = 17:20, minus 5 min = 17:15. Matches.
17:15 is before 17:30. Yes.

and then the answer: “It arrives at 17:15, which is before 17:30.” Note the habits RL encourages: a first attempt, a check by a different route, then a conclusion. A real trace is usually longer, messier and may include dead ends (“wait, that is wrong, let me redo it”).

Why writing the steps down changes the computation

Without a trace, the model must produce the first answer token after a fixed number of layers; whatever it can compute “in its head” is limited by that depth. With a trace, each written step is an intermediate result sitting in the context. The next step reads it back through attention, so many forward passes chain together. Concretely: a 32-layer model can do at most about 32 layers of sequential processing per token, but 500 thinking tokens give it up to 500 x 32 layers of chained processing, with the intermediate values stored as text (see Test-time compute — what it means and why it raises accuracy on some tasks above).

What it costs: accounting for one request

Illustrative numbers: prompt 400 tokens, thinking 3,200 tokens, visible answer 250 tokens, input $3 per million, output $15 per million, decode speed 60 tokens/s.

price_in, price_out = 3.0, 15.0           # dollars per million tokens (illustrative)
prompt, think, answer = 400, 3200, 250

with_thinking = (prompt * price_in + (think + answer) * price_out) / 1e6
no_thinking   = (prompt * price_in + answer * price_out) / 1e6
print(round(with_thinking, 4), round(no_thinking, 4))    # 0.053 0.005   about 10.7x more expensive
print(round(think / 60, 1), "seconds spent before the first answer token")   # 53.3

Reading it: the user sees only 250 tokens of answer but is billed for 3,450 output tokens; thinking is 93% of the output bill here. It also adds 3200 / 60 = 53 seconds before the answer starts streaming.

Memory cost: every thinking token also adds a KV-cache entry. With an illustrative 8B-class model at 128 KiB of KV cache per token (2 x 32 layers x 8 KV heads x 128 dims x 2 bytes), 3,200 thinking tokens occupy 3200 x 128 KiB = 400 MiB of GPU memory while the request runs (see KV cache, prompt caching, and inference economics).

Is thinking kept across turns?

This is a provider/API behaviour; do not assume. The two policies have different consequences, shown for a 4-turn chat with 400 prompt tokens, 3,200 thinking tokens and 250 answer tokens per turn (cumulative context entering turn 4 plus that turn):

policy	context after turn 1	after turn 4
thinking from earlier turns kept	3,850	15,400
thinking dropped between turns	650	2,600

Keeping it preserves the model’s earlier working but fills the context window about 6x faster here; dropping it saves space and cost but the model cannot refer back to its earlier derivations. Check the documentation for the specific model: some APIs require passing earlier thinking blocks back unchanged when tool calls are in progress.

Pitfalls

• Thinking tokens count against max_tokens: a budget that is too small can leave no tokens for the answer, producing an empty or truncated reply.
• A reasoning trace is model output, not ground truth; it can contain wrong intermediate claims and still land on a right (or wrong) answer.
• Do not paste raw traces into other prompts as if they were facts.

Non-reasoning models can do it too, by prompting

Asking “think step by step” or giving a worked example makes an ordinary instruction-tuned model emit a chain-of-thought in its visible answer and often improves multi-step accuracy.

Plain:     "17 * 24 + 9 = ?"                                      -> may answer 417 or guess wrongly
CoT:       "17 * 24 + 9 = ? Think step by step."                  -> "17*24 = 408 ... so 417"

The difference with a reasoning model is training: it was optimised with RL to use long reasoning well (next concept), rather than merely being prompted to, so it explores, backtracks and checks more reliably, and it does this by default without being asked.

The key ingredient is RL with verifiable rewards (RLVR): for domains with an automatically checkable correct answer (math with a numeric final answer, code with passing unit tests, formal proofs), the model is rewarded directly for producing a correct final answer after generating its own reasoning trace — no human preference labeling or learned reward model needed, since correctness is checked programmatically. This differs from RLHF (see Training and post-training) in the reward source: RLVR’s reward is objective and hard to hack, so the model is free to discover whatever reasoning strategies actually produce correct answers, including long backtracking, self-verification, and exploring multiple approaches — behaviors that were not explicitly demonstrated in any SFT data but emerge because they raise the reward. This is why reasoning-model traces often look meandering rather than like a clean textbook derivation: they were optimized for correctness, not readability.

The training loop, step by step

1. take a problem with a known checkable answer         ("What is 17*3 - 9?"  gold = 42)
2. policy samples G responses, each = reasoning + final answer   (G of 8 to 64 is typical; illustrative)
3. a verifier scores each response                       reward 1.0 if answer matches gold, else 0.0
4. compute each response's advantage vs its siblings     (how much better than the group average)
5. gradient step: make high-advantage responses (all their tokens) more likely, low-advantage less likely
6. repeat over many problems

A verifier, concretely

import re

def reward(completion: str, gold: float) -> float:
    """1.0 only if the completion ends with 'Answer: <number>' and the number is right."""
    m = re.search(r"Answer:\s*(-?\d+(?:\.\d+)?)\s*$", completion.strip())
    if m is None:
        return 0.0                         # no parseable final answer: no reward
    return 1.0 if float(m.group(1)) == gold else 0.0

print(reward("17*3 = 51, 51 - 9 = 42\nAnswer: 42", 42.0))   # 1.0
print(reward("I think it's 42", 42.0))                      # 0.0   right number but wrong format
print(reward("Answer: 41", 42.0))                           # 0.0   wrong

The format requirement is part of the reward: it teaches the model to end every trace with a parseable answer. The reward ignores the reasoning content, so any strategy that reaches the right answer is reinforced. (Many setups also add a small format reward; exact reward shaping varies by lab.)

Advantage in GRPO — an equation, with numbers

Group-relative methods such as GRPO (introduced in the DeepSeekMath work) avoid training a separate value network. For one prompt, sample G responses with rewards r_1 ... r_G and compute

A_i = ( r_i - mean(r) ) / std(r)

Symbol	Meaning	Where it comes from
r_i	reward of response i	the verifier
mean(r), std(r)	average and standard deviation of the G rewards for this prompt	computed from the samples
A_i	the advantage: positive if this response beat its siblings, negative if worse; in units of standard deviations	derived; no learned parameters

Every token in response i then gets the same advantage A_i in the usual clipped policy-gradient objective (the same clip idea as PPO in Training and post-training, typically with a KL term to a reference model).

import statistics as st
rewards = [1, 0, 0, 1, 0, 0, 0, 1]             # 3 of 8 samples were correct
mu, sd = st.mean(rewards), st.pstdev(rewards)
print(round(mu, 3), round(sd, 3))               # 0.375 0.484
print([round((r - mu) / sd, 2) for r in rewards])
# [1.29, -0.77, -0.77, 1.29, -0.77, -0.77, -0.77, 1.29]

Correct samples get +1.29, incorrect ones -0.77. If all 8 samples were right (or all wrong), std = 0 and the advantage is zero (implementations add a small epsilon to avoid dividing by zero): a prompt that is too easy or too hard gives no learning signal, which is why training sets are filtered or ordered by difficulty.

Why the behaviours emerge

Nothing tells the model to “double-check” or “try another approach”. But samples that happen to re-verify their work reach the correct answer more often, get positive advantage, and have all their tokens (including the checking phrases) reinforced. Over many steps, self-verification, backtracking and longer traces become common. This is an observed, reproducible phenomenon, but how much comes from RL itself versus latent ability from pretraining is still discussed.

Limits

• Only works where a reliable verifier exists (maths, code, formal logic, some structured tasks). Subjective quality (writing, advice) still needs preference-based methods.
• Reward hacking still exists, just differently: a model can special-case unit tests or exploit a weak answer parser. Verifiers need as much care as reward models.
• Models are usually further trained with SFT and preference tuning so that the final answer reads well and the model follows instructions.

More thinking is not always better. Diminishing and even negative returns show up in real ways:

• Overthinking on easy inputs — a trivial question padded with unnecessary deliberation costs latency and money with no accuracy gain, and occasionally the model talks itself out of an initially correct answer.
• Compounding errors — a long trace gives more opportunity for an early wrong turn to propagate, especially if the model does not actually verify intermediate steps.
• Distribution mismatch — reasoning depth tuned during RL for math/code benchmarks does not automatically transfer well to open-ended or subjective tasks with no verifiable answer.
• Latency-sensitive products — a chat UI or a low-latency agent step may not have the budget for extended thinking regardless of the accuracy upside.

The practical implication: reasoning effort should be a per-task decision, not a global default — route simple lookups/formatting to low or no reasoning, and reserve high effort for genuinely multi-step problems.

A shape to keep in mind (illustrative, not measured)

accuracy
   ^                         ___________  plateau
   |                   _____/
   |              ____/
   |         ____/
   |    ____/                  (sometimes dips slightly past the plateau)
   |___/
   +--------------------------------------------> thinking tokens (log scale)
   easy tasks hit the plateau almost immediately; hard tasks keep climbing for much longer

Why long traces go wrong: compounding arithmetic

If each step in a chain is correct with probability p and a single wrong step ruins the answer (no self-correction), the chance the whole chain is right is p^steps:

print(round(0.9 ** 5, 3))     # 0.59   5 steps at 90% each
print(round(0.95 ** 10, 3))   # 0.599  10 steps at 95% each
print(round(0.99 ** 20, 3))   # 0.818  20 steps at 99% each

This is a simplified model (real models do recover from some errors, and steps are not independent), but it explains why long chains need either very reliable steps or verification and backtracking, and why PRMs and parallel sampling (below) exist.

It controls (directly or via a target token count) how much of the reasoning-token budget the model is allowed or encouraged to spend before committing to a final answer — low effort biases toward a short or skipped trace, high effort allows extended deliberation, sometimes with self-checking passes built in. Setting it:

• Low/none — classification, extraction, formatting, simple factual lookup, anything latency-sensitive with a clear answer.
• Medium — typical coding tasks, moderate multi-step questions.
• High — hard math/logic, multi-file code changes with non-obvious interactions, planning tasks, anything where you have observed the lower setting getting it wrong.

Treat it like any other tunable: measure accuracy and cost/latency on your task distribution at each setting rather than assuming “higher is strictly better” — it usually is for hard tasks and usually is not worth it for easy ones.

Two styles of control

• Named levels (such as low / medium / high): the model decides how many tokens to use within a loose range.
• Explicit token budget: a maximum number of thinking tokens. It is typically a ceiling, not a target; the model may use fewer, and a too-small budget truncates thinking, which can hurt more than none at all.

Parameter names, accepted values and whether the budget is hard or soft differ by provider and model generation, so check the current documentation for the model you use rather than copying names from elsewhere.

Measure it: a small sweep

import time

def run_sweep(solve, cases, efforts=("low", "medium", "high")):
    """solve(prompt, effort) -> (answer, output_tokens); cases = [{'prompt':..., 'gold':...}]"""
    for effort in efforts:
        correct = tokens = 0
        t0 = time.time()
        for c in cases:
            answer, out_tokens = solve(c["prompt"], effort)
            correct += (answer == c["gold"])
            tokens += out_tokens
        n = len(cases)
        print(f"{effort:6s} acc={correct/n:.0%}  avg_tokens={tokens/n:.0f}  sec/case={(time.time()-t0)/n:.1f}")

solve is a placeholder wrapping your provider’s API call. The output gives accuracy, tokens per case (cost) and latency per setting on your own tasks; choose the cheapest level whose accuracy is within your tolerance of the best. Do this per task category, not globally.

Because thinking tokens are billed and generated serially, effort changes both cost and time-to-answer.

# Illustrative price and speed; replace with your provider's real numbers.
price_out = 15.0            # dollars per million output tokens
tok_per_s = 60              # decode speed

for name, tokens in [("no thinking", 300), ("medium", 300 + 1500), ("high", 300 + 8000)]:
    cost = tokens * price_out / 1e6
    secs = tokens / tok_per_s
    print(f"{name:12s} {tokens:5d} tokens  ${cost:.4f}  {secs:5.1f}s")
# no thinking    300 tokens  $0.0045    5.0s
# medium        1800 tokens  $0.0270   30.0s
# high          8300 tokens  $0.1245  138.3s

A 6x to 28x jump in cost and latency for the same visible answer. At a million requests a month, the gap between “no thinking” and “high” is thousands of dollars. This is why routing (a cheap classifier or rule decides the effort level per request) is a standard pattern. Many providers can stream the answer after the thinking completes, but the time to first answer token still includes the whole thinking phase.

• ORM (outcome reward model) — scores only the final answer’s correctness. Simple, and the reward signal is unambiguous (matches a ground truth or not), but gives no credit or blame to which step in a long chain was right or wrong — a correct final answer reached via flawed reasoning gets full reward, and a good chain that slips at the last step gets none.
• PRM (process reward model) — scores the correctness of each intermediate reasoning step, trained on step-level human or automated annotations. This gives denser, more targeted training signal and can catch “right answer, wrong reasoning” or be used to guide search (reject a branch as soon as a step is scored bad) — but step-level labels are far more expensive to collect and noisier to define for non-mathematical domains.

Much of RLVR in practice leans on ORM-style outcome rewards precisely because they are cheap and hard to game for verifiable domains, with PRMs and self-consistency checks layered in for guided search or re-ranking multiple sampled solutions.

Same trace, two kinds of score

Problem: A shirt costs $40 and is discounted 25%, then taxed 10%. Final price?

Step 1: discount = 40 * 0.25 = 10          -> price 30         PRM: correct   (1.0)
Step 2: tax = 30 * 0.10 = 3                -> 33               PRM: correct   (1.0)
Step 3: "so the final price is 30 + 3 = 33"                        PRM: correct   (1.0)
Answer: 33                                                      ORM: correct   (1.0)

Flawed chain that lands on the same number by luck:
Step 1: discount = 40 * 0.25 = 12          -> price 28         PRM: wrong     (0.0)
Step 2: tax = 28 * 0.10 = 2.8 -> 30.8
Step 3: "rounding up, 33"                                         PRM: wrong     (0.0)
Answer: 33                                                      ORM: correct   (1.0)  <- rewarded for bad reasoning

How a step-level score becomes a trace-level score

PRMs output a probability (or label) per step. Common aggregations: the minimum over steps (a chain is as good as its weakest step), the product (all steps must be right), or the last-step score. Used for re-ranking:

import math

def trace_score(step_scores, how="min"):
    if how == "min":
        return min(step_scores)
    if how == "product":
        return math.prod(step_scores)
    return step_scores[-1]

a = [0.95, 0.90, 0.92]        # candidate A, PRM probability each step is correct (illustrative)
b = [0.99, 0.40, 0.98]        # candidate B, one shaky step
print(trace_score(a, "min"), round(trace_score(a, "product"), 3))   # 0.9 0.787
print(trace_score(b, "min"), round(trace_score(b, "product"), 3))   # 0.4 0.388   -> prefer A

Trade-offs

	ORM / programmatic outcome check	PRM
Labels	cheap (just the final answer)	expensive (per step), or generated automatically by sampling continuations from each step and checking how often they succeed
Signal	sparse: one number per whole trace	dense: credit assignment per step
Main risk	rewards lucky or flawed reasoning	learned scorer can be gamed; hard to define for open-ended tasks
Typical use	RL training with a verifier	guiding search, reranking samples

A single long chain-of-thought commits early to a direction and can get stuck rationalizing a wrong initial step. Self-consistency samples several independent reasoning paths (typically at temperature > 0 for diversity) and takes the majority answer; best-of-N with a verifier samples several candidates and picks the one a separate judge or checker scores highest. Both exploit the fact that independent errors are less likely to be the same error — if most paths converge on one answer, that convergence is itself evidence of correctness, in a way a single trace’s confidence is not. The cost is proportional to N (more generation, more compute), so it is a lever for accuracy-critical, latency-tolerant tasks (a hard benchmark problem, an offline batch job) rather than an always-on default.

Why voting works: the numbers

Suppose each independent sample is right with probability p = 0.6 on a yes/no question. The chance a majority of N samples is right is the binomial tail:

from math import comb
def majority_correct(p, n):
    return sum(comb(n, k) * p**k * (1 - p)**(n - k) for k in range(n // 2 + 1, n + 1))

for n in (1, 3, 5, 11):
    print(n, round(majority_correct(0.6, n), 3))
# 1 0.6
# 3 0.648
# 5 0.683
# 11 0.753

It helps, with diminishing returns. It helps much more when wrong answers are scattered, because a plurality only needs to beat the largest wrong cluster. Simulation: one correct answer with probability 0.4, wrong answers spread over four alternatives:

import random, collections
random.seed(1)
def sample():
    return "42" if random.random() < 0.4 else random.choice(["41", "43", "40", "44"])
def vote(n):
    return collections.Counter(sample() for _ in range(n)).most_common(1)[0][0]

for n in (1, 5, 15):
    acc = sum(vote(n) == "42" for _ in range(20000)) / 20000
    print(n, round(acc, 2))
# 1  0.40
# 5  0.55
# 15 0.78    (simulated; exact values vary slightly with the seed)

A model that is right only 40% of the time per sample becomes right about 78% of the time with 15 votes, because the wrong 60% is split four ways (15% each).

The catch: errors are not independent

The maths assumes independent samples. Real samples share the same model and the same blind spots: if the model systematically misreads a problem, all N samples fail together and voting just confirms the mistake. Independence is highest with higher temperature and varied prompts, lowest for systematic gaps in knowledge.

Best-of-N and pass@k

pass@k is the probability that at least one of k samples is correct. It is an upper bound on what best-of-k can achieve with a perfect verifier. If one sample succeeds with probability 0.3: pass@1 = 0.3, pass@5 = 1 - 0.7^5 = 0.83, pass@10 = 0.97. Given n generated samples of which c are correct, the standard unbiased estimator is:

from math import comb
def pass_at_k(n, c, k):
    """P(at least one of k samples drawn without replacement from n is correct)."""
    if n - c < k:
        return 1.0
    return 1 - comb(n - c, k) / comb(n, k)

print(round(pass_at_k(10, 3, 1), 3))   # 0.3
print(round(pass_at_k(10, 3, 5), 3))   # 0.917

The gap between pass@k and what you actually get is the verifier’s quality: with a perfect verifier (unit tests) you get close to pass@k; with an imperfect model-based judge, less, and sometimes it prefers confident-looking wrong answers.

Choosing among the options

Situation	Choose
Answer is a short value (number, label)	self-consistency (majority vote), no verifier needed
Answer is free-form but checkable (code with tests)	best-of-N with the checker
Free-form and uncheckable	rank with a judge model, but beware judge bias; consider just one good long trace
Tight latency	one trace; or run N in parallel if capacity allows (latency of one, cost of N)

A product that uses a reasoning model must decide what to do with the thinking text: display it, summarise it, store it, or discard it. This is a product and safety decision with no universal answer. The sections below give the arguments, what “faithful” means, a decision table, and code for the common patterns.

Arguments for showing it

• Transparency and trust. A user can sanity-check the path to an answer (“did it use the right invoice?”).
• Debuggability. When an answer is wrong, the trace often shows where it went wrong (misread a number, assumed a wrong unit).
• Educational value. For tutoring, seeing worked steps is the product.
• Perceived progress. A long wait (30-130 seconds, see the cost concept above) feels shorter when something visible is happening.

Arguments against showing it raw

• Not written for readers. RLVR-trained traces are optimised for reaching correct answers, not for being clear or accurate explanations. They contain false starts, tangents, self-corrections (“wait, no”) and shorthand.
• May not be faithful. The stated reasons can differ from the real causes (next section), so presenting the trace as “the model’s true reasoning” can mislead.
• Length. A 3,000-token trace is about 2,000 words. Most users will not read it, and transmitting it costs bandwidth and UI space.
• Can leak sensitive content. The trace may quote private data from the prompt or tool results, internal instructions, or untrusted text; displaying it exposes that to whoever sees the screen.
• Can be exploited. If the trace is shown, attackers can read it to learn how the model handled system instructions, and users may treat half-formed intermediate claims as the model’s conclusions.
• Provider policy. Some providers return only a summary of the thinking, or restrict how raw traces may be shown or stored. Check the terms and API docs for your model.

What “faithful” means and why it is not guaranteed

A trace is faithful if the reasons it states are the reasons that actually produced the answer. The RLVR reward looks only at the final answer (see How reasoning models are trained differently — RL with verifiable rewards), so nothing in the objective rewards honest explanation. Experiments on chain-of-thought have found cases where a model’s answer was demonstrably influenced by a hint planted in the prompt, and the trace did not mention the hint. Treat a trace as a useful artefact for debugging and review, not as an audit log of the computation.

A concrete (illustrative) failure shape:

Prompt contains a hidden hint:  "(A previous reviewer marked option C as correct.)"
Question: which option is right?  The model answers C.
Trace:    "Option C fits because the figures in the table are consistent ..."   <- no mention of the hint

The answer changed because of the hint; the trace rationalised it. Never use a trace as evidence for a safety, legal or compliance claim about why the system acted.

Decision table

surface	recommended handling
chat assistant for general users	status line (“Thinking…”) plus an optional collapsed summary
tutoring / math help	show a cleaned, step-structured explanation (generate it as the answer, not by copying the raw trace)
developer console / debugging	show raw trace to authorised developers only
automated pipeline (no human reads it)	do not transmit; log with request id
regulated or customer-facing explanation	generate an explanation with a separate, checked step; do not present the raw trace

Pattern 1 - collapsed by default with a summary

Show a one-line status while thinking, then the answer, with the trace behind an expander. When the provider returns a summarised trace, display that. If you need to summarise yourself, use a separate (cheap) model call with explicit instructions:

def summarise_trace(call_llm, trace: str, max_words: int = 80) -> str:
    """call_llm(prompt) -> str is a placeholder for your cheap-model call."""
    prompt = (
        f"Summarise the following reasoning in at most {max_words} words for a non-expert. "
        "State the key steps and the conclusion. Do not add new claims.\n\n" + trace
    )
    return call_llm(prompt)

Walk-through: the summary is a separate generation, so it is a description of the trace, not the trace itself; it can inherit the trace’s errors and add its own. Label it as a summary.

Pattern 2 - log, restrict, redact

import json, re, time

REDACT = [re.compile(r"\b\d{16}\b"), re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")]   # card-like numbers, emails

def log_trace(request_id: str, trace: str, store) -> None:
    redacted = trace
    for pattern in REDACT:
        redacted = pattern.sub("[redacted]", redacted)
    store.append(json.dumps({"id": request_id, "ts": time.time(), "trace": redacted}))

Walk-through: store the trace under the request id so a developer can replay a bad answer, redact obvious sensitive patterns first, and limit who can read the store. The regexes are examples, not a complete PII solution.

More product rules

• Do not feed traces back as instructions. Text in a trace is model output and may quote untrusted content from tool results; treat it as data.
• Do not rely on the trace for safety decisions. Validate outputs with separate checks (schema validation, policy classifiers, human review for high stakes).
• Do not promise users that the trace is “how the AI thinks”. “Here are the steps it worked through” is accurate; “here is exactly why” is not.
• Plan for latency anyway. Whether or not you display the trace, thinking happens before the first answer token, so streaming the answer does not remove the wait (see the cost concept above).