Common misconceptions and gotchas

• Token — a chunk of text (a word piece) with an integer id; the unit the model reads and writes.
• Logit — the raw score the model assigns to each possible next token before it is turned into probabilities.
• Softmax — the function that converts logits into probabilities that sum to 1.
• Temperature — a setting that divides the logits before softmax; low = peaked/predictable, high = flat/varied.
• Greedy decoding / argmax — always picking the single highest-scoring token.
• Floating point (fp16 / bf16 / fp32) — the binary formats used for numbers; they have limited precision, so rounding errors exist.
• Non-associative — (a + b) + c is not always equal to a + (b + c); true for floating point.
• Batching — running several requests through the GPU together for throughput.
• Seed — the starting value of a pseudo-random number generator; the same seed gives the same random draws.
• max_tokens — the upper limit on how many tokens the model may generate in one reply.
• Context window — the maximum number of tokens the model can read in one request.
• Stateless — keeping no memory between calls; everything needed must be sent each time.
• Lost in the middle / context rot — the observed drop in recall for information placed mid-prompt, and for very long prompts generally.
• Embedding — a vector (list of numbers) representing the meaning of a piece of text.
• Cosine similarity — the cosine of the angle between two vectors: 1 = same direction, 0 = unrelated, -1 = opposite.
• Anisotropy — embeddings occupying a narrow cone of the space rather than spreading evenly, so all similarities look high.
• Pooling — combining per-token vectors into a single vector for the whole text (mean, last token, or a special token).
• Chain-of-thought (CoT) — having the model write out intermediate reasoning steps before the final answer.
• Few-shot — including worked examples of the task in the prompt.
• RAG (Retrieval-Augmented Generation) — retrieving relevant documents and placing them in the prompt.
• Chunk — a slice of a document stored and retrieved as one unit.
• Reranking — re-ordering retrieved candidates with a more accurate (slower) model.
• Parametric knowledge — facts stored in the model’s weights, as opposed to facts in the prompt.
• System prompt — operator instructions at the start of the context.
• Fine-tuning — continuing training on your own data to change the model’s weights.
• Harness (agent loop) — the ordinary program that calls the model, runs its tool requests and feeds results back.
• Quantisation — storing weights in fewer bits (fp8, int4) to save memory and cost, at some quality cost.
• Speculative decoding — a serving trick where a small model drafts tokens that the big model verifies in parallel.
• Top-p (nucleus sampling) — sampling only from the smallest set of tokens whose probabilities add up to p.
• Chat template — the exact special tokens and role markers a model was trained to see around system, user and assistant turns.
• Catastrophic forgetting — loss of previously learned abilities after training on a narrow new dataset.
• Reversal curse — a reported failure where a model trained on “A is B” cannot reliably answer questions about “B is A”.

The belief. “Answers got better when I raised max_tokens from 256 to 2048, so a bigger cap makes the model write better.”

Why it feels true. Longer limits coincide with longer, fuller answers, and fuller feels better.

The mechanism. max_tokens is only a stop condition: generation halts when the model emits its end-of-text token or hits the cap, whichever comes first. It never changes the probabilities of tokens, so a model that would naturally finish in 100 tokens finishes in 100 whether the cap is 256 or 4096. Quality appears to improve for three different reasons:

1. Truncation was happening. Outputs hit the old cap mid-sentence or mid-JSON; now they finish. Check the response’s stop/finish reason: length (or max_tokens) means it was cut off, stop/end_turn means it finished.
2. The model reasons visibly first. With step-by-step output or a reasoning mode, extra room lets it write more intermediate steps, and those tokens help (see When “think step by step” helps and when it hurts).
3. Noise. On a small sample, differences are chance (see Evaluating LLMs).

cap 256 :  "...the three causes are (1) latency, (2) cache mis"      <- finish_reason = length
cap 2048:  "...the three causes are (1) latency, (2) cache misses, (3) cold starts."  <- stop

Setting 4000 everywhere is not free: it does not make the model slower on short answers, but it removes your protection against runaway output and lets a loop burn money. Set the cap a bit above the longest legitimate answer and alert on length finishes.

The belief. “These two sentences mean the same thing, but similarity is 0.78. The embedding model is broken.”

Why it feels true. Cosine similarity is bounded by 1 and 1 sounds like “identical”, so 0.78 sounds like a poor match.

The mechanism. Cosine similarity is cos(a, b) = (a . b) / (|a| |b|): the dot product of the vectors divided by their lengths, i.e. the cosine of the angle between them. Embedding models do not spread vectors evenly over the sphere. Most vectors share a large common component (a direction all text has in common), so everything sits in a narrow cone and unrelated text already scores high. The scale is only meaningful relative to that model’s own distribution. A synthetic demonstration (illustrative: 200 random vectors that share a common offset, to imitate anisotropy):

Raw vectors with a shared component:
  unrelated pairs: min 0.66   median 0.81   max 0.90     <- "unrelated" already ~0.8
  a near-paraphrase pair (same vector + small noise): 0.996

Same vectors after subtracting the mean vector:
  unrelated pairs: min -0.52  median -0.01  max 0.50     <- spread around 0, as you'd expect

So 0.78 might be a low score for a paraphrase and 0.82 might be the 99th percentile for real matches; only the ranking matters. What to do:

• Choose the threshold empirically: score a labelled set of true-match and non-match pairs and pick the cutoff that separates them (plot both score histograms).
• Follow the model’s usage card: many models need prefixes such as query: and passage: and a specific pooling strategy; omitting them shifts all scores.
• Normalise vectors to unit length (so dot product equals cosine) and use the same metric the model was trained with.
• Never hard-code “above 0.9 = same” across models.

The belief. “A model that can write code can obviously compare two decimals; getting this wrong means it is broken or lazy.”

Why it feels true. To a human, comparing numbers is a basic operation we do with a numeric concept. We assume the model does the same.

The mechanism. The model never receives the numbers; it receives tokens, and it predicts the next token by pattern. 9.11 is typically split into pieces like 9, ., 11 (the exact split depends on the tokenizer). If the model reads the part after the dot as the integer 11, then 11 > 9, so 9.11 > 9.9. That reading is exactly the version-number / section-number convention, in which 9.11 really does come after 9.9, and such text is common in training data.

print(9.11 > 9.9)                                   # -> False  (decimal reading)
v = lambda s: tuple(map(int, s.split(".")))
print(v("9.11") > v("9.9"))                         # -> True   (version-number reading: (9, 11) > (9, 9))

Both readings are valid conventions; the model must choose from context and sometimes picks the wrong one. It is pattern completion, not a numeric operation, and no arithmetic unit exists inside to settle it.

What helps. Give it a calculator or code execution; or force digit alignment (“write both to two decimals: 9.11 and 9.90, then compare”); or state the convention (“as decimal numbers”). Arguing with the model after the fact rarely helps.

The belief. “It has a 200,000-token window, so it remembers my whole conversation and attends to all of it equally.”

Why it feels true. Chat interfaces show a continuous thread, and “window” implies the model sees all of it.

The mechanism has two separate parts.

1. “Remembers” — the model is stateless

The API keeps nothing between calls. Your application re-sends the whole history every turn; if you trim a message, it is simply gone.

history = []
def turn(user_text):
    history.append({"role": "user", "content": user_text})
    reply = call_model(history)            # the ENTIRE list is sent every time
    history.append({"role": "assistant", "content": reply})
    return reply

This also explains cost: if each turn adds roughly 500 tokens, then turns 1 to 5 send 500, 1000, 1500, 2000 and 2500 input tokens, a total of 7,500 for a five-turn chat (not 2,500), which is why prompt caching and history compaction matter.

2. “Everything” — recall is not uniform

Within the window, retrieval of a fact depends on where it sits and how much surrounds it:

accuracy of recalling an inserted fact (schematic, not measured data)
high |*                                              *
     |  *                                        *
     |     *  *                          *  *
low  |           *   *   *   *   *   *
     +-------------------------------------------------
      start            middle                       end

• Lost in the middle: facts near the start and end of the prompt are recalled better than those in the middle.
• Context rot: quality degrades with total length even if the fact is well placed.
• Distractors: text that looks similar to the answer but is wrong hurts recall more than irrelevant text.

Practical test. Insert a unique fact (“the access code is 7413”) at 10%, 50% and 90% of a long document and ask for it; repeat with and without similar-looking distractors. The window is where the model stops working, not where it stops working well; the effective context is typically a fraction of the advertised one. Put key instructions at the start and end, and retrieve rather than stuff.

The belief. “Adding ‘think step by step’ makes any answer better.”

Why it feels true. It became famous as a cheap boost on reasoning benchmarks, so it is treated as a universal switch.

The mechanism. The model produces one token at a time and has no hidden scratchpad besides the tokens it has already written. For a problem needing several dependent steps, writing the intermediate results into the output gives later steps something to attend to:

Q: A shop sells pens at 3 for 4.50. How much do 14 pens cost?

Direct answer (all steps in one pass): e.g. "19.50"   (an illustrative wrong guess; 14 is not a multiple of 3)

Step by step:
  1 pen = 4.50 / 3 = 1.50
  14 pens = 14 * 1.50 = 21.00
  Answer: 21.00

Helps

• Multi-step maths, logic, planning and code design: the intermediate state is externalised into tokens.

Neutral or harmful

Situation	Why it hurts
Simple lookup or classification	Adds latency and cost; occasionally the model talks itself out of the right label
Output must be a bare value (JSON, one word)	You must parse the answer out of the reasoning
A model with a dedicated reasoning mode	It already reasons internally; a manual “think step by step” can conflict with trained behaviour. Use the reasoning-effort setting instead
Latency-critical path	Hundreds of extra tokens before the answer

Gotcha. The written reasoning is not a guaranteed faithful account of how the answer was reached; do not treat it as proof. Verify the final answer where it matters.

The belief. “The RAG system gave a wrong answer, so the retriever must be bad; tune the embeddings.”

Why it feels true. The pipeline is retrieval then generation, and the wrong output looks like missing information.

The mechanism. Check what the model was given. If the correct passage is in the prompt, the retriever did its job and the failure is in generation. Causes:

1. The fact is buried mid-way in a long stuffed context (lost in the middle).
2. Two chunks conflict (an old policy and a new one) and the model picks the wrong one.
3. The prompt never told it to prefer the context over what it already believes, so a strong parametric prior wins.
4. The right sentence is in a chunk surrounded by many near-duplicates.

Diagnose by evaluating the two stages separately

Question: "What is the notice period for terminating the contract?"
Retrieved top-3 contained: [chunk 2: "...notice period is 60 days..."]   -> retrieval OK (recall@3 = 1)
Model answer: "30 days" (its general prior)                              -> generation failure

# Stage 1: did the gold chunk reach the prompt?  (retrieval metric)
retrieved_ids = {c.id for c in top_chunks}
retrieval_ok = gold_chunk_id in retrieved_ids

# Stage 2: given the gold chunk, is the answer faithful? (generation metric)
generation_ok = judge_faithful(answer, context=top_chunks)

print(retrieval_ok, generation_ok)   # -> True False  => fix the prompt/ordering, not the retriever

Fixes: rerank and reduce to the best few chunks, put the best chunk first or last, add “answer only from the context below; if it is not there, say so”, deduplicate conflicting versions, and quote the supporting sentence before answering.

The belief. “If three examples help, thirty must help more.”

Why it feels true. More training data usually helps, and few-shot prompting looks like training.

The mechanism. Few-shot examples do not update the weights; the model just continues the pattern in the text. Everything in the pattern, including accidents, is copied:

• Label imbalance: 8 of 10 examples are “positive” and the model skews positive.
• Recency and position effects: the last example influences the answer disproportionately.
• Surface patterns: if every example output starts with “Sure!”, so does the answer; if all inputs are short, long inputs go badly.
• Context cost: long examples push the real input toward the weakly-attended middle and cost tokens on every call.

Examples (7 of 8 negative):            Test input: "It was fine, nothing special."
  "Terrible"       -> negative         Model predicts: negative   (base-rate echo, not a judgment)
  ... (6 more negative) ...
  "Loved it"       -> positive

What to do. For instruction-tuned models the sweet spot is often 1 to 3 well-chosen, diverse examples that include an edge case and are balanced across labels and formatted exactly as you want the output. Measure with an eval (see Evaluating LLMs): try 0, 2, 5, 10 shots and look at the curve rather than assuming it rises.

The belief. “I’ll fine-tune on our 5,000 internal documents and then it will know them.”

Why it feels true. Training changes the weights, and the documents are the training data, so the facts must go in.

The mechanism. Fine-tuning (see Fine-tuning mechanics in practice) minimises next-token loss on your text. That reliably shifts style, format and surface patterns. It is a poor way to store retrievable facts, for four reasons.

1. Too few exposures. 5,000 documents of about 800 tokens each is 4 million tokens, one pass. The pretrained model learned its facts from trillions of tokens with each fact repeated in many phrasings. A fact seen once in one phrasing is a weak, fragile trace.
2. Training on a sentence is not training on the question. The loss rewards predicting the document’s wording. Recalling a fact later requires producing it from a differently worded question. Models can continue a memorised sentence yet fail when asked directly or in reverse (this is the “reversal curse” reported in the research literature: training on “A is B” does not reliably give “B is A”).
3. No separation of signal from noise. The model cannot tell the one critical number from the thousands of other tokens, and it will still generate a plausible-looking wrong value when unsure.
4. Weights are frozen at training time. A changed policy needs re-training; nothing is cited; and a narrow dataset can degrade general ability (catastrophic forgetting).

What you observe

Fine-tune on: "The notice period for terminating the Acme contract is 60 days."

Prompt (same wording):  "The notice period for terminating the Acme contract is"  -> " 60 days."   works
Prompt (question):      "How long must I give notice to end the Acme contract?"    -> "Typically 30 days."  wrong
Prompt (reverse):       "Which contract has a 60 day notice period?"               -> no Acme              wrong

The model learned the sentence’s surface form, not a retrievable fact.

Code: a probe that tests knowledge rather than memorised wording

facts = [
    {"asked_same_way": "The notice period for terminating the Acme contract is",
     "asked_as_question": "How long must I give notice to end the Acme contract?",
     "reverse": "Which contract has a 60 day notice period?",
     "expect": "60"},
]

def probe(model, facts):
    out = {"same_wording": 0, "question": 0, "reverse": 0}
    for f in facts:
        out["same_wording"] += f["expect"] in model(f["asked_same_way"])
        out["question"]     += f["expect"] in model(f["asked_as_question"])
        out["reverse"]      += "Acme" in model(f["reverse"])
    return {k: v / len(facts) for k, v in out.items()}

def stub(prompt):   # stands in for a fine-tuned model that memorised the sentence
    return " 60 days." if prompt.endswith("is") else "Typically 30 days."

print(probe(stub, facts))
# -> {'same_wording': 1.0, 'question': 0.0, 'reverse': 0.0}

Walkthrough: the same fact is tested three ways. A model that has genuinely absorbed it passes all three; a model that memorised a sentence passes only the first. Run this on a held-out set of questions your users will actually ask, never on the training sentences alone. A high same-wording score and a low question score is the signature of this misconception.

Comparing the options

Need	Right tool
Answers grounded in current docs, with citations	RAG or a retrieval tool
Consistent tone, format or schema at scale	Fine-tuning (or a good prompt plus examples)
Shrink a very long fixed prompt to save latency/cost	Fine-tuning, then compare against prompt caching
A specific new skill with many examples	Fine-tuning
Facts that change weekly	RAG; update the index, not the weights
Small, stable, critical facts	Put them in the prompt (or system prompt)

If you do want facts in the weights

It can work with many varied rewrites of each fact (question and answer pairs, paraphrases, both directions) and several epochs, but it is expensive, brittle, and still uncited. For most products retrieval is cheaper to build and easier to verify.

A common good design is both: RAG for the facts, a light fine-tune for the behaviour (answer format, tone, “say so if the context lacks it”).

The belief. “The model is dumb; it keeps calling the same broken tool.”

Why it feels true. The model is the one emitting the repeated calls.

The mechanism. The model sees the error, but the identical context produces nearly the same next action, so a persistent failure creates a fixed point: same state in, same call out. Ordinary code in the harness should detect and break such loops. Contributing factors:

• An uninformative tool error ("Error 500") gives nothing to adapt to.
• Long transcripts degrade attention, so earlier failed attempts are not noticed.
• No step limit or repeat detection.

import json

MAX_REPEATS, MAX_STEPS = 2, 20

def run_agent(model, tools, task):
    history, seen = [task], {}
    for step in range(MAX_STEPS):
        call = model(history)                       # -> {"tool": ..., "args": {...}} or a final answer
        if "final" in call:
            return call["final"]
        key = (call["tool"], json.dumps(call["args"], sort_keys=True))
        if seen.get(key, 0) >= MAX_REPEATS:         # same call, already failed twice
            history.append("Harness: this exact call failed repeatedly. "
                           "Try a different approach or report that you are blocked.")
            continue
        result = tools[call["tool"]](**call["args"])
        if result.startswith("ERROR"):
            seen[key] = seen.get(key, 0) + 1
        history.append(result)
    return "STOPPED: step limit reached; partial results: ..."

Walkthrough: each call is fingerprinted as (tool, canonical JSON of args); repeats of a failing fingerprint are counted; after two, the harness stops executing it and injects an explicit nudge; a hard step limit guarantees termination. Also return actionable errors ("file not found: /a/b.txt; did you mean /a/c.txt?") and compact long histories. “The model looped” almost always means “the harness let it loop”.

The belief. “Same model name, so same quality everywhere.”

Why it feels true. The weights are the same file, so the outputs should be too.

The mechanism. What you talk to is model + serving stack + platform prompt + default settings, and every part of that stack can differ.

what you type
   |
   v
[ platform prompt / hidden system prompt ]   <- differs per product
[ history truncation or compaction        ]  <- differs per product
[ chat template (special tokens, roles)   ]  <- must match the model's training
   |
   v
[ weights: fp16 / fp8 / int4 ]               <- differs per host
[ inference kernels, batching, speculative decoding ]
   |
   v
[ sampling: temperature, top-p, max_tokens, reasoning on/off ]   <- defaults differ
   |
   v
reply

Difference	Effect
Default sampling parameters	Different temperature/top-p changes variety and error rate
Injected system prompt	Platform adds its own rules and persona
Quantisation (fp16 vs fp8 vs int4)	Cheaper hosting, with some quality loss, mostly on hard tasks
Context truncation policy	Long inputs silently cut or compressed
Reasoning mode default	On vs off changes accuracy and latency
Speculative-decoding settings, kernels, batching	Small numerical differences
Prompt template differences	Wrong chat template can badly hurt open-weight models
Tools and retrieval	A product with web search or code execution answers factual and arithmetic questions better than the bare model
Output post-processing	Formatting, safety filters and length limits change what you see

Worked example: how temperature alone changes behaviour

Temperature divides the logits (raw scores) before softmax: p_i = exp(z_i / T) / sum_j exp(z_j / T), where z_i is the logit of token i and T is the temperature. Take three candidate next tokens with logits [4, 3, 1] (illustrative; the first is the “correct” token):

T	P(token 1)	P(token 2)	P(token 3)
0.2	0.993	0.007	0.000
0.7	0.798	0.191	0.011
1.0	0.705	0.259	0.035
1.5	0.607	0.311	0.082

Over a 20-token answer where each step looked like this and one wrong pick derails the answer, the chance of staying on the correct path is 0.993^20 = 0.87 at T = 0.2 but 0.705^20 = 0.001 at T = 1.0 (a deliberately simplified model: real steps are not all this uncertain, so the gap is smaller in practice). A provider that defaults to a higher temperature will feel less precise on exactly the same weights.

Worked example: why quantisation shows up on hard tasks

Storing weights at 4 bits instead of 16 gives each weight a rounding error. Easy questions have a large gap between the best and the second-best token, so the error does not change the choice; hard multi-step problems have many near-ties, and small perturbations flip them. That is why a heavily quantised deployment can look equal on chat and worse on maths or code. See Quantization and inference efficiency for formats and VRAM and running LLMs locally for the memory trade-off.

How to compare providers fairly

import json, statistics

def run_eval(call_provider, cases, runs=3):
    """call_provider(prompt) -> text. Fixed prompts, no extra system prompt."""
    scores = []
    for _ in range(runs):
        ok = sum(case["check"](call_provider(case["prompt"])) for case in cases)
        scores.append(ok / len(cases))
    return {"mean": round(statistics.mean(scores), 3), "runs": scores}

cases = [
    {"prompt": "What is 17 * 23? Answer with the number only.", "check": lambda t: "391" in t},
    {"prompt": "Reply with the single word: yes", "check": lambda t: t.strip().lower().startswith("yes")},
]
provider_a = lambda p: "391" if "17" in p else "Yes"
provider_b = lambda p: "about 400" if "17" in p else "Yes"
print(json.dumps({"A": run_eval(provider_a, cases), "B": run_eval(provider_b, cases)}))
# {"A": {"mean": 1.0, "runs": [1.0, 1.0, 1.0]}, "B": {"mean": 0.5, "runs": [0.5, 0.5, 0.5]}}

Walkthrough: the same prompts go to both providers, graded by code, repeated three times so run-to-run noise is visible (runs list). With real APIs, also set the same temperature and max tokens, disable any platform system prompt, state the precision if known, and use enough items for the confidence intervals in Evaluating LLMs: benchmarks, contamination, judges.

Checklist before concluding “provider X is worse”

1. Same prompt text, byte for byte? Same chat template?
2. Same temperature, top-p, max tokens, reasoning setting?
3. Is a hidden system prompt or retrieval layer in play?
4. Is the context being truncated (check token counts)?
5. Same model version string, and is it quantised?
6. Enough test items for the difference to exceed the noise?

When comparing, hold all of this fixed, or you are benchmarking the deployment, not the model.