Context, long conversations, and attention decay

• Token — the unit a model reads and writes: a word piece, punctuation mark or byte group. English averages very roughly 4 characters (about 0.75 words) per token; code and other languages vary.
• Context / prompt — the full token sequence the model conditions on for one call.
• Context window — the maximum number of tokens the model can process in one call.
• Prefill / decode — prefill processes the whole input prompt in parallel; decode then generates output tokens one at a time.
• KV cache — stored key and value vectors for every earlier token, per layer, so decode does not recompute them. Its size grows linearly with context length.
• Attention — the mechanism by which each token position weighs all earlier positions. See Self-attention and positional structure.
• Softmax — the function that turns raw scores into positive weights that sum to 1.
• Logit — a raw, pre-softmax score.
• Attention sink — a position (often the first token) that attracts a lot of attention weight regardless of its content.
• RoPE (Rotary Position Embedding) — a positional encoding that rotates query and key vectors by an angle depending on position, so attention depends on relative distance.
• Position interpolation — a technique for stretching a model trained on shorter sequences to longer ones by rescaling positions.
• Lost in the middle — the finding that facts in the middle of a long context are used less reliably than those at the start or end.
• Context rot — the general decline in answer quality as input length grows.
• Needle in a haystack — an eval that hides one fact (the needle) in a long filler document and asks the model to retrieve it.
• RULER, NoLiMa — long-context benchmarks that go beyond simple needle retrieval to estimate effective context length.
• RAG (Retrieval-Augmented Generation) — retrieving relevant text and placing it in the prompt.
• Reranking — reordering retrieved candidates with a stronger scorer so the best are first.
• Compaction — replacing old conversation turns with a shorter summary.
• Truncation — dropping the oldest content once a token budget is reached.
• Sliding window — keeping only the most recent N turns verbatim, often next to a running summary of older ones.
• Prompt cache — provider-side reuse of an identical prompt prefix, cheaper and faster than reprocessing it.
• Sub-agent — a separate model loop with its own fresh context for a bounded task.

“Context” is the exact sequence of tokens the model conditions on for the current forward pass, nothing more. For a chat application it is the fully rendered prompt: the system prompt, every prior user and assistant turn, any tool-call requests and tool results, any injected documents (RAG chunks), and the current user message, concatenated and templated into one token sequence.

What the model actually receives

+--------------------------------------------------------------------------+
| system prompt            |  1,500 tokens   (pinned, same every call)     |
| tool definitions         |  6,000 tokens   (resent every call)           |
| turn 1  user + assistant |  ~1,200 tokens                                |
| turn 2  user + assistant |  ~1,500 tokens                                |
|   (tool call + result)   |  ~3,000 tokens                                |
| ...                                                                      |
| current user message     |    200 tokens                                 |
+--------------------------------------------------------------------------+
        everything above is ONE flat token sequence for this call

The token counts are illustrative; measure your own with your provider’s tokenizer or token-counting endpoint.

What is not context

Things frequently confused with it:

• The model’s weights: its parametric knowledge from training. Not context; not editable at inference.
• Anything from a previous conversation, unless your application re-sends it. The model has no memory between API calls; “memory” features are your code storing text and re-injecting it into a later prompt.
• Files or tools “available” to the model: only their descriptions are in context; their contents enter only when a tool returns them.

Statelessness, concretely

# Call 1
reply1 = llm(messages=[{"role": "user", "content": "My name is Ada."}])
# -> "Nice to meet you, Ada."

# Call 2: a fresh request WITHOUT the earlier turns
reply2 = llm(messages=[{"role": "user", "content": "What is my name?"}])
# -> "I don't know your name."   (nothing carried over)

# Call 2, done the way chat apps do it: resend the history
reply2 = llm(messages=[
    {"role": "user", "content": "My name is Ada."},
    {"role": "assistant", "content": "Nice to meet you, Ada."},
    {"role": "user", "content": "What is my name?"},
])
# -> "Your name is Ada."

Here llm stands for any chat-completion call. The model did not remember; the history was in the prompt.

Consequence

Every call reconstructs the whole situation from tokens. Therefore (1) whatever is not in the context does not exist for the model, (2) everything that is in it costs compute, and (3) what you choose to include is the main lever you have.

The context window is the maximum number of tokens the model can process in one call (input plus output combined, in most APIs). It is a hard limit set by training, positional-encoding range and the memory available for the KV cache.

Budget arithmetic

Treat the window as a budget to be divided. For a 200,000-token window (an illustrative figure):

window                         200,000
- reserved for the answer         8,000   (max_output_tokens you allow)
- system prompt                   1,500
- tool definitions                6,000
- pinned context (e.g. rules)     2,500
= left for conversation history 182,000

This is the arithmetic a harness does before every call. In code:

WINDOW = 200_000
RESERVED_OUTPUT = 8_000
FIXED = 1_500 + 6_000 + 2_500          # system + tools + pinned

def history_budget():
    return WINDOW - RESERVED_OUTPUT - FIXED

def needs_compaction(history_tokens, threshold=0.75):
    return history_tokens > threshold * history_budget()

print(history_budget())            # -> 182000
print(0.75 * history_budget())     # -> 136500.0   compact above this
print(needs_compaction(40_000))    # -> False
print(needs_compaction(200_000))   # -> True

Why compact at 75% and not 100%? You need headroom for the next tool result (which may be tens of thousands of tokens) and for the summarisation call itself, which has to fit the material it is summarising.

Why the advertised figure is not the number to design for

The advertised figure (128k, 200k, 1M) is where the model stops functioning, not where it stops functioning well. Reasons to plan against a smaller effective budget:

• Retrieval degradation with distance and position (lost-in-the-middle, below): recall of facts deep in a long context is materially worse than at the ends.
• Length generalisation. Many models are trained mostly on much shorter sequences and extended to the headline length via position interpolation or continued training; quality often sags well before the limit.
• Cost and latency scale with tokens: prefill of a full window is slow and expensive.
• Distraction. More irrelevant tokens measurably lower accuracy even when the relevant fact is present.
• Memory. The KV cache grows linearly with length. As an illustration, a model with 32 layers, 8 key/value heads, head dimension 128 and 16-bit values stores 2 (K and V) x 32 x 8 x 128 x 2 bytes = 131,072 bytes = 128 KiB per token, so 8,000 tokens take about 1 GiB and 100,000 tokens about 12.2 GiB, per concurrent request. This is why providers limit and price long contexts; see VRAM and running LLMs locally for the full accounting.

Benchmarks such as RULER and NoLiMa probe “effective context length”, and it is routinely a fraction of the advertised one.

Counting tokens rather than guessing

A rough rule is 1 token per 4 English characters, good enough for a threshold check but wrong for code, non-English text and JSON, which tokenise less efficiently. For anything near a limit use the real tokenizer, or the provider’s count-tokens endpoint, on the exact rendered prompt including the chat template and tool definitions.

Lost in the middle is the empirical finding that when a relevant fact sits in the middle of a long context, retrieval accuracy is significantly lower than when it is near the beginning or the end. The position-versus-accuracy curve is U-shaped.

The shape

accuracy
 high |*                                           *
      | *                                         *
      |   *                                     *
      |      *  *                         *  *
  low |            *   *   *   *   *   *
      +---------------------------------------------> where the fact is placed
       start                middle                end

This is a schematic of the effect, not measured data. How deep the dip is depends on the model, the task and the context length; newer models show a smaller dip, which is exactly why you should measure yours (see the last concept on this page).

Contributing causes

• Attention sinks and recency bias. Many heads put much of their weight on the first tokens and the most recent ones, leaving less for the middle.
• Positional-encoding effects. RoPE-based models show decaying attention to distant relative offsets, and extended-context models inherit training-distribution biases toward short ranges. See Self-attention and positional structure.
• Training data. Instructions and salient content tend to sit at the start or end of training documents.

Example

A prompt contains 20 retrieved passages and one question. Passage 10 holds the answer.

Order A (as retrieved):   [p1 p2 p3 ... p10(answer) ... p20] + question   -> answer sits in the dip
Order B (after rerank):   [p10(answer) p3 p7 ... p15 p1]    + question   -> answer sits at the start

Both prompts contain identical information; Order B tends to be answered more reliably.

Mitigations

• Put the most important material at the start or end, not buried; put the actual question or instruction last.
• Rerank retrieved chunks so the best are at the edges.
• Keep retrieved context small and relevant: 5 good chunks beat 50 mediocre ones.
• For very long inputs, summarise or chunk-and-map (process each chunk separately, then combine) rather than one giant prompt.
• Test with a needle-in-haystack eval at your context sizes and positions.

There are a few separate effects, and it helps to keep them apart because they have different fixes.

1. Softmax is a partition of a fixed budget

Attention weights at a position sum to 1, because they come from softmax: for scores s_1 ... s_n the weight on position i is w_i = exp(s_i) / sum_j exp(s_j). Here s_i is the query-key score for position i (computed by the model, learned), n is the number of positions being attended over, and exp is the exponential function. See Self-attention and positional structure for where the scores come from.

With more tokens n, the average weight per token is 1/n, so it falls mechanically. A model can still focus sharply on one token if that token’s score is much larger than the rest. Let us compute how large. Suppose one relevant token has score gap g above all others, and the other n - 1 tokens all score 0. Then the relevant token’s weight is exp(g) / (exp(g) + n - 1):

import math

def weight_on_needle(n, gap):
    return math.exp(gap) / (math.exp(gap) + n - 1)

for n in (100, 10_000, 100_000):
    for gap in (5, 10):
        print(n, gap, round(weight_on_needle(n, gap), 4))
# 100     5   0.5999
# 100     10  0.9955
# 10000   5   0.0146
# 10000   10  0.6878
# 100000  5   0.0015
# 100000  10  0.1805

Reading the table: a gap of 5 gives the needle 60% of the attention among 100 tokens but under 2% among 10,000. The same gap of 10 that is enough at 100 tokens still leaves 69% at 10,000 and only 18% at 100,000. So “dilution” is real for weak signals: to keep the same focus, the model must produce larger score gaps as context grows. It is not an absolute loss of capability; well-trained heads do produce large gaps for tokens they are looking for. The toy case (all others equal at 0) is the best case; real distractors have scores above 0, which makes it worse.

2. Distraction and interference

More history means more plausible but wrong things to attend to, such as an earlier, superseded version of the same fact. Accuracy on a target task drops as you add irrelevant tokens even with the relevant content held fixed. This is the dominant practical effect, and it is worse when the distractors are similar to the target (same domain, same entities).

3. Position-encoding decay

With RoPE, attention scores tend to decay with relative distance, so content far back is intrinsically harder to pull forward than recent content: a recency bias baked into the architecture.

4. Out-of-distribution length

If the conversation exceeds the lengths the model saw in training, quality degrades for reasons beyond attention arithmetic: the whole forward pass is in a regime it saw little of.

Context rot

Context rot is the umbrella term: as input length grows, performance on the same task degrades, and not gracefully or uniformly. The model does not “forget” anything, since everything is still in context. What gets harder is reliably using the right part.

Effect	Fix
Softmax dilution of weak signals	Make the key fact strong and explicit; repeat it near the end; shorten context
Distraction	Remove irrelevant and superseded text
Positional decay	Put important content at the edges or near the question
Out-of-distribution length	Stay well below the window; compact

The same toolbox serves two related problems: quality degrading as history grows (context rot), and the hard limit — a conversation, or a RAG query whose prompt + retrieved chunks + history together risk exceeding the context window.

The techniques

• Truncation. Drop the oldest content once a budget is reached. Cheapest to implement, but it can silently lose important early context (the original goal, a constraint stated in turn 2), and it is biased toward whatever is most recent regardless of relevance. Pin the system prompt and the task statement so they are never the thing dropped.
• Sliding window. Keep only the last N turns verbatim, usually alongside a persistent, separately maintained summary of everything older: recency plus some retained long-term context.
• Compaction / summarisation. Periodically replace old turns with a compact structured summary (decisions made, facts established, open threads). It costs an extra call and risks dropping a detail, so summarise conservatively and keep recent turns verbatim.
• Externalise state. Keep the source of truth (task list, file contents, findings) in files or a store the agent re-reads on demand, rather than relying on it persisting in-context. Re-read right before acting.
• Retrieval over the history. Treat the transcript as a corpus and pull only the turns relevant to the current step.
• Sub-agents with fresh context. Delegate a bounded task to a separate context that starts clean, returns only its conclusion, and keeps intermediate tool output out of the main thread. See Multi-agent architectures in the AI agents topic.
• Prompt-cache-aware structure. Keep the stable prefix (system prompt, tool definitions, pinned context) fixed so it stays cached, and append rather than rewrite. Editing an early message invalidates the cache for everything after it.
• Put the current instruction last. Recency bias works in your favour if the actionable ask is at the end.
• Clear stale tool output. Once a 20k-token file read has been used, replace it with a stub (“[file contents elided; re-read if needed]”).

Trade-offs at a glance

Technique	Implementation cost	What it can lose
Truncation	trivial	anything old, regardless of importance
Sliding window + running summary	low	details the summary omitted
Compaction / summarisation	an extra LLM call per compaction	specific values a fluent summary glosses over
Externalised state	design work (files, store, re-read discipline)	little, if the agent actually re-reads
Retrieval over the history (RAG applied to the transcript)	highest: indexing, ranking, evaluation	turns the retriever fails to rank as relevant
Sub-agents	orchestration and result contracts	intermediate detail the sub-agent did not report

None is free: each trades implementation complexity against information loss differently, so name the cost you are accepting rather than defaulting to one technique.

A compaction implementation

def compact(messages, keep_last=6, summarise=None):
    """Replace everything except the last `keep_last` messages with one summary message."""
    head, tail = messages[:-keep_last], messages[-keep_last:]
    if not head:
        return messages
    summary = summarise(head)              # an LLM call in real use
    return [{"role": "user", "content": "Summary of earlier conversation:\n" + summary}] + tail

history = [{"role": "user", "content": "x" * 4000} for _ in range(200)]   # ~1,000 tokens each
print(len(history))                         # -> 200 messages, about 200,000 tokens
new = compact(history, summarise=lambda ms: f"({len(ms)} messages summarised)")
print(len(new))                             # -> 7  (1 summary + 6 recent)
print(new[0]["content"])                    # -> Summary of earlier conversation:\n(194 messages summarised)

The pattern: find the cut point, summarise the head, keep the tail verbatim. Keep the tail verbatim because the most recent turns contain the exact wording the next step depends on.

What a good summary contains

Summaries fail by being fluent but vague. Ask for structure:

Goal: migrate the billing service from Python 3.9 to 3.12
Decisions: keep Poetry; drop the py39 CI job; pin pydantic<3 until the invoice module is ported
Facts established: tests pass on 3.12 except tests/test_invoice.py (3 failures, datetime.utcnow removal)
Done: pyproject.toml updated, CI matrix updated
Open: fix 3 failing tests; update Dockerfile base image
Exact values to preserve: branch name `chore/py312`, ticket BILL-482

The last line matters: identifiers, paths and numbers are the details that fluent summaries drop and that you then cannot recover.

Compaction pitfalls

• A summary of a summary of a summary loses detail progressively. Where possible keep important facts in an external file and re-read it.
• Summarising right at the window limit may itself fail to fit. Compact early (see the 75% threshold above).
• Compaction changes the prefix, so the prompt cache is cold for the next call. This is a one-off cost.
• A summary can encode an earlier mistake as fact. Distinguish “verified” from “assumed” in the summary.

Every extra token is both a cost and a liability.

Cost

Because the model is stateless, each turn re-processes the history (prompt caching reduces the price of this but not the number of tokens attended over). With 9,000 fixed tokens (system prompt plus tools) and 1,500 new tokens per turn, the total input tokens processed over 50 turns is the sum over each turn t of 9000 + 1500 * t, which is 50 x 9,000 + 1,500 x (1 + 2 + … + 50) = 450,000 + 1,912,500 = 2,362,500 tokens, versus only 84,000 in the final prompt. Total work grows with the square of the number of turns.

Quality

It also dilutes weak signals, adds distractors (old, superseded decisions), pushes the actionable content further from the edges, and may run you toward the out-of-distribution tail of the model’s length training.

The alternative

A fresh conversation seeded with a tight summary of only what matters often does better than one carrying 50 turns of exploratory back-and-forth, and it is cheaper and faster.

Long thread (turn 50)                       Fresh start
---------------------------------------     ---------------------------------------
system + tools + 50 turns of exploring      system + tools
  (3 dead ends, 2 reversed decisions)       + 400-token brief: goal, decisions made,
+ the actual question                         constraints, what failed and why
                                            + the actual question
~84,000 tokens                              ~10,000 tokens

The judgement call is what to carry forward. Carry decisions, constraints, exact identifiers and why dead ends were dead; drop the exploration itself. The default of “just keep appending” is rarely optimal past a certain length.

It is strictly better in that you can fit more. It is a trap when it changes how you build.

Failure modes

• Stuffing instead of retrieving. Dumping an entire codebase or document set into the prompt when the relevant slice could be fetched: worse accuracy, higher cost, slower.
• Searching when you could look up. Relying on the model to find a needle you could have located deterministically (a grep, a SQL query, a key lookup).
• Assuming uniform recall across the window.
• Unbounded transcripts. Letting an agent history grow because “it fits”.

When a large window is genuinely the right tool

• The task is inherently global: summarising or reasoning across a whole long document, or cross-referencing many sections.
• The material is used once, not re-sent every turn, or is cached.
• Retrieval would be unreliable (no good index, highly interdependent content).

Rule of thumb

Does the task need the WHOLE text, or just the parts relevant to a question?
  whole text   -> long context (and test recall at your size)
  parts of it  -> retrieve / grep / query, then put only those parts in context

The window is a ceiling to stay well under, not a workspace to fill.

The advertised number tells you little. Build a task-representative eval.

Procedure

• Vary the needle position across the window (start, 25%, 50%, 75%, end) and plot accuracy. Expect a U-shape, or a flat line for a strong model.
• Vary total context length at each position to find where accuracy crosses your threshold (say 95%). That crossing is your effective length.
• Use realistic distractors: content from the same domain that is plausibly relevant, not random filler, since same-domain distraction is much harder.
• Test multi-fact and aggregation tasks (“list every place X is mentioned”, “reconcile these two sections”), which stress the model far more than single-fact retrieval and are closer to real workloads.
• Hold the model, tokenizer and prompt template fixed, and re-run when you change any of them.

A minimal harness

import random

def build_prompt(needle, filler_paragraphs, position_frac, question):
    """Insert the needle at a fractional position among filler paragraphs."""
    paras = list(filler_paragraphs)
    idx = int(position_frac * len(paras))
    paras.insert(idx, needle)
    return "\n\n".join(paras) + f"\n\nQuestion: {question}\nAnswer:"

def run_eval(ask_model, filler, lengths, positions, trials=20):
    results = {}
    for n in lengths:                              # number of filler paragraphs
        for pos in positions:                      # 0.0 = start, 1.0 = end
            correct = 0
            for t in range(trials):
                code = f"{random.randint(100000, 999999)}"   # new fact every trial
                needle = f"The access code for vault 7 is {code}."
                sample = random.sample(filler, n)
                prompt = build_prompt(needle, sample, pos, "What is the access code for vault 7?")
                correct += code in ask_model(prompt)
            results[(n, pos)] = correct / trials
    return results

# usage (ask_model calls your real model):
# r = run_eval(ask_model, filler, lengths=[50, 200, 800], positions=[0, .25, .5, .75, 1])
# for (n, pos), acc in sorted(r.items()): print(n, pos, f"{acc:.0%}")

Walk-through: the random code makes memorised answers impossible; the filler is sampled fresh each trial so one lucky layout does not dominate; positions and lengths form the grid. Twenty trials per cell is a small sample (the difference between 85% and 95% is within noise), so use more for decisions.

Reading the result

paragraphs  start  25%   50%   75%   end        (illustrative output, not real measurements)
50          100%   100%  100%  100%  100%
200         100%   95%   85%   95%   100%
800         95%    70%   45%   65%   90%

If your threshold is 90% everywhere, the effective length here is somewhere between 50 and 200 paragraphs, however large the advertised window. That is the number to architect around.

Limits of this test

A needle with an obviously out-of-place sentence is easier than real data, where the answer resembles its surroundings. Swap in your own documents and questions, and include a case where two similar facts conflict.