Model architecture: vectors, encoders, decoders

• Token — a chunk of text (a whole word, a word piece, a punctuation mark, a few bytes) that the model treats as one unit. Each token has an integer id.
• Vocabulary (vocab_size, V) — the fixed list of all tokens the model knows, e.g. 32,000 to 260,000 entries.
• Tokenizer — the program that converts a string into token ids and back. It is not part of the neural network; it is fixed before training.
• BPE (Byte-Pair Encoding) — a tokenizer-building algorithm: start from single characters/bytes and repeatedly merge the most frequent adjacent pair into a new token.
• Vector — an ordered list of numbers, e.g. [0.2, -1.3, 0.7]. Its length is its dimension.
• Embedding — a learned vector standing for a discrete thing (a token id). The embedding table has shape [V, d_model].
• d_model — the width of the model: how many numbers represent one position (e.g. 4096). Also called the hidden size.
• Hidden state / residual stream — the d_model-vector the model currently holds for a position. Every layer adds its output to it rather than replacing it.
• Layer / block — one repeated unit of the network: attention followed by an MLP, each with a residual connection and a normalisation. n_layers of them are stacked.
• Attention — the operation that lets a position read information from other positions. Details in Self-attention and positional structure.
• MLP (multi-layer perceptron, feed-forward network, FFN) — two or three matrix multiplications with a non-linearity between them, applied to each position independently.
• Residual connection — output = input + f(input); the layer only has to learn a correction.
• LayerNorm / RMSNorm — rescales a vector so its entries have a stable size (RMSNorm divides by the root-mean-square). Keeps training stable.
• Logit — a raw, unbounded score. The model emits one logit per vocabulary token.
• Softmax — turns a list of logits into probabilities that are positive and sum to 1.
• Cross-entropy loss — the training signal: minus the log of the probability the model gave to the correct next token.
• Chain rule (of probability) — a text’s probability is the product of each token’s probability given all earlier tokens: p(t1) * p(t2 | t1) * ....
• Nat — unit of information when logarithms are natural (base e); cross-entropy loss is usually reported in nats per token.
• Perplexity — exp(average cross-entropy per token); “as uncertain as choosing uniformly among this many tokens”. Lower is better.
• Teacher forcing — training with the true previous tokens as input at every position, so all next-token losses are computed in one parallel pass.
• Induction head — an attention pattern that completes “A B … A” with “B”; a mechanism behind in-context copying.
• Linear probe — a small linear classifier trained on hidden states to test whether they encode some property.
• Autoregressive — generating one token at a time, each conditioned on everything before it.
• Causal mask — forbids a position from attending to later positions.
• Bidirectional — every position may attend to every other position (left and right).
• MLM (masked language modelling) — hide some tokens, train the model to recover them from both sides of the context.
• Cross-attention — attention where queries come from one sequence (the decoder) and keys/values from another (the encoder output).
• Prefill / decode — processing the whole prompt in one pass / generating output one token at a time. See KV cache, prompt caching, and inference economics.
• Tied embeddings — reusing the input embedding table (transposed) as the output projection, so one [V, d_model] matrix serves both ends.
• GQA (grouped-query attention) — attention variant with fewer key/value heads than query heads; shrinks the K/V projections and the KV cache.
• FLOPs — floating-point operations, a measure of arithmetic work.

What it is

A hidden state is the vector of numbers the model currently holds for one token position at one point inside the network. It is called “hidden” because it is neither the input (the token id) nor the output (the predicted next token): it is the model’s internal working representation of “what I currently understand about this token, given everything I have read so far”.

• It always has the same width, d_model (for example 4,096 numbers per token in an 8B-class model).
• A sequence of n tokens has one hidden state per token, stacked into a matrix X of shape [n, d_model]. That is the X in the attention formulas (see Self-attention and positional structure).
• There is one such matrix after every layer, so a model with L layers produces L + 1 of them: the starting embeddings, then one output per layer.

How it flows through the model

token id for "cat"
   |   embedding lookup (row of the embedding table E)
   v
h0 = [ 0.12, -0.40,  0.03, ... ]      starting hidden state: only knows "I am the token cat"
   |   layer 1: attention (mix in other positions) + feed-forward (transform this position)
   v
h1 = [ 0.31,  0.05, -0.22, ... ]      same size, now also carries some context
   |   layer 2 ... layer L
   v
hL = [ -0.08, 0.77,  0.14, ... ]      final hidden state: what is needed to predict the NEXT token
   |   unembedding: multiply by a [d_model, vocab_size] matrix
   v
logits (one score per vocabulary token)  ->  softmax  ->  next-token probabilities

Each layer adds its result to the existing vector (h_next = h + layer_output) instead of replacing it, so information from earlier layers is carried forward. For this reason the hidden state is also called the residual stream (see Residual stream, normalisation and the transformer block). Think of it as a shared notepad that every layer reads from and writes a little more onto.

What it contains

Depth	Typical content (from interpretability studies; indicative, not exact)
Layer 0 (embedding)	Only the identity of the token, plus position information. “bank” is just “bank”.
Early layers	Local context, word pieces joined into words, simple syntax.
Middle layers	Meaning in context: “bank” has become the river kind or the money kind; “it” has picked up its referent; facts and relations about entities.
Last layers	What is needed to choose the next token: candidate continuations, formatting, style.

The individual numbers in the vector have no human-readable meaning. Information is spread across many dimensions (a distributed representation): a concept corresponds to a direction in the space, not to a single coordinate.

Seeing real hidden states

import torch
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModel.from_pretrained("gpt2")                      # 12 layers, d_model = 768

ids = tok("The cat sat on the mat", return_tensors="pt")       # 6 tokens
out = model(**ids, output_hidden_states=True)

print(len(out.hidden_states))            # 13  = embeddings + 12 layers
print(out.hidden_states[0].shape)        # torch.Size([1, 6, 768])   [batch, n, d_model]
print(out.hidden_states[-1].shape)       # torch.Size([1, 6, 768])   same shape at every depth

h = out.hidden_states[6][0, 1]           # hidden state of the token " cat" after layer 6
print(h.shape)                           # torch.Size([768])         one vector for one token

Walkthrough: the shape never changes from the first tensor to the last, which is the “constant d_model width” property. Indexing [0, 1] picks batch item 0, token position 1, giving one 768-number vector. (In Hugging Face’s GPT-2 the very last entry has the final layer norm applied, so it differs slightly from the raw output of layer 12.)

Why the distinction matters

The same word has different hidden states in different sentences, which is how the model handles ambiguity:

def vec(sentence, word_index, layer=8):
    ids = tok(sentence, return_tensors="pt")
    with torch.no_grad():
        return model(**ids, output_hidden_states=True).hidden_states[layer][0, word_index]

a = vec("I sat on the river bank", 5)       # " bank" (river sense)
b = vec("I deposited cash at the bank", 5)  # " bank" (money sense)
# The embedding-table row for " bank" is identical in both sentences, but a and b differ:
# the hidden state is *contextual*, the embedding is not.

Where hidden states are used

• Next-token prediction: the final hidden state of the last position, times the unembedding matrix, gives the logits (see Logits and the softmax).
• Embeddings for search: sentence embeddings are usually hidden states from a model’s last layer pooled (averaged, or taken from a special token) into one vector (see Embeddings and vector indexes).
• The KV cache: keys and values are linear projections of the hidden states, stored so they need not be recomputed (see KV cache and inference).
• Interpretability and probing: train a small classifier on hidden states at layer k to test whether that layer encodes some property (tense, sentiment, truthfulness); steering methods add a direction to the hidden state to change behaviour.
• Early exit and speculative decoding: some techniques read a prediction from an intermediate layer’s hidden state.

Common confusions

• Hidden state vs embedding. An embedding is the input vector for a token (the same every time). A hidden state is any vector inside the model and depends on the context. The embedding is just hidden state number 0.
• Hidden state vs “hidden layer”. Layers are the operations; hidden states are the vectors flowing between them.
• Hidden state vs KV cache. The cache holds K and V projections per layer, not the hidden states themselves.
• Hidden state in RNNs. Older recurrent networks use the term for a single vector carried from one time step to the next, summarising the whole past. In a transformer there is one hidden state per position and per layer, and attention (not recurrence) moves information between positions.
• Not interpretable per coordinate. Do not read “dimension 211 is the sentiment dimension”; features are directions spread over many dimensions.

What an embedding is, from first principles

A computer cannot do arithmetic on the word “cat”, but it can do arithmetic on a list of numbers. An embedding assigns every token id a vector (a fixed-length list of floats). The values are learned: they start random, and training nudges them so that tokens used in similar ways end up with similar vectors. “Similar” is measured geometrically: by the distance between vectors, or by the cosine similarity cos(a, b) = (a . b) / (|a| |b|) (the dot product a . b is “multiply entry by entry and add”; |a| is the vector’s length). Cosine is 1 when two vectors point the same way, 0 when they are unrelated (perpendicular), -1 when opposite.

Tiny made-up example with d_model = 3 (real models use thousands of dimensions; these numbers are chosen, not learned):

import numpy as np

vocab = ["the", "cat", "sat", "mat"]
E = np.array([[1.0, 0.0, 0.5],    # row 0 = "the"
              [0.0, 1.0, 0.5],    # row 1 = "cat"
              [0.2, 0.8, 0.0],    # row 2 = "sat"
              [0.0, 0.9, 0.4]])   # row 3 = "mat"   -> E has shape [4, 3] = [vocab_size, d_model]

ids = [0, 1, 2]                   # "the cat sat" after tokenization
X = E[ids]                        # lookup = just select rows
print(X.shape)                    # -> (3, 3)  = [n_positions, d_model]

def cos(a, b):
    return a @ b / np.linalg.norm(a) / np.linalg.norm(b)

print(round(cos(E[1], E[3]), 3))  # cat vs mat -> 0.999 (both "noun-like" in this toy)
print(round(cos(E[0], E[3]), 3))  # the vs mat -> 0.182 (function word vs noun)

Walkthrough: the “lookup” is nothing more than indexing rows of a matrix; it is mathematically the same as multiplying a one-hot vector by E. The table has vocab_size * d_model learned numbers. After training, geometric closeness reflects usage similarity, not dictionary definitions.

Where vectors appear across the stack

Vectors show up at several distinct places and it is worth not conflating them:

 text -> token ids -> [ E lookup ] -> X0 --> layer 1 --> X1 --> ... --> layer N --> XN -> [ unembed ] -> logits
                       [V, d_model]   [n, d_model]                                [n, d_model]        [n, V]
        (input embeddings)           (hidden states / residual stream)                               (one score per vocab token)

• Token embeddings — the model has a lookup table of shape [vocab_size, d_model]; each token id indexes one row. This is the model’s input representation.
• Hidden states / residual stream — after each transformer layer, every position holds a d_model-dimensional vector that has been progressively rewritten by attention and MLP blocks. The same shape as an embedding, but its content is a contextual, task-specific summary.
• Output / unembedding — the final hidden state at a position is projected back to [vocab_size] logits, one score per possible next token. If the embeddings are tied, this projection is h @ E.T: the score for token t is the dot product of the hidden state with t’s embedding, i.e. “how much does the final state look like token t?”.
• Sentence / document embeddings — a separate class of model (or a pooling of a transformer’s hidden states) that produces one vector for a whole span of text, used for retrieval and clustering. A generative decoder LLM is not automatically a good embedding model; embedding models are usually trained with a contrastive objective specifically for that.

Tied unembedding with the toy table above:

h = X[-1]               # pretend the final hidden state at the last position equals E["sat"]
logits = h @ E.T        # shape [4]: one score per vocab token
print(logits.round(2))  # -> [0.2  0.8  0.68 0.72]   (a real model's h is not simply an embedding)

A common confusion is treating “the embeddings” as a single thing — the input lookup table and a retrieval model’s output vectors are trained differently, live in different spaces, and are not interchangeable.

All three are stacks of the same transformer block; the differences are the attention mask and the training objective. The mask is a grid saying “may row i (the position asking) look at column j (the position being looked at)?”:

Bidirectional (encoder)        Causal (decoder)              Encoder-decoder
  looks at:  1 2 3 4             looks at:  1 2 3 4           decoder self-attn: causal
 pos 1       x x x x            pos 1       x . . .           decoder cross-attn: every decoder
 pos 2       x x x x            pos 2       x x . .             position -> ALL encoder positions
 pos 3       x x x x            pos 3       x x x .           encoder self-attn: bidirectional
 pos 4       x x x x            pos 4       x x x x

import numpy as np
print(np.tril(np.ones((4, 4), int)))   # causal mask: 1 = allowed
# [[1 0 0 0]
#  [1 1 0 0]
#  [1 1 1 0]
#  [1 1 1 1]]

The three families

• Encoder-only (BERT, RoBERTa). Every token attends to every other token in both directions (bidirectional, no causal mask). Trained with masked-language-modelling: hide ~15% of tokens, predict them from both-side context. Example training item: The [MASK] sat on the mat -> predict cat. Produces contextual representations of an existing sequence. Natural fit: classification, span extraction, retrieval/embeddings, token tagging. Cannot generate text autoregressively in any natural way.
• Encoder-decoder (the original Transformer, T5, BART). An encoder builds a bidirectional representation of the input; a decoder generates output autoregressively (causal self-attention) while also cross-attending to the encoder’s output. Natural fit: sequence-to-sequence tasks with a clear input/output split — translation, summarization. Example: input "translate English to German: the cat sat" is read once by the encoder; the decoder emits "die Katze sass" token by token, looking back at the encoder output at each step.
• Decoder-only (GPT, Llama, Claude). One stack, causal mask everywhere, trained on next-token prediction over the concatenation of everything. “Input” and “output” are not architecturally distinct — the prompt is just the prefix you did not sample.

How to choose

Task	Good fit	Why
Classify 1M support tickets cheaply	encoder-only	Reads the whole text both ways once; small and fast; no generation needed
Semantic search embeddings	encoder-only (contrastively trained)	One vector per text from bidirectional context
Fixed-format translation at scale	encoder-decoder (or a decoder)	Clear source/target split; input encoded once
Chat, code, agents, arbitrary instructions	decoder-only	One interface for every task; best scaling record

Gotcha: “decoder-only” models can be used for classification or embeddings (prompt them, read a logit or pool hidden states), but a small purpose-trained encoder is usually cheaper and often as accurate for those narrow jobs.

Several forces pushed the same direction:

• Task unification. Once you frame every task as “continue this text” (T5 also argued this, but decoder-only takes it furthest), you do not need an architectural input/output boundary. In-context learning, few-shot prompting, and instruction following all fall out of one objective.
• Training efficiency / density of signal. A decoder-only model gets a next-token prediction loss at every position of every training sequence. Masked LM only gets signal on the ~15% of masked positions. Per FLOP of training, decoder-only extracts more supervision.
• Simplicity of scaling. One stack, one objective, one mask. No separate encoder to size, no cross-attention to balance. The scaling-law work that mapped loss vs. parameters/data/compute was done on decoder-only and the recipe is clean.
• KV cache friendliness for serving. Causal attention means past keys/values never change as you generate, so they cache perfectly (see KV cache, prompt caching, and inference economics). Encoder-decoder still caches the decoder, but the mental model and the tooling matured around decoder-only.

Worked example of “density of signal”

Take a 1,000-token training sequence.

Masked LM (15% masked):    ~150 positions produce a loss    -> 150 supervised predictions
Causal LM (next token):    999 positions produce a loss      -> 999 supervised predictions

For the same forward pass, the causal objective supplies roughly 6-7x more training signal (999/150 is about 6.7). One forward pass with causal masking evaluates all of the predictions at once, because position i only sees 1..i and is asked for token i+1: this is why the training loop is “teacher-forced” and parallel, while generation is sequential.

Encoder-decoder is not dead — it remains competitive for narrow seq2seq — but for a model meant to do arbitrary tasks from a prompt, decoder-only is the simpler thing that scaled.

This is the prerequisite for the pipeline and the parameter count below. One decoder block, applied identically (with different weights) n_layers times:

x  (hidden state, [n, d_model])
|
+--> RMSNorm --> causal self-attention --> (+) --> x1 = x  + attn(norm(x))
|                                           ^
+-------------------------------------------+   (residual: add the input back)

x1
|
+--> RMSNorm --> MLP (up-project, nonlinearity, down-project) --> (+) --> x2 = x1 + mlp(norm(x1))
|                                                                  ^
+------------------------------------------------------------------+

• Residual connection. x_out = x + f(x). The layer does not have to reproduce its input, only learn a delta. Gradients also flow straight through the +, which is what makes 80-layer networks trainable. The running sum is called the residual stream; every layer reads from it and writes into it.
• Normalisation. Without it, activations can drift to huge or tiny magnitudes layer after layer. RMSNorm computes x / sqrt(mean(x^2) + eps) * g, where g is a learned per-dimension gain (shape [d_model]) and eps is a tiny constant to avoid dividing by zero. Most current models put the norm before each sub-block (“pre-norm”) because it trains more stably than the original post-norm layout.
• Attention sub-block. Lets positions exchange information (the only place where different positions interact). See Self-attention and positional structure.
• MLP sub-block. Processes each position independently: expand from d_model to a larger d_ff (about 4x, or about 2.7-3.5x in gated variants), apply a non-linearity, project back. Interpretability work suggests much of the model’s stored factual knowledge lives here; treat that as a useful heuristic, not a proven fact for every model.

Gotcha: the stream width d_model is constant from the first layer to the last. That is why embeddings, every hidden state, and the unembedding input all share one shape, and why the parameter count formula below is so regular.

1. Tokenization. A byte-pair-encoding (or unigram) tokenizer segments the string into subword tokens and maps each to an integer id. The vocabulary is typically 30k–260k. Whitespace and case are usually encoded into the tokens themselves.
2. Embedding lookup + positional information. Each id indexes the embedding table to get a d_model vector. Position is injected either by adding positional embeddings or, in modern models, by rotating query/key vectors inside attention (RoPE).
3. N transformer layers. Each layer: causal multi-head self-attention (every position mixes in information from earlier positions), then a position-wise MLP, each wrapped in a residual connection and normalization. Keys and values for every position are written to the KV cache.
4. Final norm + unembedding. The last layer’s hidden state at the last position is projected to vocab_size logits.
5. Sampling. Logits -> (optionally temperature-scaled) -> softmax -> probability distribution -> truncation (top-k / top-p) -> draw one token.
6. That token is appended; steps 2–5 repeat for the next position, but now only the new token is run through the stack (everything else is cached). This is the prefill -> decode split.

Shapes at each step (prompt of n tokens)

string "the cat sat"
 -> ids                [n]                    e.g. [464, 3797, 3332]
 -> X0 = E[ids]        [n, d_model]
 -> N blocks           [n, d_model]           (shape never changes)
 -> final norm         [n, d_model]
 -> take last row h    [d_model]              (only the last position predicts the next token in generation)
 -> logits = h @ U     [vocab_size]           U is [d_model, vocab_size]; U = E.T when tied
 -> softmax            [vocab_size]           sums to 1
 -> sample             one token id

Sampling in code

import numpy as np

def softmax(z, T=1.0):
    z = np.array(z) / T
    z = z - z.max()                 # shift for numerical stability (does not change the result)
    e = np.exp(z)
    return e / e.sum()

def top_k(p, k):
    keep = np.argsort(p)[::-1][:k]
    m = np.zeros_like(p); m[keep] = p[keep]
    return m / m.sum()              # renormalise after truncation

def top_p(p, P):
    order = np.argsort(p)[::-1]
    c = np.cumsum(p[order])
    n = int(np.searchsorted(c, P)) + 1   # smallest set whose mass reaches P
    m = np.zeros_like(p); m[order[:n]] = p[order[:n]]
    return m / m.sum()

p = softmax([2.0, 1.0, 0.1, -1.0])
print(p.round(3))                   # -> [0.638 0.235 0.095 0.032]
print(top_k(p, 2).round(3))         # -> [0.731 0.269 0.    0.   ]
print(top_p(p, 0.9).round(3))       # -> [0.659 0.242 0.099 0.   ]  (0.638+0.235=0.873 < 0.9, so a third token is kept)

Walkthrough: top_k keeps the k most likely tokens and rescales (0.638/(0.638+0.235) = 0.731). top_p (“nucleus”) keeps the smallest set of top tokens whose cumulative probability reaches P; the set size adapts to how confident the model is. A final random draw from the surviving probabilities picks the token. Temperature is covered in more depth in Decoding parameters: temperature, top-p, top-k.

Because the model never sees characters — it sees token ids — and the token boundaries are an artifact of a compression algorithm run on a training corpus, not of meaning.

• Arithmetic and digit tasks suffer when numbers tokenize inconsistently (1234 might be one token, 1235 three). Many models now force digit-level tokenization to fix this.
• Character-level tasks (“how many r’s in strawberry”, reversing a string) are genuinely hard because the relevant unit is invisible to the model.
• Rare languages / code / whitespace-significant formats get shredded into many tokens, wasting context and degrading quality.
• Prompt-boundary artifacts. If your prompt ends mid-word relative to how the tokenizer would segment the continuation, you can get “token healing” issues — the model’s natural next token conflicts with your partial token.
• Cost and context accounting are in tokens, not words or characters, and the ratio varies by language and content type (roughly 0.75 words/token for English prose, far worse for JSON or CJK).

Illustrative example: why “strawberry” is hard

A tokenizer might split it as str | aw | berry (illustrative; the exact split differs by tokenizer). The model receives three opaque ids. To count the letter r it must have memorised, per token, how many rs each id contains, and add them up. Spelling the word out first (s-t-r-a-w-b-e-r-r-y) makes each letter its own token and the task becomes easy. This is the same reason “write it out step by step” helps with character-level tasks.

Measuring it yourself

# pip install tiktoken      (a real tokenizer library; encoding names are stable)
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
for s in ["hello world", "1234567", "  indented", '{"a": 1}']:
    ids = enc.encode(s)
    print(repr(s), len(ids), [enc.decode([i]) for i in ids])
# The exact splits depend on the encoding: run it and compare prose, digits, JSON and whitespace.

Run it on your own prompts: len(enc.encode(text)) is the number you are billed for and that fills the context window. Different model families use different tokenizers, so token counts are not portable between providers.

The problem

Words as tokens give a huge vocabulary and cannot handle unseen words (“unfathomable-ish”). Single characters give a tiny vocabulary but very long sequences (expensive, since attention cost grows with length). Subwords are the compromise: frequent words stay whole, rare words are assembled from pieces.

Algorithm (BPE, training time)

1. Start with a vocabulary of single characters (or raw bytes, which guarantees any string can be encoded).
2. Count every adjacent pair of tokens in the training corpus.
3. Merge the most frequent pair into a new token; add it to the vocabulary and record the merge.
4. Repeat until the vocabulary reaches the target size.

At encode time the same merges are replayed, in the order they were learned, on the new string.

Worked example

Corpus: low x5, lower x2, newest x6, widest x3. </w> marks end of word.

import collections

def bpe(words, merges):
    corpus = collections.Counter(tuple(w) + ("</w>",) for w in words.split())
    for i in range(merges):
        pairs = collections.Counter()
        for w, c in corpus.items():
            for a, b in zip(w, w[1:]):
                pairs[(a, b)] += c
        best = max(pairs.items(), key=lambda kv: (kv[1], kv[0]))[0]
        print(i + 1, best, pairs[best])
        new = collections.Counter()
        for w, c in corpus.items():
            out, j = [], 0
            while j < len(w):
                if j < len(w) - 1 and (w[j], w[j + 1]) == best:
                    out.append(w[j] + w[j + 1]); j += 2
                else:
                    out.append(w[j]); j += 1
            new[tuple(out)] += c
        corpus = new
    return corpus

text = "low " * 5 + "lower " * 2 + "newest " * 6 + "widest " * 3
print(bpe(text, 6))
# 1 ('t', '</w>') 9
# 2 ('s', 't</w>') 9
# 3 ('e', 'st</w>') 9
# 4 ('o', 'w') 7
# 5 ('l', 'ow') 7
# 6 ('w', 'est</w>') 6
# Counter({('n','e','west</w>'): 6, ('low','</w>'): 5, ('w','i','d','est</w>'): 3, ('low','e','r','</w>'): 2})

Walkthrough: t</w> occurs 9 times (6 in “newest” + 3 in “widest”), so it merges first; then st</w>, then est</w> follow. After 6 merges low is a unit and est</w> is a unit (a reusable suffix), whereas rare “widest” is still partly spelled out. If you now encode the unseen word lowest, the learned merges give low | est</w> — a sensible split built from pieces never seen together. (Ties are broken arbitrarily by the implementation; real tokenizers add details such as byte-level alphabets and regex pre-splitting.)

Consequences you can use

• Vocabulary size is a trade-off: bigger vocab -> fewer tokens per text (cheaper attention, cheaper context) but a bigger embedding table (V * d_model) and rarer tokens that are poorly trained.
• Whitespace is usually fused into tokens (" cat" and "cat" are different ids). Trailing-space prompts therefore tokenize differently from what the model saw in training.
• Changing the tokenizer means retraining the model; the embedding table is indexed by these ids.

Logits are the raw, unnormalized scores the model outputs for each vocabulary token at a position — any real number, no constraint. The softmax turns a vector of logits z into a probability distribution:

p_i = exp(z_i) / sum_j exp(z_j)

Every symbol

• z — the logit vector, shape [vocab_size]. Produced by the model (h @ U); not chosen by you.
• z_i — the logit of vocabulary token i.
• exp(.) — e^x with e ~ 2.718. It maps any real number to a positive one, and amplifies differences.
• sum_j — sum over all vocabulary tokens j; it makes the outputs add to 1.
• p_i — the resulting probability of token i, in (0, 1).

Nothing here is learned: softmax has no parameters. What is learned is the network that produces z.

Numeric example

import math
z = [2.0, 1.0, 0.1, -1.0]
e = [math.exp(x) for x in z]        # [7.389, 2.718, 1.105, 0.368]
s = sum(e)                          # 11.58
p = [x / s for x in e]
print([round(x, 3) for x in p])     # -> [0.638, 0.235, 0.095, 0.032]
print(-math.log(p[0]))              # -> 0.449  (cross-entropy loss if token 0 was the correct answer)

Properties that matter: it is shift-invariant (softmax(z) == softmax(z + c): adding 10 to every logit above still gives [0.638, 0.235, 0.095, 0.032], which is why implementations subtract max(z) for numerical stability), it is monotonic (higher logit -> higher probability), and the gaps between logits, not their absolute values, determine the distribution’s sharpness. The model is trained with cross-entropy loss against this distribution, so the logits are calibrated to be interpreted through softmax — reading them any other way (e.g. raw magnitude) is not meaningful.

Temperature

Temperature T is literally a divisor on z before this step: p_i = exp(z_i / T) / sum_j exp(z_j / T). T is chosen by you at inference, not learned.

T = 0.5 : [0.862, 0.117, 0.019, 0.002]   (sharper: the top token dominates)
T = 1.0 : [0.638, 0.235, 0.095, 0.032]   (the model's own distribution)
T = 2.0 : [0.451, 0.274, 0.174, 0.101]   (flatter: more random)
T -> 0  : always the highest logit ("greedy")

Pitfalls

• Overflow: exp(1000) is infinity in float32; always subtract the max first.
• Low probability is not “wrong”: a 3% token is still sampled about once in 33 draws, so sampled output varies between runs unless T = 0.
• Softmax probabilities are not guaranteed to be well-calibrated truth probabilities; they are “how plausible is this continuation”, which is different from “how likely is this statement to be true”.

The phrase is both accurate and misleading. This concept separates what the training objective is (precisely defined, and exactly “predict the next token”) from what the network has to become to do that well. We build it up in order: the chain rule, the training loss, a worked numeric example, an experiment showing why predicting well can require more than memorising, and what that implies in practice.

Part 1 - the steelman: what “just” gets right

A text is a sequence of tokens t1, t2, ..., tn. The chain rule of probability says its probability is a product of conditionals:

p(t1, t2, ..., tn) = p(t1) * p(t2 | t1) * p(t3 | t1, t2) * ... * p(tn | t1 ... t(n-1))

Symbols: t_i is the token at position i; p(t3 | t1, t2) reads “the probability of t3 given that t1 and t2 came before”; the whole left side is the probability the model assigns to the entire text. The network’s single job is to estimate each factor: given a prefix, output a distribution over the vocabulary (the softmax output, see Logits and the softmax — definition and equation). Generation samples from that distribution, appends the token, and repeats. There is no separate planning module, goal register or world-model component wired in by hand; everything is a conditional distribution over the next token.

Training signal. For each position the loss is -log p(correct next token) (cross-entropy). Averaged over positions it is the number the optimiser lowers. It is computed in parallel for all positions of a training sequence at once (teacher forcing, see “Worked example of density of signal” above).

Worked numeric example

Five tokens of a sentence, with illustrative probabilities the model assigned to each actual next token:

import math
ps = [0.04, 0.01, 0.20, 0.30, 0.10]          # p(t_i | t_1..t_(i-1)) for the true tokens (illustrative)

total = math.prod(ps)                         # chain rule: probability of the whole text
nll = -sum(math.log(p) for p in ps)           # total negative log-likelihood (nats)
per_tok = nll / len(ps)                       # average cross-entropy per token
print(total, round(nll, 2), round(per_tok, 3), round(math.exp(per_tok), 2))
# 2.4e-06 12.94 2.588 13.3

Reading it: the model gave the whole text a probability of 0.0000024; the average loss is 2.59 nats per token; exp(2.588) = 13.3 is the perplexity, meaning “the model was as uncertain as if choosing uniformly among about 13 tokens at each step”. Lower is better. The second token (probability 0.01) contributes -ln 0.01 = 4.6 nats, far more than the fourth (-ln 0.30 = 1.2): surprising tokens dominate the loss, which is exactly where the model has most to learn.

Part 2 - the complication: loss pressure builds machinery

Minimising this loss on a rich corpus rewards whatever internal computation lowers it. A lookup table of memorised text lowers it for sentences it has seen; for sentences it has not seen, only structure helps. A runnable experiment makes the point with the smallest possible example: addition facts a+b= with a, b in 0…9, a model trained on 70 of the 100 pairs and tested on the 30 held-out pairs.

import math, random, collections
random.seed(0)
pairs = [(a, b) for a in range(10) for b in range(10)]
random.shuffle(pairs)
train, test = pairs[:70], pairs[70:]

answers = collections.Counter(a + b for a, b in train)       # how often each sum appeared in training
seen = {(a, b): a + b for a, b in train}                     # the memoriser's lookup table

def memoriser_loss(a, b):
    if (a, b) in seen:                                       # exact context seen: near-certain, correct
        return -math.log(0.99)
    return -math.log((answers[a + b] + 1) / (len(train) + 19))   # unseen: fall back to overall frequency

def rule_loss(a, b):                                         # a model that learned the addition rule
    return -math.log(0.99)

mean = lambda f, data: sum(f(a, b) for a, b in data) / len(data)
print("uniform guess over 19 sums:", round(math.log(19), 3))
print("memoriser   train/test:", round(mean(memoriser_loss, train), 3), round(mean(memoriser_loss, test), 3))
print("rule-based  train/test:", round(mean(rule_loss, train), 3), round(mean(rule_loss, test), 3))
# uniform guess over 19 sums: 2.944
# memoriser   train/test: 0.01 2.86
# rule-based  train/test: 0.01 0.01

Walk-through: both models have the same (tiny) loss on training data, but on held-out pairs the memoriser’s loss is 2.86 nats, barely better than the uniform guess (2.94), while a model that implements addition keeps 0.01. The objective never says “learn arithmetic”; it says “lower the loss on text”, and on unseen text only the model that built the rule wins. Scale that to a corpus of code, proofs, stories and dialogue, and the pressure is to build internal representations of whatever generates that text: syntax and scope for code, entity tracking for stories, procedures for arithmetic. (This is a toy; it shows the direction of pressure, not what a real model necessarily does for any given task.)

Evidence from inside models

Findings in the interpretability literature, each with its limits:

finding	what it shows	caution
induction heads	attention circuits that copy “after A B … A, predict B”; underlie in-context pattern completion	found in small and large models; how much of in-context learning they explain is debated
board-state probes in game models (Othello-GPT)	a model trained only on move sequences develops a linear-probe-readable representation of the board	one controlled setting, not proof for every capability
truthfulness / refusal directions	a direction in hidden-state space correlates with true vs false statements, or with refusing	correlation; steering along it only partly works
limited planning	some evidence that representations of upcoming tokens (e.g. a rhyme word) form before the line is written	active research, narrow tasks

So “just predicting” is true of the objective and the interface (tokens in, a distribution out) and misleading about the internal computation, which is whatever minimising that loss at scale required. Post-training (SFT, RLHF, RLVR; see Training and post-training) then reshapes the distribution further, away from pure imitation of web text towards helpful, safe and verified behaviour; it still outputs a next-token distribution.

A concrete case

Prompt: The capital of Australia is. A model that only memorised surface co-occurrence favours Sydney (it appears near “Australia” far more often). To drive the loss down on millions of such sentences the model has to represent “capital-of” as a relation and prefer Canberra. Nobody wrote a capital-of module; the objective stayed “predict the token” and the solution required knowledge. Conversely, when the fact is missing the distribution still has to put mass somewhere, and a fluent wrong continuation scores well on plausibility: this is where hallucination comes from.

Practical takeaways

• Prompt wording matters because you are changing the conditioning context of a conditional distribution.
• Hallucination follows from the same mechanism: plausible-sounding continuations are rewarded even where knowledge is absent (see Safety, prompt injection, and failure modes).
• Do not infer limits from the objective. “It only parrots” is not licensed by “it predicts tokens”; capability is judged by behaviour on held-out tasks, as the experiment above does.
• Do not infer abilities from fluency. A fluent explanation of why it did something is itself just more next-token prediction, and not guaranteed to describe the actual computation (see Reasoning models and test-time compute).
• Generation is sequential, training is parallel. Training evaluates all positions at once with the causal mask; generation must run one token at a time because each token’s input depends on the previous sample.

For a standard decoder-only model the parameter count is dominated by, per layer:

• Attention: four d_model x d_model projection matrices (Q, K, V, output) — 4 * d_model^2 (less if grouped-query attention shrinks K and V).
• MLP: two matrices, typically d_model x 4*d_model and back — 8 * d_model^2.

So roughly 12 * d_model^2 per layer, times n_layers, and the MLP is about two-thirds of it. Then the token embedding table (vocab_size * d_model) and the unembedding (often tied to it). LayerNorm/RMSNorm gains and biases are negligible in count. This is why “parameters” and “FLOPs per token” both scale as roughly ~2 * N for a forward pass (N = param count): each weight is used in one multiply-accumulate per token (a multiply and an add = 2 FLOPs). It also explains why context length does not change the parameter count — attention over a longer sequence is more compute and more KV-cache memory, but the same weights.

Symbols

• d_model — hidden width (a design choice made before training, e.g. 4096).
• n_layers — number of blocks (design choice, e.g. 32).
• vocab_size (V) — fixed by the tokenizer.
• d_ff — MLP inner width (design choice).
• n_heads, n_kv_heads, d_head — attention heads; d_head = d_model / n_heads. With GQA, K and V projections are d_model x (n_kv_heads * d_head).
• N — total parameter count, the result of those choices.

Worked count for a Llama-3-8B-like configuration

Gated MLP (three matrices: gate, up, down) as in modern models, so MLP = 3 * d_model * d_ff; the 12*d^2 rule above is the simpler classic (non-gated, d_ff = 4 d_model) approximation.

def params(d, L, V, dff, n_heads, n_kv, gated=True, tied=False):
    dh = d // n_heads
    attn = 2 * d * d + 2 * d * n_kv * dh      # Q and O are d x d; K and V are d x (n_kv*dh)
    mlp = (3 if gated else 2) * d * dff
    emb = V * d * (1 if tied else 2)          # input table (+ separate unembedding if untied)
    per_layer = attn + mlp
    return attn, mlp, L * per_layer, emb, L * per_layer + emb

print(params(4096, 32, 128256, 14336, 32, 8))
# -> (41943040, 176160768, 6979321856, 1050673152, 8029995008)
#    attn/layer  mlp/layer   32 layers   embeddings   TOTAL ~ 8.03 B
print(params(8192, 80, 128256, 28672, 64, 8))
# -> total 70,552,387,584  (~70.6 B)
print(params(768, 12, 50257, 3072, 12, 12, gated=False, tied=True))
# -> total 123,532,032  (~124 M, GPT-2-small-like shape)

Reading the 8B result: per layer, attention is about 42M parameters and the MLP about 176M, so the MLP is roughly 80% of a layer’s weights here (GQA shrinks attention, and the gated MLP is wide). The 32 layers hold 6.98B; embeddings add 1.05B (two separate 128256 x 4096 tables), giving about 8.03B. For the plain 12 * d^2 rule at d = 4096, 32 layers: 12 * 4096^2 * 32 = 6.44B — within about 8% of the layer total, good enough for back-of-envelope sizing.

What this tells you

• Memory for weights = N * bytes_per_param (about 16 GB for 8B in bf16). See VRAM and running LLMs locally for the full accounting.
• Compute per generated token is about 2N FLOPs (the ~2 comes from multiply + add); this is why a 70B model is roughly 9x slower per token than an 8B one on the same hardware.
• For small models the embedding table is a large share (for a 1B model with V = 128k, d = 2048 it is about 0.26B per table, over a quarter of the model); for large models it is a rounding error. This is why small models often tie the embeddings.
• MoE models change this arithmetic (total vs active parameters); see Mixture of experts and architecture families.