Training and post-training

• Token — a chunk of text (a word piece) with an integer id; the unit the model reads and writes.
• Logit — a raw, un-normalised score the model outputs for each possible next token, before softmax.
• Softmax — turns a list of logits into probabilities that are positive and sum to 1.
• Cross-entropy loss — -log(p) where p is the probability the model gave to the correct token; small when the model was confident and right.
• Perplexity — exp(average cross-entropy); roughly “how many equally likely tokens the model is choosing between at each step”.
• Base model — the output of pretraining: good at continuing text, not yet a helpful assistant.
• SFT (Supervised Fine-Tuning) — training on curated (instruction, ideal response) pairs with the same next-token loss.
• RLHF (Reinforcement Learning from Human Feedback) — optimising the model against a learned score that imitates human preferences.
• DPO (Direct Preference Optimization) — a way to use preference pairs without a separate reward model or RL loop.
• RLAIF — RLHF where an AI model, not a human, supplies the preference labels.
• Policy (pi) — the model being trained, seen as a function that gives a probability to every possible response y for a prompt x. Written pi(y|x).
• Reference policy (pi_ref) — a frozen copy of the model from before RL/DPO started (usually the SFT model); the anchor the new policy must not drift too far from.
• Reward model (RM) — a network that reads (prompt, response) and outputs one number, higher meaning “humans would prefer this”.
• Reward (r) — that number. Reward hacking — exploiting flaws in the reward model to get a high score without actually being better.
• Preference pair — (x, y_win, y_lose): a prompt, the response a rater preferred (y_win, also called “chosen”) and the other one (y_lose, “rejected”).
• Bradley–Terry model — a standard statistical model of pairwise comparisons: the chance that A beats B depends on the difference of their scores.
• Sigmoid (sigmoid(z)) — 1 / (1 + e^(-z)); squashes any number into (0, 1). sigmoid(0) = 0.5.
• KL divergence (KL(P || Q)) — a non-negative measure of how different distribution P is from Q; zero only when they are identical.
• beta — the strength of the KL anchor (a hyper-parameter you choose, often 0.01 to 0.5).
• PPO (Proximal Policy Optimization) — the reinforcement-learning algorithm classically used in RLHF; it limits how much each update can change the policy.
• Entropy — a measure of how spread out a probability distribution is; high entropy means varied output, low entropy means predictable output.
• Temperature — a sampling setting that divides logits before softmax; higher means flatter distribution and more varied output.
• Catastrophic forgetting — a model losing earlier capability while learning something new.
• Constitution — a written list of principles used to guide AI-generated critiques and preference labels.

A modern chat model is built in stages. Each stage uses a different kind of data and a different training signal.

 raw text (trillions of tokens)          (instruction, ideal answer) pairs        (prompt, chosen, rejected) triples
          |                                         |                                         |
          v                                         v                                         v
  +---------------+   base model   +---------+   SFT model   +-------------------+   aligned model
  |  PRETRAINING  | -------------> |   SFT   | ------------> |   RLHF  or  DPO   | ------------->  ship
  | next-token    |                | next-token on answers   | optimise toward                       
  | prediction    |                | only                    | human preference                      
  +---------------+                +---------+               +-------------------+
   knowledge + language            format + persona           tone, helpfulness, refusal calibration

• Pretraining — next-token prediction over a huge, broad text corpus. This is where almost all factual and linguistic capability comes from; it produces a base model that completes text plausibly but has no notion of “answer the user helpfully” — it will happily continue a question with more questions, because that is what raw text does.
• Supervised fine-tuning (SFT) — further trains on curated (instruction, response) pairs. This teaches the format: respond to instructions, stop at the right point, adopt an assistant persona. It reshapes behavior far more than it adds knowledge.
• RLHF / DPO — further optimizes the SFT model against a signal of human preference between candidate responses, rather than imitating a single fixed target. This is what pushes quality beyond “plausible imitation of the SFT data” toward “what humans actually rated as better” — tone, helpfulness, refusal calibration, avoiding subtly bad-but-plausible answers.

Each stage operates on the output of the previous one; skipping SFT and going straight to RLHF on a base model does not work well in practice because there is no reasonable initial policy to rate.

The same prompt through the three models (illustrative)

Prompt:  "What is the capital of France?"

Base model:      "What is the capital of Germany? What is the capital of Spain? ..."
                 (continues the *document*, which looks like a quiz sheet)

SFT model:       "The capital of France is Paris."
                 (answers, then stops)

After RLHF/DPO:  "The capital of France is Paris. Is there anything else you'd like to know
                  about France?"   (style, tone and length shaped by what raters preferred)

Why preference data at all, if SFT already exists?

SFT needs someone to write the ideal answer, which is slow and hard for subjective qualities (“be concise but warm”). Preference data only needs someone to choose between two answers the model already produced. Ranking is easier, faster and more consistent than writing, and it directly targets “which of the things this model would say is better”, including errors that a single demonstration never shows the model.

This is the training signal for the first two stages, and the baseline everything else is compared to.

What it is

At every position the model outputs one logit per vocabulary token; softmax turns them into a probability distribution over “what comes next”. The loss at that position is

loss_t = - log p_model( correct_next_token | tokens before t )

• p_model(...) is the probability (between 0 and 1) the model assigned to the token that actually came next in the training text. It comes from the model’s own forward pass.
• The correct_next_token is taken from the data, so no human labelling is needed in pretraining: the text supplies its own answers.
• -log turns “high probability” into “low loss”: p = 1 gives 0, p = 0.5 gives 0.69, p = 0.1 gives 2.30, and p -> 0 gives a huge loss. The model is punished heavily for being confidently wrong.
• The total loss is the average of loss_t over all positions in the batch.

Worked example

The model is shown “The cat sat on the” and must predict three tokens, giving them probabilities 0.5, 0.25 and 0.1:

import math

probs = [0.5, 0.25, 0.1]                      # p the model gave to each correct token
losses = [-math.log(p) for p in probs]
print([round(l, 3) for l in losses])          # [0.693, 1.386, 2.303]
mean = sum(losses) / len(losses)
print(round(mean, 3))                         # 1.461  (average cross-entropy, in nats)
print(round(math.exp(mean), 2))               # 4.31   (perplexity)

A perplexity of about 4.3 reads as “on average the model was as unsure as if choosing between roughly 4 equally likely tokens”. (Natural log gives “nats”; using log base 2 would give “bits”. Training code almost always uses natural log.)

Pretraining vs SFT: the only difference is which tokens count

Both use this exact loss. In SFT each example is a prompt plus an answer glued into one sequence, but the loss is usually computed only on the answer tokens (prompt tokens are “masked out”, often by setting their label to -100 in Hugging Face code). The model is then trained to produce good answers given prompts, not to memorise prompts.

tokens:   [ User: Capital of France? ] [ Assistant: Paris. <end> ]
loss on:       (masked - no loss)           (loss counted here)

Gotcha: forgetting the <end> token in the SFT targets means the model never learns to stop.

A reward model (RM) is a model (often the SFT model with its head replaced) that takes a (prompt, response) pair and outputs a scalar score. It is trained on pairwise human preferences — annotators are shown two responses to the same prompt and pick the better one — using a Bradley–Terry-style loss:

loss = -log( sigmoid( r(x, y_win) - r(x, y_lose) ) )

where r(x, y) is the reward model’s score for response y to prompt x. This trains the RM to assign a higher score to the preferred response without ever needing an absolute quality label — only relative comparisons, which humans are much better and more consistent at giving. The trained RM is then used as the reward signal for RL (PPO) or, in DPO, folded directly into the policy loss without being materialized as a separate network.

Every symbol

Symbol	Meaning	Where it comes from
x	the prompt (and conversation so far)	the preference dataset
y_win, y_lose	the response raters preferred / rejected	human (or AI) labels
r(x, y)	one real number: the RM’s score of y as a reply to x	the RM’s forward pass: the transformer reads x + y and a small linear “head” maps the final hidden state to one number. These weights are what is trained.
r(x, y_win) - r(x, y_lose)	the margin: how much more the RM likes the winner	difference of two RM outputs
sigmoid(m) = 1/(1+e^-m)	converts the margin into “probability the winner really beats the loser”	fixed function; this is the Bradley–Terry assumption
-log(...)	cross-entropy against the label “winner won”	fixed

Where the formula comes from

Bradley–Terry says: if two items have scores a and b, the probability that A is preferred is P(A beats B) = e^a / (e^a + e^b) = sigmoid(a - b). Only the difference matters, so absolute scores are meaningless (adding 100 to every score changes nothing) — this is why the RM is trained on comparisons. Training then maximises the likelihood of the observed human choices, i.e. minimises -log P(winner beats loser), which is the loss above.

Numeric example

import math
sigmoid = lambda z: 1 / (1 + math.exp(-z))
loss = lambda w, l: -math.log(sigmoid(w - l))

print(loss(2.0, 0.5))   # 0.2014  RM agrees with the human (margin +1.5): P(winner) = 0.82, small loss
print(loss(0.5, 2.0))   # 1.7014  RM disagrees (margin -1.5): P(winner) = 0.18, big loss
print(loss(1.0, 1.0))   # 0.6931  RM can't tell them apart: P = 0.5, loss = ln 2
print(loss(5.0, 0.0))   # 0.0067  very confident and right: almost no loss

The gradient pushes the winner’s score up and the loser’s down until the margin is comfortably positive, then the loss flattens (diminishing returns for already-correct pairs).

Pitfalls

• Scores are only comparable within a prompt family. r = 3 means nothing in absolute terms.
• Rater noise. Humans often disagree; typical preference sets have a meaningful fraction of contested pairs, so an RM cannot reach 100% agreement and should not be trusted past the point of its held-out accuracy.
• Length and style bias. If raters mildly prefer longer or more confident answers, the RM learns that and the policy later exploits it (verbose, flattering outputs). Check reward against response length.

The next two concepts (KL penalty, DPO) both use log pi(y|x). Here is where it comes from.

A language model generates a response one token at a time. The probability of a whole response y = (t_1, ..., t_T) is the product of the per-token probabilities, each conditioned on the prompt and the tokens before it:

pi(y | x) = pi(t_1 | x) * pi(t_2 | x, t_1) * ... * pi(t_T | x, t_1..t_{T-1})
log pi(y | x) = sum over t of log pi(t_t | x, t_<t)          (logs turn the product into a sum)

• pi is the model being trained (the policy); its weights are updated.
• pi_ref is the reference policy: the same architecture with the weights frozen at the start (the SFT model).
• A sequence log-probability is a large negative number (a 100-token answer with ~0.5 probability per token is about 100 * ln 0.5 = -69). It is easy to compute: run one forward pass over prompt + response, take the log-softmax at each response position, pick out the log-prob of the token that actually appears, and sum.

import math
# per-token probabilities the policy and the reference gave to the SAME 4-token response
pi_probs  = [0.60, 0.50, 0.70, 0.40]
ref_probs = [0.50, 0.50, 0.50, 0.40]

log_pi  = sum(math.log(p) for p in pi_probs)
log_ref = sum(math.log(p) for p in ref_probs)
print(round(log_pi, 3), round(log_ref, 3))    # -2.012 -2.773
print(round(log_pi - log_ref, 3))             # 0.761  = log( pi(y|x) / pi_ref(y|x) )

The quantity log pi(y|x) - log pi_ref(y|x) is the log-ratio: positive means “the new policy likes this response more than the reference did”. Both the KL penalty and DPO are functions of this log-ratio.

PPO-based RLHF optimizes the policy to maximize expected reward from the RM, but maximizing a learned reward model unconstrained leads to reward hacking — the policy drifts into a region that scores highly under the (imperfect) RM’s proxy objective while actually degrading real quality (gibberish that happens to trigger the RM, degenerate repetition, losing general capability). The fix is to add a penalty for diverging from the original SFT policy:

objective = E[ reward(y) ] - beta * KL( policy(y|x) || sft_policy(y|x) )

beta controls how far the policy is allowed to wander from the reference model to chase reward. Too small and you get reward hacking; too large and RLHF barely changes anything. This is the same core tension in every RLHF-family method, including DPO’s implicit KL term.

Every symbol

Symbol	Meaning	Where it comes from
E[ ... ]	average over prompts x from the dataset and responses y sampled from the current policy	estimated from a batch of samples
reward(y) (i.e. r(x, y))	the RM score of the response	frozen, previously trained RM
policy(y given x)	the model being optimised (pi)	trainable weights
sft_policy(y given x)	the frozen reference (pi_ref)	copy of the model before RL
KL(P, Q)	sum over y of P(y) * log( P(y) / Q(y) ): the expected log-ratio, measured under P	computed from the two models’ log-probs
beta	trade-off weight	chosen by you; commonly somewhere in 0.01 to 0.5

What KL measures, with numbers

Take a toy “model” with only three possible responses. The reference gives them [0.5, 0.3, 0.2], the tuned policy [0.7, 0.2, 0.1]:

import math
pi  = [0.7, 0.2, 0.1]
ref = [0.5, 0.3, 0.2]
kl = sum(p * math.log(p / q) for p, q in zip(pi, ref))
print(round(kl, 4))                       # 0.0851
print([round(math.log(p/q), 3) for p, q in zip(pi, ref)])   # [0.336, -0.405, -0.693]  per-response log-ratios

Properties worth knowing: KL is never negative, is zero only if the two distributions are equal, is not symmetric (KL(P||Q) differs from KL(Q||P)), and the direction used here, KL(policy || ref), punishes the policy for putting probability where the reference put little (“don’t invent behaviour the SFT model found very unlikely”).

How it is applied in practice

You cannot sum over all possible responses, so implementations estimate it from the sampled response itself. The sampled log-ratio log pi(y|x) - log pi_ref(y|x) has expectation KL when y is sampled from pi. A common implementation applies it per token: each generated token gets a small penalty -beta * (log pi(t) - log pi_ref(t)), and the RM score is added at the final token.

beta = 0.1
reward_model_score = 2.0                      # RM score for the whole response (added at the end)
per_token_log_ratio = [0.4, 0.1, -0.3]        # log pi(t) - log pi_ref(t) for a 3-token response
penalty = beta * sum(per_token_log_ratio)
print(round(penalty, 3))                      # 0.02
print(round(reward_model_score - penalty, 3)) # 1.98  the signal PPO actually optimises

Here the policy has drifted only slightly (net log-ratio 0.2), so the penalty is tiny. If the policy started producing tokens the reference found 100 times less likely, each would carry a log-ratio near 4.6 and the penalty would quickly outweigh any reward gain.

What beta does: the optimum of this objective

For a fixed reward, the policy maximising E[r] - beta * KL(pi || pi_ref) has a closed form (used again in the DPO concept):

pi*(y|x) = pi_ref(y|x) * exp( r(x,y) / beta ) / Z(x)          Z(x) = sum over y of pi_ref(y|x) * exp( r(x,y) / beta )

Z(x) is just the normaliser that makes the probabilities sum to 1. In words: reweight the reference by exp(reward / beta). Toy example with rewards [1, 0, -1] for the same three responses:

import math
ref = [0.5, 0.3, 0.2]; r = [1.0, 0.0, -1.0]
for beta in (1.0, 0.5, 0.1):
    w = [p * math.exp(x / beta) for p, x in zip(ref, r)]
    Z = sum(w); pi = [v / Z for v in w]
    print(beta, [round(v, 3) for v in pi])
# 1.0 [0.784, 0.173, 0.043]   moderate shift towards the high-reward response
# 0.5 [0.919, 0.075, 0.007]   stronger shift
# 0.1 [1.0, 0.0, 0.0]         almost all mass on the top reward (nothing holds the policy back)

Large beta keeps the policy near the reference; small beta lets it collapse onto whichever response the RM scores highest, flaws included.

Pitfalls

• KL as an early-warning metric. Log the mean KL during training; a sharp rise together with rising reward and falling quality in spot-checks is the classic reward-hacking signature.
• Not the only protection. Reward-model ensembles, early stopping and periodic human review are used alongside it.

The KL objective needs an optimiser. Classic RLHF uses PPO. This is the shape of one iteration:

1. sample prompts x from the dataset
2. policy pi generates a response y for each x                         (rollout)
3. RM scores (x, y)  ->  r;  reference scores the same tokens          (needs 4 models in memory:
4. reward per token = [-beta * log-ratio] + RM score on last token      policy, reference, RM, value net)
5. estimate how much better each token was than expected               (advantage, from a value network)
6. update policy weights with a CLIPPED objective, then repeat

Why clipping. PPO multiplies the advantage A by the ratio rho = pi_new(t) / pi_old(t) and clips it to [1 - eps, 1 + eps] (often eps = 0.2) so one batch cannot change the model too violently:

def ppo_term(rho, A, eps=0.2):
    clipped = max(min(rho, 1 + eps), 1 - eps)
    return min(rho * A, clipped * A)      # take the more pessimistic of the two

print(ppo_term(1.1, +1.0))   # 1.1   inside the trust region: full credit
print(ppo_term(1.5, +1.0))   # 1.2   good token already up-weighted by 50%: credit capped
print(ppo_term(1.5, -1.0))   # -1.5  bad token made MORE likely: penalty not capped

Why it is painful: four models, many hyper-parameters (learning rate, beta, eps, value-loss weight, sampling temperature), and unstable training that needs careful monitoring. This operational cost is what DPO, next, avoids.

DPO observes that, under the same Bradley–Terry preference model and KL-constrained objective PPO optimizes, the optimal policy has a closed-form relationship to the reward function — so instead of training a reward model and then running RL to chase it, you can substitute that relationship directly into the preference loss and optimize the policy itself on preference pairs:

loss = -log( sigmoid( beta * [ log(pi(y_win|x)/pi_ref(y_win|x)) - log(pi(y_lose|x)/pi_ref(y_lose|x)) ] ) )

This is a plain supervised loss — no reward model, no sampling rollouts, no PPO’s notorious instability and hyperparameter sensitivity — trained directly on the same (prompt, chosen, rejected) data RLHF would use. It is why DPO became the default for many teams: dramatically simpler infrastructure for often-comparable results, at the cost of being somewhat more sensitive to the quality of the offline preference data since there is no on-policy exploration correcting it live.

Derivation in three steps (where the formula comes from)

1. The RLHF optimum (from the previous concept): pi*(y|x) = pi_ref(y|x) * exp(r(x,y)/beta) / Z(x).
2. Invert it to express reward through the policy: take logs and rearrange, r(x,y) = beta * log( pi*(y|x) / pi_ref(y|x) ) + beta * log Z(x). Check with the toy numbers from the KL concept (beta = 1): the formula returned [1.0, 0.0, -1.0], exactly the rewards used.
3. Plug into Bradley–Terry: the RM loss depends only on r(x,y_win) - r(x,y_lose). In that difference the awkward beta * log Z(x) term is identical for both responses (same prompt x) and cancels. What remains is beta * [log-ratio(y_win) - log-ratio(y_lose)]. Train the policy pi so that this implied reward margin is large, and you get the DPO loss.

So DPO treats beta * log( pi(y|x) / pi_ref(y|x) ) as an implicit reward: the model is its own reward model.

Every symbol

Symbol	Meaning	Where it comes from
pi(y given x)	probability the model being trained gives response y	trainable; one forward pass over x + y, sum of token log-probs
pi_ref(y given x)	same for the frozen reference	frozen copy; can be precomputed once per pair
log(pi/pi_ref)	log-ratio: how much the policy has moved towards y	difference of the two log-probs
beta	scale of the implicit reward / strength of the KL anchor	chosen; around 0.1 is a common starting point
y_win, y_lose	chosen / rejected responses	preference dataset

Numeric example

Sequence log-probabilities for one preference pair (illustrative numbers):

import math
sigmoid = lambda z: 1 / (1 + math.exp(-z))

def dpo_loss(lp_pi_w, lp_ref_w, lp_pi_l, lp_ref_l, beta=0.1):
    reward_w = beta * (lp_pi_w - lp_ref_w)          # implicit reward of the chosen answer
    reward_l = beta * (lp_pi_l - lp_ref_l)          # implicit reward of the rejected answer
    margin = reward_w - reward_l
    return -math.log(sigmoid(margin)), margin

# Start of training: policy == reference, so every log-ratio is 0
print(dpo_loss(-14, -14, -18, -18))   # (0.6931, 0.0)     loss = ln 2, the "know nothing" value

# Policy slightly prefers chosen over rejected
print(dpo_loss(-12, -14, -20, -18))   # (0.5130, 0.4)     chosen +2 nats, rejected -2 nats

# After more training: chosen log-prob up by 4, rejected down by 5
print(dpo_loss(-10, -14, -23, -18))   # (0.3412, 0.9)     margin grows, loss falls

Walk-through: at step 0 the policy equals the reference, so the margin is 0 and the loss is ln 2 = 0.693. Gradient descent raises the probability of the chosen response relative to the reference and lowers that of the rejected one. The reference never moves, which is the KL anchor: the loss only rewards relative movement, and beta sets how steeply movement is rewarded.

Effect of beta

A small beta makes the loss change slowly with the log-ratios, so the policy can move far from the reference before the margin looks “large”; a large beta saturates quickly, so the model stops moving early and stays close to the reference. (This mirrors the RLHF behaviour above.)

Pitfalls and trade-offs

• Both log-probs of the chosen answer can fall. DPO only constrains the margin. It is common for the chosen response’s absolute likelihood to drop as well, so long as the rejected one drops more. Watch for degraded fluency; some variants add an SFT term on the chosen answer.
• Length bias. Since sequence log-probs are sums, longer responses have more mass to move; DPO can drift towards verbose or terse answers depending on the data. Length-normalised variants exist.
• Off-policy. Preference pairs were generated by some other model, not by the current policy, so there is no live exploration. Data should come from (or be close to) the SFT model’s own distribution.
• Memory. You still need two forward passes (policy and reference), but no RM, no value network and no rollouts. Reference log-probs can be precomputed once.
• When RL still wins. If you have a cheap, reliable programmatic reward (tests passing, a verifier), on-policy RL (see Reasoning models and test-time compute) is usually the better fit than preference pairs.

RLAIF (RL from AI Feedback) uses a capable LLM, guided by a written set of principles (a constitution), to generate or judge the preference data instead of (or alongside) human annotators. Constitutional AI is the specific two-phase method: the model critiques and revises its own outputs against the constitution to build an SFT dataset, then an AI judge (again guided by the constitution) produces preference labels used for RL. Motivations: human labeling is slow, expensive, and inconsistent at scale, especially for harms that are individually rare but need broad coverage; AI feedback can be regenerated cheaply as the constitution or model changes; and it makes the value system explicit and auditable as text, rather than implicit in a large opaque human labeling dataset. The obvious risk is circularity — the judge shares the base model’s blind spots — which is why human oversight and calibration checks remain part of the pipeline, not a replacement for it.

The two phases, concretely

Phase 1 - critique and revision (builds an SFT set)
  prompt:    "How do I pick a lock?"
  draft:     (model's first answer)
  critique:  "Identify ways the draft conflicts with principle P: 'avoid helping with clearly illegal acts
              while still being useful for legitimate needs'."
  revision:  (model rewrites the draft in light of its own critique)
  -> SFT on (prompt, revision)

Phase 2 - AI preference labels (builds the RL / DPO data)
  for each prompt, sample two responses A and B
  judge model sees (prompt, A, B, principle) and answers "which better follows the principle?"
  -> preference pairs (chosen, rejected)  ->  train an RM or run DPO, exactly as with human labels

Practical notes

• The AI judge is just an LLM call with a rubric; a minimal version is a prompt such as “Here are two answers and a principle. Reply with A or B.” Randomise which answer appears first, because LLM judges have position bias.
• Validate the AI labels against a small human-labelled sample; report agreement rate before trusting the pipeline.
• A written constitution is editable: changing one principle and regenerating labels is far cheaper than re-collecting human data.

Optimizing toward a reward model’s peak pushes probability mass toward whatever responses score highest, and away from the long tail of plausible-but-lower-scoring alternatives the base model would have produced — measurably lowering entropy in the output distribution (less lexical variety, more formulaic phrasing, convergent answers across paraphrased prompts). This is sometimes called the alignment tax. It is a genuine trade-off, not simply a bug: for most deployed use (support, coding, factual Q&A) consistent, safe, well-formatted output is exactly what you want, and the diversity loss is an acceptable cost. It becomes a real cost for creative-writing or brainstorming use cases, which is part of why providers tune temperature defaults and offer less-aligned or differently-tuned model variants for those workloads rather than fighting it purely at inference time.

Entropy, made concrete

Entropy H(p) = -sum( p_i * ln p_i ) is 0 when one outcome has probability 1 and is largest when all outcomes are equally likely. Imagine the model’s distribution over four ways to open a reply:

import math
H = lambda d: -sum(p * math.log(p) for p in d)

base    = [0.40, 0.30, 0.20, 0.10]    # base model: several openings plausible
aligned = [0.80, 0.15, 0.04, 0.01]    # after preference tuning: one "preferred" opening dominates
print(round(H(base), 3), round(H(aligned), 3))     # 1.28 0.638
kl = sum(p * math.log(p / q) for p, q in zip(aligned, base))
print(round(kl, 3))                                 # 0.363  how far aligned moved from base

Entropy halves (1.28 to 0.64 nats). Ask the aligned model the same question ten times and about eight answers will open identically. (Numbers are illustrative.)

Why it happens

1. Preference data rewards what raters pick on average, so the mode of the rating distribution gets reinforced.
2. The RL/DPO objective is mean-seeking: maximising expected reward gains more by concentrating on the top answer than by spreading mass.
3. The KL anchor limits but does not eliminate the effect.

Why temperature does not fully fix it

Raising temperature flattens the distribution, but it flattens it proportionally; it does not restore alternatives the model has learned to treat as near-impossible:

import math
aligned = [0.80, 0.15, 0.04, 0.01]
logits = [math.log(p) for p in aligned]
def softmax_T(z, T):
    e = [math.exp(v / T) for v in z]; s = sum(e); return [v / s for v in e]
print([round(v, 3) for v in softmax_T(logits, 1.5)])   # [0.659, 0.216, 0.089, 0.036]

At temperature 1.5 the top choice is still 66% likely, and the “variety” that appears is among options the tuned model already preferred. Diversity at the level of ideas, not just word choice, is what is lost.

Mitigations

Sample several candidates and pick among them, prompt explicitly for diverse alternatives, use a less-tuned variant for brainstorming, or add diversity-aware objectives in training.

Each additional training stage (SFT after pretraining, RLHF/DPO after SFT, or later a customer’s own fine-tune) risks overwriting capability the earlier stage established, especially on narrow or underrepresented skills relative to the new stage’s data distribution — a model heavily SFT’d on customer-support-style short answers can measurably lose long-form reasoning or coding ability it had right after pretraining, if that data was thin in the SFT set. Mitigations used across the industry: data mixing (include a healthy fraction of general/capability data at every stage, not only the new specialty), low learning rates relative to the earlier stage, KL regularization to the prior policy (the RLHF/DPO KL term above is partly a forgetting-prevention mechanism, not only a reward-hacking one), and held-out capability evals run after every stage specifically to catch regressions before shipping — the same discipline as regression-testing a prompt change, just one level down the stack.

Why it happens

Weights are shared. Gradient descent on new data moves every weight in the direction that lowers that data’s loss; nothing in the loss mentions the old skills, so any weight those skills relied on is free to move. The more the new data differs from the old, and the more steps and the larger the learning rate, the more the old function drifts.

A mixing recipe (illustrative)

Stage            Target data                        Replay / mixing
SFT (domain)     90% domain examples                10% general instruction + code + reasoning samples
Eval after:      domain held-out (should go up)  +  general suite (should NOT drop)

Detecting it

Keep a fixed regression suite of prompts outside the new task (maths, code, long-form writing, refusals) and compare scores before and after every stage. A drop of more than your noise floor is a regression even if the target metric improved. See Fine-tuning mechanics in practice for the customer fine-tuning version of this workflow.