Fine-tuning mechanics in practice

• Fine-tuning — continuing to train an already-trained model on a smaller, task-specific dataset.
• Weight / parameter — a learned number inside the model. A “7B model” has about 7 billion of them.
• Gradient — for each weight, the direction and size of the change that would reduce the loss. Computed by backpropagation.
• Optimizer state — extra numbers the optimizer keeps per weight. Adam stores two (a running mean m and a running mean of squares v of the gradient).
• Adam / AdamW — the standard optimizer for LLM training; adaptive per-weight step sizes.
• Frozen — a weight that is not updated (no gradient is kept for it).
• PEFT (Parameter-Efficient Fine-Tuning) — methods that train only a small number of extra or selected parameters; also the Hugging Face library (peft) that implements LoRA and friends.
• Transformers — Hugging Face’s library for loading and running pretrained models (AutoModelForCausalLM, AutoTokenizer).
• TRL (Transformer Reinforcement Learning) — Hugging Face’s post-training library; provides SFTTrainer plus DPO/GRPO trainers.
• bitsandbytes — the library that quantizes weights to 4-bit NF4 or 8-bit while loading (BitsAndBytesConfig); QLoRA’s backend.
• Double quantization — quantizing the per-block scale constants themselves, to cut their storage overhead.
• Paged optimizer — an optimizer whose state can spill to CPU RAM during GPU memory spikes (e.g. paged_adamw_8bit).
• DoRA / rsLoRA / LoRA+ — LoRA variants that respectively split the update into magnitude and direction, rescale it by alpha / sqrt(r), and use different learning rates for A and B.
• Bottleneck adapter — a small extra layer inserted into each transformer block; unlike LoRA it cannot be merged away.
• Prompt / prefix tuning — learning a few continuous “virtual token” embeddings prepended to the input while the model stays frozen.
• DPO / ORPO / KTO — preference-optimisation methods that learn from chosen vs rejected answers without a separate reward model.
• PPO / GRPO — reinforcement-learning algorithms for post-training; GRPO scores a group of sampled answers against each other instead of training a value model.
• Continued pretraining — further next-token training on raw domain text rather than (prompt, response) pairs.
• Distillation — training a smaller “student” model on a larger “teacher” model’s outputs.
• Gradient accumulation — summing gradients over several micro-batches before one optimizer step, to simulate a larger batch.
• Gradient clipping — capping the gradient norm to guard against loss spikes.
• Learning-rate schedule (cosine, warmup) — how the step size changes over training: ramp up briefly, then decay along a cosine curve.
• Kaiming initialisation — a standard random initialisation scaled to a layer’s input size.
• Chat template — the model-specific way role-tagged messages are rendered into one token sequence.
• EOS (end-of-sequence) token — the special token that tells generation to stop.
• DDP / FSDP / DeepSpeed ZeRO — multi-GPU training: DDP replicates the model on each GPU; FSDP and ZeRO shard weights, gradients and optimizer state across GPUs.
• accelerate / torchrun — launchers that start one training process per GPU.
• GQA (grouped-query attention) — attention where several query heads share one K/V head, so the K/V projections are narrower than Q.
• GGUF / AWQ / vLLM — a local-inference file format, a 4-bit quantization method, and a serving engine; where a merged model typically goes next (see Quantization and inference efficiency).
• LoRA (Low-Rank Adaptation) — a PEFT method that learns a small low-rank update to selected weight matrices.
• Rank (r) — for a matrix, the number of independent directions it contains; a rank-r update is the product of a tall-thin and a short-wide matrix with inner size r.
• Adapter — the small set of trained LoRA matrices; saved as its own file and loaded on top of a base model.
• alpha (lora_alpha) — a scaling constant for the LoRA update; the update is multiplied by alpha / r.
• Target modules — which weight matrices of the transformer get a LoRA adapter (e.g. q_proj, v_proj).
• QLoRA — LoRA on top of a base model stored in 4-bit precision.
• NF4 (NormalFloat 4) — a 4-bit number format whose 16 levels are placed to suit normally distributed weights.
• Quantization — storing numbers with fewer bits (e.g. 4 instead of 16) to save memory; see Quantization and inference efficiency.
• bf16 / fp16 / fp32 — 16-bit and 32-bit floating-point formats (2 and 4 bytes per number).
• Gradient checkpointing — recomputing activations during the backward pass instead of storing them, trading compute for memory.
• Catastrophic forgetting — losing earlier capabilities while learning the new task.
• SFT (Supervised Fine-Tuning) — fine-tuning on (input, desired output) pairs with the next-token loss.
• Epoch — one full pass over the training set.
• Held-out set — examples deliberately kept out of training and used only for evaluation.
• Canary — sending a small share of live traffic to a new version to check it before full rollout.
• RAG (Retrieval-Augmented Generation) — fetching relevant documents at query time and putting them in the prompt.
• SVD (singular value decomposition) — factorises a matrix into directions and strengths; keeping the largest r strengths gives the best rank-r approximation.
• Standard error — the typical size of the noise in a measured average; for accuracy p on n examples it is sqrt(p(1-p)/n).
• MiB — mebibyte, 1,048,576 bytes.
• Prefill — the phase where the model processes all prompt tokens before generating; its time grows with prompt length.
• Prompt caching — provider-side reuse of an identical prompt prefix across requests, billed at a reduced rate.
• Context rot — quality degradation as the context grows long and cluttered.
• Constrained decoding — restricting generation token by token so the output must match a schema or grammar.

“Fine-tuning” spans a spectrum from changing every weight to changing none. Two choices are independent: which parameters are trained (the method, below) and which loss they are trained with (the objective, further down).

Which parameters change

• Full fine-tuning — update every weight. Maximum expressiveness and the highest ceiling on how much behavior can change, but requires storing and updating the full parameter count (optimizer state alone is typically 2–4x model size in memory), is the most prone to catastrophic forgetting, and produces a whole new checkpoint per task/customer.
• LoRA (the maths is in the next concept) — freeze the base weights, train a small low-rank update (typically 0.1-2% of the parameters, on chosen matrices). Far cheaper to train and store (megabytes per adapter instead of the full model), multiple LoRA adapters can be swapped or even combined at inference against one shared base model, and forgetting is naturally limited since most weights never move.
• QLoRA — the same trainable adapters as LoRA, but the frozen base is stored in 4-bit NF4, which puts 7B-70B models within reach of a single 24-48 GB GPU (see Why QLoRA fits a larger model below).
• LoRA variants (DoRA, rsLoRA, LoRA+) — similar memory and parameter counts to LoRA; they change how the update is parametrised, scaled or optimised for marginal quality or stability gains.
• Prompt/prefix-tuning — freeze the entire model, learn a small set of continuous “virtual token” embeddings prepended to every input, optimized end-to-end like weights but never decoded to real text. Cheapest of all in trainable-parameter count, but the lowest ceiling — it can steer behavior within what the frozen model can already do, not teach new capability the way LoRA/full fine-tuning can, and eats into the usable context window. It is weak on generative tasks and rarely chosen now.
• Bottleneck adapters — small extra layers inserted into each transformer block. An older approach; unlike LoRA they cannot be merged into the existing weights, so they add inference latency.
• Partial fine-tuning — freeze most of the model and train only the top layers, or only the norms/biases (a few percent of parameters). Low memory; good for a cheap domain nudge.

Default in most production settings is LoRA/QLoRA: the accuracy gap to full fine-tuning is often small for typical adaptation tasks, and the operational simplicity (many cheap adapters over one base) is a large practical win.

Where full fine-tuning’s memory goes

For each weight, training with AdamW in mixed precision holds (a common layout):

bf16 weight               2 bytes
bf16 gradient             2 bytes
fp32 master weight        4 bytes
fp32 Adam m               4 bytes
fp32 Adam v               4 bytes
                         --------
                         16 bytes per parameter   (+ activations on top)

params = 6.74e9                       # a Llama-style 7B model
print(round(params * 2 / 1e9, 1))     # 13.5 GB  just to hold the weights for inference in bf16
print(round(params * 16 / 1e9, 1))    # 107.8 GB to fully fine-tune it with AdamW (before activations)

That is why the “2-4x model size” in the bullet above is, if anything, conservative: about 8x the bf16 weight size once gradients, master copies and Adam states are counted. Different frameworks use different layouts (8-bit optimizers, no master copy, ZeRO sharding across GPUs), so treat 16 bytes as a typical upper-middle figure, not a law. LoRA removes the gradient and optimizer state for all frozen weights, which is where the saving comes from. What that means for GPU choice (approximate, before activations dominate):

7B  full FT:   ~108-112 GB + activations                         -> multiple GPUs
7B  LoRA:      ~14 GB bf16 weights + ~0.2 GB adapter state       -> one 24 GB GPU
7B  QLoRA:     ~4 GB NF4 weights + adapter state + activations   -> an 8-12 GB GPU
70B QLoRA:     ~40 GB NF4 weights + ...                          -> one 48-80 GB GPU

Comparison at a glance

Method	Trainable params (7B model, illustrative)	What changes	Storage per task	Ceiling
Full fine-tune	~6.7 billion (100%)	every weight	full model (~13 GB bf16)	highest
LoRA r=16, all linear layers	~40 million (~0.6%)	low-rank additive updates	~80 MB (bf16)	high for most tasks
LoRA r=8, q_proj + v_proj only	~4 million (~0.06%)	small updates to two attention matrices	~8 MB (bf16)	moderate
Prompt tuning, 20 virtual tokens	20 x 4096 = ~82 thousand	input embeddings only	~160 KB	lowest

(Computed in the next concept for the LoRA rows; adapter sizes are parameters times 2 bytes for bf16. QLoRA has the same rows as LoRA; only the frozen base shrinks.)

The objective is a separate choice

Any of the methods above can be trained with different losses:

1. SFT — next-token loss on (prompt, response); teaches format, style, task.
2. Preference optimisation (DPO, ORPO, KTO) — learns from chosen vs rejected answers without a separate reward model.
3. RL with verifiable rewards (GRPO, PPO) — for reasoning, code, maths where an automatic check of the answer exists.
4. Continued pretraining — raw text in a new domain/language to inject knowledge; needs far more data.
5. Distillation — train a small model on a larger model’s outputs.

Choosing

• Teach a style, format, or tool-calling schema from a few thousand examples: LoRA SFT.
• Not enough VRAM: QLoRA, gradient checkpointing, a small batch with gradient accumulation.
• Inject new factual knowledge: fine-tuning is poor at this; use RAG, or continued pretraining if the corpus is large.
• Align preferences/safety: SFT, then DPO.
• Improve reasoning with checkable answers: GRPO.
• Large dataset, big budget, need the maximum change: full fine-tuning (sharded across GPUs).

Forgetting, data quality and regression evaluation are covered further down this page.

The idea

Fine-tuning changes a weight matrix W into W + dW. Empirically, the useful part of dW for adapting a pretrained model has low rank, meaning it can be described with far fewer numbers than a full matrix. LoRA therefore never stores dW directly. It parametrises it as a product of two thin matrices and trains only those.

Why low rank is a reasonable bet: fine-tuning usually teaches a narrow behaviour (a style, a format, a domain), not a whole new world model, and the weight change needed for that has low “effective rank” — a few directions carry most of it. For many tasks the bet works; for tasks that need a lot of genuinely new knowledge or skill it can fall short of full fine-tuning.

The equation

h = W0 x  +  (alpha / r) * B A x

Symbol	Shape	Meaning	Where it comes from
x	[d_in] (one token’s input vector)	the input to this linear layer	the previous layer’s output
W0	[d_out, d_in]	the original pretrained weight	frozen, loaded from the base model
A	[r, d_in]	“down-projection”: squeezes the d_in input down to r numbers	trained; initialised with small random values (Gaussian in the original paper; Kaiming-uniform in the PEFT library)
B	[d_out, r]	“up-projection”: expands the r numbers back to d_out	trained; initialised to zeros
r	scalar	the rank; the width of the bottleneck	chosen by you (commonly 4 to 64)
alpha	scalar	scaling constant	chosen by you
alpha / r	scalar	the multiplier applied to the update	derived from the two choices above
h	[d_out]	the layer’s output	computed

In words: run the input through the original layer as usual, also run it through the small bottleneck A then B, scale that, and add the two results.

            +--------- W0 (frozen, d_out x d_in) ---------+
 x --------+                                               (+)----> h
 [d_in]     \--> A (r x d_in) --> B (d_out x r) --> * alpha/r -/
              r numbers in the middle (the bottleneck)
              only A and B receive gradients

Why B starts at zero

At the first step B = 0, so B A x = 0 and h = W0 x: the model behaves exactly like the pretrained model before any training. If both were random the update would start as noise and damage the model. A cannot be zero as well, otherwise both would get zero gradients and never move; with B = 0 and A != 0, B gets a non-zero gradient immediately.

Why alpha / r

The factor decouples “how strong the update is” from “how wide the bottleneck is”. With the common habit of setting alpha to a fixed multiple of r (such as alpha = 2r), the scale stays at 2 regardless of rank. If you keep alpha fixed and increase r, the multiplier shrinks. (A stabilised variant, rsLoRA, uses alpha / sqrt(r).) Changing alpha is closely related to changing the learning rate of the adapter.

Parameter counts

The adapter on one matrix has r*d_in + d_out*r = r*(d_in + d_out) parameters instead of d_in * d_out.

For one square 4096 x 4096 projection with r = 8:

d, r = 4096, 8
full = d * d                    # 16,777,216 weights in the frozen matrix
lora = r * (d + d)              # 65,536 trainable weights (A: 8x4096, B: 4096x8)
print(full, lora, lora / full)  # 16777216 65536 0.00390625   -> 0.39% of this matrix

for r in (8, 16, 64):
    print(r, r * (d + d), round(full / (r * (d + d))))
# 8   65536  256    -> 256x fewer trainable parameters than the full matrix
# 16  131072 128
# 64  524288 32

For a whole Llama-style 7B model (4096 hidden size, 11008 MLP width, 32 layers; these architecture numbers are those of the published 7B model):

def lora_params(shapes, r):
    return sum(r * (d_in + d_out) for d_in, d_out in shapes)

d, f, layers = 4096, 11008, 32
per_layer = {
    "q_proj": (d, d), "k_proj": (d, d), "v_proj": (d, d), "o_proj": (d, d),
    "gate_proj": (d, f), "up_proj": (d, f), "down_proj": (f, d),
}
for r in (8, 16):
    attn_qv = lora_params([per_layer[k] for k in ("q_proj", "v_proj")], r) * layers
    all_lin = lora_params(per_layer.values(), r) * layers
    print(r, attn_qv, all_lin)
# 8  4194304  19988480     q,v only: 4.2M (0.06% of 6.74B)   all linear: 20.0M (0.30%)
# 16 8388608  39976960     q,v only: 8.4M                     all linear: 40.0M (0.59%)
print(sum(a * b for a, b in per_layer.values()) * layers)   # 6476005376 frozen linear weights

Walk-through: each target matrix contributes r*(d_in + d_out); the loop sums that over the matrices in one layer, and the * layers multiplies by 32 layers. The trainable fraction is under 1% in all these configurations, and doubling r doubles the adapter exactly. Models with grouped-query attention have narrower K/V projections, which shrinks the count further: on an 8B-class model (32 layers, d_model 4096, K/V width 1024), adapters on Q, K, V and the output projection at r = 16 come to about 13.6M parameters, roughly 0.17% of 8.03B.

A runnable numeric check (numpy)

import numpy as np
rng = np.random.default_rng(0)
d_out, d_in, r, alpha = 6, 8, 2, 4

W = rng.normal(size=(d_out, d_in))              # frozen pretrained weight
A = rng.normal(scale=0.1, size=(r, d_in))       # random init
B = np.zeros((d_out, r))                        # zero init
x = rng.normal(size=(d_in,))
scale = alpha / r

y0 = W @ x
y1 = W @ x + scale * (B @ (A @ x))
print(scale, np.allclose(y0, y1))               # 2.0 True    -> identical to base at step 0

B = rng.normal(scale=0.1, size=(d_out, r))      # pretend training has moved B
dW = scale * B @ A
print(dW.shape, np.linalg.matrix_rank(dW))      # (6, 8) 2    -> a 6x8 update, but rank only 2
y2 = W @ x + scale * (B @ (A @ x))
print(np.allclose(y2, (W + dW) @ x))            # True        -> "merging" gives identical outputs
print(W.size, A.size + B.size)                  # 48 28       -> 28 trainable numbers instead of 48

What it shows: (1) zero-init B makes step 0 identical to the base; (2) the update dW has full shape but rank at most r — every row of B @ A is a weighted combination of the r rows of A, which is exactly what “rank r” means; (3) because the adapter is linear, you can merge it: compute W + (alpha/r) B A once and serve a normal model with no extra latency. (The toy matrix is too small for big savings; at 4096 x 4096 the ratio is 0.4%.)

Pitfalls

• The extra forward pass through A and B is cheap (2*r*(d_in + d_out) multiply-adds vs 2*d_in*d_out, about 0.4% for the example above), but the base weights still have to be read and the activations stored, so training time falls much less than parameter count suggests.
• Merging an adapter into a quantized base is lossy: merge into a 16-bit copy, then re-quantize.
• LoRA cannot represent updates that genuinely need high rank (teaching an entirely new language, for example); see the next concept.

This concept turns the three LoRA knobs from the previous concept into decisions. Recall the update h = W0 x + (alpha / r) · B A x: r is the width of the bottleneck between A and B, alpha / r is the strength of the update, and target modules decide which W0 matrices get such a pair at all.

What rank means, with a numeric picture

The rank r caps how many independent “directions” the update dW = (alpha/r) · B A can have. An update that needs many independent directions cannot be squeezed through a narrow bottleneck. The SVD (singular value decomposition) shows this: the best rank-r approximation of a matrix keeps its r largest singular values, and the relative error tells you how much a rank-r adapter could ever capture.

import numpy as np
rng = np.random.default_rng(0)
n = 256

# Case 1: a "true" update that is really rank 8, plus a little noise
low = rng.standard_normal((n, 8)) @ rng.standard_normal((8, n)) + 0.05 * rng.standard_normal((n, n))
# Case 2: an update with no low-rank structure at all
rand = rng.standard_normal((n, n))

def rel_err(M, r):
    u, s, vt = np.linalg.svd(M)
    Mr = (u[:, :r] * s[:r]) @ vt[:r]               # best rank-r approximation
    return np.linalg.norm(M - Mr) / np.linalg.norm(M)

for r in (1, 2, 4, 8, 16, 32):
    print(r, round(rel_err(low, r), 3))
# 1 0.905   2 0.814   4 0.625   8 0.017   16 0.016   32 0.014   <- error collapses at r = 8, then flat
print(round(rel_err(rand, 8), 3), round(rel_err(rand, 32), 3))
# 0.942 0.789                                        <- unstructured update: rank 32 still misses most of it

Reading it: when the needed change is low-rank (case 1), r = 8 captures it and more rank buys nothing; below the true rank (r = 1 to 4) you lose a lot. When the change is unstructured (case 2) no small rank works. The empirical bet behind LoRA is that adapting an already-capable model to a task looks like case 1. The task decides which case you are in: formatting and style are low-rank; teaching a new language or large amounts of new facts is not. (Toy matrices; illustrative of the mechanism, not a measurement of a real model.)

Which matrices to adapt (target_modules)

A transformer layer has attention projections (q_proj, k_proj, v_proj, o_proj) and MLP projections (gate_proj, up_proj, down_proj in Llama-style models; names differ by family, so list them from the model itself). The original LoRA paper adapted only query and value. Later practice, including the QLoRA paper’s experiments, found that adapting all linear layers generally gives better quality at the same rank. The MLP holds most of the weights, so skipping it leaves most of the layer unadaptable.

# Discover the real module names before writing target_modules
import torch
names = {n.split(".")[-1] for n, m in model.named_modules() if isinstance(m, torch.nn.Linear)}
print(sorted(names))
# ['down_proj', 'gate_proj', 'k_proj', 'lm_head', 'o_proj', 'q_proj', 'up_proj', 'v_proj']   (Llama-style model)

PEFT’s LoraConfig accepts a list of names, or the string "all-linear" to target every linear layer except the output head. Exclude lm_head unless you also change vocabulary.

Adapter size by rank and target set (computed)

Llama-7B-style shapes (hidden 4096, MLP 11008, 32 layers, 6.74B parameters). Each target matrix costs r · (d_in + d_out) parameters; sizes assume bf16 (2 bytes per number).

Rank r	q and v only	all attention (q,k,v,o)	all linear layers	all linear as % of base	adapter file (all linear)
4	2.1 M	4.2 M	10.0 M	0.15%	about 19 MiB
8	4.2 M	8.4 M	20.0 M	0.30%	about 38 MiB
16	8.4 M	16.8 M	40.0 M	0.59%	about 76 MiB
32	16.8 M	33.6 M	80.0 M	1.19%	about 152 MiB
64	33.6 M	67.1 M	159.9 M	2.37%	about 305 MiB
128	67.1 M	134.2 M	319.8 M	4.75%	about 610 MiB

Size scales linearly in r. Training state for the adapter itself (gradient plus two Adam states, if kept in fp32) is about 16 bytes per adapter parameter, i.e. roughly 610 MiB for the r=16 all-linear case: small next to the base model, which is why LoRA fits where full fine-tuning does not.

Rank decision procedure

1. Start at r = 16, all linear layers. It is a robust default for instruction tuning and domain adaptation.
2. Go down (4 to 8) for narrow format, tone or classification tasks with a few hundred examples: fewer parameters means less overfitting and a tiny artifact.
3. Go up (32 to 64+) only when the training loss plateaus clearly above what a full fine-tune reaches on the same data, or the task is genuinely broad (code in a new language, a new domain at scale). Check that the held-out score improves too; extra rank on a small dataset mostly adds overfitting.
4. If memory is tight, lower rank before dropping modules.

These bands are conventions, not guarantees; run two or three ranks and compare held-out scores (one rank-8 and one rank-32 run usually tells you which way to go).

Alpha: the strength of the update

alpha is divided by r, so the multiplier on B A x is alpha / r:

r	alpha	alpha / r	rsLoRA alpha / sqrt(r)
8	16	2.0	5.66
16	16	1.0	4.0
16	32	2.0	8.0
64	16	0.25	2.0
64	128	2.0	16.0

Reading it: with alpha fixed at 16, going from r = 16 to r = 64 makes the update four times weaker, so a bigger rank would look worse for the wrong reason. Keeping alpha = 2r keeps the multiplier at 2 for any rank. The pragmatic rule: pick alpha = r or 2r, then tune the learning rate, which acts almost the same way as alpha; do not tune both at once. rsLoRA (use_rslora=True in PEFT) divides by sqrt(r) instead, which is intended to keep learning stable at high ranks.

A complete configuration

from peft import LoraConfig, get_peft_model

cfg = LoraConfig(
    r=16,
    lora_alpha=32,                    # alpha = 2r -> multiplier 2.0
    target_modules="all-linear",      # or ["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"]
    lora_dropout=0.05,                # mild regulariser, useful below ~1k examples
    bias="none",
    task_type="CAUSAL_LM",
)
peft_model = get_peft_model(model, cfg)
peft_model.print_trainable_parameters()
# trainable params: ~40M || all params: ~6.8B || trainable%: ~0.6   (Llama-7B-style; exact figures vary by model)

Other settings and pitfalls

• lora_dropout (0 to 0.1) — dropout on the adapter branch’s input; helps on small datasets.
• Learning rate — LoRA usually needs a higher rate than full fine-tuning (conventionally around 1e-4 to 3e-4 versus 1e-5), because it starts from a zero update and moves few parameters. Starting points, not requirements.
• Pitfall: wrong module names. A typo or a name from another model family either raises an error or silently adapts nothing; always print print_trainable_parameters() and check it is not zero.
• Pitfall: tuning only q and v and blaming LoRA for weak results; try all-linear first.
• Pitfall: comparing ranks with a fixed alpha (see the table above).

For the complete PEFT/TRL training code see the next concept.

How LoRA works in one picture

  x ──► W (frozen, d×d)  ───────────► + ──► y
   │                                  ▲
   └──► A (d×r) ──► B (r×d) × (α/r) ──┘          only A and B receive gradients
                    r = 8..64, much smaller than d

The update ΔW = (α/r)·B·A has rank at most r. B is initialised to zero so training starts at exactly the base model. After training, W' = W + ΔW can be merged for zero-overhead inference.

Code

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

name = "Qwen/Qwen2.5-7B-Instruct"
tok = AutoTokenizer.from_pretrained(name)

model = AutoModelForCausalLM.from_pretrained(
    name,
    quantization_config=BitsAndBytesConfig(           # drop this block for plain LoRA
        load_in_4bit=True, bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16,
    ),
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

lora = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],   # all linear layers
    task_type="CAUSAL_LM",
)

ds = load_dataset("json", data_files="train.jsonl")["train"]    # {"messages": [{"role":..., "content":...}, ...]}

trainer = SFTTrainer(
    model=model,
    train_dataset=ds,
    peft_config=lora,
    processing_class=tok,
    args=SFTConfig(
        output_dir="out",
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,       # effective batch 16
        learning_rate=2e-4,                  # LoRA tolerates ~10x higher LR than full FT
        num_train_epochs=2,
        lr_scheduler_type="cosine", warmup_ratio=0.03,
        bf16=True,
        gradient_checkpointing=True,         # trade compute for memory
        optim="paged_adamw_8bit",
        logging_steps=10,
        max_length=2048,
    ),
)
trainer.train()
trainer.save_model("out/adapter")            # a few tens of MB

Walkthrough:

1. Load the base model, optionally in 4-bit (QLoRA). The base weights are frozen.
2. LoraConfig selects which linear layers get adapters, and the rank r. Higher r gives more capacity and more memory; 8-32 covers most tasks. lora_alpha / r scales the update.
3. SFTTrainer applies the chat template, masks/pads, and runs the training loop. peft_config wraps the model with adapters.
4. Effective batch size = per-device batch × accumulation steps × number of GPUs.
5. Save only the adapter.

Merging and serving

from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16)   # NOT the 4-bit copy
merged = PeftModel.from_pretrained(base, "out/adapter").merge_and_unload()
merged.save_pretrained("out/merged"); tok.save_pretrained("out/merged")
# then: quantize to GGUF/AWQ, or serve with vLLM (which can also hot-load LoRA adapters).

Pitfalls

• Merging into a 4-bit model loses precision; merge into the bf16 original.
• Wrong chat template is the most common silent bug; train and serve with the same template.
• Training on prompt tokens wastes signal; use completion-only loss (TRL does this with assistant-only masks).
• Too-high learning rate or rank causes forgetting; too few epochs gives no behaviour change. Check eval, not just training loss.
• Padding side and EOS token mistakes lead to models that never stop generating.

Understanding the bare loop makes the abstractions less magic. This is a minimal LoRA implemented by hand plus a training loop.

A LoRA linear layer

import math, torch, torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=8, alpha=16, dropout=0.0):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad_(False)                      # freeze the pretrained weight
        self.A = nn.Parameter(torch.empty(r, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))   # zero init -> starts as identity
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.scale = alpha / r
        self.drop = nn.Dropout(dropout)

    def forward(self, x):
        return self.base(x) + self.drop(x) @ self.A.T @ self.B.T * self.scale

def add_lora(model, targets=("q_proj", "v_proj"), **kw):
    for name, module in list(model.named_modules()):
        for child_name, child in list(module.named_children()):
            if isinstance(child, nn.Linear) and child_name in targets:
                setattr(module, child_name, LoRALinear(child, **kw))
    return model

The training loop

import torch
from torch.utils.data import DataLoader

model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16).cuda()
for p in model.parameters(): p.requires_grad_(False)
add_lora(model, r=16, alpha=32)
model.gradient_checkpointing_enable(); model.enable_input_require_grads()

params = [p for p in model.parameters() if p.requires_grad]       # only A and B
opt = torch.optim.AdamW(params, lr=2e-4, weight_decay=0.0)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=total_steps)

def collate(batch):
    # each item: input_ids, labels (labels = -100 on prompt tokens so loss ignores them)
    ids = torch.nn.utils.rnn.pad_sequence([b["input_ids"] for b in batch], batch_first=True, padding_value=tok.pad_token_id)
    lab = torch.nn.utils.rnn.pad_sequence([b["labels"] for b in batch], batch_first=True, padding_value=-100)
    return ids, lab, (ids != tok.pad_token_id)

accum = 8
model.train()
for step, (ids, labels, mask) in enumerate(DataLoader(train_ds, batch_size=2, shuffle=True, collate_fn=collate)):
    out = model(input_ids=ids.cuda(), attention_mask=mask.cuda(), labels=labels.cuda())
    (out.loss / accum).backward()                                  # scale for accumulation
    if (step + 1) % accum == 0:
        torch.nn.utils.clip_grad_norm_(params, 1.0)
        opt.step(); sched.step(); opt.zero_grad(set_to_none=True)

What each piece does:

• Freezing (requires_grad_(False)) means autograd stores no gradients or optimizer state for the base model, which is where LoRA’s memory saving comes from.
• labels with -100 — PyTorch’s cross-entropy ignores that index, so only response tokens contribute to the loss. The HF model shifts labels internally for next-token prediction.
• Gradient accumulation simulates a larger batch by summing gradients over several micro-batches before opt.step().
• Gradient clipping guards against loss spikes.
• Gradient checkpointing recomputes activations during backward; typically 30% slower but cuts activation memory sharply. enable_input_require_grads() is needed so gradients flow through frozen embeddings.

Scaling beyond one GPU

• DDP — replicate the model on each GPU, average gradients; fine when the model fits on one GPU (LoRA/QLoRA).
• FSDP / DeepSpeed ZeRO-3 — shard weights, gradients, and optimizer state across GPUs; needed for full fine-tuning of 13B+ models.
• Launch with accelerate launch or torchrun --nproc_per_node=N train.py.

Verifying you did it right

1. Print trainable parameter count (sum(p.numel() for p in params)), expect well under 2%.
2. Overfit 20 examples first: loss should go to near zero in a few dozen steps.
3. Compare base vs tuned outputs on a held-out set, including general-capability prompts, to catch forgetting.

LoRA already avoids storing gradients/optimizer state for the frozen base weights, but the base weights themselves still sit in GPU memory at full precision (bf16/fp16) during training, and that static footprint is what caps model size on a given GPU. QLoRA quantizes the frozen base weights to 4-bit (dequantizing on the fly for each forward/backward computation) while keeping the trainable LoRA adapters (A, B) in full precision — since the base weights never receive a gradient, their precision only affects how faithfully the forward pass approximates the original model, not the fine-tuning dynamics. This roughly quarters the static memory the base model occupies, which is why QLoRA is what actually makes fine-tuning multi-tens-of-billions-parameter models feasible on a single consumer or single-datacenter GPU rather than a multi-GPU cluster.

Why it works at all: the capacity a fine-tune needs (a low-rank adjustment, learned in the full-precision A and B) is far smaller than the capacity needed to represent what the model already knows (the base weights, which 4-bit quantization only mildly perturbs). QLoRA decouples “how precisely must we store the existing knowledge” from “how much new information does this fine-tune add”, and spends precision only on the second.

The three ingredients of QLoRA

1. NF4 storage. Pretrained weights are roughly bell-shaped (normally distributed). A 4-bit number has only 16 possible values, so NF4 places those 16 levels at quantiles of a normal distribution, so that each level represents about the same number of weights. Weights are stored in small blocks (64 weights per block in the paper); each block keeps its own scale constant so large and small regions both use the full range.
2. Double quantization. The per-block scale constants are themselves quantized (the paper describes 8-bit constants), cutting the overhead of the scales from about 0.5 bits per weight to about 0.13.
3. Paged optimizers. Optimizer states live in memory that can spill to CPU RAM when the GPU runs out during a memory spike (e.g. a long sequence), avoiding an out-of-memory crash.

What happens on a forward pass

stored:   W0 in NF4 (4 bits/weight, frozen)      A, B in bf16 (trainable)
step 1:   dequantize this layer's W0 block-by-block to bf16 (temporary)
step 2:   h = W0_bf16 x  +  (alpha/r) * B (A x)        <- the matmul runs in bf16, the "compute dtype"
step 3:   backward: gradients flow *through* W0 to earlier layers, but are only *stored* for A and B
step 4:   discard the temporary bf16 copy of W0

Note that gradients must still flow through the frozen quantized weights (the chain rule needs them), but no gradient or optimizer state is kept for them.

Memory arithmetic (7B model, approximate)

params = 6.74e9
print(round(params * 2 / 1e9, 1))              # 13.5 GB   bf16 base (plain LoRA)
print(round(params * 4.127 / 8 / 1e9, 1))      # 3.5 GB    NF4 with double quantization (~4.127 bits/weight)
print(round(params * 4.5 / 8 / 1e9, 1))        # 3.8 GB    NF4 with fp32 block scales and no double quant (~4.5 bits)
adapter = 39_976_960                           # r=16 on all linear layers (computed above)
print(round(adapter * 2 / 1e6), "MB bf16 adapter weights")        # 80 MB
print(round(adapter * 16 / 1e6), "MB if weights+grads+Adam in fp32")  # 640 MB  (still small)

So the base shrinks by about 4x (13.5 GB to about 3.5 GB; for an 8B-class model, roughly 15 GiB to 4 GiB) and the trainable part is a few hundred MB. Activations, which depend on batch size and sequence length, then become the dominant variable cost; gradient checkpointing (recompute activations in the backward pass) is routinely enabled to reduce them. Real usage also includes CUDA context and fragmentation, so budget a GB or two above the sum. Figures here are approximate; see VRAM and running LLMs locally for the full memory budget method.

Trade-offs and pitfalls

• Speed. Dequantizing on every forward and backward pass makes QLoRA slower per step than 16-bit LoRA on a GPU that could hold the bf16 model. Use plain LoRA when it fits; use QLoRA when it does not.
• Quality. The QLoRA paper reports that NF4 + LoRA recovers the quality of 16-bit LoRA on the benchmarks it studied; treat that as “close in practice” and verify on your own eval rather than assume equality.
• Baked-in quantization error. The base model’s NF4 rounding error is present throughout training; the adapter can partly compensate for it but does not remove it.
• Merging. The adapter was trained against dequantized NF4 weights. Merge into a bf16 copy of the original base for deployment (merging into the quantized base means dequantize, add, requantize, which adds rounding error), or serve the adapter on top of the same quantized base, and re-evaluate after any merge.

A model fine-tuned heavily on one narrow task/domain (a specific support workflow, a specific coding style) can measurably regress on capabilities the fine-tuning data never touched — general reasoning, following unrelated instructions, or even basic formatting it previously handled fine — because gradient updates optimized purely for the new data have no signal telling them to preserve everything else. Guards: mix in general-purpose data alongside the task-specific data rather than training on task data exclusively; use a low learning rate and often few epochs (over-training on a small narrow set is the fastest way to forget broadly); prefer LoRA over full fine-tuning when forgetting risk is a concern, since a low-rank update structurally limits how much of the base function can shift; and run held-out capability evals covering both the target task and general capability before/after, because a fine-tune that improves the target metric while silently breaking something else is a net loss you will not see without measuring it.

Signs

• Target-task score rises while a general eval (instruction following, maths, a second language) falls.
• The model starts using the training set’s phrasing, length or template in unrelated answers (it is “stuck in the fine-tune’s voice”).
• Refusals or safety behaviour change; formatting that used to be reliable breaks.
• Training loss keeps falling to near zero while held-out loss rises: overfitting, which usually accompanies forgetting.

A before/after table to keep for every run (illustrative numbers)

Eval                      Base    After fine-tune   Verdict
-------------------------------------------------------------
Target task (held-out)    61%     84%               good: the point of the run
General instruction set   78%     77%               ok: within noise
Maths word problems       52%     41%               REGRESSION: investigate / mix in more data
Safety refusals           96%     89%               REGRESSION: do not ship as is

Why LoRA helps, and its limits

LoRA’s update has rank at most r and the base weights are untouched, so the fine-tuned function can only move within a small subspace; the same property that limits what it can learn also limits how much it can destroy. It reduces forgetting, but does not eliminate it: a high rank, high learning rate and many epochs can still damage general behaviour. You can also switch the adapter off at inference to recover the base model exactly.

• Format consistency — every example should match the exact input/output shape the model will actually see in production (same instruction style, same delimiters); inconsistent formatting between training and deployment quietly caps achievable quality.
• Diversity over redundancy — many near-duplicate examples teach the model less than fewer, more varied ones covering the real range of inputs (edge cases, unusual phrasing, adversarial inputs), and can bias the model toward overfitting the redundant pattern.
• Correctness of the label itself — a fine-tuning dataset with even a modest fraction of wrong or low-quality target outputs measurably drags down quality, more than an equivalent fraction of missing coverage would, since the model is directly imitating those labels.
• Balanced difficulty — a set dominated by easy cases under-trains the model on the genuinely hard cases that actually motivated fine-tuning in the first place.
• Representative of real distribution, not synthetically convenient — if production traffic skews toward short, informal queries and your dataset is all long, formal ones, the fine-tune generalizes poorly to what it will actually see.

Quality and format discipline consistently beats raw volume for SFT-scale fine-tuning.

What one training example looks like

Chat-style SFT data is commonly stored as JSON Lines, one conversation per line (field names such as messages, role, content are the widespread convention, but check what your trainer expects):

{"messages": [
  {"role": "system", "content": "You extract invoice fields as JSON."},
  {"role": "user", "content": "Invoice 4471 from Acme Ltd, total 1,250.00 EUR, due 2026-11-30."},
  {"role": "assistant", "content": "{\"invoice_id\": \"4471\", \"vendor\": \"Acme Ltd\", \"total\": 1250.00, \"currency\": \"EUR\", \"due\": \"2026-11-30\"}"}
]}

(In the file this is a single line; it is broken up here for reading.) The system and user messages must match what production sends; the assistant message is the only target the loss is computed on, as described in Training and post-training.

A cheap pre-flight check

import json, collections

rows = [json.loads(line) for line in open("train.jsonl", encoding="utf-8")]
users = [r["messages"][-2]["content"] for r in rows]
answers = [r["messages"][-1]["content"] for r in rows]

dupes = [u for u, n in collections.Counter(users).items() if n > 1]
print(len(rows), "examples;", len(dupes), "duplicated prompts")
bad = []
for r in rows:
    try:
        json.loads(r["messages"][-1]["content"])      # our task requires valid JSON answers
    except json.JSONDecodeError:
        bad.append(r)
print(len(bad), "examples whose target is not valid JSON")   # -> fix or drop these before training
lengths = sorted(len(a) for a in answers)
print("answer length min/median/max:", lengths[0], lengths[len(lengths)//2], lengths[-1])

Walk-through: it loads every example, counts prompts that appear more than once (redundancy), checks that every target meets the format contract (label correctness), and prints the length distribution so you can compare it with production traffic. Replace the validity check with whatever rule defines a correct answer for your task; the principle is to run an automated check on every label before spending GPU hours.

There is no universal number, because “how much” depends on how far the target behaviour is from what the base model already does, how clean the labels are, and how strict the success criterion is. The table gives rough, commonly observed regimes when adapting an already instruction-tuned model; these are rules of thumb, not measurements.

Goal	Examples	Why that much	Typical example
A narrow output format or tone	low hundreds (100 to 500)	the model already can; you are selecting a mode of behaviour	always reply as strict JSON with these 6 keys; company voice
A well-scoped domain task	1,000 to a few thousand	needs coverage of the input variety and edge cases	classify support tickets into 25 categories; extract fields from invoices
Broad or new skill	tens of thousands and up	you are teaching, not nudging	a new programming language, a new language, long-form domain reasoning
New factual knowledge	not a data-volume problem	weights learn facts unreliably and cannot be updated cheaply	use retrieval (see Tool vs RAG vs fine-tuning below)

Past the first thousand or two, label quality and coverage matter more than count: 500 carefully checked examples often beat 5,000 noisy ones. The practical approach is not to target a number up front but to fine-tune on what you have, evaluate, and add data aimed at the failures the evaluation reveals. Volume without a feedback loop is a guess.

What the numbers cost

Training cost is measured in tokens processed: examples x tokens per example x epochs.

print(1000 * 400 * 3)    # 1,200,000 tokens   (1k examples, 400 tokens each, 3 epochs) -- minutes to an hour on one GPU for a 7B QLoRA run
print(5000 * 500 * 2)    # 5,000,000 tokens

So the data collection and labelling effort, not GPU time, is usually the real cost at the low-thousands scale. (GPU time depends on hardware and model size; measure with a short run instead of trusting a formula.)

The loop, step by step

1. Fine-tune on a small seed set (for example 200 to 500 clean examples).
2. Evaluate on a held-out set and read 30 to 50 failures by hand.
3. Group the failures (wrong format, missing edge case, wrong tone, factual gap).
4. Write or collect examples aimed at the largest group; add them; retrain from the same base (not from the previous fine-tune, to avoid compounding drift).
5. Stop when the held-out score stops improving, or when the remaining failures are knowledge gaps (which more data will not fix; use retrieval).

The held-out set must be big enough to see the change

A held-out set is a sample, so the measured accuracy has noise. For accuracy p measured on n examples, the standard error is sqrt(p · (1 - p) / n) and a rough 95% interval is plus or minus 1.96 times that.

Held-out size n	Standard error at p = 0.80	95% interval half-width
50	0.057	about 11 points
100	0.040	about 8 points
300	0.023	about 4.5 points
1,000	0.013	about 2.5 points

With 100 held-out examples, a “5 point improvement” from fine-tuning is inside the noise. (Comparing two models on the same examples is less noisy than this independent-sample figure, but the order of magnitude holds.) Size the held-out set to the smallest improvement you care about, and never train on it.

A learning-curve check

Train on 25%, 50% and 100% of the data you have and plot the held-out score. If the step from 50% to 100% still climbs steeply, more data of the same kind helps. If it is flat, the limiter is label quality, task ambiguity or model capacity, not quantity.

import random

def learning_curve(train_rows, eval_rows, train_fn, score_fn, fractions=(0.25, 0.5, 1.0), seed=0):
    rng = random.Random(seed)
    rows = train_rows[:]
    rng.shuffle(rows)                          # same shuffled order, nested subsets
    results = {}
    for f in fractions:
        subset = rows[: int(len(rows) * f)]
        model = train_fn(subset)               # always start from the SAME base model
        results[f] = score_fn(model, eval_rows)
    return results
# e.g. {0.25: 0.71, 0.5: 0.78, 1.0: 0.80}  (illustrative) -> gain from 50% to 100% is only 2 points: more of the same data will not help much

You supply train_fn (one fine-tuning run returning a model) and score_fn (your evaluation). Using nested subsets of one shuffle makes the points comparable.

Epochs and hyper-parameters on small sets

With a few hundred examples, 2 to 3 epochs is a common choice; more epochs make the model memorise the training set (training loss near zero, held-out loss rising). Prefer early stopping on held-out loss over a fixed epoch count. Conventions, not guarantees.

Pitfalls

• Duplicates and near-duplicates inflate the example count without adding information, and leak between train and held-out sets so scores look better than reality.
• Adding data that contradicts earlier examples (two annotators following different rules) lowers quality; write the labelling rules down.
• If failures are “the model does not know this fact”, more examples of that fact teach it poorly; put the fact in context or a retrieval layer instead.

1. Hold out a slice of the fine-tuning task’s own data never trained on, to measure genuine target-task improvement rather than memorization of the training set.
2. Run a fixed general-capability eval suite (instruction-following, reasoning, formatting, whatever your product also relies on) before and after, to catch forgetting the target-task metric alone would never reveal.
3. Compare against the base (un-fine-tuned) model on the same prompts, side by side, not just against an aggregate score — some regressions are only visible reading actual outputs.
4. Test the exact production prompt/harness, not the fine-tuning data’s raw format — a fine-tune that scores well on its own held-out set can still underperform once wrapped in the real system prompt and tool definitions it will actually run inside.
5. Canary in production — route a small percentage of real traffic to the fine-tuned model and compare live metrics (user feedback, task success, escalation rate) before a full rollout.

A minimal comparison harness

def evaluate(generate, cases, check):
    """generate(prompt) -> str; check(output, case) -> bool"""
    passed = [check(generate(c["prompt"]), c) for c in cases]
    return sum(passed) / len(passed)

suites = {"target_task": target_cases, "general": general_cases, "safety": safety_cases}
for name, cases in suites.items():
    base = evaluate(base_model_generate, cases, check_fn[name])
    tuned = evaluate(tuned_model_generate, cases, check_fn[name])
    flag = "REGRESSION" if tuned < base - 0.02 else ""          # 2-point tolerance = your noise floor
    print(f"{name:12s} base={base:.2%} tuned={tuned:.2%} {flag}")

This is a sketch: base_model_generate, tuned_model_generate, the case lists and check_fn are placeholders for your own model calls and graders. The important structure is the same cases, both models, every suite, with an explicit tolerance. With small suites, random variation is large: a 100-case suite has a standard error around 5 percentage points near 50%, so a 2-point tolerance there is optimistic. Use larger suites or repeat runs before deciding that a small change is real.

When the task is narrow, high-volume, and latency/cost-sensitive — classification, extraction, formatting, a fixed style — a small model fine-tuned specifically for it can match or beat a much larger prompted general model on that exact task, at a fraction of the per-request cost and latency, because the fine-tune compresses “how to do this one thing well” directly into the weights instead of re-deriving it from instructions every call. It is the wrong call when the task is broad, evolving, or knowledge-dependent — fine-tuning does not solve a knowledge-access problem (that is RAG, see the retrieval topic) and a narrowly fine-tuned small model can be brittle outside the distribution it was trained on, where a larger prompted model degrades more gracefully. The decision hinges on whether the task is stable and narrow enough to be worth committing to weights, versus needing the flexibility of being redirected purely through prompt changes.

Why the economics work

A prompted large model pays for the instructions and few-shot examples on every call. A tuned small model has them in its weights. With purely illustrative numbers:

# Illustrative only: substitute your own prices and token counts.
calls = 5_000_000                       # per month
prompt_tokens_large = 1_800             # long instructions + 5 few-shot examples
prompt_tokens_small = 150               # fine-tuned model needs only the input
price_large, price_small = 3.0, 0.2     # dollars per million input tokens (made-up)

cost_large = calls * prompt_tokens_large * price_large / 1e6
cost_small = calls * prompt_tokens_small * price_small / 1e6
print(cost_large, cost_small)           # 27000.0 150.0

The gap comes from two multipliers (shorter prompt, cheaper model). Against that, add the one-off costs: data creation, training runs, evaluation, and the maintenance whenever requirements change, since changing a prompt is instant but changing a fine-tune needs a new training run.

Decision table

Situation	Better choice
Fixed output schema, millions of calls, tight latency	fine-tune a small model
Answers depend on documents that change weekly	RAG (fine-tuning will go stale)
Requirements change often; few labelled examples	prompt a large model, iterate on the prompt
Need a consistent brand voice across many tasks	fine-tune (or system prompt first; escalate if it does not hold)
Need new factual knowledge	RAG first; fine-tuning is unreliable for adding facts

A capability can live in four places: in the prompt (plain context), behind a tool (called on demand), in a retrieval layer (RAG: fetched text placed in the prompt), or in the weights (fine-tuning). Each has a different cost profile, freshness, and failure mode. Choosing is mostly a matter of asking six questions about the capability.

The default order: prompt, then retrieval or tools, then weights

Reach for prompt engineering (adjusting the instructions and examples in the prompt) first: it is the cheapest lever and iterates in minutes. Move to RAG or a tool when the underlying problem is that the model lacks access to particular knowledge — specific facts, current information, proprietary or internal content it was never trained on — rather than a behaviour or style problem. Reach for fine-tuning when you need to shape behaviour, format or style consistently at scale, or teach a narrow, specialised task pattern. It is expensive and slow to iterate on compared with a prompt change, and it does not solve a knowledge-access problem: a fine-tuned model has no better access to facts it was never trained on than the base model, because fine-tuning shapes behaviour, not the fresh, query-time knowledge that RAG injects. The six questions below decide the cases where the order is not obvious.

The six questions

1. How often does it change? Per second (balance, stock level), per week (policy docs), or almost never (tone of voice)? Weights are stale the moment training ends; a tool is always current.
2. How big is it? If it fits in context cheaply, put it in the prompt. If it is far larger than the context, it needs retrieval or a search tool.
3. Is it a read or an action? Reading can be done by tools, RAG, or context. A side effect (send, create, charge) can only be done by a tool.
4. Is it knowledge or behaviour? Facts belong in context/RAG/tools. Format, style, and consistent decision rules can be put in weights.
5. What is the per-request cost and latency budget? Every extra token costs money and prefill time; every tool round trip adds a full model call.
6. Does it need an audit trail or exact answers? Tools and RAG can cite what they fetched; weights cannot.

Decision table

	Plain context	RAG	Tool	Fine-tuning
Freshness	as fresh as you paste it	as fresh as the index	live	frozen at training time
Size it can handle	up to the context window	corpus of any size	any size, behind an API	not for bulk facts
Can perform actions	no	no	yes	no
Per-request token cost	high if the data is big	low (only top chunks)	low (only the result)	lowest (shorter prompts)
Latency added	prefill time of all tokens	one retrieval (tens of ms) plus prefill	one extra model round trip per call	none at inference
Upfront effort	none	build index, chunking, evaluation	write and secure the tool	data, training, evaluation, redeploy
Typical failure	context rot on huge inputs	retrieval misses the right chunk	wrong tool or arguments	forgets, memorises, goes stale
Best for	small, stable reference	large reference text	live data, actions, big databases	format, tone, repeated behaviour

Worked scenarios (illustrative prices)

The figures use illustrative prices of $3 per million input tokens and $15 per million output tokens; substitute your provider’s real prices. The point is the shape of the comparison, not the absolute dollars.

IN, OUT = 3.0, 15.0                           # $ per million tokens (illustrative)
def cost(inp, out): return (inp * IN + out * OUT) / 1e6

# Scenario 1: question over a 400-page handbook (about 200,000 tokens)
stuff_all = cost(200_000 + 300, 300)          # paste the whole handbook every time
rag       = cost(3_000, 300)                  # question + 5 retrieved chunks of ~500 tokens + instructions
print(round(stuff_all, 4), round(rag, 4))     # 0.6054 0.0135   -> 45x cheaper per question
print(round(stuff_all * 1000), round(rag * 1000))   # 605 14    -> dollars per 1,000 questions per day

# Scenario 2: "What is my current account balance?" through a tool
call1 = cost(1_500, 50)                       # system prompt + tool definitions + question -> tool call
call2 = cost(1_700, 100)                      # same + tool result -> final answer
print(round(call1 + call2, 5))                # 0.01185 per question, two model round trips

# Scenario 3: strict-JSON extraction with few-shot examples in the prompt vs a fine-tune
few_shot  = cost(1_400, 80)                   # 1,200 tokens of examples + 200 for the document
fine_tuned = cost(300, 80)
saving = few_shot - fine_tuned
print(round(saving, 4), round(2000 / saving))  # 0.0033 per request; 606,061 requests to repay a $2,000 project

Scenario 1 — handbook Q&A. Pasting 200,000 tokens per question costs about 45 times more than retrieving about 3,000 tokens, and makes every request pay the prefill time of the full document (seconds, not milliseconds) with the added risk of context rot. With provider prompt caching the repeated handbook prefix may be billed at a fraction of the normal rate (some providers charge roughly a tenth for cache reads; check yours), which narrows the gap, but RAG is still smaller and faster. Choose RAG, and go back to plain context only if the whole document is a few thousand tokens. Freshness: update the index when the handbook changes.

Scenario 2 — account balance. The data changes constantly and lives in a system of record, so only a tool is correct; the cost is a second model call (about 1 extra second of latency in a typical setup plus the API call itself) and a few thousand tokens per question. RAG over stale snapshots or a fine-tune would give confident wrong numbers. The tool result should return just the balance, not the customer record (see Designing and bounding tool results in Tool use and agent loops, AI agents topic).

Scenario 3 — format at volume. If a task runs a million times a month and the prompt carries 1,200 tokens of few-shot examples, a fine-tune that internalises the format saves about $0.0033 per request, about $3,300 per million requests, and cuts prefill latency. Whether it is worth it depends on project cost: with $2,000 of effort the break-even is roughly 600 thousand requests. At 10,000 requests a month, keep the prompt (or use constrained decoding for guaranteed JSON, see Structured output and constrained decoding).

Scenario 4 — vague search over 10 million products. Far beyond any context, and the query often needs refinement (filter by price, then by brand). A tool wrapping a search index lets the model iterate; single-shot RAG is an acceptable cheaper version if one query is enough.

Scenario 5 — company tone of voice. Stable, behavioural, small: start with the prompt; fine-tune only when prompt size or consistency becomes a measured problem.

Combining them

These compose. A typical production assistant has a short system prompt (behaviour), a fine-tuned or prompted format, a retrieval tool over documentation (read), and action tools (write). The common mistakes:

• Fine-tuning to inject facts — the single most common mistake: treating a knowledge/retrieval problem as a training problem. Facts in weights go stale and are recalled unreliably; RAG or a tool is dramatically cheaper, faster to build and easier to keep current.
• A tool for static reference text that would fit in plain context: you pay a round trip for nothing.
• RAG for live data: the index is always slightly behind.
• A tool for something that needs a guaranteed format: tool arguments are validated by schema, but the model can still choose the wrong tool; use constrained decoding for hard format guarantees.
• Skipping the evaluation. Pick on the six questions, then confirm with a small test set per option rather than on intuition.