Quantization and inference efficiency

• Weight (parameter) — a number learned in training and then frozen; an LLM has billions of them, mostly in matrices.
• Activation — a number computed during a forward pass from the input (hidden states, attention outputs). Unlike weights, activations differ for every input.
• Quantization — storing numbers with fewer bits (e.g. 4 or 8 instead of 16) at the cost of some rounding error.
• bf16 / fp16 / fp32 — floating-point formats with 16 / 16 / 32 bits. bf16 keeps fp32’s range with less precision; fp16 has more precision but a narrow range.
• fp8 — 8-bit floating-point formats (commonly E4M3 and E5M2: sign + exponent bits + mantissa bits).
• Sign / exponent / mantissa — the three fields of a float: the sign bit, which power of two (exponent), and the binary digits after the leading 1 (mantissa). Exponent bits set range; mantissa bits set precision.
• Bias — a constant (2^(E-1) - 1) subtracted from the stored exponent so it can represent negative powers of two.
• Subnormal — a tiny float with exponent field 0 and no implicit leading 1, filling the gap between the smallest normal number and zero.
• Overflow / underflow — a value too large for the format (becomes infinity) / too small (rounds to zero).
• int8 / int4 — 8-bit / 4-bit integers: evenly spaced values, no exponent.
• Scale — the real-number step size between adjacent integer levels.
• Zero-point — the integer that represents real 0.0 in an asymmetric mapping.
• Symmetric / asymmetric quantization — range centred on zero (no zero-point needed) / range from min to max with a zero-point.
• Per-tensor / per-channel / per-group — how many weights share one scale: a whole matrix / one output row or column / a block of e.g. 32-128 consecutive weights.
• Group size — how many consecutive weights share one scale in per-group quantization (typically 32, 64, or 128).
• Dynamic range — the ratio between the largest and smallest magnitudes a format can represent.
• Outlier — a value much larger than the rest of its tensor, which stretches the quantization range.
• Calibration — running a few hundred sample inputs through the model to measure activation ranges or sensitivity.
• PTQ (post-training quantization) — quantize after training, no gradient updates.
• QAT (quantization-aware training) — simulate quantization during training so the model adapts.
• GPTQ — a PTQ method that rounds weights one at a time and corrects the remaining weights using second-order (Hessian) information.
• Hessian — the matrix of second derivatives of the error; here, it says which weights the layer’s output is most sensitive to.
• AWQ (activation-aware weight quantization) — a PTQ method that protects the weight channels multiplied by large activations.
• GGUF — the single-file model format used by llama.cpp; k-quants / i-quants are its quantization families (Q4_K_M, IQ3_XS …) using block-wise scales and, for i-quants, codebooks (small lookup tables of representative values).
• NF4 — “NormalFloat 4”, a 4-bit format whose levels are placed to suit normally distributed weights; used by QLoRA.
• Perplexity — exp(average cross-entropy loss); a measure of how surprised the model is by held-out text. Lower is better.
• LoRA (low-rank adaptation) — fine-tuning by training two small matrices whose product is added to a frozen weight.
• QLoRA — LoRA on top of a 4-bit-quantized frozen base model.
• Rank (r) — the inner dimension of the LoRA factors; controls how many parameters are trained.
• Distillation — training a small student to imitate a large teacher.
• Pruning — removing weights (or whole neurons/heads) judged unimportant.
• HBM bandwidth — speed of reading weights from GPU memory; the limit during token-by-token decoding.
• bpw (bits per weight) — the effective storage cost of one weight including its share of the scales; always a little above the nominal bit count.
• EXL2 / EXL3 — the quantization formats of ExLlama (a GPU inference engine), which mix bit widths per layer to hit a target average bpw.
• Unsloth — a fine-tuning toolkit that also publishes “dynamic” GGUF quants keeping sensitive layers at higher precision.
• bitsandbytes — a library that quantizes weights to int8 / NF4 / FP4 on the fly while a model loads; the usual backend for QLoRA.
• MXFP4 / NVFP4 — microscaling 4-bit float formats: small blocks of values (32 for MXFP4) share one 8-bit exponent scale.
• Ada / Hopper / Blackwell — NVIDIA GPU generations; Ada and Hopper added native fp8 arithmetic, Blackwell adds native fp4.
• llama.cpp / Ollama / LM Studio — the C/C++ local inference engine that reads GGUF, and two desktop apps built on it.
• vLLM / TensorRT-LLM / TGI — data-centre serving engines (TGI = Hugging Face Text Generation Inference) that load GPTQ, AWQ and fp8 checkpoints.
• Transformers / PEFT — Hugging Face’s model-loading library, and its Parameter-Efficient Fine-Tuning library (LoRA and related methods).
• safetensors — Hugging Face’s safe, memory-mappable tensor file format; GPTQ/AWQ checkpoints ship as a folder of these files.

Every quantization argument is about how many distinct values a format can hold and where they sit on the number line. With b bits there are only 2^b distinct bit patterns, so 16 bits give 65,536 values, 8 bits give 256, 4 bits give 16. This concept builds the formats from their bit layouts, decodes one number in each, computes ranges and precision, and shows what rounding does to real values.

Anatomy of a floating-point number

A float stores a number in scientific notation, in binary: a sign bit, an exponent (which power of two), and a mantissa (also called fraction or significand: the digits after the leading 1).

value = (-1)^sign  x  (1 + mantissa / 2^M)  x  2^(exponent - bias)            (normal numbers, exponent > 0)
value = (-1)^sign  x  (mantissa / 2^M)      x  2^(1 - bias)                    (subnormal numbers, exponent = 0)

Symbols: sign is 0 or 1; exponent is the unsigned integer stored in the exponent field; E is the number of exponent bits and bias = 2^(E-1) - 1 (shifts the stored exponent so it can represent negative powers); M is the number of mantissa bits; mantissa is the unsigned integer stored in those bits. The “1 +” is the implicit leading 1: it is not stored, which is why M bits of mantissa give M + 1 bits of precision. Subnormals (exponent field all zeros) drop the implicit 1 so that values can fade smoothly toward zero instead of jumping from the smallest normal number to 0.

Bit layouts

fp32   [ 1 | 8 bits exponent | 23 bits mantissa ]    32 bits   bias 127
fp16   [ 1 | 5 bits exponent | 10 bits mantissa ]    16 bits   bias 15
bf16   [ 1 | 8 bits exponent |  7 bits mantissa ]    16 bits   bias 127
fp8    E4M3: [ 1 | 4 bits exp | 3 bits mant ]         8 bits   bias 7
fp8    E5M2: [ 1 | 5 bits exp | 2 bits mant ]         8 bits   bias 15
int8   [ 8 bits two's complement ]                    8 bits   -128 .. 127   (no exponent; times a scale)
int4   [ 4 bits two's complement ]                    4 bits   -8 .. 7       (no exponent; times a scale)

Note that bf16 is simply the top 16 bits of an fp32 (same sign and exponent, mantissa cut from 23 to 7 bits). That is why converting fp32 to bf16 is trivial and why bf16 has the same range as fp32.

Decoding one number in every format: 0.15625

0.15625 = 1.25 x 2^-3, so the leading digits are 1 + 0.25 and the power is -3. Each format stores exponent = -3 + bias and mantissa = 0.25 x 2^M:

format	stored fields (sign, exponent, mantissa)	decode
fp32	0, 01111100 (124 = -3 + 127), 01000000000000000000000 (0.25 x 2^23)	1.25 x 2^(124-127)
bf16	0, 01111100, 0100000 (0.25 x 2^7 = 32)	same
fp16	0, 01100 (12 = -3 + 15), 0100000000 (0.25 x 2^10 = 256)	1.25 x 2^(12-15)
fp8 E4M3	0, 0100 (4 = -3 + 7), 010 (0.25 x 2^3 = 2)	1.25 x 2^(4-7)
fp8 E5M2	0, 01100, 01 (0.25 x 2^2 = 1)	1.25 x 2^(12-15)

def decode(sign, exp, mant, m_bits, bias):
    """Value of a float with the given fields (normal numbers and subnormals)."""
    if exp == 0:                                         # subnormal: no implicit leading 1
        return (-1) ** sign * (mant / 2**m_bits) * 2 ** (1 - bias)
    return (-1) ** sign * (1 + mant / 2**m_bits) * 2 ** (exp - bias)

print(decode(0, 4, 0b010, 3, 7))                         # e4m3: 0 0100 010                  -> 0.15625
print(decode(0, 12, 0b01, 2, 15))                        # e5m2: 0 01100 01                  -> 0.15625
print(decode(0, 12, 0b0100000000, 10, 15))               # fp16: 0 01100 0100000000          -> 0.15625
print(decode(0, 124, 0b0100000, 7, 127))                 # bf16: 0 01111100 0100000          -> 0.15625
print(decode(0, 124, 0b01000000000000000000000, 23, 127))  # fp32                            -> 0.15625

Walk-through: all five formats represent 0.15625 exactly because it is 1.25 x 2^-3 and 0.25 needs only two mantissa bits. A number like 0.1 needs infinitely many binary digits, and each format cuts it off at a different point; that cut-off is the rounding error.

Range and precision, computed

format	largest finite value	smallest normal	smallest subnormal	relative step (spacing near 1.0)	distinct bit patterns
fp32	3.4e38	1.18e-38	1.4e-45	1.19e-7 (2^-23)	about 4.3 billion
bf16	3.39e38	1.18e-38	9.2e-41	7.8e-3 (2^-7)	65,536
fp16	65,504	6.1e-5	6.0e-8	9.8e-4 (2^-10)	65,536
fp8 E4M3	448	1.56e-2 (2^-6)	1.95e-3 (2^-9)	0.125 (2^-3)	256
fp8 E5M2	57,344	6.1e-5	1.5e-5	0.25 (2^-2)	256
int8	127 x scale	uniform step = scale		absolute step = scale	256
int4	7 x scale	uniform step = scale		absolute step = scale	16

How to read it: the exponent bits set the range (how large and small a value can be); the mantissa bits set the precision (how finely values are spaced within each power of two). bf16 spends its bits on range (8 exponent bits) at the cost of precision (7 mantissa bits, about 2-3 significant decimal digits); fp16 does the opposite (5 exponent bits, 10 mantissa bits). Spacing doubles with each power of two: near 1.0 an E4M3 value steps by 0.125, near 256 it steps by 32 (2^8 x 2^-3). That is relative precision: always the same fraction of the number’s size (E4M3: between 6.25% and 12.5% per step).

Floating-point E4M3 has no infinity and only one NaN pattern per sign, which is how it reaches a maximum of 448 rather than 240; E5M2 follows IEEE conventions (infinities and NaNs) and so tops out at 57,344. Formats and special values differ between hardware vendors, so check the exact variant your hardware supports.

What rounding actually does

import struct
import numpy as np

def to_bf16(x):                                          # round-to-nearest-even on the top 16 bits of an fp32
    u = struct.unpack(">I", struct.pack(">f", x))[0]
    u = (u + 0x7FFF + ((u >> 16) & 1)) >> 16 << 16
    return struct.unpack(">f", struct.pack(">I", u))[0]

for x in (3.14159265, 70000.0, 1e-8):
    with np.errstate(over="ignore"):
        print(x, "fp16", float(np.float16(x)), "bf16", to_bf16(x))
# 3.14159265 fp16 3.140625 bf16 3.140625              both keep ~3 digits; fp32 would give 3.1415927
# 70000.0    fp16 inf      bf16 70144.0               fp16 OVERFLOWS (max 65,504); bf16 rounds to 70144 (step 512 there)
# 1e-08      fp16 0.0      bf16 1.0011717677116394e-08   fp16 UNDERFLOWS to 0; bf16 keeps the magnitude, loses digits

Walk-through: the three lines are the three failure modes. Pi loses digits in both 16-bit formats (relative error of about 0.03%). 70000 shows why fp16 training needs loss scaling and care: a value above 65,504 becomes infinity, destroying the computation. bf16 keeps the value but with only 7 mantissa bits: at 70,000 neighbouring bf16 values are 512 apart. 1e-8 shows underflow: gradients and small activations can vanish in fp16 but survive in bf16. This is the core reason LLMs are trained and served in bf16 rather than fp16.

Integers: uniform spacing times a scale

Integers have no exponent, so the integer grid is evenly spaced. To represent real weights, a scale s maps integer q back to a real value q x s (the next concept covers the mapping fully). Illustration with six weights [0.31, -0.12, 0.05, 0.9, -0.44, 0.02] and a symmetric scale from the largest magnitude:

import numpy as np
w = np.array([0.31, -0.12, 0.05, 0.9, -0.44, 0.02])

s4 = np.abs(w).max() / 7                                 # int4 range -8..7, so 7 steps cover 0..max
q4 = np.clip(np.round(w / s4), -8, 7).astype(int)
print(round(s4, 4), q4, np.round(q4 * s4, 3), round(np.abs(q4 * s4 - w).max(), 3))
# 0.1286 [ 2 -1  0  7 -3  0] [ 0.257 -0.129  0.     0.9   -0.386  0.   ] 0.054

s8 = np.abs(w).max() / 127
q8 = np.round(w / s8).astype(int)
print(round(s8, 5), q8, round(np.abs(q8 * s8 - w).max(), 4))
# 0.00709 [ 44 -17   7 127 -62   3] 0.0018

w2 = w.copy(); w2[3] = 9.0                               # one outlier, 10x larger
s = np.abs(w2).max() / 7
print(np.clip(np.round(w2 / s), -8, 7).astype(int))
# [0 0 0 7 0 0]    every other weight rounded to zero

Walk-through: with 16 levels (int4) the step is 0.1286, so small weights like 0.05 and 0.02 collapse to 0 and the worst error is 0.054. With 256 levels (int8) the step is 0.0071 and the worst error 0.0018. In the last case a single outlier of 9.0 stretches the scale to 1.29 and every other weight rounds to 0. That is why outliers are the central enemy of integer quantization, why scales are stored per small group of weights (so an outlier only damages its group), and why float formats, with relative precision, cope with outliers better than int4 (see fp8 vs int4 for inference — why format shape matters as much as bit count).

Floating point vs integer, in one comparison

fp8 (E4M3) values near 1:      1.0   1.125   1.25  ...  (step 0.125)    values near 100:  96  104  112 ... (step 8)
int8 values with scale 0.5:    ... 0.5   1.0   1.5 ...  (step 0.5 everywhere, range limited to 127 x 0.5 = 63.5)

Floats are dense near zero and sparse for large magnitudes (good for values spanning many orders of magnitude); integers are equally spaced (good for tensors whose values lie in a narrow band, and cheap to compute with).

Why LLMs run in bf16, not fp32

Training and inference in fp32 doubles memory and bandwidth for essentially no accuracy gain. bf16 keeps fp32’s 8 exponent bits (it never overflows where fp32 would not) and drops mantissa bits, which neural networks tolerate; accumulations inside matrix multiplies are usually still done in fp32 and rounded on output. Quantization pushes the same idea further: 8 or 4 bits for the stored weights, sometimes with the arithmetic itself in a wider format.

Effective bits per weight

A format’s nominal bit count understates its real storage when scales are involved: a group of 32 weights at 4 bits each plus one fp16 scale costs 32 x 4 + 16 = 144 bits for 32 weights, or 4.5 bits per weight. For group size 128 it is (128 x 4 + 16) / 128 = 4.125 bits. This is the “~4.5 bits” figure used in the memory table in the next concept.

Two separate wins, both from moving fewer bytes:

1. Fits in memory. Weight memory is N * bits / 8 bytes for N parameters.
2. Faster decoding. Token-by-token generation is limited by how fast weights stream from memory (see KV cache, prompt caching, and inference economics). Half the bytes means up to roughly twice the tokens per second.

for N, name in [(8.03e9, "8B"), (70.55e9, "70B")]:
    for bits, label in [(16, "bf16"), (8, "int8/fp8"), (4, "4-bit"), (4.5, "4-bit + scales (~4.5 bits)")]:
        print(f"{name:4s} {label:28s} {N * bits / 8 / 2**30:6.1f} GiB")
# 8B   bf16                          15.0 GiB
# 8B   int8/fp8                       7.5 GiB
# 8B   4-bit                          3.7 GiB
# 8B   4-bit + scales (~4.5 bits)     4.2 GiB
# 70B  bf16                         131.4 GiB
# 70B  int8/fp8                      65.7 GiB
# 70B  4-bit                         32.9 GiB
# 70B  4-bit + scales (~4.5 bits)    37.0 GiB

Reading it: a 70B model that needs two or more 80 GiB GPUs in bf16 fits on one in 4-bit (with room left for the KV cache). The “~4.5 bits” line is the honest figure: each group of weights also stores a scale (and sometimes a zero-point), which adds fractions of a bit per weight (the next concepts compute this). VRAM and running LLMs locally turns these figures into hardware choices.

It re-represents each floating-point value with a lower-precision number, typically via a linear scale + zero-point mapping per tensor or per smaller group of values:

q = round( x / scale ) + zero_point         (quantize:   float -> int)
x_hat = ( q - zero_point ) * scale          (dequantize: int -> approx. float)

scale is chosen from the value range being quantized (e.g. scale = (max - min) / (2^bits - 1) for the block being quantized). The error x - x_hat is the quantization noise — bounded by roughly half a quantization step, and it accumulates across the many multiply-accumulates in a forward pass. Fewer bits (int8, int4) mean a coarser scale and more noise per value; weights, not activations, are usually the primary target because they are static and dominate memory, while activations (computed per input) are quantized more cautiously since their distribution shifts by input.

Every symbol

• x — an original floating-point value (a weight). Given by the trained model.
• q — the stored integer (e.g. 0…255 for 8 bits unsigned). This is what lives in memory.
• scale — a float, computed from the data (not learned by gradient descent): the real-number distance between two adjacent integer levels.
• zero_point — an integer, also computed from the data: the integer that maps back to exactly 0.0, so that zero (extremely common in weights) is represented without error.
• round(.) — round to the nearest integer; this is the only lossy step.
• x_hat — the reconstructed value used at compute time. x - x_hat is the error.

Worked example (asymmetric int8)

Six weights: [-0.9, -0.2, 0.0, 0.4, 1.1, 0.05]. Range min = -0.9, max = 1.1.

import numpy as np
x = np.array([-0.9, -0.2, 0.0, 0.4, 1.1, 0.05])
mn, mx = x.min(), x.max()
scale = (mx - mn) / 255                  # 8 bits -> 256 levels -> 255 steps: 2.0 / 255 = 0.007843
zp = round(-mn / scale)                  # integer that represents 0.0: 114.75 -> 115
q = np.clip(np.round(x / scale) + zp, 0, 255).astype(int)
x_hat = (q - zp) * scale
print(round(scale, 5), zp)               # -> 0.00784 115
print(q)                                 # -> [  0  89 115 166 255 121]
print(x_hat.round(4))                    # -> [-0.902 -0.2039  0.  0.4  1.098  0.0471]
print(np.abs(x - x_hat).max(), scale / 2)  # -> 0.0039 0.0039  (max error equals half a step)

Walkthrough for x = -0.2: -0.2 / 0.00784 = -25.5, which rounds to -26 (numpy rounds exact halves to the even neighbour; either neighbour is equally good), plus zero-point 115 gives q = 89. Dequantizing: (89 - 115) * 0.00784 = -0.2039. The error (0.0039) is at most scale / 2, as promised. The minimum and maximum map exactly to 0 and 255.

Worked example (symmetric int4)

Symmetric quantization drops the zero-point: scale = max(|x|) / 7 (use the positive limit of a signed 4-bit range -8…7) and q = round(x / scale).

s = np.abs(x).max() / 7                  # 1.1 / 7 = 0.1571
q4 = np.clip(np.round(x / s), -8, 7).astype(int)
print(q4, (q4 * s).round(3))
# -> [-6 -1  0  3  7  0]  [-0.943 -0.157  0.  0.471  1.1  0.]
print(np.abs(x - q4 * s).max())          # -> 0.0714  (half step is 0.0786)

With only 16 levels, the weight 0.05 is rounded all the way to zero and -0.2 becomes -0.157: the step is 20x coarser than int8’s. Symmetric is simpler and faster to compute (no zero-point term in the matmul); asymmetric uses the range better when values are skewed.

Granularity — how many weights share one scale

per-tensor : 1 scale for the whole matrix              cheapest, worst if any outlier exists
per-channel: 1 scale per output row                     common for int8
per-group  : 1 scale per 32..128 consecutive weights    standard for 4-bit

Smaller groups follow local ranges, so each scale is less distorted by outliers, at the price of storing more scales: with groups of 128 and one fp16 scale each, the overhead is 16 / 128 = 0.125 extra bits per weight (0.156 with a 4-bit zero-point as well), which is where the “~4.5 bits” figure comes from.

Why outliers hurt, with numbers

y = np.array([0.1, -0.2, 0.15, 0.05, -0.1, 0.2, -0.05, 8.0])   # one outlier
s = np.abs(y).max() / 7                                          # 8.0 / 7 = 1.143 per level
print(np.round(y / s) * s)
# -> [0. -0. 0. 0. -0. 0. -0. 8.]   every ordinary weight collapsed to zero

One outlier stretches the range so that the seven normal weights (all under 0.2, far less than half a step of 1.14) round to zero. Group-wise scales or protecting outlier channels are the standard remedies.

Group size experiment

Random weights from a bell curve with a few large outliers (illustrative), 4-bit symmetric:

rng = np.random.default_rng(0)
W = rng.normal(0, 0.02, 4096 * 64).astype(np.float32); W[::1000] *= 15   # sprinkle outliers

def quant_dequant(W, group, bits=4):
    qmax = 2**(bits - 1) - 1
    Wg = W.reshape(-1, group)
    s = np.abs(Wg).max(1, keepdims=True) / qmax
    q = np.clip(np.round(Wg / s), -qmax - 1, qmax)
    return (q * s).reshape(-1)

for g in [W.size, 4096, 128, 32]:
    print(g, float(np.sqrt(np.mean((W - quant_dequant(W, g)) ** 2))))
# 262144 0.01986    (one scale for everything)
# 4096   0.01527
# 128    0.00435
# 32     0.00267    (RMS error ~7x smaller than per-tensor)

The root-mean-square error falls by about 7x going from one global scale to groups of 32. That is the mechanism behind “k-quants”, AWQ/GPTQ group sizes of 128, and so on: you pay a little storage for much lower error.

• PTQ — quantize an already-trained model’s weights (and optionally activations) after the fact, with at most a small calibration pass (running a few hundred sample inputs to estimate good scales/ranges). Cheap and fast — the dominant approach for large pretrained LLMs, since re-training a 70B+ model just to quantize it is rarely worth the cost.
• QAT — simulate quantization during training (fake-quantize weights/activations in the forward pass, keep full-precision gradients) so the model learns weights that are robust to the eventual rounding. Higher accuracy at a given bit-width, but requires training compute you may not have or want to spend, and is more common at the pretraining stage of very-low-bit models than as a retrofit.

Most “quantize this open-weights checkpoint” workflows in practice are PTQ.

How QAT works in one step

Rounding has zero gradient almost everywhere, so QAT uses the straight-through estimator: in the forward pass the weight is replaced by its quantize-dequantize version x_hat; in the backward pass the gradient is passed through as if rounding were the identity. The full-precision “master” weights keep being updated; the model sees the rounded ones.

import torch

class FakeQuant(torch.autograd.Function):
    @staticmethod
    def forward(ctx, w, scale):
        return torch.round(w / scale).clamp(-8, 7) * scale   # quantize then dequantize (4-bit symmetric)
    @staticmethod
    def backward(ctx, grad_out):
        return grad_out, None                                 # straight-through: pretend round() was identity

w = torch.tensor([0.31, -0.74, 0.05], requires_grad=True)
wq = FakeQuant.apply(w, 0.1)
wq.sum().backward()
print(wq.detach(), w.grad)    # -> tensor([ 0.3000, -0.7000,  0.0000]) tensor([1., 1., 1.])

The forward value has been rounded (0.31 -> 0.30, 0.05 -> 0.0) while the gradient reaches the underlying float weight unchanged, so training can keep adjusting it.

Choosing

Situation	Choice
Open-weights model, 8-bit or good 4-bit target, no training budget	PTQ (GPTQ/AWQ/GGUF)
Need quality at very low bits (3-bit or below) and you control training	QAT or train natively at low precision
Model already fine-tuned with LoRA	quantize the base (QLoRA) or merge then PTQ

Gotcha: PTQ calibration data should resemble real use. Calibrating on English web text then deploying for code or another language can leave the scales mis-tuned for the real activation ranges.

Naive round-to-nearest quantization treats every weight as equally important, but empirically a small fraction of weights (and, more so, activation channels) have outsized impact on output quality — quantizing them coarsely causes disproportionate error.

• GPTQ quantizes weights layer by layer, and after rounding each weight, adjusts the remaining unquantized weights in that layer to compensate for the error just introduced (using second-order / Hessian-based information from a calibration set), so the cumulative error across the whole layer is minimized rather than each weight being rounded independently and errors just piling up.
• AWQ (activation-aware quantization) observes that certain weight channels matter more because of the activation magnitudes they multiply against, and protects those specific channels (via per-channel scaling) from precision loss rather than treating all channels uniformly.

Both meaningfully outperform naive int4 rounding at the same bit-width, which is why they became the standard recipes for 4-bit LLM deployment rather than “just round every weight.”

What the layer is really trying to preserve

A linear layer computes y = W x. Quantization replaces W with W_hat. What matters is not |W - W_hat| but the error in the output: |W x - W_hat x| over typical inputs x. A weight multiplied by large inputs contributes more output error for the same rounding error. Both methods exploit this: they use a small calibration set to learn which directions in x are large.

Why AWQ’s channel scaling works (numbers)

Suppose one input channel is consistently large (say activations about 10) and the weight on it is 0.30; another channel has activations about 0.1 and weight 0.30. With a step of 0.1 the weights round to 0.3 in both cases, error 0 here, but at weight 0.34 the rounding error 0.04 turns into an output error of 0.04 * 10 = 0.4 on the large channel versus 0.004 on the small one. AWQ multiplies the important channel’s weights by a factor s > 1 before quantizing (so they occupy more of the quantization grid, and their relative rounding error shrinks) and divides the matching activation by s so the product is unchanged mathematically. Illustrative: with s = 2, the weight 0.34 becomes 0.68, which rounds on the 0.1 grid to 0.7; undoing the scale gives 0.35, an error of 0.01 instead of 0.04. In general the rounding error (up to 0.05 on the grid) is halved once the scale is undone, in exchange for a coarser relative grid on the other channels.

GPTQ in a sentence you can picture

for each column of W in order:
    round that column to the grid -> get error e
    spread e over the not-yet-rounded columns, in the proportions the Hessian says reduce output error

Because later columns are adjusted to absorb earlier errors, the total error in W x is much smaller than rounding each weight independently.

Using them

# Illustrative: AutoAWQ / GPTQModel / llama.cpp each have their own CLI and API; check the project's docs.
# Typical workflow shape:
#   1. pick a calibration set (a few hundred representative samples)
#   2. run the quantizer (minutes to a few hours for 70B)
#   3. evaluate on YOUR tasks before and after (see the evaluation concept below)

A concrete AutoAWQ run is in Quantization formats compared below.

Gotchas: results vary by group size, calibration set, and model; 4-bit recipes occasionally leave particular layers (embeddings, the final layer norm, sometimes lm_head) in higher precision for stability.

GGUF (used by llama.cpp and its ecosystem) is a single-file format bundling the quantized weights, tokenizer, and architecture metadata needed to load and run a model with no external framework dependency — the point is portability and local/CPU-friendly inference (laptops, consumer GPUs, edge devices) rather than the data-center serving stack (vLLM/TensorRT-LLM) most production APIs use. It supports a range of named quantization schemes (various k-quants) trading size against quality, and its popularity comes from making “download one file and run a capable model locally” trivial. Its trade-off is that it targets breadth of hardware and simplicity of deployment over the throughput-maximizing techniques (continuous batching, PagedAttention) that data-center serving stacks focus on.

What is inside the file

GGUF file
+-- header: magic bytes, version, counts
+-- metadata (key-value): architecture ("llama"), context length, head counts,
|                         tokenizer vocab and merges, chat template, quantization type ...
+-- tensor index: name, shape, quantization type, offset  for every weight tensor
+-- tensor data (aligned, ready to memory-map)

Because everything is in one file, llama.cpp can memory-map it and start quickly, and the same file works on CPU, Apple silicon, and GPUs.

Reading the quantization names

Names like Q4_K_M and Q8_0 encode: the approximate bits per weight (Q4 = 4-bit class), the scheme (_0/_1 = simple block quantization with a scale (and minimum), K = k-quant with hierarchical block scales, an IQ prefix = i-quant with codebooks), and a size/quality variant (S, M, L, small/medium/large, which keep more sensitive tensors at higher precision).

Qwen2.5-32B-Instruct-Q4_K_M.gguf
                     | | |
                     | | +-- M = medium (mix of block sizes; S small, L large)
                     | +---- K = k-quant (block-wise scales with super-block scales)
                     +------ 4 = roughly 4 bits per weight (real: ~4.8 bpw)

Effective bits are higher than the label because the scales are stored too: Q4_K_M is about 4.8 bpw, Q5_K_M about 5.7, Q6_K about 6.6, Q8_0 about 8.5. Typical rule of thumb (approximate, model-dependent): Q8_0 is nearly indistinguishable from bf16, Q5_K_M and Q4_K_M are the usual sweet spots, and below about 3 bits quality drops noticeably, fastest on small models. File sizes are close to the “4-bit + scales” row in the memory table above, plus a little.

Using one

# Illustrative llama.cpp usage (check the project's current docs for exact flags)
llama-cli -m ./model-Q4_K_M.gguf -p "Explain KV caching in two sentences." -n 128 -ngl 99
#   -m    path to the GGUF file
#   -n    number of tokens to generate
#   -ngl  how many layers to offload to the GPU (99 = all that fit)

For the full hardware sizing workflow see VRAM and running LLMs locally; for converting and quantizing a model to GGUF yourself, and how GGUF sits next to the other formats, see Quantization formats compared below.

When GGUF vs a data-center format

Need	Prefer
One user, laptop/desktop, CPU or small GPU, easy setup	GGUF + llama.cpp (or an app built on it)
Many concurrent users on server GPUs, max throughput	vLLM / TensorRT-LLM with AWQ, GPTQ, or fp8 weights
Offloading part of the model to CPU RAM when VRAM is short	GGUF (partial layer offload is a first-class feature)

fp8 keeps a floating-point representation (a small exponent plus mantissa) at 8 bits, so it retains a wide dynamic range — it represents both very small and very large magnitudes reasonably, which matters because weight and activation distributions in transformers have real outliers. int4 is a linear fixed-point format — uniform step size across its range — so it represents a narrow range well but clips or badly resolves outliers unless mitigated (grouping, AWQ/GPTQ-style protection, or per-channel scales). This is why int4 needs materially more careful calibration to match fp8’s accuracy despite using half the bits: the format’s shape, not only its bit count, determines how well it fits the actual value distribution. In practice fp8 is increasingly used for both training and inference on modern accelerators with native fp8 support, while int4 remains the choice when memory/bandwidth savings must go further than fp8 allows and some additional accuracy loss is acceptable.

Where the 256 or 16 levels sit

int4 (symmetric, scale = 0.1):   -0.8 ... -0.2 -0.1  0  0.1  0.2 ... 0.7        evenly spaced, 16 levels, nothing beyond +-0.8
fp8 E4M3:        ... 0.0156 0.0312 0.0625 0.125 0.25 0.5 1 2 4 8 ... 448       log-spaced: tiny numbers AND big ones
                      dense near zero, sparse for large magnitudes (relative step 6-12%)

Floating-point levels are dense near zero (where most weights live) and sparse far out, so an outlier of 50 times the typical weight still lands on a level within a few percent of its size. int4 has no such tail: either the scale is stretched for the outlier (small weights collapse to zero, as in the worked example above) or the outlier is clipped.

Practical comparison

	fp8 (E4M3 weights)	int4 (grouped)
Memory vs bf16	0.5x	~0.28x (with group scales)
Accuracy loss	usually small, little tuning	needs good method (AWQ/GPTQ), noticeable if done naively
Hardware	needs native fp8 support (recent data-centre GPUs) for speed	dequantized in kernels; runs on many GPUs and CPUs
Typical use	server inference, sometimes training	local/edge, or fitting a very large model on few GPUs

Gotchas: fp8 “weights only” and fp8 “weights and activations” are different (the latter also needs activation scales and gives compute speedups, not just memory savings). Many int4 deployments are “W4A16”: weights in 4-bit, but dequantized to 16-bit for the arithmetic; the win is bandwidth, not compute. A related fp4 family exists on the newest hardware; treat it as hardware-specific.

The concepts above explain the maths (scale and zero-point, GPTQ/AWQ, GGUF internals, float vs integer shape). This one is the map: which format you will meet and which tool consumes it — they are not interchangeable.

Format	Typical bits	Made by	Runs in	Notes
GGUF (Q8_0, Q5_K_M, Q4_K_M, Q3_K_S, IQ2_XS …)	2-8	llama.cpp tools	llama.cpp, Ollama, LM Studio	Single file with weights, tokenizer, metadata. Block-wise scales. CPU, Apple, and GPU.
GPTQ	3-4, 8	Calibration with approximate second-order (Hessian) error minimisation	vLLM, Transformers, ExLlama	GPU only; needs calibration data; quantizes layer by layer to compensate rounding error.
AWQ	4	Activation-aware: protects ~1% of salient weight channels by scaling	vLLM, Transformers, TGI	Fast, accurate at 4-bit; GPU only.
EXL2 / EXL3	2-8, mixed per layer	ExLlama	ExLlama	Variable bit-rate to hit a target average (e.g. 4.65 bpw).
bitsandbytes (int8, NF4, FP4)	4, 8	On-the-fly at load time	Transformers + PEFT	No calibration, no pre-made file; mainly for training (QLoRA). Slower inference.
fp8 (E4M3 / E5M2)	8	Cast or scaled-cast	vLLM, TensorRT-LLM; Hopper/Ada/Blackwell GPUs	Hardware-native float; near-lossless; 2x vs fp16.
MXFP4 / NVFP4	4	Microscaling floats	Blackwell-class GPUs; some open models ship natively	Float levels plus per-block scale; some models (e.g. gpt-oss) are released directly in MXFP4.
Dynamic / mixed quants	mixed	e.g. Unsloth dynamic GGUF	llama.cpp	Keeps sensitive layers (embeddings, attention output, first/last layers) at higher bits.

How to read a GGUF name such as Q4_K_M (and its real bpw) is in the GGUF concept above; the group-size storage overhead (4.125 bpw at group 128, 4.5 at group 32) is worked out under “Effective bits per weight” and the scale/zero-point concept.

Quantizing a model yourself

# GGUF: convert HF weights, then quantize
python convert_hf_to_gguf.py ./Qwen2.5-7B-Instruct --outfile qwen-f16.gguf
llama-quantize qwen-f16.gguf qwen-Q4_K_M.gguf Q4_K_M

# AWQ with AutoAWQ (calibration-based, outputs a safetensors folder)
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

src = "Qwen/Qwen2.5-7B-Instruct"
model = AutoAWQForCausalLM.from_pretrained(src)
tok = AutoTokenizer.from_pretrained(src)
model.quantize(tok, quant_config={"zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM"})
model.save_quantized("qwen-awq")

# bitsandbytes NF4: quantizes while loading, no separate file
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,        # also quantize the scales (saves ~0.4 bit/weight)
    bnb_4bit_compute_dtype=torch.bfloat16, # matmuls run in bf16 after dequantizing
)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct", quantization_config=bnb, device_map="auto")

Choosing

• Local CPU/Mac/consumer GPU -> GGUF.
• vLLM serving on NVIDIA -> AWQ (4-bit) or fp8 (8-bit if hardware supports it).
• Fine-tuning on a small GPU -> bitsandbytes NF4 (QLoRA).
• Never fine-tune on top of a GGUF/GPTQ/AWQ artifact directly; train from the original weights, then quantize the merged result.

It rarely announces itself as an obvious crash — it shows up as quiet degradation: slightly worse reasoning on hard multi-step tasks, more frequent subtle factual slips, degraded performance specifically on long-tail/rare inputs (exactly where precision margins were thinnest), or occasional repetition/incoherence from noisier logits shifting the sampling distribution. It is also non-uniform across capabilities — a 4-bit quantization might barely affect casual chat quality while measurably hurting code correctness or arithmetic, because those tasks are more sensitive to precision in specific layers/channels. Catching it requires task-specific evals run before/after quantization (the same regression-testing discipline as any other model change) — comparing perplexity alone is a weak proxy and can look fine while downstream task accuracy has genuinely dropped.

Why small average error can still break a task

Errors compound across layers and tokens. A multi-step arithmetic answer requires every step to be correct, so a small per-step error rate multiplies: if each of 10 dependent steps is right 99% of the time the chain is right about 90% of the time; at 97% per step it is about 74% (0.99^10 = 0.904, 0.97^10 = 0.737). A model whose average log-loss moved by a hair can therefore lose several points on a long-chain benchmark.

Perplexity: useful, but a weak proxy

Perplexity = exp(mean cross-entropy). Illustrative: a loss rising from 2.00 to 2.05 nats per token moves perplexity from 7.39 to 7.77, about +5%. That looks small and may be reported as “negligible”, yet specific tasks (code, maths, other languages) can fall much further. Use it as a quick smoke test (a large jump means something is broken), never as the acceptance test.

A before/after check you can run

# Pseudocode-level harness: plug in your own model loaders and task set.
def evaluate(generate, cases):
    ok = 0
    for prompt, check in cases:          # check(output) -> bool, e.g. unit tests for code, exact match for extraction
        ok += bool(check(generate(prompt)))
    return ok / len(cases)

base = evaluate(base_model_generate, my_cases)
quant = evaluate(quant_model_generate, my_cases)
print(f"base {base:.1%}  quant {quant:.1%}  delta {quant - base:+.1%}")
# Decide in advance what delta you will accept (e.g. no worse than -1 point) and use enough cases
# that a 1-point change is distinguishable from noise (hundreds, not dozens).

Checklist

1. Evaluate on your tasks and languages, including the hardest 10%.
2. Compare at the same sampling settings (greedy or fixed seed) to reduce noise.
3. Test long-context and structured-output cases; they are often the first to wobble.
4. Compare several quantization levels (8-bit, 5-bit, 4-bit) and pick the smallest that passes, not the smallest available.

Quantization and pruning compress an existing model’s parameters while trying to preserve its learned function as closely as possible — bounded by how much redundancy that specific architecture and training run actually has. Distillation trains a genuinely smaller student model from scratch (or from a smaller pretrained checkpoint) to match a larger teacher’s outputs (soft labels, or the teacher’s generated data) — it is not compressing the same weights, it is teaching a different, smaller function to approximate the teacher’s behavior. Distillation is the better choice when you need a target size quantization/pruning cannot reach without unacceptable quality loss, when you want the smaller model specialized on a narrower task distribution than the teacher covers (distilling only on your domain’s data), or when you want a materially different, cheaper architecture rather than a shrunk version of the same one. It costs real training compute and data curation, unlike PTQ’s near-free retrofit, so it is the heavier-weight option reserved for when quantization/pruning alone do not hit the target.

How distillation works

Two common recipes:

1. Sequence-level (data) distillation. Have the teacher generate answers (often with reasoning) for many prompts, then fine-tune the student on those outputs as ordinary training data. Simple and works across any model pair, even through an API.
2. Logit (soft-label) distillation. Train the student to match the teacher’s probability distribution over the next token, not just the chosen token. Needs access to the teacher’s logits and a shared tokenizer, but carries much more information per example: a hard label says “the answer is token A”; a soft label says “A 85%, B 11%, C 3%”, which tells the student what is nearly right.

The soft-label loss, explained

Loss: L = KL( softmax(z_t / T) || softmax(z_s / T) ), usually multiplied by T^2 and often mixed with ordinary cross-entropy on real labels.

• z_t, z_s — teacher and student logits (outputs of the models; the student’s are what is trained).
• T — a temperature you choose (typically 1-4). Dividing logits by T before softmax flattens the distribution, so small probabilities (the “dark knowledge” of which wrong answers are near-misses) become visible.
• KL(p || q) = sum_i p_i * log(p_i / q_i) — how much extra surprise you incur using q when the truth is p; zero only when they are identical.
• T^2 — compensates for gradients shrinking as T grows.

import numpy as np
def softmax(z, T): e = np.exp((z - z.max()) / T); return e / e.sum()
z_t = np.array([4.0, 2.0, 0.5, -1.0])           # teacher logits
z_s = np.array([3.0, 2.5, 0.0, -0.5])           # student logits (still imperfect)
print(softmax(z_t, 1).round(3))                 # -> [0.853 0.115 0.026 0.006]  nearly one-hot, little to learn from
print(softmax(z_t, 4).round(3))                 # -> [0.433 0.263 0.18  0.124]  T=4 reveals the ranking of the wrong answers
pt, ps = softmax(z_t, 2), softmax(z_s, 2)
print(pt.round(3), ps.round(3))                 # -> [0.616 0.227 0.107 0.051] [0.46  0.358 0.103 0.08 ]
print((pt * np.log(pt / ps)).sum())             # -> 0.058  (KL at T=2; training pushes this toward 0)

Choosing between the shrinking tools

Tool	What changes	Cost	Typical gain	Main risk
Quantization (PTQ)	bits per weight	minutes to hours	2-4x smaller, faster decode	quiet degradation on hard tasks
Pruning (unstructured)	weights set to zero	calibration plus some fine-tuning	needs sparse kernels to be faster	speed gain not guaranteed
Structured pruning (e.g. 2:4 sparsity, dropping heads/layers)	fewer or patterned weights	calibration plus recovery training	hardware-supported speedups for patterned sparsity	quality loss, needs recovery
Distillation	a new smaller model	substantial training compute and data	any target size or architecture	student may miss teacher’s rare skills

They compose: distil to a smaller model, then quantize that student to 4-bit. Pruning is the least common for LLM serving, since unstructured sparsity rarely yields speed on common hardware unless the sparsity pattern is one the hardware supports.

Gotchas

• Licences: distilling from a model’s outputs may be restricted by its terms of use. Check before training on a provider’s outputs.
• A student trained only on one domain is brittle outside it; measure on held-out prompts of the real use case.
• Distillation transfers the teacher’s mistakes and biases along with its skills.