Mixture of experts and architecture families

• MoE (Mixture of Experts) — a model whose feed-forward block is split into many “experts”, with only a few used per token.
• Router / gate — the small linear layer that scores experts for each token and picks the top-k.
• Top-k routing — selecting the k highest-scoring experts per token (commonly k = 2 to 8).
• Active parameters — the parameters actually multiplied for one token; total parameters minus the unselected experts.
• FFN (feed-forward network) / MLP — the per-token two- or three-matrix block after attention; holds most of a dense model’s parameters.
• MHA / MQA / GQA / MLA — Multi-Head, Multi-Query, Grouped-Query, and Multi-head Latent Attention: variants that trade KV-cache size against expressiveness.
• SWA (sliding-window attention) — attention limited to the last W tokens.
• SSM (state-space model) / linear attention — layers that keep a fixed-size recurrent state instead of a growing KV cache.
• Gated DeltaNet (GDN) — a linear-attention/recurrent layer type with a gated, delta-rule state update.
• RoPE (rotary position embedding) — position encoding applied by rotating query/key vectors.
• SwiGLU — the gated activation used in most modern FFNs.
• RMSNorm — a cheaper layer normalisation without mean-centering.
• MTP (multi-token prediction) — extra heads that predict several future tokens, usable for speculative decoding.
• n-gram embedding — an embedding table indexed by a short sequence of tokens (bigram, trigram) rather than a single token.
• Host memory — CPU RAM, as opposed to GPU VRAM.
• Pre-norm — applying the normalisation before attention/FFN inside each residual branch rather than after.
• Residual connection — adding a block’s input to its output (x + f(x)) so information and gradients flow around the block.
• QK-norm — normalising query and key vectors before their dot product to keep attention scores well-scaled.
• config.json — the file shipped with a Hugging Face model that lists its architecture hyper-parameters (layers, heads, sizes, vocabulary).
• intermediate_size — the hidden width of the FFN, between its input and output projections.
• Tokenizer / vocabulary — the rule and table that split text into token ids; vocab_size is the number of distinct tokens.

The idea

In a dense transformer layer, every token goes through the same FFN, and the FFN holds roughly two-thirds of the layer’s parameters. A Mixture-of-Experts layer replaces that one FFN with many smaller FFNs (“experts”) and a router. Each token is sent to only a handful of them.

            token hidden state h
                    |
              [ router ]  -> scores for each of E experts
                    |
        pick top-k experts (e.g. 8 of 128)
        /        |        |        \
   expert 3  expert 17  expert 52  expert 90     (the other 124 stay idle)
        \        |        |        /
        weighted sum by router probabilities
                    |
              output hidden state

Attention layers, embeddings, and norms stay dense and shared; only the FFN is sparse. A few architectures also add one or more shared experts that every token always uses, to hold common knowledge so the routed experts can specialise.

The arithmetic

Suppose a model has 48 MoE layers, 128 experts per layer, 8 routed per token, each expert about 0.9B params across the stack:

experts_total_params = 125e9                 # all experts + attention + embeddings, roughly
n_experts, top_k = 128, 8
dense_part = 3e9                             # attention, embeddings, shared parts (always active)
expert_part = experts_total_params - dense_part

active = dense_part + expert_part * top_k / n_experts
print(f"{active/1e9:.1f}B active of {experts_total_params/1e9:.0f}B total")   # ~10.6B  (illustrative)

Real configurations (finer-grained experts, a smaller dense part, k/E closer to 3-5%) reach ratios like 125B total / 6B active. The compute per token scales with the active count, so the model runs about as fast as a ~6B dense model, while its capacity (what it can store and know) scales with the total count.

Why it works

• Knowledge is sparse. Facts and skills do not all need to touch every token. Experts specialise loosely (by syntax, language, domain), so each token needs only the relevant slice.
• Scaling laws favour parameters. Quality improves with total parameters and training tokens; compute-per-token is the expensive part. MoE lets you buy parameters without buying proportional FLOPs.

The router and load balancing

The router is a linear layer producing one logit per expert, then softmax (or sigmoid) and top-k:

import torch, torch.nn.functional as F

def moe_layer(x, router, experts, k=2):          # x: [tokens, d]
    logits = router(x)                           # [tokens, E]
    weights, idx = torch.topk(F.softmax(logits, -1), k, dim=-1)
    weights = weights / weights.sum(-1, keepdim=True)
    out = torch.zeros_like(x)
    for e, expert in enumerate(experts):
        mask = (idx == e)                        # which tokens chose expert e
        rows = mask.any(-1)
        if rows.any():
            w = (weights * mask)[rows].sum(-1, keepdim=True)
            out[rows] += w * expert(x[rows])
    return out

Walkthrough: score all experts, keep the top k per token, renormalise those weights, run each expert only on the tokens routed to it, and sum the weighted outputs. Production kernels batch this with grouped GEMMs rather than a Python loop.

Failure mode: routing collapse — the router sends most tokens to a few favoured experts, which then train more and get favoured further. Remedies: an auxiliary load-balancing loss, per-expert capacity limits, or bias-based balancing without an auxiliary loss, as in DeepSeek-V3.

The catch — memory is still total, not active

All experts must be available because any token might route to any of them. So:

Quantity	Scales with
FLOPs per token, decode speed (if weights are in fast memory)	Active parameters
VRAM / RAM to hold the model	Total parameters
Training compute	Active parameters × tokens
Training memory	Total parameters (+ optimizer state)

A 125B-total model at 4-bit still needs about 70 GB of memory, even though each token touches only 6B worth of weights. This is why MoE models suit unified-memory Macs, multi-GPU servers, and CPU-RAM + GPU hybrids: the experts sit in big, cheap RAM, and only the few selected ones are read per token.

At batch size 1, each token reads just its experts’ weights, so decode is fast. At large batch, different tokens select different experts, so almost every expert gets read each step, and the bandwidth advantage shrinks. MoE shines for compute efficiency and low-concurrency local use; at serving scale it needs expert parallelism (experts sharded across GPUs, tokens shuffled via all-to-all communication).

For inference, memory splits into weights and a per-token state that grows with context. Architecture decides how big that state is.

Attention variants and the KV cache

Per token, per layer, the cache stores keys and values for each KV head:

KV bytes per token = 2 × n_layers × n_kv_heads × head_dim × bytes

Variant	KV heads	Idea	Example per-token KV (80 layers, head_dim 128, fp16)
MHA	= query heads (64)	every head has its own K and V	2.6 MB
GQA	groups (8)	query heads share K/V in groups	327 KB
MQA	1	all query heads share one K/V	41 KB
MLA	latent vector	cache a compressed latent (e.g. 512-d) and expand at use	~100 KB-ish, with quality close to MHA

GQA (Llama 3, Qwen, Mistral) is the common compromise: 8x less cache than MHA with little quality loss. MLA (DeepSeek) compresses further by storing a low-rank latent, trading extra matmuls for much less memory.

Limiting what attention looks at

• Sliding-window attention — each token attends to the last W tokens; cache capped at W per layer. Mistral 7B used W = 4096. Often interleaved with occasional global layers (as in Gemma) to retain long-range recall.
• Sparse attention — attend to selected blocks of the past, chosen by a small indexer (used in newer DeepSeek and Qwen models).

Replacing attention with fixed-size state

SSM / linear-attention layers (Mamba, Gated DeltaNet) keep a constant-size recurrent state regardless of context. Hybrids interleave them with a few full-attention layers:

layer:  1    2    3    4    5    6    7    8
type:  GDN  GDN  GDN  ATTN GDN  GDN  GDN  ATTN       # 3:1 hybrid
cache:  O(1) O(1) O(1) O(n) O(1) O(1) O(1) O(n)       # KV only for 1 in 4 layers

A 3:1 hybrid cuts KV-cache memory by about 4x on top of GQA, which is why such models are cheap at 256K-1M context. The cost: weaker exact long-range retrieval from the recurrent layers, mitigated by the remaining full-attention layers.

Other levers

• Depth vs width — deeper models have more layers (more KV); wider ones have fatter matmuls. Same parameter count, different cache and latency.
• Vocabulary size — a 250K vocab makes the embedding and output matrices huge (250K × 4096 × 2 bytes = 2 GB each); small models spend a visible fraction of their parameters there. Tied input/output embeddings halve it.
• Tied embeddings, MoE vs dense, FFN expansion ratio all move parameters between “always active” and “stored”.

Reading a config.json for memory

import json
cfg = json.load(open("config.json"))
L, H_kv = cfg["num_hidden_layers"], cfg["num_key_value_heads"]
d_head = cfg.get("head_dim", cfg["hidden_size"] // cfg["num_attention_heads"])
per_token = 2 * L * H_kv * d_head * 2          # bytes in fp16
print(f"{per_token/1024:.0f} KiB of KV cache per token")

num_key_value_heads vs num_attention_heads tells you immediately whether the model is MHA or GQA, and num_experts / num_experts_per_tok tells you the MoE ratio.

At the level of the core block, these families are close cousins: decoder-only transformers with RMSNorm, RoPE, SwiGLU FFNs, and GQA. The differences are in the details (head counts, vocabulary, bias terms, window sizes) and in the direction each lab has taken. This concept first explains the shared recipe, then compares concrete configurations you can reproduce from config.json, then the family-level differences.

The shared recipe, piece by piece

Each decoder layer, in order (pre-norm layout):

x ──> RMSNorm ──> attention (GQA, RoPE on Q and K) ──> + x   (residual)
x ──> RMSNorm ──> FFN (SwiGLU: down( silu(gate(x)) * up(x) )) ──> + x

• RMSNorm rescales a vector by the inverse of its root-mean-square, x / sqrt(mean(x^2) + eps) times a learned gain. Cheaper than LayerNorm (no mean subtraction, no bias) and used in all three families.
• RoPE rotates query and key vectors by position-dependent angles so attention depends on relative distance (see Self-attention and positional structure). Families differ in the rotation base (rope_theta) and context-extension scaling.
• SwiGLU FFN uses three matrices (gate, up, down) instead of two: down( silu(gate x) * up x ). The intermediate width is a design choice (not a fixed 4x), which is why FFN sizes look irregular (14336, 18944).
• GQA shares each K/V head across a group of query heads to shrink the KV cache.

Because the recipe is the same, the differences live in a handful of config numbers.

Concrete configurations

Values from the published config.json of these specific models (verify against the file for the exact release you use; different versions of a family differ):

	Llama-3-8B	Mistral-7B-v0.1	Qwen2.5-7B
num_hidden_layers	32	32	28
hidden_size	4096	4096	3584
num_attention_heads	32	32	28
num_key_value_heads	8	8	4
head dimension (hidden / heads)	128	128	128
FFN intermediate_size	14336	14336	18944
vocab_size	128256	32000	152064
Attention bias on Q/K/V	no	no	yes
Sliding window	none	4096	none
Computed total parameters	8.03 B	7.24 B	7.62 B
KV cache per token (fp16)	128 KiB	128 KiB (capped, see below)	56 KiB
KV cache at 32,768 tokens	4.0 GiB	0.5 GiB (window cap)	1.75 GiB

Reading it:

• The depth/width trade differs. Qwen2.5-7B has fewer layers (28 vs 32) and a smaller hidden size but a wider FFN (18944 vs 14336), moving parameters from attention into the feed-forward blocks.
• KV heads drive memory. Llama and Mistral use 8 KV heads; Qwen2.5-7B uses only 4, so per-token cache is 56 KiB against 128 KiB, even though it has almost the same parameter count. At a given context Qwen2.5-7B needs less than half the cache.
• Vocabulary changes where parameters live. Mistral 7B’s 32,000-token vocabulary puts about 32000 x 4096 x 2 = 0.26 B parameters in the two embedding matrices; Llama 3’s 128,256 puts about 1.05 B there; Qwen2.5’s 152,064 about 1.09 B (at hidden 3584, input and output untied). A larger vocabulary encodes text in fewer tokens (cheaper, shorter context use, helps non-English text) at the price of a larger output layer.
• Sliding window (Mistral 7B v0.1): each token attends to the last 4,096 tokens, so the cache is capped at 128 KiB x 4096 = 0.5 GiB per sequence whatever the context. Later Mistral models mostly dropped the window and use full attention.
• QKV bias (Qwen2/2.5) adds a small bias vector to the Q, K and V projections; Llama and Mistral have none. Qwen3 replaced the bias with QK-norm (normalising Q and K before the dot product, which stabilises training).

Reproducing the parameter counts

You can verify any config yourself; if the formula reproduces the published parameter count, you have read the config correctly.

def count_params(L, hidden, n_heads, n_kv, ffn, vocab, qkv_bias=False, tied=False):
    hd = hidden // n_heads
    attn = hidden * n_heads * hd        # W_Q
    attn += 2 * hidden * n_kv * hd      # W_K and W_V (fewer columns under GQA)
    attn += n_heads * hd * hidden       # W_O
    if qkv_bias:
        attn += n_heads * hd + 2 * n_kv * hd
    mlp = 3 * hidden * ffn              # gate, up, down
    norms = 2 * hidden                  # two RMSNorm gains per layer
    emb = vocab * hidden * (1 if tied else 2)   # input embedding + output head
    return L * (attn + mlp + norms) + emb + hidden   # + final norm

print(count_params(32, 4096, 32, 8, 14336, 128256) / 1e9)                    # 8.03  Llama-3-8B
print(count_params(32, 4096, 32, 8, 14336, 32000) / 1e9)                     # 7.24  Mistral-7B-v0.1
print(count_params(28, 3584, 28, 4, 18944, 152064, qkv_bias=True) / 1e9)     # 7.62  Qwen2.5-7B

Walkthrough: attention parameters are the four projection matrices; GQA shrinks only W_K and W_V (their output width is n_kv x head_dim, not hidden). The FFN dominates: 3 x 4096 x 14336 = 176 M per layer against about 42 M for attention in Llama-3-8B. Embeddings are counted twice unless input and output are tied.

Larger sizes

At 70B-class, Llama-3-70B and Qwen2.5-72B both use 80 layers, hidden size 8192, 64 query heads and 8 KV heads, so their KV cache is identical at 320 KiB per token (fp16); they differ in FFN width (28672 vs 29568) and vocabulary. Computed totals: 70.5 B and 72.7 B.

Family-level differences

	Llama (Meta)	Mistral (Mistral AI)	Qwen (Alibaba)
Core block	Pre-norm, RMSNorm, RoPE, SwiGLU, GQA	Same recipe	Same recipe, plus QKV bias in earlier versions; QK-norm in Qwen3
Attention	GQA (8 KV heads for larger sizes)	GQA + sliding-window in 7B; full attention in later models	GQA; Qwen3-Next and later move to hybrid linear + full attention
Dense models	1B-405B, dense only in Llama 3	7B, 12B (Nemo), 24B (Small), 123B (Large)	0.5B-72B dense across Qwen2.5 / Qwen3
MoE	Llama 4 introduced MoE (Scout, Maverick)	Mixtral 8x7B / 8x22B, later Mistral Large 3-class MoE	Qwen3 30B-A3B, 235B-A22B; Qwen3-Next / 3.8 ultra-sparse (6B active of 125B)
Vocabulary	~128K tokens (Llama 3)	32K in early models, 131K in Tekken (Nemo+)	~151K tokens, strong on Chinese and code
Context	128K	32K -> 128K	32K -> 128K native, 256K-1M in newer
Notable emphasis	Scale of dense training data, open ecosystem	Efficiency per parameter, sliding window, early MoE	Multilingual, code, wide size ladder, rapid architecture experiments

How to read this practically

• Memory per context token at equal size is similar when all use GQA with 8 KV heads; the difference comes from the KV-head count (Qwen’s 4 above), Mistral’s sliding window (cache capped) and Qwen’s newer hybrid layers (cache shrunk further).
• Mixtral-style MoE (8 big experts, top-2) differs from Qwen / DeepSeek-style MoE (many small, “fine-grained” experts, top-8, plus shared experts). Finer experts give the router more combinations and better specialisation.
• Tokenizer matters for cost. A bigger vocabulary encodes the same text in fewer tokens, which means fewer KV entries and fewer decode steps for the same output. Qwen’s and Llama 3’s larger vocabularies help non-English text. Token counts for the same text are not comparable between families.
• Treat details as versions, not identity. All three labs have shifted designs between releases, so check the specific model’s config.json instead of assuming the family’s reputation. The comparison table above is a snapshot; the numbers in the config table are for specific named checkpoints.
• Chat templates differ. Each family has its own special tokens and message format; using the wrong template silently degrades output (see Messages, roles, and prompt structure).

Qwen3.8-Flash-Next is presented by Qwen as a preview of its next architecture. Per the paper and model recipe, it combines several ideas on top of a standard MoE: 125B total parameters with 6B active, a 3:1 hybrid of Gated DeltaNet and full-attention layers, sparse attention for long context, MTP for speculative decoding, and an extra n-gram embedding table of roughly 51B parameters.

What a normal embedding is, and what this one adds

A standard embedding is a lookup table with one row per vocabulary token:

token id 8421 ("cat")  ->  row 8421 of E  ->  vector of size d

An n-gram embedding adds a second table indexed by a short sequence of tokens ending at the current position:

tokens: ... "New"  "York"  "City" ...
standard lookup   : "City"                    -> vector
bigram lookup     : ("York","City")           -> vector
trigram lookup    : ("New","York","City")     -> vector

The number of possible bigrams and trigrams is astronomically larger than the vocabulary, so the key is hashed into a fixed number of rows (multi-head hashing: several independent hash functions per n-gram, so a collision in one is unlikely to repeat in all). Sources describe about 20 million bigram/trigram entries; the table scale in the ablations ranges from 20x to 200x the base vocabulary.

Where it goes and how it combines

The retrieved vectors are injected into the residual stream at an early layer (layer 2), through a learned contextual gate that decides how much of the looked-up vector to mix in given the current hidden state. Conceptually:

def ngram_inject(h, token_ids, tables, hash_fns, gate):
    # h: [seq, d] hidden state entering layer 2
    keys = [hash_fn(token_ids) for hash_fn in hash_fns]        # indices computed from tokens only
    e = sum(table[k] for table, k in zip(tables, keys))        # pure lookup: no matmul
    return h + gate(h) * e                                     # gated residual addition

The model gets cheap, pre-learned representations of common phrases and local patterns (“New York City”, “in order to”, if __name__) without spending attention and FFN compute to rediscover them.

Why a lookup table is “free” capacity

A lookup reads one row; a matmul reads (and multiplies) a whole weight matrix. So adding 51B parameters as a table adds almost no FLOPs per token, unlike adding 51B of expert weights. The table is a third scaling axis next to depth/width and expert count: memory capacity that costs storage but not compute.

Why it can live in CPU RAM

The key property: the indices are known before any computation, because they are a function of the input token ids only, not of hidden states. So the system can:

1. Hash the upcoming tokens as soon as they are known.
2. Start copying only the needed rows from host RAM to the GPU asynchronously.
3. Overlap that transfer with the compute of layers 0-1 (the reason for placing the lookup at layer 2).

GPU timeline:   [ embed ][ layer 0 ][ layer 1 ][ inject n-gram ][ layer 2 ... ]
CPU->GPU copy:       [ prefetch only needed rows over PCIe ]  --^
host RAM:       51B-parameter table (not occupying VRAM)

Contrast with MoE experts: the router’s choice depends on the hidden state at that layer, so you cannot prefetch experts in advance as reliably. This is a structural reason lookup memory is easier to offload than expert weights. The vLLM recipe reportedly supports this offload on NVIDIA GPUs and asks for at least ~51 GB of free host memory on top of VRAM; check the recipe for current flags.

Quantization and counting caveats

• The table is not a matmul weight, so quantization schemes that optimise matmul accuracy do not apply in the same way; at least one GGUF release keeps it at a fixed ~4.5 bits per weight while the rest of the model varies.
• Sources disagree on headline size (125B with 6B active, versus roughly 176B if the table is counted). When sizing hardware, add the table to the total.
• The paper notes that growing the n-gram vocabulary lowers training loss steadily while downstream accuracy saturates, so more table is not automatically better.

Related idea

DeepSeek’s “Engram” work follows the same principle of conditional memory via hashed n-gram lookups. Per-layer embeddings in Gemma 3n are a sibling idea: large lookup tables offloaded from accelerator memory, since a lookup is cheap to fetch.

Pitfall

Do not confuse the n-gram embedding table with an n-gram language model, a KV cache, or RAG retrieval. It is learned during pretraining, fixed at inference, and is part of the forward pass.