VRAM and running LLMs locally

• VRAM — the dedicated memory on a GPU. Weights must sit here (or be streamed in) to be used at full speed.
• HBM (High Bandwidth Memory) — the stacked DRAM on datacenter GPUs (H100, A100); several TB/s of bandwidth.
• GDDR — the memory on consumer GPUs; cheaper, lower bandwidth than HBM.
• Unified memory — CPU and GPU share one pool of RAM (Apple Silicon, some APUs), so “VRAM” is just system RAM.
• KV cache — the stored attention keys and values for every token so far (see KV cache, prompt caching, and inference economics).
• bpw (bits per weight) — average storage cost of one weight after quantization, including scales.
• GGUF — the single-file model format used by llama.cpp and its ecosystem.
• OOM (out of memory) — the allocation failure you get when weights + cache + activations exceed VRAM.
• TTFT (time to first token) — latency until the first output token; dominated by prefill.
• tok/s — generation throughput, tokens per second.
• PagedAttention — vLLM’s technique of storing the KV cache in fixed-size blocks, like OS virtual-memory pages.
• CUDA / ROCm / Metal / Vulkan — GPU compute backends for NVIDIA, AMD, Apple, and vendor-neutral hardware.
• GQA (grouped-query attention) — attention where several query heads share one key/value head, shrinking the KV cache.
• KiB / GiB — 1,024 bytes / 1,024^3 bytes (binary units, slightly larger than KB / GB).
• Inference engine — the program that loads the weights and runs the forward pass (llama.cpp, vLLM, MLX).
• ggml — the C tensor library underneath llama.cpp.
• k-quant / i-quant — llama.cpp quantization families (Q4_K_M, IQ3_XS): block-wise scales with mixed precision / importance-aware.
• Prefill / decode — processing the whole prompt in parallel / generating one token at a time.
• Flash attention — a memory-efficient attention kernel that avoids materialising the n x n score matrix.
• Slot — one concurrent conversation in llama-server; slots share the configured context.
• MoE (Mixture of Experts) — a model with many expert sub-networks of which only a few run per token.

Three facts combine to make VRAM the thing everyone fights over:

1. A model is mostly one giant read-only array of weights, and every generated token reads (almost) all of it. A 70B model in fp16 is 140 GB; producing one token means streaming those weights through the compute units. If the weights are not in fast memory, you wait for them.
2. Decode is memory-bandwidth-bound, not compute-bound. Per token, each weight is used for roughly one multiply-add. The GPU can do far more arithmetic than it can feed from memory, so speed is set by bytes moved per second, not FLOPs.
3. VRAM is expensive, fixed, and not upgradeable. You cannot add DIMMs to a GPU. A 24 GB card is a hard wall: 24.1 GB is not a slightly slower run, it is an OOM or a drastic fallback.

The bandwidth rule of thumb

Each decoded token must read the active weights once, so:

max tok/s  ≈  memory bandwidth (GB/s)  /  bytes of weights read per token (GB)

Memory	Bandwidth	8B model @ 4-bit (≈4.5 GB)	70B @ 4-bit (≈40 GB)
Dual-channel DDR5 (CPU)	~80 GB/s	~18 tok/s	~2 tok/s
Apple M-series Max (unified)	~400-550 GB/s	~90-120 tok/s	~10-13 tok/s
RTX 4090 GDDR6X	~1000 GB/s	~220 tok/s	does not fit (24 GB)
H100 HBM3	~3350 GB/s	~700 tok/s	~80 tok/s

These are upper bounds; real numbers land at 50-80% of them. The pattern is the lesson: the same model is an order of magnitude slower on CPU RAM than on GPU memory, and that gap is why people pay for VRAM.

Why spilling to system RAM hurts so much

When a model does not fit, runtimes can offload some layers to CPU RAM. But those layers are then computed at DDR bandwidth, or streamed over PCIe (~32 GB/s on PCIe 4 x16, 64 GB/s on PCIe 5), which is 15-30x slower than GDDR. One offloaded layer drags every token down to its speed because layers run in sequence. Offloading 10% of layers can cut throughput by well over half.

Total memory at inference is the sum of four terms:

VRAM  =  weights  +  KV cache  +  activations/workspace  +  runtime overhead

1. Weights

weights_bytes = n_params × bytes_per_weight

Precision	bytes/weight	7B	13B	70B
fp32	4	28 GB	52 GB	280 GB
fp16 / bf16	2	14 GB	26 GB	140 GB
int8 / fp8	1	7 GB	13 GB	70 GB
4-bit (≈4.5 bpw real)	0.56	3.9 GB	7.3 GB	39 GB

For Mixture-of-Experts models, all experts count toward weight memory, even though only a few run per token (see Mixture of experts).

2. KV cache

Grows linearly with context length and with the number of concurrent sequences:

kv_bytes = 2 × n_layers × n_kv_heads × head_dim × seq_len × batch × bytes_per_elem

Worked example, Llama-3-70B-like (80 layers, 8 KV heads via GQA, head_dim 128, fp16):

layers, kv_heads, head_dim, bytes_ = 80, 8, 128, 2
per_token = 2 * layers * kv_heads * head_dim * bytes_      # 327,680 B = 320 KiB
print(per_token / 1024, "KiB/token")                        # 320.0
print(per_token * 128_000 / 2**30, "GiB for one 128k ctx")  # ~39 GiB
print(per_token * 8_192 * 32 / 2**30, "GiB for 32 users @ 8k")  # ~80 GiB

So one 128k-token conversation costs about as much as the 4-bit weights, and serving 32 users at 8k costs more than the weights. For a server, the KV cache, not the weights, is usually what limits concurrency.

3. Activations and workspace

Intermediate tensors during a forward pass. Small at decode (one token), but prefill processes the whole prompt at once and can spike: attention score matrices, MLP intermediate (typically 4x hidden size) for every prompt token. A long prompt can OOM a setup that was fine at steady state; chunked prefill bounds this.

4. Runtime overhead

CUDA context (~300-800 MB), allocator fragmentation, CUDA graphs, and the framework itself. Budget 1-2 GB, and leave ~10% headroom. vLLM’s --gpu-memory-utilization 0.90 exists for this reason.

Quick sizing procedure

1. Pick a quantization -> weights GB.
2. Decide max context and concurrency -> KV GB (halve it with fp8 KV cache).
3. Add ~1.5 GB overhead + 10% headroom.
4. Compare to VRAM. If over: smaller model, lower-bit quant, shorter context, KV-cache quantization, or fewer concurrent users.

Weights are not the only thing that can be compressed. When a setup does not fit, you have several levers; pull them in order of quality cost, cheapest first.

The levers, ranked

Lever	Saves	Quality cost	When to reach for it
Shorter max context	KV cache, linearly	None unless you need the context	Context is set far above real use (131k on a chat that never passes 8k)
fp8 / int8 KV cache	KV 2x	Usually negligible	Long context or many concurrent users
8-bit weights	weights 2x	Negligible	Model almost fits
4-bit weights (Q4_K_M, AWQ)	weights about 3.5x	Small; noticeable on reasoning and code for small models	The standard default
4-bit KV cache	KV 4x	Measurable on long-context retrieval and reasoning; test it	Memory-critical, long context
3-bit and below	weights 5x or more	Clear degradation	Last resort
Partial CPU offload	any	Large speed cost	Must run a model that does not fit
Smaller model, higher precision	all	Often the better trade than a 2-bit big model	Instead of extreme quantization

A frequently quoted heuristic: a larger model at 4-bit generally beats a smaller model at fp16 of the same memory footprint, until you go below roughly 3 bits.

What “quantizing the KV cache” means

The KV cache stores one K and one V vector per token, per layer, per KV head, normally as 16-bit numbers. Cache quantization stores them as 8-bit or 4-bit numbers (with a small scale factor per group of values) and converts back when they are read for attention. It changes only the stored activations, not the model weights, and it is independent of the weight format: a 4-bit weight model can have an fp16, fp8 or int4 cache.

kv_bytes = 2 x n_layers x n_kv_heads x head_dim x seq_len x batch x bytes_per_element

Symbols: the 2 counts K and V; n_layers is the transformer depth; n_kv_heads is the number of key/value heads (fewer than query heads under GQA); head_dim is the vector width per head; seq_len is tokens in the context; batch is the number of concurrent sequences; bytes_per_element is 2 for fp16, 1 for fp8/int8, about 0.5 for 4-bit (a little more once scales are counted). Where the formula comes from is in Self-attention and positional structure.

Computed examples

def kv_bytes(layers, kv_heads, head_dim, tokens, bytes_per_elem, batch=1):
    return 2 * layers * kv_heads * head_dim * tokens * batch * bytes_per_elem

GiB = 2**30
shapes = {"Llama-3-8B-like": (32, 8, 128), "Llama-3-70B-like": (80, 8, 128)}   # layers, KV heads, head_dim
for name, (L, g, d) in shapes.items():
    for fmt, b in (("fp16", 2), ("fp8", 1), ("int4", 0.5)):
        print(name, fmt,
              round(kv_bytes(L, g, d, 1, b) / 1024), "KiB/token,",
              round(kv_bytes(L, g, d, 8192, b) / GiB, 2), "GiB at 8k,",
              round(kv_bytes(L, g, d, 131072, b) / GiB, 1), "GiB at 128k")
# Llama-3-8B-like  fp16 128 KiB/token, 1.0 GiB at 8k,  16.0 GiB at 128k
# Llama-3-8B-like  fp8   64 KiB/token, 0.5 GiB at 8k,   8.0 GiB at 128k
# Llama-3-8B-like  int4  32 KiB/token, 0.25 GiB at 8k,  4.0 GiB at 128k
# Llama-3-70B-like fp16 320 KiB/token, 2.5 GiB at 8k,  40.0 GiB at 128k
# Llama-3-70B-like fp8  160 KiB/token, 1.25 GiB at 8k, 20.0 GiB at 128k
# Llama-3-70B-like int4  80 KiB/token, 0.62 GiB at 8k, 10.0 GiB at 128k

(The shapes are those of the published Llama-3 8B and 70B models; int4 here ignores the scale overhead, which adds roughly 10% in real formats.)

Worked scenario: 8B model on a 24 GB card. The card has about 22.4 GiB. Weights at 4.5 bits per weight: 8.03e9 x 0.5625 bytes = 4.2 GiB. Reserve 10% headroom and 1.5 GiB runtime overhead: 22.4 x 0.9 - 4.2 - 1.5 = 14.4 GiB left for the cache. Divide by bytes per token:

Cache type	KiB per token	Max tokens in 14.4 GiB (single sequence)	Equivalent to
fp16	128	about 118,000	one long conversation
fp8	64	about 236,000	one very long conversation, or 2x the users
int4	32	about 472,000	beyond the model’s trained context; spend it on concurrency

For a server with 16 sequences of 8k tokens each, the cache needs 16 GiB in fp16 (does not fit in the 14.4 GiB left), 8 GiB in fp8 (fits). The cache dtype is what turns “OOM at 16 users” into “fits with room to spare”. (Note most models cannot use beyond their trained context length no matter how much memory is free; the 472,000 is arithmetic, not a recommendation.)

How to switch it on

# llama.cpp: quantized K and V cache (a quantized V cache needs flash attention enabled; check --help for your build)
llama-server -m model-Q4_K_M.gguf -c 32768 -ngl 99 -fa --cache-type-k q8_0 --cache-type-v q8_0

# vLLM: fp8 KV cache
vllm serve Qwen/Qwen2.5-7B-Instruct --kv-cache-dtype fp8 --max-model-len 32768

# Ollama: via environment variables on the server process
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

Flag names change between versions; confirm with --help or the project docs for the version you run.

Gotchas

• Keys are more sensitive than values. Quantization error in K changes the attention weights themselves, so some setups quantize K to 8 bits and V to 4. Test, do not assume.
• Evaluate on your task. The quality cost of a 4-bit cache shows up as lost needles in long contexts and weaker multi-step reasoning, not on short chat; run your own long-context test before committing.
• The levers stack: 4-bit weights plus 8-bit cache plus a context cap often turns an impossible plan into a comfortable one, and each lever’s cost is separate.
• Not all engines or models support it (some attention kernels or architectures do not); the engine usually errors at load rather than silently ignoring it, but check the log.

Prefix caching and sliding-window / hybrid attention (see Mixture of experts and architecture families) attack the KV term from the architecture side; see Quantization and inference efficiency (its Quantization formats compared concept) for weight-quantization formats.

Option 1: Ollama (simplest)

# install from ollama.com, then:
ollama pull llama3.1:8b
ollama run llama3.1:8b "Explain KV cache in two sentences"

# it also exposes a local HTTP API on :11434
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "Hello"}],
  "stream": false
}'

Ollama bundles llama.cpp, downloads GGUF models from its own registry, picks a quantization (usually Q4_K_M) for you, and manages model load/unload.

Option 2: llama.cpp directly (most control)

# grab a GGUF (e.g. from Hugging Face), then:
llama-server -m ./Qwen2.5-7B-Instruct-Q4_K_M.gguf \
  -c 16384 \          # context window (drives KV size)
  -ngl 99 \           # number of layers to offload to GPU (99 = all)
  --flash-attn \
  --port 8080         # OpenAI-compatible server at /v1/chat/completions

-ngl is the VRAM dial: lower it until the model fits, and the remaining layers run on CPU.

Option 3: Python with Hugging Face Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "Qwen/Qwen2.5-7B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(
    name, torch_dtype=torch.bfloat16, device_map="auto"  # spreads layers across GPUs/CPU
)

msgs = [{"role": "user", "content": "What is a KV cache?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=200)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Easy and flexible, but slower and more memory-hungry than llama.cpp or vLLM; best for experiments and fine-tuning, not serving.

Option 4: GUI front-ends

LM Studio, Jan, GPT4All, and koboldcpp wrap llama.cpp (and on Mac, MLX) behind a UI and a local OpenAI-compatible server.

Which hardware, practically

Hardware	What it runs well
CPU only, 16-32 GB RAM	1-8B at 4-bit, a few tok/s
8-12 GB GPU	7-8B at 4-bit comfortably, 13B tight
24 GB GPU (3090/4090)	up to ~32B at 4-bit, 8B at fp16 with long context
Apple Silicon 64-128 GB unified	70B at 4-bit, large MoEs; slower than NVIDIA per GB but huge capacity
Multi-GPU	tensor/pipeline parallelism via vLLM or llama.cpp split

They solve different problems: one user on one box vs many users on a GPU fleet.

	Ollama	llama.cpp	vLLM
Primary goal	Easy local use	Efficient inference anywhere	Max throughput serving
Target	Laptop/desktop	CPU, Apple, consumer GPU, edge	NVIDIA/AMD datacenter GPUs
Weight format	GGUF (own registry)	GGUF	HF safetensors; AWQ, GPTQ, fp8, (experimental GGUF)
Batching	Basic	Parallel slots	Continuous batching, high concurrency
KV management	Per-request	Contiguous, slot-based	PagedAttention, prefix caching
Multi-GPU	Limited	Layer split	Tensor + pipeline parallel
CPU / Apple support	Yes	Excellent	Poor
Setup	One command	Build or download binary	Python env, CUDA version matching
Strength	Zero friction	Runs on anything	Throughput under load

Why vLLM wins at scale — PagedAttention and continuous batching

Naive serving reserves a contiguous KV buffer of max_seq_len per request, so most of it is empty (60-80% waste) and fragmented. vLLM splits the KV cache into fixed-size blocks and keeps a per-request block table, like virtual memory pages:

request A block table:  [7] -> [2] -> [9]        (logical order)
request B block table:  [4] -> [1]
physical GPU blocks:    0:free 1:B 2:A 3:free 4:B 5:free 6:free 7:A 8:free 9:A

Blocks are allocated as tokens are generated and freed immediately, so memory use tracks actual tokens, and shared prefixes (a long system prompt) can point at the same physical blocks. Combined with continuous batching (new requests join the batch every step instead of waiting for the batch to finish), this commonly gives 2-4x or more throughput over naive serving.

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --tensor-parallel-size 1
# OpenAI-compatible server at http://localhost:8000/v1

Why Ollama/llama.cpp win for one user

A single stream of tokens cannot exploit batching, so vLLM’s machinery buys nothing, while its setup cost, GPU-only focus, and higher baseline memory use are real. llama.cpp’s hand-tuned quantized kernels, CPU SIMD paths, Metal backend, and partial offload let it run a model on hardware where vLLM cannot start at all.

Rule of thumb

• Personal use, laptop, experimenting -> Ollama (or LM Studio).
• Squeezing performance or odd hardware, custom quantization -> llama.cpp.
• Serving an API to many concurrent users on GPUs -> vLLM (or SGLang).

An inference engine is the program that loads the weights and runs the forward pass to produce tokens. Many tools are built on, or compete with, the same ideas; knowing which layer each one sits at makes the landscape much less confusing.

The layers

 you / app            chat UI, IDE plugin, script
      |
 front end            Ollama, LM Studio, koboldcpp, llamafile      <- model management, UI, API
      |
 engine / library     llama.cpp (ggml)  |  vLLM  |  MLX  |  ExLlama  |  TensorRT-LLM
      |
 weights format       GGUF  |  safetensors (+AWQ/GPTQ/fp8)  |  EXL2/EXL3  |  MLX format
      |
 hardware backend     CUDA | ROCm | Metal | Vulkan | CPU (AVX2/AVX512/NEON)

When two tools “use the same engine”, they can usually load the same weight files. That is why a GGUF downloaded for Ollama also works in LM Studio or llama-server, but not in vLLM’s mainstream path.

What llama.cpp is

llama.cpp is a C/C++ inference library (built on the ggml tensor library) plus command-line programs. Its design goals: run on anything (CPU, Apple Silicon, consumer GPUs, phones), with minimal dependencies, using quantized weights stored in a single GGUF file (weights, tokenizer, and metadata together). Programs you will meet:

Program	Purpose
llama-cli	Interactive or one-shot generation in the terminal
llama-server	An OpenAI-compatible HTTP server (/v1/chat/completions) with a small web UI
llama-bench	Measures prompt-processing and generation speed (tok/s) for a given model and settings
llama-quantize	Converts a high-precision GGUF to a quantized one (Q4_K_M, Q5_K_M, Q8_0, …)
llama-perplexity	Measures perplexity, to compare the quality of different quantizations

The flags that matter most

llama-server -m Qwen2.5-7B-Instruct-Q4_K_M.gguf -c 16384 -ngl 99 -np 2 -fa --port 8080
# then:
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json"   -d '{"messages":[{"role":"user","content":"Hi"}]}'

Flags: -m is the GGUF file to load; -c the context length in tokens (drives KV cache size); -ngl the number of layers to place on the GPU (99 means all); -np the number of parallel slots, which share the context; -fa enables flash attention (needed for a quantized V cache).

Walkthrough: -ngl is the main speed dial: layers on the GPU run at GPU memory bandwidth, layers left on the CPU run at system RAM bandwidth (typically several times slower). -c sets how much KV cache is allocated; with -np 2 the 16,384 tokens are divided between two slots, so each conversation gets about 8,192, a classic surprise. Check llama-server --help for your build, since options change.

Built on llama.cpp (same GGUF files)

• Ollama — Go wrapper that adds a model registry (ollama pull), automatic configuration, and a background daemon with its own HTTP API. It has increasingly moved to its own engine for some newer models.
• LM Studio — desktop GUI; llama.cpp plus Apple MLX engines.
• koboldcpp — llama.cpp fork aimed at roleplay and story writing, single executable, extra sampler options.
• llamafile — packs model and runtime into one cross-platform executable.
• ik_llama.cpp — fork focused on faster CPU and hybrid inference and newer quant types (i-quants, trellis quants); popular for running big MoE models with the expert weights in RAM and attention on the GPU.
• llama-cpp-python — Python bindings to the same library.

Independent engines

• vLLM — throughput-oriented datacenter serving (PagedAttention, continuous batching; above).
• SGLang — same class as vLLM; RadixAttention prefix cache and strong structured-output and multi-turn performance.
• TensorRT-LLM — NVIDIA’s compiled engine; fastest on NVIDIA hardware, most painful to build and deploy.
• Hugging Face TGI — production server, now in maintenance mode in favour of other engines.
• MLX / mlx-lm — Apple’s array framework; native unified-memory inference on Mac.
• ExLlamaV2/V3 — NVIDIA-only, very fast single-user inference with the EXL2/EXL3 quantization formats.
• MLC-LLM — compiles models for phones, browsers (WebGPU), and diverse GPUs.

GGUF quant names, decoded

Name	Roughly	Use
Q8_0	about 8.5 bits per weight, near-lossless	When it fits and quality matters most
Q6_K	about 6.6 bits	Very close to Q8 at 25% less memory
Q5_K_M	about 5.7 bits	Good quality/size
Q4_K_M	about 4.8 bits	The common default
Q3_K_M and IQ3/IQ2	2 to 3.5 bits	Only when forced; quality drops visibly

The _K means “k-quant” (block-wise scales with mixed precision); S/M/L is small/medium/large variant; IQ are “i-quants” tuned with importance information. Exact bits per weight vary by model; check the file size. A rough size is n_params x bpw / 8 bytes, e.g. 7e9 x 4.8 / 8 = 4.2 GB.

Choosing among them — a decision list

1. Apple Silicon -> MLX, or llama.cpp / Ollama / LM Studio.
2. CPU-only, or mixed CPU+GPU for a giant MoE -> llama.cpp or ik_llama.cpp.
3. One consumer NVIDIA GPU, maximum single-user speed -> ExLlama or llama.cpp.
4. Multi-user API server -> vLLM or SGLang; TensorRT-LLM if you can afford the engineering.
5. Need an OpenAI-compatible endpoint locally -> any of the above; all expose one.

Pitfalls

• Format mismatch. A GGUF will not load in vLLM’s normal path; an AWQ or GPTQ checkpoint will not load in llama.cpp. Pick the engine first, then download the matching format.
• Benchmark the way you will use it. llama-bench numbers for batch 1 say nothing about throughput under 20 concurrent users, where vLLM’s batching shines.
• Fork lag. A fork may trail upstream on support for a brand-new architecture, or add features that upstream lacks; check both before assuming.
• Wrapper defaults. Ollama and LM Studio choose context length, offload and sampling for you; verify them instead of assuming the engine’s maximum.

Each pitfall below is given as symptom, cause, how to confirm it, and the fix. Most come from the same root: VRAM is a hard wall and several things you did not choose consume it.

1. Context length silently drives memory

• Symptom: the model loads, then runs out of memory on a long prompt, or it answers as if it never saw the start of your document.
• Cause: the KV cache grows with context (see the VRAM budget concept). For a Llama-3-8B-like model in fp16 it is 128 KiB per token, so a 131,072-token context needs 16 GiB for the cache alone, more than the 4-bit weights (about 4 GiB). Some engines pre-allocate the whole cache at load; others grow it and fail later. The opposite problem also exists: Ollama’s default context is small (a few thousand tokens; check your version), so a long prompt can be silently truncated and the model never sees the beginning.
• Confirm: look at the engine log for the context size it actually used, and for a “truncating input” warning.
• Fix: set the context explicitly to what you need (-c in llama.cpp, num_ctx in an Ollama Modelfile or request, --max-model-len in vLLM), then re-check memory. Use a cache dtype of fp8/int8 if you need long contexts.

# Ollama Modelfile
FROM qwen2.5:7b
PARAMETER num_ctx 16384

2. “Fits” is not “fast” — partial offload

• Symptom: tokens per second is a fraction of what benchmarks show, though the model “runs”.
• Cause: layers that do not fit in VRAM run on the CPU, whose memory bandwidth is several times lower. Decode speed is roughly bandwidth divided by bytes read per token, so the slow part dominates. With illustrative bandwidths of 1,000 GB/s (GPU) and 60 GB/s (system RAM) for a 4.5 GB model:

w = 4.5                                   # GB of weights read per token
for cpu_share in (0.0, 0.1, 0.2, 0.5):
    t = (1 - cpu_share) * w / 1000 + cpu_share * w / 60       # seconds per token
    print(cpu_share, round(1 / t), "tok/s")
# 0.0  222 tok/s   all on GPU
# 0.1   87 tok/s   10% of layers on CPU: 2.5x slower
# 0.2   54 tok/s
# 0.5   25 tok/s

Moving just 10% of the layers to the CPU cuts speed by more than half in this model. (Real numbers depend on hardware and kernels; the shape of the curve is the lesson.)

• Confirm: ollama ps shows the GPU/CPU split; llama.cpp prints how many layers were offloaded at load (-ngl); nvidia-smi shows VRAM used.
• Fix: a smaller model or a lower-bit quant that fits fully, a smaller context, a quantized KV cache; offload only deliberately (MoE expert weights in RAM is the one case where it works well, see below).

3. Quantization names differ by tool

Q4_K_M (GGUF), AWQ-4bit, GPTQ and NF4 are different formats made for different engines. You cannot load a GGUF in vanilla Transformers workflows or in vLLM’s mainstream path, nor an AWQ checkpoint in llama.cpp. Pick the engine, then the matching format. Two files both called “4-bit” can also differ in quality and size (4.8 versus 4.25 bits per weight is a big difference at 70B: about 42 GB versus 37 GB).

4. Batch size and concurrency multiply KV memory

The cache formula has a batch term. The same 8B setup with 8k contexts needs about 1 GiB of cache for one request and about 8 GiB for eight (fp16), so a configuration that is fine for you alone can OOM under load. In llama.cpp -np N divides -c among N slots, so each user gets c / N tokens, not c. Plan with kv_bytes x concurrent sequences, then leave headroom.

5. MoE models need all expert weights resident

A “3B-active” model is not a 3B-memory model. A 30B-total MoE at about 4.5 bits per weight needs 30e9 x 0.5625 / 2^30 = 15.7 GiB for weights, and a 235B-total model about 123 GiB, regardless of how few parameters are active per token. Speed follows the active parameters; memory follows the total. The hybrid trick: keep attention and shared layers on the GPU and the expert tensors in system RAM (in llama.cpp this is done with a tensor-override option; check --help for your build), because only the few chosen experts are read per token. See Mixture of experts and architecture families.

6. Prefill spikes and first-token latency

Processing a long prompt at once needs temporary memory and time proportional to the prompt (and attention cost grows faster than linearly). A model that runs fine on short chats can OOM or take tens of seconds to the first token on a 100k-token prompt. Chunked prefill (a smaller physical batch size, -ub in llama.cpp) bounds the memory.

7. Thermal and power limits

Laptops and small-form-factor machines throttle under sustained load, so a short benchmark overstates throughput. Measure a few minutes of continuous decode, not one reply.

Pre-flight checklist

1. Weights GB for your chosen quant (n_params x bits / 8).
2. KV GB for your maximum context and concurrent users, at your cache dtype.
3. Plus 1.5 GB overhead and 10% headroom; compare with free VRAM (nvidia-smi).
4. After loading, confirm the context length and GPU/CPU split in the log; then run one maximum-length prompt as a test before depending on it.