Multimodal models: images, audio, video

• Modality — a kind of data: text, image, audio, video.
• Token — a unit the transformer reads; for text a word piece, for images a patch embedding, for audio a short time slice.
• Embedding — a vector (list of numbers) representing a token; the model works only on embeddings.
• d_model — the width of the LLM’s embedding vectors, e.g. 4096.
• Pixel / RGB channel — an image is a grid of pixels, each with 3 numbers (red, green, blue), usually 0 to 255.
• Patch — a small square block of pixels (e.g. 14x14) treated as one input unit.
• Patch size P — the side length of a patch in pixels.
• ViT (Vision Transformer) — a transformer that takes image patches as its input tokens.
• Vision encoder — the ViT (or similar) that turns an image into a set of vectors.
• CLIP — a model family trained to put an image and its caption close together in one vector space (contrastive learning).
• Contrastive learning — training so matching pairs (image, its caption) score higher than mismatched pairs.
• Projector / adapter — a small trained layer or MLP that converts vision vectors to the LLM’s embedding width.
• MLP (Multi-Layer Perceptron) — a small network of linear layers with non-linearities between them.
• VLM (Vision-Language Model) — an LLM that accepts images as well as text.
• VQA (Visual Question Answering) — training/evaluation data of images with questions and answers.
• Prefill — the phase where the model reads the whole prompt in parallel before generating; its cost grows with prompt length.
• KV cache — stored key/value vectors of all earlier tokens, so each new token need not recompute them; it grows with context length.
• Tiling — cutting a large image into several fixed-size crops that are each encoded separately.
• Resampler / Q-Former — a module with a fixed number of learned queries that compress many vision tokens into few via cross-attention.
• Cross-attention — attention where the queries come from one sequence (text) and keys/values from another (image features).
• Early fusion — mixing modalities in one model from the start of training, rather than attaching a vision module to a finished LLM.
• RoPE (Rotary Position Embedding) — a way of encoding token position by rotating query/key vectors; see Self-attention and positional structure.
• OCR (Optical Character Recognition) — software that extracts text characters from images.
• Mel spectrogram — a picture of audio: time on one axis, frequency bands (on a perceptual “mel” scale) on the other.
• Audio codec / tokenizer — a model that compresses audio into a stream of discrete codes and can reconstruct it.
• ASR (Automatic Speech Recognition) — converting speech to text (e.g. Whisper-style models).
• Grounding — keeping the model’s statements tied to what is actually in the input.

A transformer only consumes a sequence of vectors. Text gets there through a token-id lookup table. An image has no ids, so it is converted into vectors in four stages.

image 224x224x3
   |  (1) cut into 14x14 patches -> 16 x 16 = 256 patches
   v
[ patch 1 ][ patch 2 ] ... [ patch 256 ]       each patch = 14*14*3 = 588 raw numbers
   |  (2) vision encoder (ViT): linear patch embedding + transformer layers
   v
256 vectors, each 1024-d (the encoder's width)
   |  (3) projector: map 1024 -> 4096 (the LLM's d_model)
   v
256 "image tokens", each 4096-d  -- look like text-token embeddings
   |  (4) splice into the text sequence where the image occurred
   v
[ "Describe" ][ "this" ][ img_1 ... img_256 ][ "image" ][ "." ]  -> ordinary LLM stack

Step 1 and 2 in code: patchify, then embed

import numpy as np

H = W = 224; P = 14
img = np.arange(H * W * 3, dtype=np.float32).reshape(H, W, 3)       # stand-in for real pixels

# split into non-overlapping PxP patches and flatten each one
patches = img.reshape(H // P, P, W // P, P, 3).transpose(0, 2, 1, 3, 4).reshape(-1, P * P * 3)
print(patches.shape)                     # -> (256, 588)   256 patches, 14*14*3 = 588 numbers each

rng = np.random.default_rng(0)
W_patch = rng.standard_normal((P * P * 3, 1024)).astype(np.float32) * 0.01   # learned in a real model
emb = patches @ W_patch
print(emb.shape)                         # -> (256, 1024)  one 1024-d vector per patch

W_proj = rng.standard_normal((1024, 4096)).astype(np.float32) * 0.01         # the projector (learned)
tok = emb @ W_proj
print(tok.shape)                         # -> (256, 4096)  now the same width as text-token embeddings

Walkthrough: the reshape cuts the 224 rows into 16 groups of 14 and the 224 columns into 16 groups of 14, the transpose brings each 14x14x3 block together, and the final reshape flattens each block to 588 numbers. The patch-embedding matrix (588 x 1024, about 603 thousand parameters including bias) is what turns raw pixels into the encoder’s vectors; in a real ViT transformer layers then mix information between patches. The projector multiplies by a 1024 x 4096 matrix so the result can sit in the LLM’s embedding space. Weights here are random stand-ins; in a real model they are learned.

What is trained and what is frozen

The vision encoder is often pretrained separately with a contrastive image-text objective (CLIP-style) and kept frozen or lightly tuned. Most multimodal capability comes from training the projector and the LLM on interleaved image-text data (captions, VQA, instructions), so the LLM learns to attend across the image tokens.

Why a projector is needed at all

The encoder’s vectors live in a space with its own coordinates; the LLM’s embedding table lives in another. Without a learned map between them, image vectors would be meaningless noise to the LLM. A linear projector (1024 x 4096, bias included, about 4.2 million parameters) or a two-layer MLP (about 21 million) is enough because the encoder already encodes semantics; the projector only needs to translate coordinates.

An image is not one token; it becomes (width / P) * (height / P) tokens, rounded as the encoder dictates.

The arithmetic

tokens = (W / P) * (H / P)            W, H = image width/height in pixels (after resizing)
                                      P    = patch side in pixels (fixed by the encoder)

Image (after resize)	Patch P	Grid	Tokens
224 x 224	14	16 x 16	256
224 x 224	16	14 x 14	196
336 x 336	14	24 x 24	576
448 x 448	14	32 x 32	1,024
1024 x 768	16	64 x 48	3,072

Doubling the side length (224 to 448) multiplies tokens by 4, because they scale with pixel area.

Token merging

Many models shrink the count after encoding by merging neighbouring patches: for example merging each 2x2 block of the 24x24 = 576 grid into one token gives 144 tokens (the merged token carries the four patches’ information in a wider vector or through a small MLP). Others use a resampler to produce a fixed number of tokens regardless of image size (see below).

Tiling for large images

Many systems split a big image into fixed-size tiles plus a downsampled overview. With 576 tokens per tile (illustrative), an image cut into 5 tiles plus 1 overview costs 5 * 576 + 576 = 3,456 tokens: six times a single tile. This preserves small text but multiplies cost.

Money and memory, with illustrative numbers

Take a hypothetical 576-token image and a price of $3 per million input tokens (a made-up price for the arithmetic):

one image : 576 tokens * $3 / 1,000,000       = $0.0017
1,000 images                                  = $1.73
a 20-page PDF rendered at 1,000 tokens/page   = 20,000 tokens  (a long text prompt)

Memory: the KV cache stores a key and a value per layer per token. For an illustrative model with 32 layers, 8 KV heads, head dimension 128 and fp16 (2 bytes):

bytes per token = 2 (K and V) * 32 layers * 8 heads * 128 dim * 2 bytes = 131,072 B = 128 KiB
576 image tokens  -> 72 MiB       1,024 tokens -> 128 MiB       3,456 tokens -> 432 MiB

And prefill attention is quadratic in total length: adding a 576-token image to a 500-token text prompt turns 500^2 = 250,000 score entries into 1,076^2 = 1,157,776, about 4.6x the attention work. See VRAM and running LLMs locally for how this competes with weights for GPU memory.

Practical advice

• Resize or crop to the resolution the task needs: a photo caption does not need 4K; reading a dense receipt or a chart legend might.
• For screenshots with small text, crop the relevant region rather than raising resolution everywhere.
• Count image tokens in budgets and rate limits exactly as you do text tokens.
• Sending a video as frames multiplies the cost: 60 s sampled at 1 frame per second with 256 tokens per frame is 60 * 256 = 15,360 tokens before any text.

Three ways to let the language model use visual information.

(a) Interleaved tokens (LLaVA-style)      (b) Cross-attention adapter (Flamingo-style)     (c) Early fusion / native
 text + image tokens in ONE sequence       text tokens stay text-only                       one model trained on all
 [t t t I I I I t t]                       [t t t t t] <--cross-attn--> [vision features]   modalities from scratch
       |  self-attention                         ^ extra layers inserted between LLM layers       [t I a t I a ...]
       v                                                                                          shared weights
   unchanged LLM

• Interleaved token fusion. Project patches into the LLM’s embedding space and concatenate them with text tokens. The existing self-attention handles cross-modal interaction with no architectural change. Simple and reuses a pretrained LLM, but image tokens consume the context budget, KV cache and compute exactly like text.
• Cross-attention adapters. The LLM’s own sequence stays text-only. New cross-attention layers let text tokens query a separate set of vision features, often compressed by a resampler: a fixed set of, say, 64 learned queries attends to all 576 patch vectors and returns just 64 vectors. Cost of that step is about 576 * 64 = 36,864 attention scores per head, rather than the 576 * 576 = 331,776 that full self-attention among the patches would need, and the LLM’s context sees 64 tokens or none. Decouples vision cost from the context budget, at the price of more architecture and training complexity, and some lost fine detail.
• Early fusion / native multimodal pretraining. Train one model on interleaved text, image and audio tokens from the start, potentially with a shared embedding space or tokenizer. Highest ceiling for deep cross-modal reasoning; most expensive to train and requires large multimodal data.

	Interleaved	Cross-attention	Early fusion
Reuses a pretrained text LLM	Yes	Yes	No
Image cost in LLM context	Full token count	Small or none	Full (native tokens)
Training cost	Low	Medium	High
Cross-modal depth	Good	Limited by adapter	Highest

Frontier models have moved toward native multimodal pretraining because adapter-style bolt-ons cap how deeply the model can reason jointly across modalities.

Traditional OCR (a text detector plus a recogniser) is purpose-built for extracting characters. It is faster, cheaper per page, deterministic and reliable on clean, well-scanned text at scale, and it does not need a full LLM forward pass per page.

VLM extraction wins when the task requires understanding layout and meaning jointly.

Task	OCR	VLM
Transcribe a clean scanned contract page	Best (cheap, exact)	Works, costs more
Fill a schema (vendor, date, total, line items) from a messy invoice	Needs extra parsing code per layout	Strong
“What is the trend in this chart?”	Reads axis labels only	Can answer
Handwriting, skewed scan, mixed languages	Brittle	More robust
Exact digits in a financial table	Reliable if the scan is clean	Can output plausible but wrong digits

A robust hybrid for numbers

def extract_total(image, vlm, ocr):
    v = vlm(image, "Return the invoice total as a number only.")      # semantic read
    ocr_text = ocr(image)                                              # literal characters
    candidates = {t.replace(",", "") for t in find_amounts(ocr_text)}
    return {
        "value": v,
        "verified": str(v) in candidates,     # the number appears literally on the page
    }
# verified == False -> send to human review instead of trusting the VLM

Walkthrough: the VLM supplies the semantic judgment (“which number is the total”), while OCR provides literal characters. If the VLM’s number is not among the amounts actually printed on the page, it is likely a hallucinated digit and is routed to a person. Use the same pattern, a cheap independent check, for any high-stakes field.

Cascade (transcribe, then text)

A speech-recognition model (Whisper-style) converts audio to text; the LLM only sees text.

audio -> ASR -> "please cancel my order" -> LLM -> text reply -> (optional) TTS -> audio

Simple, reuses text-only infrastructure. But it drops everything non-lexical: tone (“please” said sarcastically), emphasis, pauses, overlapping speakers, background sounds, and transcription errors propagate uncorrected into the LLM.

Native audio tokens

Audio is converted to a stream of vectors or discrete codes that go into the model beside text.

waveform (e.g. 16,000 samples/s)
   -> mel spectrogram (about 100 frames per second, one column of frequency energies each)
   -> audio encoder (strided convolutions + transformer; stride 2 halves the rate)
   -> about 50 audio tokens per second

A Whisper-style encoder, for example, processes 30-second windows: 3,000 spectrogram frames become 1,500 encoder positions. At about 50 tokens per second, one minute of audio is 60 * 50 = 3,000 tokens and a 30-second clip is 1,500, so audio costs context like images do. Codec-based approaches instead quantise audio into discrete codes (often several parallel streams per time step) so the model can also generate audio tokens that a codec decodes back to sound.

	Cascade	Native
Tone, emphasis, speaker overlap	Lost	Available to the model
Real-time voice conversation	Added latency across 3 stages	Lower latency possible
Debuggability	Easy (inspect transcript)	Harder
Complexity	Low	High

Rule of thumb: for transcription-style tasks (meeting notes, search) a cascade is usually enough and cheaper; for tone-aware or real-time voice assistants, native audio is the better fit.

Positions: text is 1D, images are 2D

RoPE and similar schemes assign each token one index along a line, so distance means “how many tokens apart”. Flatten a 16x16 patch grid row by row and patch (row 0, col 5) gets index 5 while the patch directly below it, (row 1, col 5), gets index 21: spatial neighbours are 16 positions apart in the sequence, while the left neighbour is 1 apart. A 1D scheme therefore misrepresents vertical adjacency.

grid indices (16 wide)          flattened order
  0  1  2 ... 15                0 1 2 ... 15 | 16 17 18 ... 31 | ...
 16 17 18 ... 31                 above/below are 16 apart, left/right are 1 apart

Some architectures give patches a pair of coordinates (row, col) and apply positional rotations separately per axis (2D-aware or factorised schemes); video adds a third axis for time.

KV cache: bursty and repeated

• A single image drops hundreds to thousands of tokens into the cache at once (e.g. 72 MiB for 576 tokens in the illustrative model above), unlike text generation, which adds one token at a time.
• A user often asks several questions about the same attached image or document. If the serving layer is not prefix-cache aware for the image portion, each follow-up re-encodes the image and re-prefills its tokens. Cache the image tokens (or their KV) keyed by the image content and keep the image at a stable position at the start of the prompt so a prefix cache can match.

Both are fluent output that is not supported by evidence, but the missing evidence is different.

• Text-only hallucination is typically about facts absent from training data or context: the model asserts something ungrounded in any source it has (an invented citation, a made-up API). The right information was never available to it.
• VLM hallucination is often about misreading what is in front of it: the information is in the input (the pixels), but the model reports something else. It is a grounding failure.

Where the evidence gets lost

photo 4000x3000, a serial number "8472" printed 20 px tall
   |  (1) resize to the encoder's input, e.g. 336 wide      scale 336/4000 = 0.084  -> digits ~1.7 px tall
   v
336x336 pixels
   |  (2) cut into 14x14 patches (24 x 24 = 576 patches)    one patch >> one digit
   v
576 patch vectors      <- a digit is smaller than one patch; its shape is blurred into the patch vector
   |  (3) projector + LLM
   v
LLM reads 576 image tokens + your question
   - where the visual evidence is weak, the language prior ("serial numbers look like 8472... or 3472?") fills in

Arithmetic check: a 20-pixel digit scaled by 336/4000 = 0.084 becomes about 20 * 0.084 = 1.7 pixels, much smaller than the 14-pixel patch, so the model effectively cannot see the digit. To keep that digit at 14 pixels (one patch tall) you need a scale of 0.7. Cropping a 400 x 300 region around the number and scaling it by 0.7 gives a 280 x 210 image, which is (280 / 14) * (210 / 14) = 20 * 15 = 300 tokens, versus 30,000 tokens to encode the whole 2800 x 2100 photo at that scale. Crop-and-zoom is far cheaper than raising resolution everywhere.

Typical failures, each with the mechanism

Failure	Example	Why it happens
Object hallucination	Describes “a fork next to the plate” when no fork is visible	Forks commonly co-occur with plates in training captions; the language prior supplies the likely object
Miscounting	Says 7 apples in a photo with 9	Counting is not a native operation; many small or overlapping objects are blurred into patches
Wrong spatial relation	Swaps left and right, or says the cup is “on” the table when it is beside it	1D token order and patch merging weaken precise geometry (see Why multimodal context complicates positional encoding and KV-cache management)
Small-text misread	Reads 8 as 3 in a low-resolution serial number, or misses a footnote	Glyphs are smaller than a patch
Chart misreading	Reports a bar’s value from its label colour rather than its height	Values must be measured against axes; the model pattern-matches typical chart text instead
Attribute leak	“The red car” when the only car is blue and a red sign is nearby	Features of neighbouring regions blend
Question-led agreement	“What colour is the dog?” answered “brown” when there is no dog	Presupposition in the question is trusted over the image

Mechanism

It is a grounding failure rather than a knowledge gap: the language model’s prior about what typically appears in such scenes overrides the visual evidence the encoder extracted. Reasons include coarse patches (small things fall below patch resolution), the language side being much stronger than the visual side (the LLM was trained on far more text than the VLM on image-text pairs), and image tokens being a minority of the context so the text of the question dominates attention.

Text vs vision hallucination, side by side

	Text-only	VLM
Is the right info in the input?	Usually no	Often yes, but not resolved
Typical symptom	Invented fact or citation	Wrong object, count, position, digit
First fix	Retrieval, “I don’t know”	Crop, zoom, higher resolution, region-by-region description
Independent check	Source lookup	Detector, OCR, second crop
Detection	Compare to a reference text	Compare to the image or a second view

Measuring it: a polling probe

A standard way to measure object hallucination is to ask yes/no questions about objects that are present and objects that are absent, then check how often the model says “yes” to absent ones (this idea is used by published benchmarks of this kind; build your own on your own images).

def object_probe(vlm, items):
    """items: (image, object_name, truly_present). Yes/no questions about present and absent objects."""
    tp = fp = fn = tn = 0
    for image, obj, present in items:
        said_yes = vlm(image, f"Is there a {obj} in the image? Answer yes or no.").strip().lower().startswith("yes")
        if present and said_yes: tp += 1
        elif present: fn += 1
        elif said_yes: fp += 1
        else: tn += 1
    n = tp + fp + fn + tn
    return {"accuracy": (tp + tn) / n, "recall": tp / (tp + fn),
            "false_yes_rate": fp / (fp + tn), "yes_rate": (tp + fp) / n}

def stub(image, question):   # a biased stand-in: says yes whenever a table-scene object is mentioned
    return "yes" if ("fork" in question or "plate" in question) else "no"

items = ([("img", "plate", True)] * 5 + [("img", "fork", True)] * 3 + [("img", "fork", False)] * 6 +
         [("img", "knife", False)] * 2 + [("img", "cup", True)] * 2)
print(object_probe(stub, items))
# -> {'accuracy': 0.556, 'recall': 0.8, 'false_yes_rate': 0.75, 'yes_rate': 0.778}   (rounded)

Walkthrough: tp counts correct “yes”, fn missed objects, fp hallucinated objects (the key quantity), tn correct “no”. false_yes_rate = fp / (fp + tn) is the fraction of absent objects the model claimed to see: here 6 of 8, or 0.75. A high yes_rate with decent recall means the model leans on co-occurrence instead of looking. Balance present and absent objects so a model that always answers “yes” cannot score well.

A second test: remove the image

Ask the same question with a blank or unrelated image. If the answer is unchanged, the model was answering from text priors and the image did not matter for that question. Questions that pass this test (answers change when the image changes) are the ones that measure vision.

Mitigations

• Ask the model to describe specific regions before answering (“List every object on the table, then answer”).
• Use higher resolution or crop and zoom on the area of interest when fine detail matters (the arithmetic above).
• Ask for evidence (“Quote the text you read in the bottom-left corner”).
• Avoid leading questions: ask “What is on the table?” rather than “What colour is the fork?”
• For counting and verification, use an independent check (a detector, OCR as in the hybrid above, a second pass at a different crop) rather than trusting one answer.
• Evaluate on your own images with known answers, as in Evaluating LLMs: benchmarks, contamination, judges; a public VQA score says little about your document type.