Late interaction and multi-modal retrieval

A standard bi-encoder compresses an entire query and an entire document each into one vector, so all fine-grained token-level information is collapsed before any comparison happens — a document that matches every query term individually but in a diffuse way, and a document with one incidentally- similar sentence, can end up with similar single-vector similarity despite very different true relevance. ColBERT instead keeps one vector per token for both query and document (encoded independently, so documents are still pre-computable and indexable, preserving bi-encoder-style scalability), and scores a (query, document) pair with MaxSim:

score(Q, D) = sum over query tokens q_i of:  max over document tokens d_j of  ( q_i . d_j )

Each query token finds its single best-matching document token and those best-matches are summed — so the score genuinely reflects fine-grained token-level alignment (this word of the query really did match this word of the document) rather than one blurred whole-sequence comparison. It is called “late interaction” because the interaction between query and document representations happens after both are independently encoded (late), unlike a cross-encoder where they interact during encoding (early/full) and unlike a bi-encoder where they never token-level interact at all.

Storing one vector per token instead of one vector per document multiplies storage and index size by roughly the average token count per document — often one to two orders of magnitude more vectors to index and search over, plus the MaxSim scoring itself is more expensive per candidate than a single dot product. Production systems manage this with compression (quantizing the per-token vectors, since individual token-level precision matters less than aggregate relevance) and by treating it as a first-stage or re-ranking component over a smaller candidate set rather than searching the full corpus with full per-token vectors naively. It is worth the cost specifically when retrieval quality on fine-grained term matching genuinely matters more than infrastructure simplicity — benchmarks consistently show ColBERT-family models beating single-vector bi-encoders on precision, closing much of the gap to cross-encoders, while still being searchable at a fraction of true cross-encoder query- time cost (no query-document pair needs a fresh transformer pass at query time).

	Bi-encoder	ColBERT (late interaction)	Cross-encoder
Query-document interaction	none (single vectors compared)	token-level, post-encoding	full, during encoding
Pre-computable document index	yes	yes (token vectors)	no
Relative accuracy	lowest	middle, close to cross-encoder	highest
Relative cost/storage	lowest	higher (per-token index)	highest (per-pair inference)
Typical role	first-stage retrieval at scale	first-stage or re-rank when precision matters more than storage	re-rank top ~100

Use a bi-encoder as the default first-stage retriever when storage/latency dominates the decision; reach for ColBERT when retrieval precision has a real, measured gap versus a bi-encoder on your data and you can afford the storage; keep a cross-encoder as the final re-ranking pass regardless, since its accuracy ceiling is the highest of the three and it only runs on a small candidate set.

CLIP-family models train an image encoder and a text encoder jointly with a contrastive objective: given a batch of (image, caption) pairs, push each image’s embedding close to its true caption’s embedding and far from every other caption in the batch (and symmetrically for text to image), using something like an InfoNCE loss over the batch’s similarity matrix. Neither encoder ever produces the other’s native representation — the point is that both are trained to land in one shared embedding space where cosine similarity is meaningful across modalities, so a text query’s embedding can be compared directly against pre-computed image embeddings with the same distance metrics used for text-only retrieval (see the distance-metrics subtopic). This is what makes “search images by a text description” and “search text by an example image” both possible with the same index and no separate translation step.

• Weak fine-grained/compositional understanding — CLIP-style models are trained on broad image-caption pairs and are often much better at coarse “what is this image generally about” than precise compositional queries (“a red cube on top of a blue sphere, not the reverse”) — word order and relational structure in the query do not always translate to correspondingly precise image matching.
• Text-in-image confusion — a model can retrieve based on text visible within an image (a sign, a label) when the query resembles that text, rather than the image’s actual visual content, since training data often correlates the two.
• Caption-distribution bias — retrieval quality mirrors the style of captions the model was trained on (often short, generic alt-text-style descriptions); queries phrased very differently from that distribution (highly technical, very long, or unusually structured) retrieve worse than a well-tuned text-only retriever would on comparable text queries.
• No native notion of recency or provenance the way a lexical/metadata system has — purely embedding-based image search needs the same metadata-filtering layer as text RAG to scope by date, source, or rights.

Mitigation mirrors the text case: hybrid with metadata filters, and add a re-ranking stage (potentially a more capable VLM scoring top candidates) when precision on compositional queries matters.

Code search has a specific character that argues for late interaction or hybrid-with-lexical over a plain single-vector bi-encoder: relevance often hinges on specific identifiers and exact tokens (a function name, a variable, an error string) which a single pooled vector tends to blur the same way it blurs any fine-grained term match, and exact-token behavior is closer to what BM25 and ColBERT’s token-level matching both preserve better than mean-pooled embeddings. A practical architecture: BM25 or a learned-sparse method for exact-identifier recall, a bi-encoder or ColBERT pass for semantic/paraphrase matches (“find the retry logic” without the literal word “retry”), fused (RRF), then a cross-encoder or LLM re-ranker over the top candidates for final precision on the handful of results actually shown or fed into an agent’s context — the same layered pattern as general hybrid RAG retrieval, specialized for code’s exact-match-heavy relevance signal.