Distance and similarity metrics on vectors

For vectors a, b in R^d:

Dot / inner product:    a . b            = sum_i a_i * b_i
Cosine similarity:       cos(a, b)        = (a . b) / (||a||_2 * ||b||_2)
Euclidean (L2) distance: d2(a, b)         = sqrt( sum_i (a_i - b_i)^2 )
Squared Euclidean:       d2sq(a, b)       = sum_i (a_i - b_i)^2
Manhattan (L1) distance: d1(a, b)         = sum_i |a_i - b_i|
Chebyshev (L-inf):       dinf(a, b)       = max_i |a_i - b_i|
Minkowski (L-p):         dp(a, b)         = ( sum_i |a_i - b_i|^p )^(1/p)

For binary / bit vectors:

Hamming distance:        number of differing bits
Jaccard distance:        1 - |A ∩ B| / |A ∪ B|     (sets / binary)

Statistically weighted:

Mahalanobis distance:    sqrt( (a - b)^T * S^-1 * (a - b) )   (S = covariance)

“Similarity” (higher = closer, e.g. cosine, dot) and “distance” (lower = closer, e.g. L2) are inverse framings; an index is configured for one specific metric.

If ||a|| = ||b|| = 1, expand squared Euclidean distance:

||a - b||^2 = ||a||^2 - 2 (a . b) + ||b||^2
            = 1 - 2 (a . b) + 1
            = 2 - 2 (a . b)
            = 2 - 2 cos(a, b)          (since a.b = cos on unit vectors)

So ||a - b||^2 = 2(1 - cos(a, b)). Euclidean distance is a monotonically decreasing function of cosine similarity on the unit sphere — ranking by smallest L2 distance and ranking by largest cosine give the identical order. This is why, once you normalize, it does not matter whether your ANN index uses inner product, cosine, or L2: same neighbors. It also means a cosine threshold converts directly to an L2 threshold.

Cosine measures only direction — the angle between vectors, ignoring magnitude. Right when magnitude is noise: document length, repeated terms, or embedding norm varying for reasons unrelated to meaning. This is the default for text similarity, and most sentence-embedding models are trained and evaluated with cosine.

Raw dot product lets magnitude matter. Right when the model deliberately encodes something in the norm — some models make more “confident” or more information-dense passages have larger norms, so dot product rewards them; MIPS (maximum inner product search) is also the native formulation for some two-tower recommender and retrieval models where the norm carries a learned prior (e.g. popularity). If vectors are normalized, dot product is cosine, so the distinction only exists for un-normalized vectors.

Rule: use whatever the embedding model’s card specifies. Mixing (normalize for the query, not the docs, or use L2 when the model expects cosine) silently degrades recall.

As d grows, under many distributions the distances between a point and its nearest and farthest neighbors concentrate — the ratio (d_max - d_min) / d_min tends toward 0, so “nearest neighbor” becomes weakly defined and absolute distance values stop being informative. Contributing effects: volume concentrates near the surface of a hypersphere; random vectors become nearly orthogonal. Practical consequences for retrieval:

• Absolute L2 or cosine scores are not comparable across queries — a cosine of 0.7 might be a great match for one query and mediocre for another. Calibrate thresholds per-model, empirically, and prefer ranking over absolute cutoffs.
• Learned embedding spaces are far better behaved than random ones (the model shapes the manifold), but score calibration is still model-specific.
• It is part of why re-ranking with a cross-encoder helps — it does not rely on distances in the embedding space at all.

L1 = sum of absolute per-dimension differences; L2 squares them first. Consequences:

• L2 penalizes one large per-dimension gap much more than several small ones (squaring); L1 treats a total “budget” of difference the same however it is distributed.
• L1 is more robust to outlier dimensions — a single wildly-off coordinate dominates L2 but contributes linearly to L1.
• L1 balls are diamonds, L2 balls are spheres; L1 tends to favor axis-aligned sparsity (the same reason Lasso uses an L1 penalty).

For dense learned embeddings L2/cosine are the norm and L1 is rarely better; L1 (or Hamming on binarized vectors) shows up in some quantized / memory-constrained setups and in classic locality-sensitive hashing.

• Binary embeddings (each dim → 1 bit via sign): compare with Hamming distance (popcount of XOR) — extremely fast, SIMD-friendly, ~32x smaller than float32. Recall drops materially, so the standard pattern is binary search for a large candidate set then re-score with full-precision vectors.
• Scalar-quantized (float32 → int8): use the same metric as the float version (L2 / inner product) on the integers with a scale factor; error is small (~4x compression).
• Product-quantized: distance is approximated via per-subspace lookup tables (asymmetric distance computation) rather than computed directly.
• Jaccard / Tanimoto: for genuine set-valued or sparse-binary representations (e.g. chemical fingerprints, sparse lexical vectors), not for dense sign-bit vectors.

You can, and it is common (mean-pooling token vectors into a sentence vector; averaging a few query paraphrases; centroid of a cluster). What to watch:

• Averaging pushes toward the mean of the space and shrinks norm; if you then use cosine it is usually fine (direction survives), if you use dot product the shrinking norm distorts scores.
• Averaging semantically different things produces a vector near neither — the mean of “cat” and “quantum mechanics” is not a useful concept; it can retrieve unrelated middle-ground content.
• Weighted averages (e.g. IDF-weighted token vectors, or down-weighting a paraphrase you trust less) generally beat uniform means.
• Linear arithmetic (“king - man + woman”) works far less cleanly on modern contextual embeddings than the old word2vec demos suggested.

1. Start from the embedding model’s documentation — it states the intended metric and whether to normalize. That is right ~95% of the time; deviating needs a reason and a benchmark.
2. Normalize consistently at index and query time if using cosine, and confirm your vector DB is configured for the same metric.
3. Validate on a labeled set — build query→relevant-doc pairs, compute recall@k / MRR / nDCG under each candidate metric, and pick empirically rather than by theory.
4. Check score calibration if you plan to threshold (e.g. “only answer if top score > X”) — plot score distributions for known-relevant vs known-irrelevant pairs; the overlap tells you whether a threshold is even viable or whether you need a re-ranker.
5. Re-validate whenever the embedding model changes — metric behavior and calibration are model-specific.