Understanding Embeddings and Vector Similarity

Text becomes a vector

An embedding model maps text to a fixed-length vector. Texts with related meanings often occupy nearby directions in that model's vector space. The coordinates are learned representations, not individually interpretable labels.

Compare direction with cosine similarity

For nonzero vectors a and b, cosine similarity is their dot product divided by the product of their lengths. Parallel vectors score 1; orthogonal vectors score 0. Negative scores are possible. Similarity is not a probability or proof of correctness.

public static double Cosine(double[] a, double[] b)
{
    if (a.Length == 0 || a.Length != b.Length)
        throw new ArgumentException("Equal non-empty dimensions required.");
    double dot = 0, aa = 0, bb = 0;
    for (int i = 0; i < a.Length; i++)
    {
        dot += a[i] * b[i];
        aa += a[i] * a[i];
        bb += b[i] * b[i];
    }
    if (aa == 0 || bb == 0) return 0;
    return dot / (Math.Sqrt(aa) * Math.Sqrt(bb));
}

Work through small examples

For [1,0] and [1,0], the score is 1. For [1,0] and [0,1], it is 0. For [1,0] and [-1,0], it is -1. Our convention returns 0 for a zero vector, whose mathematical cosine is undefined. Production systems should instead reject or separately handle invalid embeddings.

Respect the model boundary

Index documents and queries using the same model and compatible preprocessing. Mixing embeddings from different models or dimensions produces meaningless comparisons. If a model changes, version and rebuild the index.

Exercise

Write unit tests for identical vectors, perpendicular vectors, a zero vector, and dimension mismatch. Then explain why two semantically similar documents might disagree factually.