An interactive explainer, from first principles to what our experiments found. Everything on this page runs locally: drag, slide, and press things. No model downloads here; the lab, visualizer and VLM playground are where you run the real encoders.
An embedding is just a list of numbers, a vector, that positions something in a space. A 512-dimensional CLIP embedding is a point in a 512-dimensional space; things the model considers similar end up pointing in similar directions. We can't draw 512 dimensions, but the geometry that matters is already visible in two: similarity between vectors is measured by the cosine of the angle between them: 1 when they point the same way, 0 at right angles, −1 when opposite. Length doesn't matter (embeddings are usually normalized to length 1), only direction.
Add one dimension and nothing conceptually changes: the two vectors still span a flat plane between them, and the cosine is still measured inside that plane. That's the mental trick for 512 dimensions too; any two vectors, however many dimensions they live in, meet in a 2D plane of their own.
A vision transformer (ViT) can't look at pixels one by one, so it chops the image into a grid of patches and treats each patch like a word in a sentence. Each patch is flattened, passed through a learned projection, and becomes a token: a vector describing that patch. CLIP-B/32 uses 32-pixel patches (a 224×224 image → 7×7 = 49 tokens); SigLIP-B/16 uses 16 (196 tokens); Florence-2's DaViT works at 768×768. Smaller patches = more tokens = more detail = more compute.
A transformer is naturally order-blind, a bag of tokens. So before anything else, each token gets a position embedding added to it: a learned "I am patch (3,5)" tag baked into the token's numbers. From that moment on, what a patch shows and where it sits are mixed together in the same vector. This single design decision is why the question "does an embedding encode what-is-where?" is interesting at all.
Attention layers let every token look at every other token; after a dozen layers, each token has absorbed context from the whole image. (If you've heard that attention "only looks backwards": that's causal attention, the variant GPT-style decoders use because they generate text left-to-right and must not peek at words they haven't produced yet. An image has no reading order — all patches exist at once — so vision encoders use full bidirectional attention, and patch (0,0) freely attends to patch (6,6). Florence-2 contains both: a bidirectional vision encoder and a causal language decoder.) But you still have 49–577 token vectors and you want one embedding. That last step, pooling, quietly decides what survives:
Mean-pooling averages all tokens. Averaging is order-blind: rearrange the tokens and the mean is identical. CLS pooling instead uses one special summary token that attends to the others, and attention weights can depend on position, so arrangement leaks into the summary.
Architecture decides what can be represented; the training objective decides what is. The three families we test were trained to do different jobs:
Self-supervised (DINOv2): different crops/augmentations of the same image are pulled together, with no language anywhere, so "similar" means visually, structurally similar. VLM encoders (Florence-2's DaViT) aren't trained to make a nice space at all: their tokens feed a language decoder that must answer "read the text" and "where is the dog", so what-is-where must survive or the decoder starves.
Here's a trap: cosine similarities from different models live on different scales. Two unrelated images score ≈0.15 on DINOv2 but ≈0.69 on Florence. A "0.7 similarity" is impressive for one model and means nothing for the other. So every experiment reports the different-image floor (the average similarity of unrelated images) and normalizes against it: (sim − floor)/(1 − floor). In high dimensions random directions are nearly orthogonal (try the slider), so these high floors are a property of the model's space, not of geometry.
The practical question behind all of this: given two embeddings, what cosine value lets you claim they're the same image with high confidence? It's a separation problem. Plot two distributions: similarities of different images (the impostors) and similarities of an image to its own transformed self. If a threshold cleanly splits them, everything above it is "same image"; the highest impostor score is your safety line. Drag the threshold and watch both error rates; switch models and transform groups.
The dragging you just did has a formal name: you were choosing an operating point on a ROC curve. Sweep the threshold from 1 down to 0 and plot, at every value, the false-positive rate (fraction of impostor pairs you'd wrongly call "same") against the true-positive rate (fraction of transformed pairs you'd catch). The curve summarizes every possible threshold at once, and the area under it (AUC) is a single threshold-free score for how separable the two distributions are: 1.0 means a perfect split; 0.5 means the score is useless. Where you should sit on the curve depends on which error costs you more; deduplicating a photo library wants a low false-positive rate (don't merge different photos), while copyright matching may accept more false positives to miss fewer copies.
An embedding model reduces an image to a vector. A VLM reads it: Florence-2's DaViT encoder produces 577 tokens, and a small language decoder consumes them like a sentence, writing captions or transcribing text. We can still steal an embedding from it (mean-pool the 577 tokens), but the encoder's real customer is the decoder, and the decoder's output is a second, sharper window into what the representation keeps. Below are the model's actual captions for our corpus under rotation (from the caption-stability experiment; live data, not mock):
With the machinery above, the seven experiments stop being isolated numbers and become one story. Charts load the live results JSONs from this repo.
Read with §4 in mind: CLIP (the only CLS-projection here) moves the most; the mean-pooled models barely flinch; the retrieval-tuned Gemini flinches least of all. The chart ranks poolings and objectives, not "how much layout the encoder saw".
Everyone beats 25% chance: the orientation information is in every vector, including the ones whose cosine barely moved. Invariance-of-cosine and absence-of-information are different claims (§4 again). The ordering, Florence > DINOv2 > SigLIP > CLIP, is the thesis ordering: encoders trained to feed a decoder keep the most where.
Working at the token level (before pooling), we rotate token positions back after rotating the image. DINOv2 recovers a third of the lost similarity: its tokens are portable patch descriptions with position carried alongside (equivariance). Florence gets worse: position is entangled into the token features themselves. Layout isn't metadata for a VLM encoder — it's part of the percept.
A mirrored beach is still a beach; mirrored text is unreadable. The flip penalty on text scales with how much the model reads: DINOv2 (no language) can't tell; Florence (does OCR) treats a mirrored document like a different image. "Semantic invariance" is a property of model × content, not of a model.
The image must find its own caption among 15 (image↔text cosine, top-1). CLIP survives shuffling that wrecks its image↔image cosine: the vector moved, but stayed in the right neighborhood. Gemini Embedding 2 is transform-proof here. Meanwhile (§8) Florence's captions were destroyed by rotations its cosine shrugged at: semantic robustness and vector robustness are different axes.
All numbers come from the repo's results-*.json, generated by scripts/experiment-*.mjs (see the layout report for method and caveats: n=15, one seed, closed-model caveats for Gemini). Charts follow a validated colorblind-safe palette; every chart has a data table (view data). Built as part of the image-embedding-lab learning project.