Embedding Invariance Report
15 diverse images × 3 encoders × 10 transform groups. Cosine similarity between each transformed embedding and the original — mean-pooled + L2-normalized, so the models are compared like-for-like. Each cell is the raw-pixel mean; the different-image floor tells you how low "low" is.
Summary: mean similarity by transform (raw pixels)
| Transformation |
CLIP (OpenAI) |
SigLIP (Google) |
DINOv2 (Meta) |
|---|
| Rotation (90/180/270) |
|
|
|
| Slight rotation (15/45) |
|
|
|
| Crop (20/35%) |
|
|
|
| Scale (80/120%) |
|
|
|
| Flip horizontal |
|
|
|
| Flip vertical |
|
|
|
| Grayscale |
|
|
|
| Brightness (±30%) |
|
|
|
| Blur |
|
|
|
| Occlude 25% |
|
|
|
| Different-image floor |
58% |
52% |
15% |
Key findings
Patch size does not rank the encoders. With embeddings pooled, SigLIP (patch 16) matches CLIP (patch 32) on rotation, and DINOv2 (patch 14) is close behind. An earlier version of this report compared CLIP's pooled vector against SigLIP/DINOv2's raw unpooled patch grids (rotating an image scrambles a patch grid's token order, which manufactured a fake "small patch = rotation-sensitive" effect). Pooling removes it.
The baseline changes the ranking. The different-image floor differs a lot: CLIP 58%, SigLIP 52%, DINOv2 15%. So CLIP's high absolute scores are partly a compressed embedding space (two unrelated images already sit at 58%). Normalized to each model's own floor, rotation robustness is CLIP 64%, SigLIP 70%, DINOv2 67%, comparable. "CLIP is the most invariant" doesn't survive the baseline.
JPEG q0.9 is negligible. Every transform is embedded twice (raw pixels and a JPEG round-trip); the mean difference is ~1 point. Compression is not what these numbers measure.
Rotation is the hardest geometric transform, but not catastrophic. 72–85% raw, comfortably above every floor. Photometric changes (grayscale, brightness, blur, occlusion) are nearly free (high 80s–90s): these encoders care about content layout far more than exact pixels.
Still true: no encoder is fully transformation-invariant: a rotated image is not treated as identical. ViT positional embeddings are absolute, so a turn is a different input. That framing was right in the earlier write-up; only the magnitudes were wrong.
Per-image breakdown (rotation / crop / flip-H, raw)
| Image |
CLIP (OpenAI) |
SigLIP (Google) |
DINOv2 (Meta) |
|---|
| photo-cafe.jpg |
74% / 90% / 99% |
75% / 84% / 99% |
51% / 88% / 94% |
| photo-phone-photographer.jpg |
95% / 82% / 99% |
95% / 82% / 99% |
87% / 66% / 98% |
| photo-road-tunnel.jpg |
93% / 81% / 98% |
95% / 84% / 98% |
86% / 70% / 97% |
| photo-floating-market.jpg |
90% / 83% / 98% |
87% / 82% / 98% |
71% / 77% / 97% |
| photo-forest-path.jpg |
89% / 88% / 100% |
83% / 86% / 100% |
65% / 85% / 96% |
| photo-sea-cliff.jpg |
82% / 76% / 98% |
82% / 84% / 99% |
62% / 81% / 95% |
| photo-open-book.jpg |
85% / 90% / 98% |
88% / 93% / 100% |
66% / 75% / 99% |
| photo-mountain-lake.jpg |
80% / 89% / 99% |
81% / 90% / 99% |
56% / 85% / 99% |
| photo-skateboard.jpg |
92% / 83% / 97% |
91% / 78% / 98% |
69% / 71% / 92% |
| photo-ocean-waves.jpg |
93% / 84% / 100% |
83% / 85% / 99% |
70% / 77% / 97% |
| text-readme.png |
90% / 84% / 94% |
82% / 84% / 90% |
83% / 89% / 99% |
| webpage-example.png |
83% / 78% / 88% |
84% / 74% / 83% |
80% / 86% / 100% |
| webpage-hn.png |
87% / 63% / 88% |
84% / 81% / 85% |
77% / 89% / 100% |
| webpage-sotw.png |
75% / 74% / 86% |
88% / 83% / 82% |
81% / 93% / 100% |
| webpage-wikipedia.png |
63% / 71% / 76% |
85% / 80% / 84% |
79% / 87% / 99% |
Embedding method: each image is rendered to a square on a mid-gray background, passed to the vision encoder, and its output is reduced to ONE vector, then L2-normalized; that vector is what the cosine compares. The pooling step matters:
image-feature-extraction returns a pooled
[1,512] for CLIP but raw patch-token grids for SigLIP
[1,196,768] and DINOv2
[1,257,384]; mean-pooling every model's tokens to one vector is what makes them comparable (skipping it compares flattened patch grids, where rotation just reshuffles token order (the artifact that produced the earlier "patch size" result). Every transform is embedded at raw pixels and three JPEG qualities (q0.9 / q0.5 / q0.2); tiling optionally splits the image into crops, embeds each, and pools them. The floor is the mean cosine between distinct images' originals.
Same method on a VLM: a vision-language model's encoder builds the same token grid, so you can tap it and pool it identically; Florence-2's
encode_image gives
[1,577,768] → mean-pool → a 768-d image embedding. An embedding is just one pooled hidden state; the alternative is running that hidden state through the output layer to write text. Try it in the
VLM playground.
Reproduce:
node cli.mjs test-images/* --model clip|siglip|dinov2|siglip384|clip-l --tiles N --json. Data + code:
GitHub.