Layout report: do embeddings encode what-is-where?

Seven experiments (2026-07-15) probing how much spatial layout lives inside an image embedding, across five encoders: CLIP ViT-B/32 (contrastive, 512-d CLS-projection), SigLIP B/16 (contrastive, mean-pooled 768-d), DINOv2-S (self-supervised, mean-pooled 384-d), Florence-2's DaViT (VLM vision encoder, mean-pooled 768-d at its native 768px), and Gemini Embedding 2 (hosted, VLM-native, 3072-d — closed model, so pooling and true input resolution are outside our control; treat it as an extra column, not a controlled comparison). All similarities are cosine; "normalized" = floor-normalized (sim − floor)/(1 − floor) against the different-image floor, so models with wildly different baselines are comparable. 15-image corpus, fp32 vision encoders in Node.

Patch shuffle: destroy layout, keep content

Each image's g×g tiles are permuted with one fixed seeded shuffle (content preserved pixel-for-pixel). A "bag of semantics" embedding shouldn't move; a layout-encoding one should collapse toward the floor.

model	2×2	4×4	8×8	floor
CLIP	0.689	0.500	0.277	0.577
SigLIP	0.835	0.651	0.348	0.518
DINOv2	0.838	0.718	0.497	0.146
Florence-2	0.825	0.674	0.409	0.694
Gemini Emb. 2	0.837	0.788	0.652	0.655

CLIP's pooled vector is the MOST layout-sensitive, the opposite of the naive prediction. The confound is what gets pooled: CLIP's output is its attention-based CLS projection (which integrates global arrangement), while the others are mean-pools over patch tokens, and mean-pooling is blind to position by construction. What you pool determines what invariance you measure. Gemini Embedding 2, a hosted embedding product tuned for retrieval, is the most shuffle-robust of all: barely a fifth of its similarity range is lost to full 8×8 scrambling.

Orientation probe: is rotation still in the vector?

A softmax probe predicts the rotation angle {0°, 90°, 180°, 270°} from the pooled embedding (leave-one-image-out CV, chance 25%). This asks about information, not movement.

model	probe accuracy
CLIP	45%
SigLIP	57%
DINOv2	62%
Florence-2	63%
Gemini Emb. 2	57%

Orientation is decodable from every embedding: "invariant" never means the information is gone. And the ordering matches the thesis: the VLM encoder retains the most orientation information, contrastive CLIP the least. Note the tension with experiment 1: CLIP's vector moves the most yet tells the least. Movement and information are different measurements.

Equivariance: did the representation break, or just move?

Compare unpooled token grids under rotation, then rotate the token positions back and compare again. Recovery ⇒ the encoder is equivariant (tokens stayed faithful, only positions moved). CLIP's pipeline output is already pooled, so it has no grid here. Values are means over 90/180/270°.

model	raw grid cos	position-compensated	recovery
SigLIP	0.270	0.426	+0.21
DINOv2	0.397	0.611	+0.35
Florence-2	0.437	0.348	−0.16

DINOv2 is the most equivariant: put its tokens back and a third of the lost similarity returns (the rest is patch-internal orientation). Florence goes negative: relocating its tokens makes things worse, meaning position is entangled into each token's features rather than carried alongside them. That's the strongest form of what-is-where coupling, and plausibly what a decoder that must answer "where is the text" needs.

Same content, swapped layout

Composite two images side-by-side as A|B and B|A: identical content, different arrangement. Floor = cosine between composites of different pairs.

Arrangement carries little weight in any pooled embedding (A|B ≈ B|A everywhere), but CLIP is again the most arrangement-sensitive, consistent with experiment 1. Mean-pooled representations barely notice a layout swap at all.

model	cos(A\|B, B\|A)	normalized
CLIP	0.913	0.832
SigLIP	0.977	0.949
DINOv2	0.950	0.937
Florence-2	0.986	0.951
Gemini Emb. 2	0.970	0.915

Caption stability: what does the VLM say under rotation?

Florence-2 captions every image at 0/90/180/270°. If the encoder moves but the decoder shrugs, captions should survive.

angle	caption Jaccard vs 0°	exact matches	embedding cos
90°	0.32	0 / 15	0.80
180°	0.24	0 / 15	0.86
270°	0.24	0 / 15	0.81

webpage-wikipedia.png · upright: "A computer screen with a page of information about cats." → 90°: "A screenshot of a computer screen with a cat on the bottom left." Rotated, the model stops reading the page and starts seeing it.

Hypothesis refuted: the decoder does not shrug. Not one caption survives any rotation, and many are wholesale mis-recognitions, while embedding cosine sits at a benign-looking ~0.8. Rotation genuinely changes what the model recognizes: photos have a canonical "up" in training data, and an upside-down scene is out-of-distribution content, not the same content re-posed. Embedding cosine understates the semantic damage.

Mirrored text: invariance depends on content × training

Horizontal flip keeps photographic semantics but destroys textual semantics. Normalized flip-similarity per category, with per-category floors:

model	photos	text/webpages	gap
CLIP	0.96	0.63	0.34
SigLIP	0.98	0.44	0.54
DINOv2	0.96	0.98	−0.02
Florence-2	0.92	−0.00	0.92
Gemini Emb. 2	0.95	0.50	0.45

The flip penalty on text scales with how much the model reads: DINOv2 (no language) can't tell mirrored text from text; CLIP and SigLIP punish it moderately; Florence, which does OCR, treats a mirrored document as no more similar than a different image (normalized ≈ 0). "Semantic invariance" isn't a property of a model; it's a property of model × content.

Caption retrieval: do the semantics survive the movement?

CLIP image↔text: after each transform, does the image still retrieve its own caption (top-1 of 15)?

transform	own-caption top-1	mean margin
identity	13 / 15	0.049
flip H	13 / 15	0.045
shuffle 4×4	12 / 15	0.017
rotate 180°	11 / 15	0.030
rotate 45°	11 / 15	0.018
crop 10%	14 / 15	0.047
crop 20%	14 / 15	0.049
crop 35%	11 / 15	0.028
rot 90° + crop 20%	11 / 15	0.028
rot 90° + crop 35%	6 / 15	0.003
rot 45° + crop 20%	9 / 15	0.012
blur	11 / 15	0.022
rotate 90°	10 / 15	0.032

CLIP's semantics are far more robust than its vector movement suggests: even a 4×4 patch shuffle (which drops image↔image cosine to ~0.79) keeps own-caption retrieval at 12/15. The vector moves through space, but it stays in the right neighborhood. Margins shrink under every transform, though: robustness is thinning, not free.

The same protocol on Gemini Embedding 2 (its text and images share a space):

transform	own-caption top-1	mean margin
identity	14 / 15	0.126
rotate 90°	15 / 15	0.111
rotate 180°	14 / 15	0.106
rotate 45°	15 / 15	0.128
crop 10%	13 / 15	0.123
crop 20%	13 / 15	0.113
crop 35%	13 / 15	0.048
rot 90° + crop 20%	14 / 15	0.082
rot 90° + crop 35%	10 / 15	0.010
rot 45° + crop 20%	13 / 15	0.084
shuffle 4×4	15 / 15	0.112
flip H	14 / 15	0.131
blur	14 / 15	0.133

Transform-proof retrieval, with one revealing exception. Gemini Embedding 2 keeps own-caption top-1 at 13–15/15 under every transform, with margins (~0.12) more than double CLIP's. Its weakest transform is the deep crop: at 35% the margin halves (0.048). That is the one transform in the set that removes content instead of rearranging it — the retrieval-tuned space shrugs at rearrangement and notices deletion, which is the thesis in one row. CLIP shows the mirror image: mild crops (10–20%) beat identity (14/15), since cropping can discard distracting background. Stack the transforms and even the strongest space finally bends: rotate 90° + crop 35% drops CLIP to 6/15 (margin 0.003, effectively chance) and Gemini to 10/15 — compound transforms multiply, they don't add.

Synthesis

Layout is in every embedding. Orientation probes beat chance everywhere (the probe), and the ordering is thesis-shaped: Florence > DINOv2 > SigLIP > CLIP on retained spatial information.

"How much the pooled vector moves" measures the aggregation, not the encoder. CLIP's CLS-projection moves most under layout changes (shuffle, layout swap) while carrying the least layout information (the probe), because mean-pooling (the other three) cancels positional structure that provably still exists at the token level (probe, equivariance). Any invariance claim must state what was pooled.

Encoders differ in how they carry position. DINOv2 is substantially equivariant (position rides alongside content and can be factored back out); Florence entangles position into the token features themselves (equivariance). Layout isn't metadata for a VLM encoder, it's part of the percept.

Semantic robustness and vector robustness are different axes. CLIP's meaning survives shuffling that wrecks its cosine (caption retrieval); Florence's cosine survives rotations that wreck its meaning (caption stability). And whether "meaning" survives a transform depends on the content: mirrored beaches are beaches, mirrored documents are noise (mirrored text).

"VLM-native" describes the lineage, not the embedding. Florence's raw encoder entangles position into its features, but Gemini Embedding 2, a retrieval-tuned embedding product from a VLM lineage, is the most layout-insensitive pooled vector in the suite (shuffle, retrieval) while still carrying decodable orientation (probe, 57%). Layout lives in the encoder; whether it survives into the embedding is decided by the head's training objective.

Reproduce

Caveats: n = 15 images, one shuffle seed, one run. Gemini Embedding 2 is a closed hosted model (public preview): its pooling, true input resolution, and any server-side preprocessing are unknown, and results may change as the preview evolves; the local models are the controlled comparison, Gemini is context. Treat orderings as robust, exact values as indicative. The CLS-vs-mean-pool confound (shuffle) is a finding here but also a limitation: a cleaner follow-up compares CLS and mean-pool within one model. The probe has 56 training samples against 384–768 dims; regularized, cross-validated, but small. Florence's language decoder runs q8 (its vision encoder fp32). Shared pipeline: scripts/lib-harness.mjs renders every transform at each encoder's native resolution per AGENTS.md. GitHub

Do embeddings encode what-is-where, or just what?