Global Geometry Is Not Enough for Vision Representations

TL;DR

Global geometry is insufficient to capture compositional relations in vision representations, so the authors introduce a synthetic attribute-binding probe to test shape–position bindings. They evaluate 21 encoders spanning eight objectives and find that static geometric metrics correlate poorly with binding performance, whereas a Jacobian-based metric called Jacobian Effective Rank (JER) robustly tracks binding. They provide an analytic account showing that many objectives constrain embedding geometry but leave local input–output sensitivity underconstrained, explaining the mismatch and linking objective design to functional sensitivity. The work advocates treating functional sensitivity as a complementary axis for modeling composition and suggests future objectives that regulate the Jacobian spectrum to improve relational reasoning and binding across vision representations.

Abstract

A common assumption in representation learning is that globally well-distributed embeddings support robust and generalizable representations. This focus has shaped both training objectives and evaluation protocols, implicitly treating global geometry as a proxy for representational competence. While global geometry effectively encodes which elements are present, it is often insensitive to how they are composed. We investigate this limitation by testing the ability of geometric metrics to predict compositional binding across 21 vision encoders. We find that standard geometry-based statistics exhibit near-zero correlation with compositional binding. In contrast, functional sensitivity, as measured by the input-output Jacobian, reliably tracks this capability. We further provide an analytic account showing that this disparity arises from objective design, as existing losses explicitly constrain embedding geometry but leave the local input-output mapping unconstrained. These results suggest that global embedding geometry captures only a partial view of representational competence and establish functional sensitivity as a critical complementary axis for modeling composite structure.

Figures (6)

• Figure 1: Global geometry is not enough for composition modeling.(a) Global isotropy does not strongly correlate with compositional binding across different types of vision encoders. (b) In contrast, our functional metric (Jacobian Effective Rank + Discriminability) accurately predicts performance. Statistics in \ref{['tab:correlations']}.
• Figure 2: (a) Existing visual representation learning objectives and metrics often assume that favorable geometric properties, such as uniformity, yield optimal representations. (b) However, we show that geometry is not everything, as functional sensitivity can differ between geometrically well-distributed representations. (c) To examine the downstream implications of this difference, we present the synthetic attribute-binding evaluation setting used in this work, designed to isolate confounding variables such as object detection.
• Figure 3: Normalized singular value spectrum of the input-output Jacobian for four models. Barlow Twins maintains a flat spectrum indicating uniform sensitivity across input directions. CLIP, DINOv2, and MAE show rapid spectral decay, concentrating sensitivity in few directions.
• Figure 4: Evolution of Jacobian effective rank across network depth. Effective rank is computed at each block using random noise inputs. Barlow Twins preserves high rank throughout the backbone. CLIP and DINOv2 exhibit mid-network rank collapse (blocks 5–8). MAE maintains high rank in the backbone but collapses at the projection layer.
• Figure 5: Geometry-based readout comparison results. Zero-shot cosine similarity, local PCA, and kNN are evaluated on frozen embeddings. Barlow Twins and VICReg perform well with cosine similarity, while CLIP, DINOv2, and MAE remain near chance under all geometric readouts.