Left-Right Symmetry Breaking in CLIP-style Vision-Language Models Trained on Synthetic Spatial-Relation Data

TL;DR

This work introduces a controllable 1D image–text testbed to probe how CLIP-style transformer models acquire left–right spatial relations. Through end-to-end contrastive training, the authors show that relational understanding emerges and generalizes best when label diversity is high, with layout variation playing a secondary role. A mechanistic analysis reveals that interactions between token and positional embeddings generate a horizontal attention gradient, breaking left–right symmetry and enabling unseen-pair generalization; ablations confirm the critical role of the positional contribution to attention. The study also shows that vision and text representations align up to a rotation, and demonstrates how RoPE-based mechanisms can yield similar generalization via bias-induced gradients or low-rank subspace alignment. Collectively, the results provide a concrete mechanistic account of when and how CLIP-style models can acquire relational spatial competence, with implications for scaling to richer 2D data and more complex relational tasks.

Abstract

Spatial understanding remains a key challenge in vision-language models. Yet it is still unclear whether such understanding is truly acquired, and if so, through what mechanisms. We present a controllable 1D image-text testbed to probe how left-right relational understanding emerges in Transformer-based vision and text encoders trained with a CLIP-style contrastive objective. We train lightweight Transformer-based vision and text encoders end-to-end on paired descriptions of one- and two-object scenes and evaluate generalization to unseen object pairs while systematically varying label and layout diversity. We find that contrastive training learns left-right relations and that label diversity, more than layout diversity, is the primary driver of generalization in this setting. To gain the mechanistic understanding, we perform an attention decomposition and show that interactions between positional and token embeddings induce a horizontal attention gradient that breaks left-right symmetry in the encoders; ablating this contribution substantially reduces left-right discrimination. Our results provide a mechanistic insight of when and how CLIP-style models acquire relational competence.

Figures (23)

• Figure 1: Schematic of our problem: how spatial and relational capabilities are, or are not, acquired in vision-language models?
• Figure 2: Schematic of CLIP model training of the toy dataset of 1D images and the corresponding texts. For two object images, left and right textual representations denote inverse relations that describe the same configuration.
• Figure 3: Three types of generalization observed in a CLIP-style training setup. This experiment is performed only with the left textual representation. (Left) Accuracy is shown for three types of generalization (a–c). (Right) Cosine similarity maps between image and text embeddings from the output layers are shown ($N_{\rm tot} = 20, N_{\rm pair}=15, N_{\rm val}=5, n_2=10$). In the similarity map of (a), $n_1$ images sharing the same text representation are repeated $N_{\rm tot}$ times along the image axis. In (b) and (c), $n_2$ images with the same text representation are repeated for all ordered pairs from $N_{\rm pair}$ and $N_{\rm val}$ labels, respectively. Hyperparameters: $M_B=2, M_{\rm rep}=2, M_h=4, d_{\rm head}=32, d_{\rm MLP}=512, d_{\rm model}=128$.
• Figure 4: Analysis of reduced 1-layer model with $M_B=M_{\rm rep}=1, M_h=4$. The other parameters are the same as those used in Fig. \ref{['fig:fig_2layer_full']}. (a) Accuracy for unseen-pair generalization. (b) Examples of the attention pattern of the vision encoder. "C" in the 1D image corresponds to the class token. The red rectangles highlight the class token row in each attention map. The model trained with $N_{\rm tot} = 20, N_{\rm pair}=15, n_2=10$ are used for the visualization. (c) Probability that each head attends to the left or right object, computed from the class token's attention weights. The red dashed line indicates random guess $0.5$. This analysis is performed on images containing label pairs not seen during training ( i.e. pairs formed from labels 16–20).
• Figure 5: Attention gradient emerging in the attention pattern. (a) Contribution of decomposed terms to logit of the attention is shown for head 2 as a representative example. The red rectangles highlight the class token row in each attention map. The model for Fig. \ref{['fig:fig_1layer_model_results']}(b,c) is used for this analysis (a-d). (b) Spatial profiles of the total and each component of the class token row for two images with swapped object positions; colors denote the total and the four decomposed terms. (c) Definition of $\Delta_{\rm p.e.}$ and $\Delta_{\rm label}$ is schematically shown. $\Delta_{\rm p.e.}$ is plotted against $\Delta_{\rm label}$ for head 2. Each scatter point represents a test 1D image. (d) The probability that each head attends to the two objects based on their spatial relationship ($|\Delta_{\rm p.e.}|<|\Delta_{\rm label}|$) versus in a label-specific manner ($|\Delta_{\rm p.e.}|<|\Delta_{\rm label}|$). (e) Effect of ablating positional-embedding–derived logit components. Accuracy for unseen object pairs generalization is shown for different ablation conditions. Baseline is the model from Fig. \ref{['fig:fig_1layer_model_results']}(a) ($n_2=10$). At inference, we zero specific pre-softmax attention logit terms for all four heads: EP, PP, PP terms and the BP term $B_Q^TW_KP^T$.