Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Models

TL;DR

This paper investigates whether text encoders in vision-language models (TE-VLMs) encode syntactic structure and how pre-training objectives, model size, and data volume influence this ability. Using DepProbe to recover Universal Dependencies trees from representations of TE-VLMs (e.g., CLIP, FLAVA) and unimodal and sentence-language models, the authors compare across layers and data. They find that unimodal language models encode syntactic information more effectively than TE-VLMs, and that pre-training objectives strongly shape syntactic knowledge, with contrastive-only training yielding particularly weak syntax signals. The results motivate incorporating auxiliary objectives into VLM pre-training to improve linguistic structure understanding, with implications for downstream tasks that depend on syntax.

Abstract

Vision-language models (VLMs), serve as foundation models for multi-modal applications such as image captioning and text-to-image generation. Recent studies have highlighted limitations in VLM text encoders, particularly in areas like compositionality and semantic understanding, though the underlying reasons for these limitations remain unclear. In this work, we aim to address this gap by analyzing the syntactic information, one of the fundamental linguistic properties, encoded by the text encoders of VLMs. We perform a thorough analysis comparing VLMs with different objective functions, parameter size and training data size, and with uni-modal language models (ULMs) in their ability to encode syntactic knowledge. Our findings suggest that ULM text encoders acquire syntactic information more effectively than those in VLMs. The syntactic information learned by VLM text encoders is shaped primarily by the pre-training objective, which plays a more crucial role than other factors such as model architecture, model size, or the volume of pre-training data. Models exhibit different layer-wise trends where CLIP performance dropped across layers while for other models, middle layers are rich in encoding syntactic knowledge.

Figures (7)

• Figure 1: Four images sampled from the StableDiffusion 3.5 model for the same prompt A cat chases a dog, and the Universal Dependency tree for the prompt. StableDiffusion 3.5 uses CLIP-L, CLIP-bigG and T5-XXl models to encode the input prompt. Accessed via https://huggingface.co/spaces/stabilityai/stable-diffusion-3.5-large-turbo. We experimented with variations of this prompt and various other prompts, with similar issues observed in the results.
• Figure 2: Example predictions. FLAVA and RoBERTa predict the gold tree (left), CLIP predicts the tree on the right
• Figure 3: Comparison of the layer-wise performance between RoBERTa and CLIP.
• Figure 4: Example predictions with different types of errors.
• Figure 5: Sentence length distribution in thee filtered EWT training split.