Object-centric Binding in Contrastive Language-Image Pretraining

TL;DR

OC-CLIP tackles the binding problem in vision-language models by introducing an object-centric binding mechanism that aligns a slot-based image representation with a text-derived scene graph. It couples an inverted cross-attention binding module with a structured similarity score that separately handles object presence and relational constraints, enabling robust multi-object compositional understanding without hard-negative data. Empirically, OC-CLIP achieves substantial gains on both synthetic binding tasks and real-world compositional benchmarks, and demonstrates scalable benefits when trained from scratch on noisy data, including notable zero-shot ImageNet improvements. The work highlights the value of inductive biases from object-centric learning for vision-language alignment and suggests pathways for more sample-efficient, compositional VLMs in practical settings.

Abstract

Recent advances in vision language models (VLM) have been driven by contrastive models such as CLIP, which learn to associate visual information with their corresponding text descriptions. However, these models have limitations in understanding complex compositional scenes involving multiple objects and their spatial relationships. To address these challenges, we propose a novel approach that diverges from commonly used strategies, which rely on the design of hard-negative augmentations. Instead, our work focuses on integrating inductive biases into pre-trained CLIP-like models to improve their compositional understanding without using any additional hard-negatives. To that end, we introduce a binding module that connects a scene graph, derived from a text description, with a slot-structured image representation, facilitating a structured similarity assessment between the two modalities. We also leverage relationships as text-conditioned visual constraints, thereby capturing the intricate interactions between objects and their contextual relationships more effectively. Our resulting model not only enhances the performance of CLIP-based models in multi-object compositional understanding but also paves the way towards more accurate and sample-efficient image-text matching of complex scenes.

Figures (16)

• Figure 1: Object-Centric CLIP (OC-CLIP) overview. OC-CLIP begins with scene parsing, where we utilize a text parser (e.g., Llama3-based) to extract objects and relations from the input caption. The extracted text objects and relations are then fed into a text encoder, which generates distinct text embeddings for both nodes and relations. In parallel, the corresponding image is processed by an image encoder to produce patch-level image embeddings. These image embeddings are then combined with the text entity embeddings and passed through a binding module, which outputs visual token slots embeddings. Both modality are aligned in a new space using a structured similarity score that matches nodes embeddings to visual slots and models relational constraints between them.
• Figure 2: Efficiency and effectiveness of OC-CLIP: Analysis on synthetic data. Performance of the finetuned OpenCLIP and OC-CLIP models on a binary classification task between a caption and its corresponding hard-negative given a synthetic image, as shown in (a). Performance is shown as a function of the percentage of animal pairs (y-axis) seen during training and the proportion of hard-negatives used in the training data (x-axis). Results shown for (a) seen and (b) unseen object pairs.
• Figure 3: ViT features layer ablation.
• Figure 4: Score dimensionality ablations In this ablations we keep the initialization seed fixed and vary the dimensionality of the relation score $d_{\text{rel}}$ (x-axis) and object score $d_{\text{obj}}$(y-axis) and report the performance on the swap and replace splits of sugarcrepe.
• Figure 5: Downstream Compositional Understanding of OC-CLIP when trained on different parsing of COCO-Captions.