Compositional Image-Text Matching and Retrieval by Grounding Entities

TL;DR

This paper tackles the poor compositional generalization of vision-language models like CLIP by introducing Grounding CLIP (GCLIP), a training-free method that refines image embeddings through explicit noun-phrase grounding. It uses GPT-3.5-turbo to decompose captions into entities and relations and Grounding DINO to localize corresponding image regions, then fuses sub-image CLIP embeddings with the global image embedding using normalized similarity weights. The approach achieves state-of-the-art results on compositional benchmarks (ComVG and SVO-Probes) and significant gains on natural-image retrieval benchmarks (Flickr30K and MS-COCO), without additional training. The work demonstrates that grounding individual entities and relations can substantially enhance fine-grained visual-text alignment and retrieval, offering a practical, scalable enhancement for real-world multi-modal retrieval systems.

Abstract

Vision-language pretraining on large datasets of images-text pairs is one of the main building blocks of current Vision-Language Models. While with additional training, these models excel in various downstream tasks, including visual question answering, image captioning, and visual commonsense reasoning. However, a notable weakness of pretrained models like CLIP, is their inability to perform entity grounding and compositional image and text matching~\cite{Jiang2024ComCLIP, yang2023amc, Rajabi2023GroundedVSR, learninglocalizeCVPR24}. In this work we propose a novel learning-free zero-shot augmentation of CLIP embeddings that has favorable compositional properties. We compute separate embeddings of sub-images of object entities and relations that are localized by the state of the art open vocabulary detectors and dynamically adjust the baseline global image embedding. % The final embedding is obtained by computing a weighted combination of the sub-image embeddings. The resulting embedding is then utilized for similarity computation with text embedding, resulting in a average 1.5\% improvement in image-text matching accuracy on the Visual Genome and SVO Probes datasets~\cite{krishna2017visualgenome, svo}. Notably, the enhanced embeddings demonstrate superior retrieval performance, thus achieving significant gains on the Flickr30K and MS-COCO retrieval benchmarks~\cite{flickr30ke, mscoco}, improving the state-of-the-art Recall@1 by 12\% and 0.4\%, respectively. Our code is available at https://github.com/madhukarreddyvongala/GroundingCLIP.

Figures (9)

• Figure 1: For the caption A man is holding a sign (positive image on right side and negative image on left side) Grounding CLIP scores are 0.2259 and 0.2151 ✓; CompCLIP are 0.2152 and 0.2226 ✗, CLIP are 0.2112, 0.2123 ✗. Note that according to Grounding CLIP the positive images associated with the caption have higher similarity score than negative images.
• Figure 2: Overview of the proposed Grounding CLIP (GCLIP) framework. Given a caption such as “A man holding a sign,” we prompt the GroundingDINO with 'man' and 'sign' to localize relevant regions in the image. The resulting sub-images are embedded using CLIP and fused with the global image embedding through a weighted sum based on cosine similarity with their corresponding text phrases. This adjusted image embedding enables more accurate compositional alignment with the caption.
• Figure 3: Two-stage decomposition of the caption “A gray dog plays in the sand at the ocean” using GPT-3.5. In Stage 1, the model identifies objects, attributes, and their relations in structured JSON format. In Stage 2, it generates short natural phrases combining attributes and objects (e.g., “gray dog”, “sand”, “ocean”) and extracts subject-verb-object connections.
• Figure 4: Examples of bounding boxes returned by Grounding DINO, when prompted by man and sign on the left and man and umbrella on the right
• Figure 5: Examples of bounding boxes returned by Grounding DINO, when prompted by man, and water on the left and cat and water on the right.