ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matching

TL;DR

<3-5 sentence high-level summary> CLIP excels at global image-text alignment but struggles with fine-grained attribute-object binding in complex scenes. The authors introduce ABE-CLIP, a training-free framework that combines Local Token-Patch Alignment with a Semantic Refinement Mechanism to strengthen attribute binding without retraining the backbone, and they fuse refined local scores with global CLIP similarity via a binding-difference-based adjustment. Empirical results on attribution benchmarks (ARO-A, SugarCrepe, ABC-6K) and zero-shot retrieval tasks show substantial gains over baselines and competitive performance with trained methods, validating the approach’s effectiveness and robustness. The work offers a practical, plug-and-play path to improve compositional understanding in CLIP-like models, with ablations confirming the contribution of each component and potential for broader compositional reasoning extensions.

Abstract

Contrastive Language-Image Pretraining (CLIP) has achieved remarkable performance in various multimodal tasks. However, it still struggles with compositional image-text matching, particularly in accurately associating objects with their corresponding attributes, because its inherent global representation often overlooks fine-grained semantics for attribute binding. Existing methods often require additional training or extensive hard negative sampling, yet they frequently show limited generalization to novel compositional concepts and fail to fundamentally address the drawbacks of global representations. In this paper, we propose ABE-CLIP, a novel training-free Attribute Binding Enhancement method designed to strengthen attribute-object binding in CLIP-like models. Specifically, we employ a Semantic Refinement Mechanism to refine token embeddings for both object and attribute phrases in the text, thereby mitigating attribute confusion and improving semantic precision. We further introduce a Local Token-Patch Alignment strategy that computes similarity scores between refined textual tokens and their most relevant image patches. By aggregating localized similarity scores, ABE-CLIP computes the final image-text similarity. Experiments on multiple datasets demonstrate that ABE-CLIP significantly improves attribute-object binding performance, even surpassing methods that require extensive training.

Figures (5)

• Figure 1: This illustration shows that CLIP fails to bind the attribute "green" to the corresponding object "vintage car". (a) The positive captions and the negative captions contain identical words while the attributes are swapped; (b) The CLIP similarity between the picture and the phrases "green vintage car" and "blue vintage car".
• Figure 2: Overview of ABE-CLIP. Given a query caption, we first parse it to extract attribute--object phrases, then refine the attribute embedding and object embedding via Representation Refinement Mechanism. Subsequently, the Local Token-Patch Local Alignment is introduced to aggregate token--patch similarities.
• Figure 3: Representation refinement sharpens token--patch alignment for "cube". (a) Input image "A yellow cylinder and a red cube." (b,c) Similarity matrix visualizations before and after refinement between the attribute "red" and each image patch.
• Figure 4: A comparison of similarity matrix visualizations for CLIP and FG-CLIP. We compute the similarity matrix using the words "floor" and "strawberry" with each image patch, respectively. It can be observed that CLIP fails to accurately identify the target "floor" and "strawberry", while FG-CLIP captures some relevant visual patches.
• Figure 5: Results of ablation study on different components. Incorporating Local Token–Patch Alignment, the Semantic Refinement Mechanism, and Binding Difference Scores yields progressive, incremental performance gains.