Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples

TL;DR

The paper tackles the limitation in multimodal contrastive learning where random negatives hinder fine-grained conceptual understanding. It proposes hard negative sampling by keyword substitution in captions and introduces InpaintCOCO, an inpainting-based challenge set to isolate and evaluate fine-grained visual-language alignment. Empirically, hard negatives yield substantial gains in fine-grained concept understanding across object, color, location, and size with only minor reductions in general retrieval, and InpaintCOCO provides a controlled benchmark to probe cross-modal concept representations. The approach is data-efficient, requiring minimal domain knowledge to design the negatives, and demonstrates strong generalization across several VL datasets, suggesting practical benefits for domain adaptation and downstream vision-language tasks.

Abstract

Current multimodal models leveraging contrastive learning often face limitations in developing fine-grained conceptual understanding. This is due to random negative samples during pretraining, causing almost exclusively very dissimilar concepts to be compared in the loss function. Consequently, the models struggle with fine-grained semantic differences. To address this problem, we introduce a novel pretraining method incorporating synthetic hard negative text examples. The hard negatives permute terms corresponding to visual concepts, leading to a more fine-grained visual and textual concept alignment. Further, we introduce InpaintCOCO, a new challenging dataset for assessing the fine-grained alignment of colors, objects, and sizes in vision-language models. We created the dataset using generative inpainting from COCO images by changing the visual concepts so that the images no longer match their original captions. Our results show significant improvements in fine-grained concept understanding across a wide range of vision-language datasets, including our InpaintCOCO dataset.

Figures (7)

• Figure 1: Classical contrastive learning approaches use $(I_{1}, T^{pos}_{1})$ as positive pairs in combination with negative samples like $T^{neg}_{2}$ and $T^{neg}_{3}$ to learn an image-text alignment. A bag of words (e.g., nouns) is often sufficient to extract the correct text that matches a given image, resulting in only broad concepts learned. We also use hard negatives like $T^{hn}_{1}$ so that fine-grained semantic concepts are learned for visual and textual alignment.
• Figure 2: Hard negative contrastive learning: Keyword substitution produces hard negative text samples, which are then randomly injected for each image $u_i$, replacing a simple negative sample in InfoNCE loss.
• Figure 3: Create hard negative image samples using open vocabulary segmentation for the masking prompt and text-to-image generation for the inpainting prompt. Additionally, a new correct caption is created manually. The InpaintCOCO dataset was created for concepts object, color, and size.
• Figure 4: Fine-grained Concept Understanding vs. General Image Retrieval: Results for four different concepts trained on corresponding dataset subsets. Checkpoints are evaluated after every 10% of the data (circles); checkpoints at epoch ends are marked with the respective numbers. The results are also in a table form in Table \ref{['tab:results']} in the Appendix.
• Figure 5: Samples from visual and cross-model datasets.