MAFA: Managing False Negatives for Vision-Language Pre-training

TL;DR

The paper tackles the problem of false negatives in Vision-Language Pre-training caused by many-to-many image-text pairings in web data. It introduces MAFA, which combines Efficient Connection Mining (ECM) to uncover missing positives and Smoothed ITC (S-ITC) to stabilize contrastive learning, building on GRIT sampling. Across 4M-Noisy and 4M-Clean datasets, MAFA significantly improves IRTR, VQA, NLVR2, and zero-shot tasks, often outperforming models trained on substantially larger data, and remains compatible with BLIP-2. The findings highlight the importance of managing false negatives in VLP and demonstrate the method’s general applicability to modern multi-modal pre-training pipelines.

Abstract

We consider a critical issue of false negatives in Vision-Language Pre-training (VLP), a challenge that arises from the inherent many-to-many correspondence of image-text pairs in large-scale web-crawled datasets. The presence of false negatives can impede achieving optimal performance and even lead to a significant performance drop. To address this challenge, we propose MAFA (MAnaging FAlse negatives), which consists of two pivotal components building upon the recently developed GRouped mIni-baTch sampling (GRIT) strategy: 1) an efficient connection mining process that identifies and converts false negatives into positives, and 2) label smoothing for the image-text contrastive (ITC) loss. Our comprehensive experiments verify the effectiveness of MAFA across multiple downstream tasks, emphasizing the crucial role of addressing false negatives in VLP, potentially even surpassing the importance of addressing false positives. In addition, the compatibility of MAFA with the recent BLIP-family model is also demonstrated. Code is available at https://github.com/jaeseokbyun/MAFA.

Figures (8)

• Figure 1: Examples of positives, negatives, and false negatives among image-text pairs.
• Figure 2: Comparison of IRTR average scores and false negatives (%) in the 4M-Noisy dataset for GRIT-VLP and MAFA across different search spaces ($M$) when applying GRIT sampling. Here, IRTR average score is defined as the average image-text retrieval accuracy across (TR/R@1, TR/R@5, TR/R@10, IR/R@1, IR/R@5, IR/R@10) on COCO 5k test set. For all models, we set the batch size $B$ as $96$. Thus, when $M=96$, GRIT sampling becomes equivalent to random sampling.
• Figure 3: Overall framework of MAFA.
• Figure 4: Efficient Connection Mining (ECM).
• Figure 5: FNR on 4M-Clean dataset in random sampling scenario. Each annotated number is a mean of two ratios, w.r.t. image and text.