FALCON: False-Negative Aware Learning of Contrastive Negatives in Vision-Language Alignment

TL;DR

This work tackles false negatives in vision-language pretraining by introducing FALCON, a learning-based mini-batch construction method that adaptively schedules negative samples via a negative mining scheduler. The scheduler operates on a unified I2T/T2I similarity distribution and uses a Beta-distributed hardness model to tailor per-anchor negative selection, optimizing cross-modal alignment by maximizing MLM loss reduction. Across three VLP frameworks (ALBEF, BLIP-2, SigLIP-2) and diverse downstream tasks, FALCON consistently outperforms heuristic negative mining approaches, especially in noisy or large-scale web data where false negatives are prevalent. While promising, the approach relies on proxy signals that primarily reflect cross-modal alignment from the vision side, suggesting future work to integrate signals from both vision and text encoders and to extend applicability to broader LVLM settings.

Abstract

False negatives pose a critical challenge in vision-language pretraining (VLP) due to the many-to-many correspondence between images and texts in large-scale datasets. These false negatives introduce conflicting supervision signals that degrade the learned embedding space and diminish the effectiveness of hard negative sampling. In this paper, we propose FALCON (False-negative Aware Learning of COntrastive Negatives), a learning-based mini-batch construction strategy that adaptively balances the trade-off between hard and false negatives during VLP. Rather than relying on fixed heuristics, FALCON employs a negative mining scheduler that dynamically selects negative samples of appropriate hardness for each anchor instance during mini-batch construction, guided by a proxy for cross-modal alignment improvement. Experimental results demonstrate that FALCON significantly improves performance across three vision-language learning frameworks (ALBEF, BLIP-2, SigLIP-2) and a broad range of downstream tasks and evaluation settings, underscoring its effectiveness and robustness in mitigating the impact of false negatives.

Figures (10)

• Figure 1: Illustration of semantic distance between an anchor text and multiple image samples in the vision-language embedding space.
• Figure 2: (Top Left) Risk of false negatives for different similarity quantiles $q$ used in mini-batch construction during VLP byun2022grit. False negatives are identified using the pretrained BLIP-129M model li2022blip. (Bottom Left) False negative ratio during VLP when false negative filtering is applied using the pretrained model’s ITM score at varying thresholds. On the right, Text-to-Image (Top Right) and Image-to-Text (Bottom Right) examples are shown where the pretrained model fails to assign high ITM scores to false negatives (highlighted in red).
• Figure 3: Overview of FALCON, a learning-based mini-batch construction strategy for VLP. Starting from a randomly selected anchor, the scheduler $\pi_\phi$ predicts hardness quantile values to iteratively sample additional candidates, forming a mini-batch index set $\mathcal{I}$. This batch is used to update the VLP model. The reduction in masked language modeling loss $\mathcal{L}_\text{MLM}$ serves as a proxy for enhanced cross-modal alignment, providing a learning signal that guides $\pi_\phi$ toward constructing more informative mini-batches in subsequent training steps.
• Figure 4: Visualization of normalized cosine similarity distributions during training. Each plot shows the similarity at different quantile levels for I2T (left), T2I (center), and I2T+T2I (right), with the color bar indicating training progression. Step values are shown in thousands (K).
• Figure 5: Image-to-Text (Left) and Text-to-Image (Right) examples illustrating FALCON’s quantile-based negative sampling strategy. For each anchor query (shown adjacent to each plot), the normalized similarity distribution $\widehat{\mathbf{S}}$ over the candidate set is displayed alongside the scheduler’s predicted distribution on quantiles (blue-shaded density curve). Sampled negatives are annotated with their (one-way similarity / pretrained ITM score), and color-coded by hardness level as defined in Figure \ref{['fig:relationship']}.