Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining

TL;DR

The paper tackles the batch-size bottleneck in contrastive multimodal learning by proposing B3, a batch mining strategy that constructs batches from a full dataset using a teacher-based ranking to form a sparse graph, followed by METIS clustering to create batches rich in in-batch negatives. It provides theoretical justification through a bound linking global and batch losses and demonstrates that batching from cohesive communities yields stronger learning signals with reduced computational cost. The authors introduce B3 and an enhanced variant B3++ that adds hard negatives, achieving state-of-the-art results on the MMEB benchmark across 36 tasks, including strong performance at batch sizes as small as $|B|=64$. Empirically, B3 is robust to teacher strength, scales well across batch sizes, and generalizes to both multimodal and text-only domains, with offline preprocessing enabling practical deployment and integration with existing pipelines.

Abstract

Contrastive learning (CL) is a prevalent technique for training embedding models, which pulls semantically similar examples (positives) closer in the representation space while pushing dissimilar ones (negatives) further apart. A key source of negatives are 'in-batch' examples, i.e., positives from other examples in the batch. Effectiveness of such models is hence strongly influenced by the size and quality of training batches. In this work, we propose 'Breaking the Batch Barrier' (B3), a novel batch construction strategy designed to curate high-quality batches for CL. Our approach begins by using a pretrained teacher embedding model to rank all examples in the dataset, from which a sparse similarity graph is constructed. A community detection algorithm is then applied to this graph to identify clusters of examples that serve as strong negatives for one another. The clusters are then used to construct batches that are rich in in-batch negatives. Empirical results on the MMEB multimodal embedding benchmark (36 tasks) demonstrate that our method sets a new state of the art, outperforming previous best methods by +1.3 and +2.9 points at the 7B and 2B model scales, respectively. Notably, models trained with B3 surpass existing state-of-the-art results even with a batch size as small as 64, which is 4-16x smaller than that required by other methods. Moreover, experiments show that B3 generalizes well across domains and tasks, maintaining strong performance even when trained with considerably weaker teachers.

Figures (5)

• Figure 1: The batch mining mechanism of B3. Initially, a teacher model generates a rank matrix $R$ over the training set, indicating potential negative relationships. From these rankings (specifically ranks in the range $[p:p+m]$ for each query), a undirected sparse preference graph $S$ is constructed. Then, METIS clustering is applied to identify communities of mutually strong negatives. Finally, diverse training batches of size |B| are formed by sampling examples from $|B|/K$ distinct communities.
• Figure 2: We compare B3 and Random Batches (Qwen2-2B, 2 epochs) across batch different sizes. The performance gap is highest at smaller batch sizes and remains significant even as batch size increases. At a batch size of 64, B3 surpasses the 2B state-of-the-art, LLaVE (Llava-OV-2B).
• Figure 3: Variants of B3 hyper-parameters and the GBM baseline are evaluated across multiple batch sizes. B3 consistently outperforms GBM at all batch sizes. At smaller batch sizes, the impact of false negatives—introduced by lower values of $p$ in B3—is more apparent.
• Figure 4: Average top-10000 scores for MSCOCO_i2t with VLM2Vec-Qwen2B
• Figure 5: Average top-10000 scores for VisualNews_t2i with VLM2Vec-Qwen2B

Theorems & Definitions (2)

• Theorem 1
• Theorem 2: saunshi2019theoretical