Sigmoid Loss for Language Image Pre-Training

TL;DR

SigLIP introduces a sigmoid-based loss for language–image pre-training that eliminates the need for global batch normalization across all pairs, enabling memory-efficient, large-batch training. The approach achieves strong zero-shot and cross-modal retrieval performance, often surpassing softmax-based baselines at small-to-moderate batch sizes and saturating near 32k even with massive data. It further demonstrates robustness to noise and data diversity, and extends efficiently to multilingual settings via bottlenecked embeddings. Practically, SigLIP enables competitive language–image pre-training on modest hardware and provides scalable pathways for future research in cross-modal representation learning.

Abstract

We propose a simple pairwise Sigmoid loss for Language-Image Pre-training (SigLIP). Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. Combined with Locked-image Tuning, with only four TPUv4 chips, we train a SigLiT model that achieves 84.5% ImageNet zero-shot accuracy in two days. The disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient. We release our models at https://github.com/google-research/big_vision and hope our research motivates further explorations in improving the quality and efficiency of language-image pre-training.

Figures (8)

• Figure 1: Efficient loss implementation demonstrated via a mock setup with 3 devices and a global batch size of 12. There are no all-gathers, and at any point in time only the bright yellow square (size$4 \times 4$ ) is materialized in memory.
• Figure 2: The effect of pre-training batch size. Left: SigLiT results, trained for 18 B seen examples. Sigmoid loss outperforms the softmax loss significantly with small batch sizes, and performs similarly at larger batch sizes. We successfully trained an SigLiT model with up to one million batch size. However, performance for both sigmoid and softmax saturate at around 32 k batch size. Middle: SigLIP results, trained for 9B seen examples. Both sigmoid loss and softmax loss saturate at a reasonable batch size, while the peak of the sigmoid loss comes earlier and slightly outperforms the peak of the softmax loss. A very large batch size hurts both losses. Right: mSigLIP results, trained for 30B seen examples. With a multilingual setup using over 100 languages, 32 k batch size is surprisingly sufficient and scaling beyond that hurts performance on a 36-language cross-modal retrieval task.
• Figure 3: SigLiT ImageNet$\mathbf{0}$-shot transfer results with different training durations. Large batch size results in a big performance boost, but needs a sufficiently long schedule to ramp up, as for short schedules, very large batch size results in a small number of gradient update steps.
• Figure 4: Top: SigLIP with pre-trained encoders ramps up quickly. However, only disabling weight decay on the pretrained encoder weights leads to stable behavior and good ImageNet 0 -shot transfer results. Bottom: ImageNet 10shot transfer results, where decaying the pre-trained weights leads to deterioration of the pre-trained model visual representation quality. Disabling weight decay flattens the curve.
• Figure 5: The effect of Adam and AdaFactor's$\beta_{2}$. As we increase batch-size, we observe more frequent training instability. This instability seen in the loss curves (top) is caused by spikes in gradient norm (middle) leading to large parameter updates (bottom). Decreasing the $\beta_{2}$ momentum stabilizes training. Occasional gradient spikes still happen (see step at 2B), but do not destabilize the training process.