Scaling Pre-training to One Hundred Billion Data for Vision Language Models

TL;DR

This work investigates scaling vision-language pretraining to 100 billion web-sourced image-text pairs and analyzes its impact beyond traditional benchmarks. Using the WebLI-100B dataset and SigLIP contrastive pretraining across multiple ViT backbones, the study reveals that Western benchmarks show limited gains at this scale, while cultural diversity and multilingual performance improve substantially, aided by long-tail concept coverage and language rebalancing. However, aggressive data filtering can inadvertently reduce cultural representation and fairness, underscoring the data-quality vs. diversity trade-off. Overall, the 100B-scale regime is crucial for building inclusive multimodal systems, even as it poses challenges for bias mitigation and transferability to generative tasks.

Abstract

We provide an empirical investigation of the potential of pre-training vision-language models on an unprecedented scale: 100 billion examples. We find that model performance tends to saturate at this scale on many common Western-centric classification and retrieval benchmarks, such as COCO Captions. Nevertheless, tasks of cultural diversity achieve more substantial gains from the 100-billion scale web data, thanks to its coverage of long-tail concepts. Furthermore, we analyze the model's multilinguality and show gains in low-resource languages as well. In addition, we observe that reducing the size of the pretraining dataset via quality filters like using CLIP, typically used to enhance performance, may inadvertently reduce the cultural diversity represented even in large-scale datasets. Our results highlight that while traditional benchmarks may not benefit significantly from scaling noisy, raw web data to 100 billion examples, this data scale is vital for building truly inclusive multimodal systems.

Figures (6)

• Figure 1: left: Scaling the data from 10 billion to 100 billion examples enhances cultural diversity and multilingual capabilities more prominently than other metrics. The numbers represent the improved accuracy (in absolute terms) when data scale is increased, averaged across all tasks. See details in Section \ref{['sect:results']}. right: Illustrative examples of the impact of data scale. The leftmost two are Western-centric metrics, which do not benefit much by scaling the data to 100 billion, while the rightmost two are illustrative of cultural diversity and multilinguality. The language Telugu, for example, makes up $<0.04\%$ of the web and benefits a lot from the 100 billion data scale.
• Figure 2: Association bias between gender and occupation, evaluated in scaled models and data.
• Figure 3: Scaling up to 100B examples leads to more notable improvements in low-resource languages. $\Delta$ denotes the improved accuracy when scaling from 10B examples to 100B.
• Figure 4: Quality filtering can hinder cultural diversity (middle) and fairness (right), even when it benefits Western-centric (left) tasks. This observation holds for both the widely-used CLIP filter and a classifier filter trained on web data.
• Figure 9: Rebalancing low-resource languages leads to significant improvements on corresponding benchmarks and slight improvements on aggregated multilingual/cultural diversity tasks. However, other tasks may experience decreased performance due to less Western-centric examples.