FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resources

TL;DR

FastCLIP tackles the resource bottleneck of CLIP training by introducing a distributed framework based on Finite-sum Coupled Compositional Optimization (FCCO) that optimizes global contrastive losses with limited compute. It advances three optimization components—inner learning-rate scheduling, temperature update rules, and optimizer choice—through systematic comparisons and a gradient-reduction strategy that lowers communication overhead. Across multiple data scales (2.7M to 315M image-text pairs) and compute scales (1–8 nodes), FastCLIP consistently outperforms OpenCLIP while delivering substantial speedups, and achieves competitive ImageNet performance in extreme-scale settings. The work enables faster, more accessible CLIP development on modest hardware and provides actionable guidance for practitioners deploying resource-limited CLIP training.

Abstract

Existing studies of training state-of-the-art Contrastive Language-Image Pretraining (CLIP) models on large-scale data involve hundreds of or even thousands of GPUs due to the requirement of a large batch size. However, such a large amount of resources is not accessible to most people. While advanced compositional optimization techniques for optimizing global contrastive losses have been demonstrated effective for removing the requirement of large batch size, their performance on large-scale data remains underexplored and not optimized. To bridge the gap, this paper explores several aspects of CLIP training with limited resources (e.g., up to tens of GPUs). First, we introduce FastCLIP, a general CLIP training framework built on advanced compositional optimization techniques while designed and optimized for the distributed setting. Our framework is equipped with an efficient gradient reduction strategy to reduce communication overhead. Second, to further boost training efficiency, we investigate three components of the framework from an optimization perspective: the schedule of the inner learning rate, the update rules of the temperature parameter and the model parameters, respectively. Experiments on different strategies for each component shed light on how to conduct CLIP training more efficiently. Finally, we benchmark the performance of FastCLIP and the state-of-the-art training baseline (OpenCLIP) on different compute scales up to 32 GPUs on 8 nodes, and three data scales ranging from 2.7 million, 9.1 million to 315 million image-text pairs to demonstrate the significant improvement of FastCLIP in the resource-limited setting. We release the code of FastCLIP at https://github.com/Optimization-AI/fast_clip .

Figures (9)

• Figure 1: Zero-shot accuracy curves on ImageNet & its variants of OpenCLIP and FastCLIP-v3 trained on 1 to 8 node(s) with 4 GPUs per node on medium and large-scale settings (c.f. Section \ref{['sec:components']}).
• Figure 2: Comparison between OpenCLIP and FastCLIP-v3. The numbers in between represent the improvement of FastCLIP-v3 over OpenCLIP.
• Figure 5: Datacomp average performance of FastCLIP-v3 with $\gamma$ decay epochs 16 (145 million samples seen) and different $\gamma_{\mathrm{min}}$ in the large-scale setting. Batch size denotes global batch size. The vertical dashed lines divided the plot into three parts (c.f. Choice of $\gamma_{\mathrm{min}}$ in the xlarge-scale Setting in Appendix \ref{['sec:app_exp']}).
• Figure 6: ImageNet-1k top 1 accuracy plots. 'bsz' denotes batch size and 'acc' denotes accuracy.
• Figure 7: ImageNet-1k Top 1 accuracy (left) and Datacomp Average performance (right) of FastCLIP-v3 with different $\varepsilon$ in the xlarge-scale setting.