MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training

TL;DR

This work tackles the challenge of deploying CLIP-like image-text models on mobile devices by designing MobileCLIP, a family of efficient architectures, and introducing multi-modal reinforced training (DataCompDR) to boost learning efficiency without extra training-time cost. DataCompDR enriches the training data with synthetic captions and embeddings from a teacher ensemble, stored in a reinforced dataset to enable rapid experimentation across architectures. The authors show that MobileCLIP achieves state-of-the-art latency-accuracy tradeoffs on zero-shot classification and retrieval across multiple benchmarks, with substantial gains in learning efficiency (up to $10\times$–$1000\times$) and robustness (ARO benchmark). The combination of Text-RepMixer, MCi image encoders, and reinforced training yields mobile-friendly models that outperform prior compact CLIPs while maintaining competitive accuracy, making on-device vision-language tasks more practical at scale.

Abstract

Contrastive pretraining of image-text foundation models, such as CLIP, demonstrated excellent zero-shot performance and improved robustness on a wide range of downstream tasks. However, these models utilize large transformer-based encoders with significant memory and latency overhead which pose challenges for deployment on mobile devices. In this work, we introduce MobileCLIP -- a new family of efficient image-text models optimized for runtime performance along with a novel and efficient training approach, namely multi-modal reinforced training. The proposed training approach leverages knowledge transfer from an image captioning model and an ensemble of strong CLIP encoders to improve the accuracy of efficient models. Our approach avoids train-time compute overhead by storing the additional knowledge in a reinforced dataset. MobileCLIP sets a new state-of-the-art latency-accuracy tradeoff for zero-shot classification and retrieval tasks on several datasets. Our MobileCLIP-S2 variant is 2.3$\times$ faster while more accurate compared to previous best CLIP model based on ViT-B/16. We further demonstrate the effectiveness of our multi-modal reinforced training by training a CLIP model based on ViT-B/16 image backbone and achieving +2.9% average performance improvement on 38 evaluation benchmarks compared to the previous best. Moreover, we show that the proposed approach achieves 10$\times$-1000$\times$ improved learning efficiency when compared with non-reinforced CLIP training. Code and models are available at https://github.com/apple/ml-mobileclip .

Figures (7)

• Figure 1: MobileCLIP models are fast and accurate. Comparison of publicly available CLIP models with MobileCLIP trained on our DataCompDR dataset. Latency is measured on iPhone12 Pro Max.
• Figure 2: DataCompDR dataset improves all metrics. Zero-shot performance of CLIP models with ViT-B/16 image encoder.
• Figure 3: Illustration of multi-modal dataset reinforcement with one image augmentation and one synthetic caption. In practice, we use multiple image augmentations and synthetic captions.
• Figure 4: Architecture of convolutional and reparameterizable blocks, called Text-RepMixer used in MobileCLIP's text encoder MCt.
• Figure 5: Real vs synthetic captions.