Image Captioners Are Scalable Vision Learners Too

TL;DR

The paper tackles whether plain image captioning can train scalable vision encoders as effectively as contrastive pretraining (CLIP) when data and compute are matched. It introduces Cap and CapPa, an encoder-decoder framework that learns from image captions, with CapPa enabling parallel caption token prediction to accelerate training. Across a broad suite of evaluations, captioning-based pretraining matches or exceeds CLIP on vision-language tasks, surpasses it on captioning-related benchmarks, and demonstrates favorable scaling and strong performance on fine-grained and relational tasks (ARO, SugarCrepe). The work also shows that coupling Cap/CapPa with pretrained text decoders (LiT, T5, GPT-2) can further improve zero-shot and multimodal capabilities, while remaining a competitive alternative to contrastive pretraining for multimodal models. Overall, plain image captioning emerges as a powerful, scalable pretraining strategy for vision backbones with strong potential for multimodal applications.

Abstract

Contrastive pretraining on image-text pairs from the web is one of the most popular large-scale pretraining strategies for vision backbones, especially in the context of large multimodal models. At the same time, image captioning on this type of data is commonly considered an inferior pretraining strategy. In this paper, we perform a fair comparison of these two pretraining strategies, carefully matching training data, compute, and model capacity. Using a standard encoder-decoder transformer, we find that captioning alone is surprisingly effective: on classification tasks, captioning produces vision encoders competitive with contrastively pretrained encoders, while surpassing them on vision & language tasks. We further analyze the effect of the model architecture and scale, as well as the pretraining data on the representation quality, and find that captioning exhibits the same or better scaling behavior along these axes. Overall our results show that plain image captioning is a more powerful pretraining strategy than was previously believed.

Figures (11)

• Figure 1: Contrastive models (left) use two separate Transformer encoders to extract vector representations from image-text pairs, which are then matched across a potentially large batch clip. Cap (middle) uses a Transformer encoder-decoder architecture transformer and predicts text tokens autoregressively. During training, all tokens are predicted in parallel by shifting the expected output by one token and applying a causal self-attention mask (teacher forcing). In parallel decoding (right) the Transformer decoder has to predict all tokens at once, conditioned only on the image. CapPa trains a single model switching between autoregressive and parallel decoding. D: model width, M: number of image tokens, N: number of text tokens, V: vocabulary size.
• Figure 2: 10-shot classification accuracy on the frozen pre-logit representation (left two columns); captioning and VQA performance with a decoder (right two columns). Top row: Performance of vision backbones pretrained with captioning (Cap/CapPa) and contrastively (CLIP${}^*$) as a function of the model size/FLOPs (we compare ViT-S/16, M/16, B/16, and L/14). CapPa exhibits favorable scaling behavior on captioning and VQA tasks. Bottom row: Performance of CapPa and CLIP${}^*$ as a function of the number of training examples seen. The behavior is similar as for model scale.
• Figure 3: 10-shot linear evaluation accuracy on the pre-logit representation. CapPa outperforms Cap and achieves overall comparable results with CLIP${}^*$ trained with a batch size of 16k.
• Figure 3: Linear probing makes cap pre-trained image encoders seem worse, but when learning the pooling (MAP probe), the gap is essentially closed.
• Figure 4: Absolute improvement of CapPa over CLIP${}^*$ in various settings. Left: CapPa pairs significantly better with decoders in image-language tasks, especially when the decoder is a pre-trained and frozen language model. Right: CapPa seems to be a noticeably better frozen feature extractor for fine-grained classification tasks (we show L/14 results, see appendix for B/16).