Classification Done Right for Vision-Language Pre-Training

TL;DR

This work introduces SuperClass, a super simple classification method for vision-language pre-training on image-text data that directly utilizes tokenized raw text as supervised classification labels, without the need for additional text filtering or selection.

Abstract

We introduce SuperClass, a super simple classification method for vision-language pre-training on image-text data. Unlike its contrastive counterpart CLIP who contrast with a text encoder, SuperClass directly utilizes tokenized raw text as supervised classification labels, without the need for additional text filtering or selection. Due to the absence of the text encoding as contrastive target, SuperClass does not require a text encoder and does not need to maintain a large batch size as CLIP does. SuperClass demonstrated superior performance on various downstream tasks, including classic computer vision benchmarks and vision language downstream tasks. We further explored the scaling behavior of SuperClass on model size, training length, or data size, and reported encouraging results and comparisons to CLIP. https://github.com/x-cls/superclass

Figures (3)

• Figure 1: (left) CLIP uses two separate Transformer encoders to extract vector representations from image-text pairs. The text encoder operates on a subword-level tokenizer. (right) The proposed bag of subwords classification both only use the single Transformer encoder.
• Figure 2: Performance of frozen visual representations on different classification datasets. 10-shot linear evaluation accuracy on the pre-logit representation. *results from the paper.
• Figure 2: Zero-shot classification accuracy and linear probing accuracy on ImageNet-1k dataset (left two columns); Performance of VQAv2 and T-VQA with LLaVA training recipe (right two columns). Top row: We compare the performance of vision backbones—ViT-S/16, B/16, and L/16—pretrained via classification and contrastive methods with the same batch size of 16k and 512 million seen samples, focusing on their size and computational cost. SuperClass demonstrates better scaling on zero-shot classification and VQAv2, T-VQA tasks. Bottom row: Comparing SuperClass and CLIP, performance increases with more training examples, mirroring the effects of model scaling. All methods are trained the same batch size of 16k and ViT-L/16 as backbone.