CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment

TL;DR

<3-5 sentences> CLIP's zero-shot vision-language capabilities are shown to transfer to vision-language understanding tasks, enabling zero-shot VQA and zero-shot cross-modality transfer to visual entailment. The paper introduces TAP-C, a two-step prompt generation and answer-filtering approach that bridges natural language questions to CLIP-style prompts, achieving strong zero-shot VQA performance. To enable few-shot VL learning, a parameter-efficient BiNor fine-tuning strategy updates only bias and normalization parameters, yielding robust improvements on VQA with limited data. A complementary zero-shot cross-modality transfer study demonstrates language-to-vision and vision-to-language transfers, underscoring the alignment of CLIP's modalities and its potential for low-resource VL reasoning. Overall, the work establishes CLIP as a viable vision-language few-shot learner and provides practical methods to exploit its capabilities without additional pre-training.</3-5 sentences>

Abstract

CLIP has shown a remarkable zero-shot capability on a wide range of vision tasks. Previously, CLIP is only regarded as a powerful visual encoder. However, after being pre-trained by language supervision from a large amount of image-caption pairs, CLIP itself should also have acquired some few-shot abilities for vision-language tasks. In this work, we empirically show that CLIP can be a strong vision-language few-shot learner by leveraging the power of language. We first evaluate CLIP's zero-shot performance on a typical visual question answering task and demonstrate a zero-shot cross-modality transfer capability of CLIP on the visual entailment task. Then we propose a parameter-efficient fine-tuning strategy to boost the few-shot performance on the vqa task. We achieve competitive zero/few-shot results on the visual question answering and visual entailment tasks without introducing any additional pre-training procedure.

Figures (4)

• Figure 1: Examples of the two vision-language understanding tasks. For VQA, language prompts are used. For visual entailment, caption and hypothesis, i.e., text-text pairs, are used in training, while image and hypothesis, i.e., image-text pairs, are used at inference.
• Figure 2: CLIP consists of a visual encoder $\mathbb V$, a text encoder $\mathbb T$, and a dot product between their outputs. It is trained to align images and texts with a contrastive loss. The dot product is used as an alignment score.
• Figure 3: The overall framework of the proposed TAP-C method for zero-shot VQA. TAP-C first generates a masked template from the question by demonstrating examples to T5 and then filters out impossible answers according to the language model. Infilling the masked template with selected answers results in prompts, which could be paired with images to calculate image-text alignment scores by the CLIP. The dashed line denotes the process of prompts generation (\ref{['sec:t_vqa']}), and the solid line denotes prompting CLIP to conduct zero-shot VQA (\ref{['sec:tap-c']}).
• Figure 4: Zero-shot cross-modality transfer on visual entailment task. The red line denotes the text-only training process, and the blue line denotes the image-text, i.e., cross-modality, evaluation process. The MLP is shared between training and evaluation, while both encoders in CLIP models are not updated.