Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conditional Prompt Learning for Vision-Language Models

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei Liu

TL;DR

Prompt learning for vision-language models often overfits base classes when adapting to new categories. CoCoOp introduces an instance-conditioned prompting scheme via a lightweight Meta-Net, enabling per-image prompts to accompany shared context vectors. Across 11 datasets, CoCoOp delivers strong improvements on unseen classes and better domain generalization compared to CoOp and CLIP, including cross-dataset transfer. The work demonstrates a simple, parameter-efficient approach to generalizable prompting for foundation-model adaptation with practical impact for open-set recognition.

Abstract

With the rise of powerful pre-trained vision-language models like CLIP, it becomes essential to investigate ways to adapt these models to downstream datasets. A recently proposed method named Context Optimization (CoOp) introduces the concept of prompt learning -- a recent trend in NLP -- to the vision domain for adapting pre-trained vision-language models. Specifically, CoOp turns context words in a prompt into a set of learnable vectors and, with only a few labeled images for learning, can achieve huge improvements over intensively-tuned manual prompts. In our study we identify a critical problem of CoOp: the learned context is not generalizable to wider unseen classes within the same dataset, suggesting that CoOp overfits base classes observed during training. To address the problem, we propose Conditional Context Optimization (CoCoOp), which extends CoOp by further learning a lightweight neural network to generate for each image an input-conditional token (vector). Compared to CoOp's static prompts, our dynamic prompts adapt to each instance and are thus less sensitive to class shift. Extensive experiments show that CoCoOp generalizes much better than CoOp to unseen classes, even showing promising transferability beyond a single dataset; and yields stronger domain generalization performance as well. Code is available at https://github.com/KaiyangZhou/CoOp.

Conditional Prompt Learning for Vision-Language Models

TL;DR

Prompt learning for vision-language models often overfits base classes when adapting to new categories. CoCoOp introduces an instance-conditioned prompting scheme via a lightweight Meta-Net, enabling per-image prompts to accompany shared context vectors. Across 11 datasets, CoCoOp delivers strong improvements on unseen classes and better domain generalization compared to CoOp and CLIP, including cross-dataset transfer. The work demonstrates a simple, parameter-efficient approach to generalizable prompting for foundation-model adaptation with practical impact for open-set recognition.

Abstract

With the rise of powerful pre-trained vision-language models like CLIP, it becomes essential to investigate ways to adapt these models to downstream datasets. A recently proposed method named Context Optimization (CoOp) introduces the concept of prompt learning -- a recent trend in NLP -- to the vision domain for adapting pre-trained vision-language models. Specifically, CoOp turns context words in a prompt into a set of learnable vectors and, with only a few labeled images for learning, can achieve huge improvements over intensively-tuned manual prompts. In our study we identify a critical problem of CoOp: the learned context is not generalizable to wider unseen classes within the same dataset, suggesting that CoOp overfits base classes observed during training. To address the problem, we propose Conditional Context Optimization (CoCoOp), which extends CoOp by further learning a lightweight neural network to generate for each image an input-conditional token (vector). Compared to CoOp's static prompts, our dynamic prompts adapt to each instance and are thus less sensitive to class shift. Extensive experiments show that CoCoOp generalizes much better than CoOp to unseen classes, even showing promising transferability beyond a single dataset; and yields stronger domain generalization performance as well. Code is available at https://github.com/KaiyangZhou/CoOp.
Paper Structure (32 sections, 3 equations, 4 figures, 6 tables)

This paper contains 32 sections, 3 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Models
  4. Prompt Learning
  5. Zero-Shot Learning (ZSL)
  6. Methodology
  7. Reviews of CLIP and CoOp
  8. Contrastive Language-Image Pre-training
  9. Context Optimization (CoOp)
  10. CoCoOp: Conditional Context Optimization
  11. Experiments
  12. Datasets
  13. Baselines
  14. Training Details
  15. Generalization From Base to New Classes
  16. ...and 17 more sections

Figures (4)

  • Figure 1: Motivation of our research: to learn generalizable prompts. The images are randomly selected from SUN397 xiao2010sun, which is a widely-used scene recognition dataset.
  • Figure 2: Our approach, Conditional Context Optimization (CoCoOp), consists of two learnable components: a set of context vectors and a lightweight neural network (Meta-Net) that generates for each image an input-conditional token.
  • Figure 3: Comprehensive comparisons of CoCoOp and CoOp in the base-to-new generalization setting. (a) CoCoOp is able to gain consistent improvements over CoOp in unseen classes on all datasets. (b) CoCoOp's declines in base accuracy are mostly under 3%, which are far outweighed by the gains in generalization.
  • Figure 4: Ablation studies.