Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning

Zhengqing Gao, Xiang Ao, Xu-Yao Zhang, Cheng-Lin Liu

TL;DR

This work tackles open-set adaptation for vision-language models by addressing the overfitting of learned prompts to base classes. It introduces test-time prompt tuning (TTPT) that dynamically fuses learned prompts with hand-crafted prompts using input-specific Maximum Concept Matching (MCM) weights, yielding input-conditioned prompts for each image. Across 11 diverse datasets, TTPT achieves superior harmonic mean accuracy on base and new classes, outperforming CoOp, CoCoOp, and adapter-based methods, with strong gains especially on ViT backbones. The approach is simple, effective, and complementary to existing OOD detection techniques, and code is released to facilitate adoption and further study.

Abstract

Adapting pre-trained models to open classes is a challenging problem in machine learning. Vision-language models fully explore the knowledge of text modality, demonstrating strong zero-shot recognition performance, which is naturally suited for various open-set problems. More recently, some research focuses on fine-tuning such models to downstream tasks. Prompt tuning methods achieved huge improvements by learning context vectors on few-shot data. However, through the evaluation under open-set adaptation setting with the test data including new classes, we find that there exists a dilemma that learned prompts have worse generalization abilities than hand-crafted prompts. In this paper, we consider combining the advantages of both and come up with a test-time prompt tuning approach, which leverages the maximum concept matching (MCM) scores as dynamic weights to generate an input-conditioned prompt for each image during test. Through extensive experiments on 11 different datasets, we show that our proposed method outperforms all comparison methods on average considering both base and new classes. The code is available at https://github.com/gaozhengqing/TTPT

Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning

TL;DR

This work tackles open-set adaptation for vision-language models by addressing the overfitting of learned prompts to base classes. It introduces test-time prompt tuning (TTPT) that dynamically fuses learned prompts with hand-crafted prompts using input-specific Maximum Concept Matching (MCM) weights, yielding input-conditioned prompts for each image. Across 11 diverse datasets, TTPT achieves superior harmonic mean accuracy on base and new classes, outperforming CoOp, CoCoOp, and adapter-based methods, with strong gains especially on ViT backbones. The approach is simple, effective, and complementary to existing OOD detection techniques, and code is released to facilitate adoption and further study.

Abstract

Adapting pre-trained models to open classes is a challenging problem in machine learning. Vision-language models fully explore the knowledge of text modality, demonstrating strong zero-shot recognition performance, which is naturally suited for various open-set problems. More recently, some research focuses on fine-tuning such models to downstream tasks. Prompt tuning methods achieved huge improvements by learning context vectors on few-shot data. However, through the evaluation under open-set adaptation setting with the test data including new classes, we find that there exists a dilemma that learned prompts have worse generalization abilities than hand-crafted prompts. In this paper, we consider combining the advantages of both and come up with a test-time prompt tuning approach, which leverages the maximum concept matching (MCM) scores as dynamic weights to generate an input-conditioned prompt for each image during test. Through extensive experiments on 11 different datasets, we show that our proposed method outperforms all comparison methods on average considering both base and new classes. The code is available at https://github.com/gaozhengqing/TTPT
Paper Structure (23 sections, 8 equations, 4 figures, 3 tables)

This paper contains 23 sections, 8 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Vision-language models
  4. Parameter-efficient fine-tuning of vision-language models
  5. Out-of-distribution (OOD) detection of vision-language models
  6. Method
  7. Problem Setup
  8. Preliminaries
  9. Proposed Approach
  10. Experiments
  11. Setup
  12. Datasets
  13. Training details
  14. Baselines
  15. Comparison with state-of-the-art methods
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Overview of our proposed method. Given a test input $\mathbf{x}$, we first compute the MCM scores using learned prompt and hand-crafted prompt on base and new classes, respectively. Then, a dynamic weighting strategy is employed to generate the input-dependent prompt that combines the advantages of learned prompt and hand-crafted prompt. It is worth noting that the hand-crafted prompt is dataset-specific, which is "a photo of a [CLASS]." for ImageNet while "a photo of a [CLASS], a type of flower." for Flowers102, for example. The learned prompt has the same dimension and length with the corresponding hand-crafted prompt.
  • Figure 2: Comparisons of our method with CoOp and CoCoOp.
  • Figure 3: Performance of different methods with varying number of shots.
  • Figure 4: Performance of different methods with diverse vision backbones.