Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Advancing Prompt Learning through an External Layer

Fangming Cui, Xun Yang, Chao Wu, Liang Xiao, Xinmei Tian

TL;DR

This work proposes a paradigm called EnPrompt with a novel External Layer (EnLa), which proposes a textual external layer and learnable visual embeddings for adapting VLMs to downstream tasks and outperforms the existing prompt learning method.

Abstract

Prompt learning represents a promising method for adapting pre-trained vision-language models (VLMs) to various downstream tasks by learning a set of text embeddings. One challenge inherent to these methods is the poor generalization performance due to the invalidity of the learned text embeddings for unseen tasks. A straightforward approach to bridge this gap is to freeze the text embeddings in prompts, which results in a lack of capacity to adapt VLMs for downstream tasks. To address this dilemma, we propose a paradigm called EnPrompt with a novel External Layer (EnLa). Specifically, we propose a textual external layer and learnable visual embeddings for adapting VLMs to downstream tasks. The learnable external layer is built upon valid embeddings of pre-trained CLIP. This design considers the balance of learning capabilities between the two branches. To align the textual and visual features, we propose a novel two-pronged approach: i) we introduce the optimal transport as the discrepancy metric to align the vision and text modalities, and ii) we introduce a novel strengthening feature to enhance the interaction between these two modalities. Four representative experiments (i.e., base-to-novel generalization, few-shot learning, cross-dataset generalization, domain shifts generalization) across 15 datasets demonstrate that our method outperforms the existing prompt learning method.

Advancing Prompt Learning through an External Layer

TL;DR

This work proposes a paradigm called EnPrompt with a novel External Layer (EnLa), which proposes a textual external layer and learnable visual embeddings for adapting VLMs to downstream tasks and outperforms the existing prompt learning method.

Abstract

Prompt learning represents a promising method for adapting pre-trained vision-language models (VLMs) to various downstream tasks by learning a set of text embeddings. One challenge inherent to these methods is the poor generalization performance due to the invalidity of the learned text embeddings for unseen tasks. A straightforward approach to bridge this gap is to freeze the text embeddings in prompts, which results in a lack of capacity to adapt VLMs for downstream tasks. To address this dilemma, we propose a paradigm called EnPrompt with a novel External Layer (EnLa). Specifically, we propose a textual external layer and learnable visual embeddings for adapting VLMs to downstream tasks. The learnable external layer is built upon valid embeddings of pre-trained CLIP. This design considers the balance of learning capabilities between the two branches. To align the textual and visual features, we propose a novel two-pronged approach: i) we introduce the optimal transport as the discrepancy metric to align the vision and text modalities, and ii) we introduce a novel strengthening feature to enhance the interaction between these two modalities. Four representative experiments (i.e., base-to-novel generalization, few-shot learning, cross-dataset generalization, domain shifts generalization) across 15 datasets demonstrate that our method outperforms the existing prompt learning method.
Paper Structure (25 sections, 11 equations, 4 figures, 10 tables)

This paper contains 25 sections, 11 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Preliminaries
  4. EnLa with Frozen Text Embeddings
  5. Alignment with Optimal Transport
  6. Alignment with Strengthening Feature
  7. Experiments
  8. Benchmark Setting
  9. Effect of Components
  10. Few-shot Learning Experiments
  11. Base-to-Novel Generalization Experiments
  12. Cross Dataset Evaluation
  13. Domain Generalization Experiments
  14. Ablative Analysis
  15. Comparison of Different Templates
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Performance comparison on base-to-novel generalization. EnPrompt (Ours) outperforms previous state-of-the-art methods on 11 datasets.
  • Figure 2: Comparison of our method with CoOp and MaPLe. Our method freezes the textual embeddings and learns an External Layer (EnLa) on top of it. Optimal transport is introduced to align visual and textual modalities apart from the conventional cross entropy (CE) loss. The textual and visual encoders are connected through a strengthening feature. This feature and the learnable visual prompt are fused and utilized for deep prompting of the visual encoder.
  • Figure 3: Few-shot Learning Experiments. All methods are trained on the ViT-B/16 CLIP backbone. EnPrompt (Ours) demonstrates consistent improvements over existing methods, specifically for minimal training data such as K=2. On average, EnPrompt provides the highest performance gains for all shots (K = 1,2,4,8,16).
  • Figure 4: Ablation study for embedding length and learning depth of image encoder on Flowers102 and UCF101.