Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting

Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, Hanwang Zhang

TL;DR

This work proposes a label-Free prompt distribution learning and bias correction framework, dubbed as **Frolic, which boosts zero-shot performance without the need for labeled data and is not only training-free but also circumvents the necessity for hyper-parameter tuning.

Abstract

Vision-language models, such as CLIP, have shown impressive generalization capacities when using appropriate text descriptions. While optimizing prompts on downstream labeled data has proven effective in improving performance, these methods entail labor costs for annotations and are limited by their quality. Additionally, since CLIP is pre-trained on highly imbalanced Web-scale data, it suffers from inherent label bias that leads to suboptimal performance. To tackle the above challenges, we propose a label-Free prompt distribution learning and bias correction framework, dubbed as **Frolic**, which boosts zero-shot performance without the need for labeled data. Specifically, our Frolic learns distributions over prompt prototypes to capture diverse visual representations and adaptively fuses these with the original CLIP through confidence matching. This fused model is further enhanced by correcting label bias via a label-free logit adjustment. Notably, our method is not only training-free but also circumvents the necessity for hyper-parameter tuning. Extensive experimental results across 16 datasets demonstrate the efficacy of our approach, particularly outperforming the state-of-the-art by an average of $2.6\%$ on 10 datasets with CLIP ViT-B/16 and achieving an average margin of $1.5\%$ on ImageNet and its five distribution shifts with CLIP ViT-B/16. Codes are available in https://github.com/zhuhsingyuu/Frolic.

Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting

TL;DR

This work proposes a label-Free prompt distribution learning and bias correction framework, dubbed as **Frolic, which boosts zero-shot performance without the need for labeled data and is not only training-free but also circumvents the necessity for hyper-parameter tuning.

Abstract

Vision-language models, such as CLIP, have shown impressive generalization capacities when using appropriate text descriptions. While optimizing prompts on downstream labeled data has proven effective in improving performance, these methods entail labor costs for annotations and are limited by their quality. Additionally, since CLIP is pre-trained on highly imbalanced Web-scale data, it suffers from inherent label bias that leads to suboptimal performance. To tackle the above challenges, we propose a label-Free prompt distribution learning and bias correction framework, dubbed as **Frolic**, which boosts zero-shot performance without the need for labeled data. Specifically, our Frolic learns distributions over prompt prototypes to capture diverse visual representations and adaptively fuses these with the original CLIP through confidence matching. This fused model is further enhanced by correcting label bias via a label-free logit adjustment. Notably, our method is not only training-free but also circumvents the necessity for hyper-parameter tuning. Extensive experimental results across 16 datasets demonstrate the efficacy of our approach, particularly outperforming the state-of-the-art by an average of on 10 datasets with CLIP ViT-B/16 and achieving an average margin of on ImageNet and its five distribution shifts with CLIP ViT-B/16. Codes are available in https://github.com/zhuhsingyuu/Frolic.

Paper Structure

This paper contains 20 sections, 2 theorems, 35 equations, 5 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methods
  4. Setup
  5. Label-Free Prompt Distribution Learning
  6. Prediction Fusion via Adaptive Calibration.
  7. Correcting Pre-training Label Bias via Label-Free Logit Adjustment
  8. Experiments
  9. Setup
  10. Main Results
  11. Ablation Studies and Further Analysis
  12. Societal Impact, Limitation and Conclusion
  13. Theoretical Analysis
  14. Proof of Eq. (\ref{['eq:sigma']}): Estimation of Class Covariance from Marginal Second Order Moment
  15. Proof of Eq. (\ref{['eq:est_pi']}): Estimation of the Priors of Gaussian Mixture Models
  16. ...and 5 more sections

Key Result

Proposition 1

(Modified from Theorem 1 in HongHCSKC21). Let $\mathbb{P}_\mathsf{pt}(y|\mathbf{x})$ and $\mathbb{P}_\mathsf{ds}(y|\mathbf{x})$ be the distributions of the pre-train and downstream data, respectively. Let $\beta_y=\mathbb{P}_\mathsf{pt}(y)$ and $\pi_y=\mathbb{P}_\mathsf{ds}(y)$ denote the priors of where $\bm{\beta}=[\beta_1,...,\beta_K]$ and $\bm{\pi}=[\pi_1,...,\pi_K]$.

Figures (5)

  • Figure 1: Illustration of prompt distribution learning and label bias correction on ImageNet using CLIP ViT-B/16. (a) Existing zero-shot models simple_zeroCuPL. (b) Our prompt distribution learning (c) Average probability prediction of original CLIP. (d) Average probability prediction of our $\text{Frolic}$.
  • Figure 2: Comparison of confidence.
  • Figure 3: Relation between gains and confidence differences.
  • Figure 4: Convergence of accuracy and $\ell_1$ error of on ImageNet.
  • Figure : Pipeline of our Frolic

Theorems & Definitions (4)

  • Proposition 1
  • proof
  • Proposition 2
  • proof