Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bayesian Principles Improve Prompt Learning In Vision-Language Models

Mingyu Kim, Jongwoo Ko, Mijung Park

TL;DR

This work tackles overfitting in prompt learning for vision–language models by introducing a Bayesian objective that balances task adaptation with retention of pre-trained knowledge. It combines a Gaussian-process-like prior over class logits, a one-vs-each softmax likelihood, and Polya–Gamma data augmentation to yield tractable, conjugate updates, with the mean of the prior anchored to pre-trained logits. Empirical results across ten unseen-prompt datasets and ViT-based setups show improved unseen/generalization and cross-dataset transfer, with robust performance across a moderate regularization strength. The method remains simple and compatible with existing prompt-learning approaches, offering a practical route to better generalization without extra network parameters. These findings highlight the value of integrating distributional logit learning and knowledge distillation in a unified Bayesian framework for VLMs.

Abstract

Prompt learning is a popular fine-tuning method for vision-language models due to its efficiency. It requires a small number of additional learnable parameters while significantly enhancing performance on target tasks. However, most existing methods suffer from overfitting to fine-tuning data, yielding poor generalizability. To address this, we propose a new training objective function based on a Bayesian learning principle to balance adaptability and generalizability. We derive a prior over the logits, where the mean function is parameterized by the pre-trained model, while the posterior corresponds to the fine-tuned model. This objective establishes a balance by allowing the fine-tuned model to adapt to downstream tasks while remaining close to the pre-trained model.

Bayesian Principles Improve Prompt Learning In Vision-Language Models

TL;DR

This work tackles overfitting in prompt learning for vision–language models by introducing a Bayesian objective that balances task adaptation with retention of pre-trained knowledge. It combines a Gaussian-process-like prior over class logits, a one-vs-each softmax likelihood, and Polya–Gamma data augmentation to yield tractable, conjugate updates, with the mean of the prior anchored to pre-trained logits. Empirical results across ten unseen-prompt datasets and ViT-based setups show improved unseen/generalization and cross-dataset transfer, with robust performance across a moderate regularization strength. The method remains simple and compatible with existing prompt-learning approaches, offering a practical route to better generalization without extra network parameters. These findings highlight the value of integrating distributional logit learning and knowledge distillation in a unified Bayesian framework for VLMs.

Abstract

Prompt learning is a popular fine-tuning method for vision-language models due to its efficiency. It requires a small number of additional learnable parameters while significantly enhancing performance on target tasks. However, most existing methods suffer from overfitting to fine-tuning data, yielding poor generalizability. To address this, we propose a new training objective function based on a Bayesian learning principle to balance adaptability and generalizability. We derive a prior over the logits, where the mean function is parameterized by the pre-trained model, while the posterior corresponds to the fine-tuned model. This objective establishes a balance by allowing the fine-tuned model to adapt to downstream tasks while remaining close to the pre-trained model.

Paper Structure

This paper contains 49 sections, 1 theorem, 64 equations, 3 figures, 9 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Contrastive Language-Image Pretraining
  4. Prompt Learning
  5. Pólya-Gamma Data Augmentation
  6. Method
  7. Prior over the Logit Function
  8. Likelihood with the Logit Function
  9. Posterior
  10. Parameter Estimation
  11. Empirical Validation with Synthetic Examples.
  12. Empirical Validation with ViT.
  13. Related Work
  14. Vision-Language Models (VLMs).
  15. Prompt Learning in VLMs.
  16. ...and 34 more sections

Key Result

Lemma 1

The marginal distribution of $\textbf{a}$ and the conditional distribution of $\textbf{b}$ given $\textbf{a}$ are given by where $\boldsymbol{\Sigma} = (\textbf{S} + \textbf{W}^{T} \boldsymbol{\Phi} \textbf{W})^{-1}$ and $\boldsymbol{\mu} = \boldsymbol{\Sigma} (\textbf{W}^{T} \boldsymbol{\Phi} \textbf{a} + \textbf{S} \textbf{m})$.

Figures (3)

  • Figure 1: 1D synthetic classification comparison between SoftMax, SoftMax_OVE and SoftMAX_OVE_PG based on the gradient-descent. [Column 1] illustrates the arrangement of training samples (top) and the underlying ground-truth distributions (bottom). [Columns 2-5] show the optimized functions from SoftMax, SoftMax_OVE, and SoftMax_OVE_PG with the precision $\alpha$ values of 1 and 100, respectively. In these columns, the top row displays the class distributions at every iteration, while the bottom row shows the class distribution at the last iteration.
  • Figure 2: Effect of OVE-PG on ViT using the EMNIST and MNIST datasets. Our proposed OVE-PG (green line) achieves better generalization compared to Softmax (blue line) and OVE-Softmax (orange line) on unseen datasets (i.e., EMNIST), while slightly compromising performance on seen datasets (i.e., MNIST), resulting in higher overall performance.
  • Figure 3: Sensitivity analysis of the hyperparameter $\beta$ for controlling KLD regularization in OVE-PG (blue solid line), compared to the softmax baseline for CoOp (red dotted line). The results demonstrate robustness within a moderate range of $\beta$ values ($0.2$–$0.5$) across all datasets.

Theorems & Definitions (2)

  • Definition 1
  • Lemma 1