Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Test-time Loss Landscape Adaptation for Zero-Shot Generalization in Vision-Language Models

Aodi Li, Liansheng Zhuang, Xiao Long, Minghong Yao, Shafei Wang

TL;DR

This work tackles zero-shot generalization under distribution shift for vision–language models by rethinking test-time adaptation through loss landscapes. It introduces Test-time Loss Landscape Adaptation (TLLA), a two-stage approach that first uses Sharpness-Aware Prompt Tuning (SAPT) to locate a training flat minimum, then applies Sharpness-based Test Sample Selection (STSS) to pick augmented test views whose landscapes align with that minimum, avoiding test-time backpropagation. Empirical results on domain generalization and cross-dataset benchmarks show state-of-the-art performance with substantial reductions in computation compared to prompt-tuning baselines, e.g., gains of $5.32$–$6.98$ percentage points on ImageNet variants and improvements across multiple datasets. Theoretical analysis provides a generalization bound framed by the loss-landscape distance between training and test distributions, supporting the intuition that alignment of flat minima yields more reliable predictions, especially for data closer to the training distribution.

Abstract

Test-time adaptation of pre-trained vision-language models has emerged as a technique for tackling distribution shifts during the test time. Although existing methods, especially those based on Test-time Prompt Tuning (TPT), have shown promising results, their high computational cost associated with parameter optimization presents challenges for scalability and practical application. This paper unveils the unnecessary nature of backpropagation in existing methods from a loss landscape perspective. Building on this insight, this paper proposes a simple yet effective framework called Test-time Loss Landscape Adaptation (TLLA). TLLA leverages the relative position between the training minimum and test loss landscapes to guide the adaptation process, avoiding the update of model parameters at test time. Specifically, it mainly consists of two main stages: In the prompt tuning stage, a Sharpness-Aware Prompt Tuning (SAPT) method is introduced to identify the training flat minimum, setting the foundation for the subsequent test-time adaptation; In the test stage, a Sharpness-based Test Sample Selection (STSS) approach is utilized to ensure the alignment of flat minima within the training loss landscape and each augmented test sample's loss landscape. Extensive experiments on both domain generalization and cross-dataset benchmarks demonstrate that TLLA achieves state-of-the-art performances while significantly reducing computational overhead. Notably, TLLA surpasses TPT by an average of 5.32\% and 6.98\% on four ImageNet variant datasets when employing ResNet50 and ViT-B/16 image encoders, respectively. The code will be available soon.

Test-time Loss Landscape Adaptation for Zero-Shot Generalization in Vision-Language Models

TL;DR

This work tackles zero-shot generalization under distribution shift for vision–language models by rethinking test-time adaptation through loss landscapes. It introduces Test-time Loss Landscape Adaptation (TLLA), a two-stage approach that first uses Sharpness-Aware Prompt Tuning (SAPT) to locate a training flat minimum, then applies Sharpness-based Test Sample Selection (STSS) to pick augmented test views whose landscapes align with that minimum, avoiding test-time backpropagation. Empirical results on domain generalization and cross-dataset benchmarks show state-of-the-art performance with substantial reductions in computation compared to prompt-tuning baselines, e.g., gains of – percentage points on ImageNet variants and improvements across multiple datasets. Theoretical analysis provides a generalization bound framed by the loss-landscape distance between training and test distributions, supporting the intuition that alignment of flat minima yields more reliable predictions, especially for data closer to the training distribution.

Abstract

Test-time adaptation of pre-trained vision-language models has emerged as a technique for tackling distribution shifts during the test time. Although existing methods, especially those based on Test-time Prompt Tuning (TPT), have shown promising results, their high computational cost associated with parameter optimization presents challenges for scalability and practical application. This paper unveils the unnecessary nature of backpropagation in existing methods from a loss landscape perspective. Building on this insight, this paper proposes a simple yet effective framework called Test-time Loss Landscape Adaptation (TLLA). TLLA leverages the relative position between the training minimum and test loss landscapes to guide the adaptation process, avoiding the update of model parameters at test time. Specifically, it mainly consists of two main stages: In the prompt tuning stage, a Sharpness-Aware Prompt Tuning (SAPT) method is introduced to identify the training flat minimum, setting the foundation for the subsequent test-time adaptation; In the test stage, a Sharpness-based Test Sample Selection (STSS) approach is utilized to ensure the alignment of flat minima within the training loss landscape and each augmented test sample's loss landscape. Extensive experiments on both domain generalization and cross-dataset benchmarks demonstrate that TLLA achieves state-of-the-art performances while significantly reducing computational overhead. Notably, TLLA surpasses TPT by an average of 5.32\% and 6.98\% on four ImageNet variant datasets when employing ResNet50 and ViT-B/16 image encoders, respectively. The code will be available soon.

Paper Structure

This paper contains 16 sections, 4 theorems, 28 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminaries
  5. Test-time Loss Landscape Adaptation
  6. Sharpness-Aware Prompt Tuning
  7. Sharpness-based Test Sample Selection
  8. Theoretical Analysis
  9. Experiments
  10. Experimental Setup
  11. Main Results
  12. Ablation Study
  13. Visualization of Loss Landscapes
  14. Conclusion
  15. Proof to Theorem 3.1
  16. ...and 1 more sections

Key Result

Theorem 1

Consider real-valued function class $\mathcal{F}=\{f_{\boldsymbol{\theta}}(\cdot)\}$, and a bounded loss function $\ell: \mathbb{R} \times \mathbb{R} \to [0,M]$. Define $\ell^{\rho}$ as: Assume that $\ell^{\rho}$ is $\mu$-Lipschitz with respect to $f$ : Denote the training and test distribution as $\mathcal{S}$ and $\mathcal{T}$, respectively. Then, with probability at least $1-\delta$, the foll

Figures (5)

  • Figure 1: Comparison of conventional TTA methods and our Test-time Loss Landscape Adaptation (TLLA). (a) Traditional TTA methods treat the test landscape as static, aiming to optimize model parameters to achieve the test flat minimum (★), using the training minimum (★) as an initialization. (b) Our TLLA keeps model parameters unchanged during testing. Instead, it adjusts the test landscape to a position where the training minimum (★) is already very close to the minimum (★) of the adapted test landscape.
  • Figure 2: Overview of the proposed Test-time Loss Landscape Adaptation (TLLA). (a) Sharpness-aware Prompt Tuning: It optimizes the model parameters to reduce the sharpness of the loss landscape, enabling stable and effective adaptations during test time without the need for large-scale retraining. (b) Sharpness-based Test Sample Selection: It introduces a selection mechanism to identify augmented test samples that ensure the training flat minimum aligns with those in their loss landscapes, enabling more confident and accurate predictions, particularly for out-of-distribution data.
  • Figure 3: Ablation study on main components of the proposed TLLA.
  • Figure 4: The influence of the key hyperparameter $\rho'$ on the test accuracy.
  • Figure 5: 2D Visualization of loss landscapes associated with different augmented test samples.

Theorems & Definitions (10)

  • Theorem 1: Generalization Bound
  • Definition 2: $\beta$-tightness
  • Definition 3: $\gamma$-separability
  • Theorem 4
  • Definition 5: $\mathcal{F} \Delta \mathcal{F}$ distance HOH
  • Lemma 1: SWAD
  • proof
  • Lemma 2: Theorem 6.31 itbook
  • proof : Proof to Theorem 3.1
  • proof : Proof to Theorem 3.4