Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PromptSync: Bridging Domain Gaps in Vision-Language Models through Class-Aware Prototype Alignment and Discrimination

Anant Khandelwal

TL;DR

PromptSync tackles domain gaps in vision-language models by introducing class-aware prototype alignment and discrimination during test-time prompting. It updates prompts on both text and visual branches, using offline per-class prototypes derived from a proxy dataset and weighting alignment by mean class probabilities across confident augmented views. The approach combines a prototype discrimination loss with a weighted prototype alignment loss to prevent class collapse and bridge distribution shifts, achieving state-of-the-art or competitive gains across domain generalization, base-to-novel generalization, and cross-dataset transfer, while offering controllable latency via a reusable prototype-discrimination step. The method demonstrates robust zero-shot generalization, with practical impact for deploying V-L models across diverse, imbalanced environments.

Abstract

The potential for zero-shot generalization in vision-language (V-L) models such as CLIP has spurred their widespread adoption in addressing numerous downstream tasks. Previous methods have employed test-time prompt tuning to adapt the model to unseen domains, but they overlooked the issue of imbalanced class distributions. In this study, we explicitly address this problem by employing class-aware prototype alignment weighted by mean class probabilities obtained for the test sample and filtered augmented views. Additionally, we ensure that the class probabilities are as accurate as possible by performing prototype discrimination using contrastive learning. The combination of alignment and discriminative loss serves as a geometric regularizer, preventing the prompt representation from collapsing onto a single class and effectively bridging the distribution gap between the source and test domains. Our method, named PromptSync, synchronizes the prompts for each test sample on both the text and vision branches of the V-L model. In empirical evaluations on the domain generalization benchmark, our method outperforms previous best methods by 2.33% in overall performance, by 1% in base-to-novel generalization, and by 2.84% in cross-dataset transfer tasks.

PromptSync: Bridging Domain Gaps in Vision-Language Models through Class-Aware Prototype Alignment and Discrimination

TL;DR

PromptSync tackles domain gaps in vision-language models by introducing class-aware prototype alignment and discrimination during test-time prompting. It updates prompts on both text and visual branches, using offline per-class prototypes derived from a proxy dataset and weighting alignment by mean class probabilities across confident augmented views. The approach combines a prototype discrimination loss with a weighted prototype alignment loss to prevent class collapse and bridge distribution shifts, achieving state-of-the-art or competitive gains across domain generalization, base-to-novel generalization, and cross-dataset transfer, while offering controllable latency via a reusable prototype-discrimination step. The method demonstrates robust zero-shot generalization, with practical impact for deploying V-L models across diverse, imbalanced environments.

Abstract

The potential for zero-shot generalization in vision-language (V-L) models such as CLIP has spurred their widespread adoption in addressing numerous downstream tasks. Previous methods have employed test-time prompt tuning to adapt the model to unseen domains, but they overlooked the issue of imbalanced class distributions. In this study, we explicitly address this problem by employing class-aware prototype alignment weighted by mean class probabilities obtained for the test sample and filtered augmented views. Additionally, we ensure that the class probabilities are as accurate as possible by performing prototype discrimination using contrastive learning. The combination of alignment and discriminative loss serves as a geometric regularizer, preventing the prompt representation from collapsing onto a single class and effectively bridging the distribution gap between the source and test domains. Our method, named PromptSync, synchronizes the prompts for each test sample on both the text and vision branches of the V-L model. In empirical evaluations on the domain generalization benchmark, our method outperforms previous best methods by 2.33% in overall performance, by 1% in base-to-novel generalization, and by 2.84% in cross-dataset transfer tasks.
Paper Structure (20 sections, 8 equations, 2 figures, 8 tables)

This paper contains 20 sections, 8 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Proposed Method: PromptSync
  5. Class-aware Prototype Generation
  6. Prototype Discriminating Loss
  7. Prototype Alignment Loss
  8. Algorithm Details
  9. Experiments
  10. Domain Generalization
  11. Base to Novel Generalization
  12. Cross-Dataset Transfer
  13. Ablation
  14. Performance and Latency
  15. Sensitivity Comparison
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Architecture of proposed PromptSync method for zero-shot generalization in CLIP. During test time, we updates the learnable prompts using discriminative and alignment of class prototypes. For a single test example, we obtain multiple augmented views and obtain the mean class probabilities after parameter updates with discriminating loss. Mean class probabilities act as weights in the class-prototype alignment with filtered augmented views. Gradient are accumulated over multiple iterations before final update to the learnable prompts.
  • Figure 2: Sensitivity Comparison. (a) Top-1 accuracy improves with number of augmented views (b) Top-1 accuracy improves consistently with number of prompt update steps.