Training-Free Test-Time Adaptation with Brownian Distance Covariance in Vision-Language Models

TL;DR

TaTa targets domain shift in vision-language models by offering a training-free test-time adaptation method that avoids backpropagation or access to source data. It introduces Brownian Distance Covariance ($\mathrm{dCov}^2$) to capture joint, nonlinear dependencies for vision–vision inference, and employs attribute-informed prompting for vision–language inference, augmented by dynamic multimodal clustering with pseudo-label refinement and a soft-voting mechanism. The approach maintains a dynamic dictionary of class prototypes, updates centroids with correctly classified samples, and fuses multimodal predictions to improve stability. Empirical results across diverse benchmarks show TaTa achieves strong domain and cross-dataset generalization with substantially lower computational cost than existing TTA methods.

Abstract

Vision-language models suffer performance degradation under domain shift, limiting real-world applicability. Existing test-time adaptation methods are computationally intensive, rely on back-propagation, and often focus on single modalities. To address these issues, we propose Training-free Test-Time Adaptation with Brownian Distance Covariance (TaTa). TaTa leverages Brownian Distance Covariance-a powerful statistical measure that captures both linear and nonlinear dependencies via pairwise distances-to dynamically adapt VLMs to new domains without training or back-propagation. This not only improves efficiency but also enhances stability by avoiding disruptive weight updates. TaTa further integrates attribute-enhanced prompting to improve vision-language inference with descriptive visual cues. Combined with dynamic clustering and pseudo-label refinement, it effectively recalibrates the model for novel visual contexts. Experiments across diverse datasets show that TaTa significantly reduces computational cost while achieving state-of-the-art performance in domain and cross-dataset generalization.

Figures (2)

• Figure 1: The process involves using Pseudo-Labeling with Multimodal-Assisted Clustering to categorize nouns into semantic centers and create a discriminative text space. Nouns for each image are matched in this space, and by concatenating image and text features, K-means clustering forms centroids and clusters.
• Figure 2: An overview of our proposed TaTa. The final inference in TaTa combines Vision-Vision and Vision-Language Inference.