D-TPT: Dimensional Entropy Maximization for Calibrating Test-Time Prompt Tuning in Vision-Language Models

TL;DR

The paper investigates calibration challenges in test-time prompt tuning for vision-language models, identifying a modality gap driven by a single dominant feature dimension in text and image embeddings. It introduces D-TPT, a dimensional entropy maximization approach that regularizes the intra-text feature distribution via a new regularization term L_DEM, integrated with the TPT objective. Empirical results across ImageNet variants, fine-grained datasets, and multiple CLIP backbones show consistent calibration improvements (ECE, AECE, MCE, AURC) and competitive accuracy, with a geometric interpretation on a hypersphere explaining the reductions in modality gap. The work offers a simple, plug-in solution to enhance reliability of VLMs under domain shifts, with limitations acknowledged and directions for theory-backed future work.

Abstract

Test-time adaptation paradigm provides flexibility towards domain shifts by performing immediate adaptation on unlabeled target data from the source model. Vision-Language Models (VLMs) leverage their generalization capabilities for diverse downstream tasks, and test-time prompt tuning has emerged as a prominent solution for adapting VLMs. In this work, we explore contrastive VLMs and identify the modality gap caused by a single dominant feature dimension across modalities. We observe that the dominant dimensions in both text and image modalities exhibit high predictive sensitivity, and that constraining their influence can improve calibration error. Building on this insight, we propose dimensional entropy maximization that regularizes the distribution of textual features toward uniformity to mitigate the dependency of dominant dimensions. Our method alleviates the degradation of calibration performance in test-time prompt tuning, offering a simple yet effective solution to enhance the reliability of VLMs in real-world deployment scenarios.

Figures (8)

• Figure 1: Correlation between text diversity and calibration.
• Figure 2: Comparison of feature values across dimensions for text and image features. In CLIP, the modality gap causes features from each modality to be positioned in different spaces, a phenomenon that manifests as the dominant influence of a few dimensions.
• Figure 3: Analysis of the impact of dominant dimensions. (a) We present the top-10 values and their indices for sensitivity, which is the change in the prediction distribution when masking the values of each dimension. For both modalities, the dominant dimensions TDD and IDD significantly influence predictions. (b) Accuracy and ECE are reported with TDD and IDD replaced by their class-wise mean values. For zero-shot CLIP (left), replacing TDD leads to average improvements in ECE across datasets. For TPT (right), replacing TDD also yields a consistent reduction in mean ECE.
• Figure 4: Conceptual illustration of D-TPT. Compared to inter-feature diversity based methods, we achieve improved calibration capability by regularization the distribution of intra-text features.
• Figure 5: Geometric interpretation of prompt tuning methods on the hypersphere. On the hypersphere, the tangent direction represents the geodesic from a text feature toward its corresponding image feature, the normal direction corresponds to the radial axis from the center of the hypersphere to the feature, and the binormal direction is defined as orthogonal to both tangent and normal directions. In this context, TPT aligns text features along the tangent direction of the hypersphere, which expands the logit range and leads to overconfidence. Meanwhile, C-TPT encourages outward displacement from the text centroid, thereby improving isotropic diversity. In contrast, proposed D-TPT maximizes entropy across feature dimensions, amplifying binormal components and reducing the modality gap.