SoC: Semantic Orthogonal Calibration for Test-Time Prompt Tuning

TL;DR

Vision-language models require reliable uncertainty estimates during test-time prompt tuning. The authors show that full orthogonality constraints in O-TPT inflate confidence and harm calibration for semantically related classes, and they introduce Semantic Orthogonal Calibration (SoC), a Huber-based regularizer that enforces smoother prototype separation while preserving semantic proximity. They derive a confidence bound linked to cosine coherence and perform a first-order analysis to explain why SoC yields better calibration than O-TPT. Empirical evaluation across 11 datasets and multiple backbones demonstrates that SoC improves calibration (ECE) with competitive accuracy and robustness to prompts, distribution shifts, and post-hoc calibration methods, highlighting its practical impact for safe VLM adaptation.

Abstract

With the increasing adoption of vision-language models (VLMs) in critical decision-making systems such as healthcare or autonomous driving, the calibration of their uncertainty estimates becomes paramount. Yet, this dimension has been largely underexplored in the VLM test-time prompt-tuning (TPT) literature, which has predominantly focused on improving their discriminative performance. Recent state-of-the-art advocates for enforcing full orthogonality over pairs of text prompt embeddings to enhance separability, and therefore calibration. Nevertheless, as we theoretically show in this work, the inherent gradients from fully orthogonal constraints will strongly push semantically related classes away, ultimately making the model overconfident. Based on our findings, we propose Semantic Orthogonal Calibration (SoC), a Huber-based regularizer that enforces smooth prototype separation while preserving semantic proximity, thereby improving calibration compared to prior orthogonality-based approaches. Across a comprehensive empirical validation, we demonstrate that SoC consistently improves calibration performance, while also maintaining competitive discriminative capabilities.

Figures (10)

• Figure 1: Motivation for SoC. With O-TPT, ambiguity inherent to the class semantics is lost due to the aggressive orthogonality constraint, leading to artificially high confidence, even when predictions are incorrect. Let us take this image as an example, whose correct class is "annual crop land", and whose closest semantic class across all categories is "permanent crop land". The zero-shot CLIP prediction incorrectly classifies the image, but its prediction remains uncertain, as the softmax for those two closely related categories remains close. In contrast, due to its orthogonality constraints, O-TPT sharifdeen2025tpt pushes the text class prototypes apart, making the model become more confident, even if the prediction is wrong. Our proposed SoC addresses this issue with a smoother orthogonality enforcement.
• Figure 2: ECE per class pair as a function of the zero-shot cosine similarity. We compute the ECE for the wrong predictions across each class pair (i.e., the model predicted class $i$ when the label was class $j$) and analyze the relation with the zero-shot similarity between both classes on EuroSAT. For classes with high initial semantic similarity, O-TPT is overconfident, caused by the underlying drawbacks of enforcing orthogonality across all pairs. Circle size indicates the number of samples in each $(i, j)$ pair.
• Figure 3: Reliability diagrams of O-TPT vs SoC. Plots showing the calibration error across the Flowers102, EuroSAT and FGVC Aircraft datasets for O-TPT (top row) and SoC (bottom row).
• Figure 4: ECE and accuracy for one and two gradient steps. Standard one-step and two-step gradient updates for C-TPT, O-TPT, and SoC averaged across 11 datasets. Hatched bars indicate the result when applying two gradient updates over text prompts.
• Figure 5: Calibration sensitivity of O-TPT vs. SoC for various initial text prompts. ECE on the DTD (left) and Aircraft (right) datasets for 18 different prompts from CLIP radford2021clip.

Theorems & Definitions (2)

• Proposition 1: Confidence floor via cosine coherence
• Corollary 1: Confidence increases under full orthogonality