Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Confidence-calibrated covariate shift correction for few-shot classification in Vision-Language Models

Behraj Khan, Rizwan Qureshi, Nouman Muhammad Durrani, Tahir Syed

TL;DR

CalShift tackles covariate shift and confidence misalignment in vision-language models under few-shot settings by jointly regularizing the CLIP objective with a Fisher information penalty $I(theta)$ and a Confidence Misalignment Penalty CMP. The final objective is $L_{CalShift} = L_{CLIP} + lambda1 I(theta) + lambda2 CMP$, producing more robust and calibrated predictions without extra data. Empirical results across 19 vision datasets show improvements in accuracy (up to 3.5%) and calibration (ECE reduction up to 5.82%) under distribution shifts, with ablations confirming complementary contributions from FIM and CMP. The work offers a practical, training-free path for reliable deployment of vision-language models in real-world, low-shot scenarios.

Abstract

Since the establishment of vision-language foundation models as the new mainstay in low-shot vision classification tasks, the question of domain generalization arising from insufficient target data is assuming more importance. This scarcity challenge induces sampling bias and amplifies model sensitivity to variations and shifts in data distributions. While fine-tuning on multiple domains could mitigate such domain generalization issues, it is resource-intensive and demands diverse data sources. In this work, we systematically analyze two critical challenges: (1) covariate shift between the pre-training distribution and the underspecified target distribution, and (2) confidence misalignment, where predictions on novel data are overconfident. To address both challenges simultaneously, we introduce \textbf{Confidence-Calibrated Covariate Shift Correction (CalShift)} -- a unified approach that combines a Fisher information penalty to mitigate covariate shift and a Confidence Misalignment Penalty (CMP) to reduce overconfidence in misclassified examples. Experimental evaluations across various vision and covariate shift benchmarks demonstrate that CalShift significantly improves model calibration, achieving up to a 5.82\% reduction in Expected Calibration Error (ECE). Furthermore, CalShift enhances robustness, improving accuracy by 3.5\% on challenging datasets impacted by covariate shifts. Our results highlight CalShift as a promising strategy for building robust and reliable low-shot vision-language systems for real-world applications.

Confidence-calibrated covariate shift correction for few-shot classification in Vision-Language Models

TL;DR

CalShift tackles covariate shift and confidence misalignment in vision-language models under few-shot settings by jointly regularizing the CLIP objective with a Fisher information penalty and a Confidence Misalignment Penalty CMP. The final objective is , producing more robust and calibrated predictions without extra data. Empirical results across 19 vision datasets show improvements in accuracy (up to 3.5%) and calibration (ECE reduction up to 5.82%) under distribution shifts, with ablations confirming complementary contributions from FIM and CMP. The work offers a practical, training-free path for reliable deployment of vision-language models in real-world, low-shot scenarios.

Abstract

Since the establishment of vision-language foundation models as the new mainstay in low-shot vision classification tasks, the question of domain generalization arising from insufficient target data is assuming more importance. This scarcity challenge induces sampling bias and amplifies model sensitivity to variations and shifts in data distributions. While fine-tuning on multiple domains could mitigate such domain generalization issues, it is resource-intensive and demands diverse data sources. In this work, we systematically analyze two critical challenges: (1) covariate shift between the pre-training distribution and the underspecified target distribution, and (2) confidence misalignment, where predictions on novel data are overconfident. To address both challenges simultaneously, we introduce \textbf{Confidence-Calibrated Covariate Shift Correction (CalShift)} -- a unified approach that combines a Fisher information penalty to mitigate covariate shift and a Confidence Misalignment Penalty (CMP) to reduce overconfidence in misclassified examples. Experimental evaluations across various vision and covariate shift benchmarks demonstrate that CalShift significantly improves model calibration, achieving up to a 5.82\% reduction in Expected Calibration Error (ECE). Furthermore, CalShift enhances robustness, improving accuracy by 3.5\% on challenging datasets impacted by covariate shifts. Our results highlight CalShift as a promising strategy for building robust and reliable low-shot vision-language systems for real-world applications.

Paper Structure

This paper contains 9 sections, 4 theorems, 25 equations, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Experiments and Results
  5. Experiments
  6. Results
  7. Conclusions
  8. Theoretical Background
  9. Ablation study

Key Result

Proposition 3.1

The Fisher information $I(\theta)$ improves generalization by controlling the curvature of the loss landscape and CMP improves calibration by penalizing overconfidence and redistributing log-likelihood to the true class.

Figures (1)

  • Figure 1: Workflow of the proposed CalShift framework: The sub-figure (a) illustrates the confidence misalignment problem caused by covariate shift. The (bottom right) part of the sub-figure (a) show misaligned predictions. The subfigure (b) middle section represents the two components as recipe in method: Fisher Information Penalty ($I(\theta)$) for covariate shift correction and Confidence Misalignment Penalty (CMP) for calibration. Both FIM and CMP are integrated into the CLIP text encoder. The final loss, $\mathcal{L}_{\text{CalShift}} = \mathcal{L}_{\text{CLIP}} + \lambda_1 I(\theta) + \lambda_2 \text{CMP}$, combines the original CLIP loss with FIM and CMP to produce robust and aligned predictions, as shown in (bottom right) as output.

Theorems & Definitions (7)

  • Proposition 3.1
  • Proposition A.1
  • Proof A.1
  • Proposition A.2
  • Proof A.2
  • Proposition A.3
  • Proof A.3