Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Calibration of Large Vision-Language Adapters

Balamurali Murugesan, Julio Silva-Rodriguez, Ismail Ben Ayed, Jose Dolz

TL;DR

The paper investigates severe miscalibration in CLIP-based adaptations (adapters, prompts, and test-time prompt tuning) under distribution shifts. It identifies logit-range magnification, not logit-norm, as a key driver of miscalibration and proposes three model-agnostic remedies—Zero-shot Logit Normalization (ZS-Norm), a penalty-based constraint, and Sample-adaptive Logit Scaling (SaLS)—that can be applied during training or inference. Across multiple CLIP backbones and a suite of OOD benchmarks, these methods substantially reduce calibration errors (ECE) while preserving or improving accuracy, demonstrating strong, model-agnostic improvements for zero-shot and few-shot CLIP adaptation. The work provides a practical, scalable approach to produce reliable uncertainty estimates in CLIP adapters, prompting safer deployment in real-world, open-world tasks.

Abstract

This paper addresses the critical issue of miscalibration in CLIP-based model adaptation, particularly in the challenging scenario of out-of-distribution (OOD) samples, which has been overlooked in the existing literature on CLIP adaptation. We empirically demonstrate that popular CLIP adaptation approaches, such as Adapters, Prompt Learning, and Test-Time Adaptation, substantially degrade the calibration capabilities of the zero-shot baseline in the presence of distributional drift. We identify the increase in logit ranges as the underlying cause of miscalibration of CLIP adaptation methods, contrasting with previous work on calibrating fully-supervised models. Motivated by these observations, we present a simple and model-agnostic solution to mitigate miscalibration, by scaling the logit range of each sample to its zero-shot prediction logits. We explore three different alternatives to achieve this, which can be either integrated during adaptation or directly used at inference time. Comprehensive experiments on popular OOD classification benchmarks demonstrate the effectiveness of the proposed approaches in mitigating miscalibration while maintaining discriminative performance, whose improvements are consistent across the three families of these increasingly popular approaches. The code is publicly available at: https://github.com/Bala93/CLIPCalib

Robust Calibration of Large Vision-Language Adapters

TL;DR

The paper investigates severe miscalibration in CLIP-based adaptations (adapters, prompts, and test-time prompt tuning) under distribution shifts. It identifies logit-range magnification, not logit-norm, as a key driver of miscalibration and proposes three model-agnostic remedies—Zero-shot Logit Normalization (ZS-Norm), a penalty-based constraint, and Sample-adaptive Logit Scaling (SaLS)—that can be applied during training or inference. Across multiple CLIP backbones and a suite of OOD benchmarks, these methods substantially reduce calibration errors (ECE) while preserving or improving accuracy, demonstrating strong, model-agnostic improvements for zero-shot and few-shot CLIP adaptation. The work provides a practical, scalable approach to produce reliable uncertainty estimates in CLIP adapters, prompting safer deployment in real-world, open-world tasks.

Abstract

This paper addresses the critical issue of miscalibration in CLIP-based model adaptation, particularly in the challenging scenario of out-of-distribution (OOD) samples, which has been overlooked in the existing literature on CLIP adaptation. We empirically demonstrate that popular CLIP adaptation approaches, such as Adapters, Prompt Learning, and Test-Time Adaptation, substantially degrade the calibration capabilities of the zero-shot baseline in the presence of distributional drift. We identify the increase in logit ranges as the underlying cause of miscalibration of CLIP adaptation methods, contrasting with previous work on calibrating fully-supervised models. Motivated by these observations, we present a simple and model-agnostic solution to mitigate miscalibration, by scaling the logit range of each sample to its zero-shot prediction logits. We explore three different alternatives to achieve this, which can be either integrated during adaptation or directly used at inference time. Comprehensive experiments on popular OOD classification benchmarks demonstrate the effectiveness of the proposed approaches in mitigating miscalibration while maintaining discriminative performance, whose improvements are consistent across the three families of these increasingly popular approaches. The code is publicly available at: https://github.com/Bala93/CLIPCalib
Paper Structure (40 sections, 18 equations, 8 figures, 10 tables)

This paper contains 40 sections, 18 equations, 8 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision language models
  4. Prompt based learning
  5. Black-box Adapters
  6. Model calibration
  7. Background
  8. CLIP Zero-Shot Classification
  9. Adaptation to novel tasks
  10. Constraining logits during adaptation
  11. Impact of adaptation in logits
  12. Our solution
  13. Zero-shot logit normalization during training (${\textbf{ZS-Norm}}$)
  14. Integrating explicit constraints in the learning objective (${\textbf{Penalty}}$)
  15. Sample-adaptive logit scaling (SaLS)
  16. ...and 25 more sections

Figures (8)

  • Figure 1: CLIP-based adaptation methods are severely miscalibrated on Out-of-distribution (OOD) samples. Three families of popular approaches to adapt CLIP under different scenarios, i.e., Prompt Learning (CoOp zhou2022coop), Adapters (Clip-Ad gao2021clip) and Test-time prompt tuning (TPT tpt), significantly degrade the miscalibration of the zero-shot baseline, despite improving its discriminative performance.
  • Figure 2: Logit norm or logit range as the source of miscalibration? These figures clearly show that when the calibration of the zero-shot (ZS) model is degraded, the logit norm of its predictions is reduced (top), which discards an increase of the logit norm as the main cause for miscalibration. In contrast, there exists a correlation between the increase of the logit ranges and miscalibration (bottom).
  • Figure 3: Effect of calibrating adapted CLIP models. Mean of the distribution of logit norms (top) and logit ranges (bottom) across the four ImageNet OOD datasets for a relevant Adapter-based (CLIP-Ad), Prompt Learning (CoOp) and TPT approach.
  • Figure 4: Additional Logit studies for few-shot Adapters. Analysis of average Logit norm and range after improving the calibration of the Adapter model TaskRes yu2023task using the proposed logit range regularization methods for improved calibration, i.e., ${\textbf{ZS-Norm}}$ (left), ${\textbf{Penalty}}$ (middle) and ${\textbf{SaLS}}$ (right).
  • Figure 5: Additional Logit studies for Prompt Learning. Analysis of average Logit norm and range after improving the calibration of the Prompt learning technique CoOp zhou2022coop using the proposed logit range regularization methods for improved calibration, i.e., ${\textbf{ZS-Norm}}$ (left), ${\textbf{Penalty}}$ (middle) and ${\textbf{SaLS}}$ (right).
  • ...and 3 more figures