Post-hoc Probabilistic Vision-Language Models

TL;DR

This work tackles the problem of unreliable uncertainty estimates in pre-trained vision-language models (VLMs) by introducing BayesVLM, a post-hoc Bayesian framework that requires no retraining or architectural changes. It employs a Laplace approximation to obtain Gaussian posteriors over the last-layer projections of image and text encoders, enabling analytical propagation of uncertainty to cosine similarities via ProbCosine. The approach yields well-calibrated predictive uncertainties and improves data-efficiency in active learning for cross-domain tasks, while maintaining competitive accuracy and incurring minimal computational overhead. These results demonstrate that reliable uncertainty quantification can be achieved for large, pre-trained VLMs without costly modifications, facilitating safer, more trustworthy deployment in real-world settings.

Abstract

Vision-language models (VLMs), such as CLIP and SigLIP, have found remarkable success in classification, retrieval, and generative tasks. For this, VLMs deterministically map images and text descriptions to a joint latent space in which their similarity is assessed using the cosine similarity. However, a deterministic mapping of inputs fails to capture uncertainties over concepts arising from domain shifts when used in downstream tasks. In this work, we propose post-hoc uncertainty estimation in VLMs that does not require additional training. Our method leverages a Bayesian posterior approximation over the last layers in VLMs and analytically quantifies uncertainties over cosine similarities. We demonstrate its effectiveness for uncertainty quantification and support set selection in active learning. Compared to baselines, we obtain improved and well-calibrated predictive uncertainties, interpretable uncertainty estimates, and sample-efficient active learning. Our results show promise for safety-critical applications of large-scale models.

Figures (16)

• Figure 1: We introduce an efficient and effective post-hoc method to provide uncertainty estimates for vision-language models (e.g., CLIP, SigLIP) using a Laplace approximation. We demonstrate that uncertainty estimates derived from this approximation improve the calibration of these models on several zero-shot classification benchmarks (\ref{['sec:results_uncertainties']}) and are effective in active learning (\ref{['sec:results_activelearning']}).
• Figure 2: Predictive error vs. uncertainty (entropy) on the EuroSAT data set helber2019eurosat for the OpenCLIP ViT-H-14 model. The zero-shot comparison (left side) of the original model () and its Bayesian counterpart () indicates that our Bayesian model exhibits better calibration and substantially reduces overconfident predictions. Active Learning results (right side) show that those improvements lead to a substantially reduced misclassification rate after adaptation; quadrant (b).
• Figure 3: Illustration of uncertainty propagation in BayesVLMs: We estimate uncertainties over the last layers of both encoders using a Laplace approximation, which induces probabilistic feature embeddings. We then approximate the distribution over cosine similarities by estimating the expected value and variance. The cosine similarity distribution is then propagated to the VLM output.
• Figure 4: Can we select informative data for fine-tuning using BayesVLM uncertainty estimates? Yes. On the OfficeHome data set (OH) and ImageNet variants (IN), when using uncertainty-based scores (EPIG () and BALD ()) to select the fine-tuning data, we achieve better performance compared with Entropy (targeted) (), Entropy (), Random selection (targeted) (), and Random selection (). Thus, highlighting the benefits of using uncertainties from BayesVLM.
• Figure 5: Relative trace of the image Hessian $B$-factor for varying base batch sizes $K$ (2048 (), 8192 (), 32768 ()) and 1–5 random batches. Error bars show $\pm1$ std over five trials.