Epistemic Uncertainty Quantification for Pre-trained VLMs via Riemannian Flow Matching

TL;DR

This paper addresses the lack of intrinsic epistemic uncertainty in pre-trained Vision-Language Models by proposing REPVLM, a manifold-native density estimator on the embedding hypersphere. By extending Riemannian Flow Matching to a unified conditional model, REPVLM learns modality-conditioned embedding distributions and computes exact log-likelihoods via a continuity equation, providing a principled epistemic-uncertainty score U_ep = -log p(z|c). Empirically, REPVLM shows near-perfect correlation between uncertainty and prediction error and robust performance for out-of-distribution detection and data curation, while maintaining computational efficiency relative to ensemble methods. The work thus offers a scalable, intrinsic mechanism to quantify model confidence in VLMs, with clear implications for selective classification and reliable deployment in real-world settings.

Abstract

Vision-Language Models (VLMs) are typically deterministic in nature and lack intrinsic mechanisms to quantify epistemic uncertainty, which reflects the model's lack of knowledge or ignorance of its own representations. We theoretically motivate negative log-density of an embedding as a proxy for the epistemic uncertainty, where low-density regions signify model ignorance. The proposed method REPVLM computes the probability density on the hyperspherical manifold of the VLM embeddings using Riemannian Flow Matching. We empirically demonstrate that REPVLM achieves near-perfect correlation between uncertainty and prediction error, significantly outperforming existing baselines. Beyond classification, we also demonstrate that the model also provides a scalable metric for out-of-distribution detection and automated data curation.

Figures (5)

• Figure 1: Overview of REPVLM. The framework estimates the probability density $p(z)$ of pre-trained VLM embeddings on the hypersphere $\mathbb{S}^{d-1}$. A unified model learns a vector field $v_t$ that transports a simple uniform base distribution $P_0 = \operatorname{Unif}(\mathbb{S}^{d-1})$ to the empirical modality-specific distributions $P_1$ (Image and Text). As illustrated, standard inputs map to high-density regions (yellow), while ambiguous or out-of-distribution inputs such as the distorted image or nonsensical text reside in low-density regions (purple). The negative log-likelihood $- \log p(z|c)$ thus serves as a principled proxy for epistemic uncertainty $U_\text{ep}(z)$, reflecting the model's confidence.
• Figure 2: Accuracy-Rejection Curves for Zero-Shot Classification. We evaluate the utility of epistemic uncertainty estimates for selective classification across three proxy datasets (rows) and six downstream benchmarks (columns). The $x$-axis represents the fraction of samples rejected based on high uncertainty, and the $y$-axis shows the Top-1 Accuracy of the remaining retained data. A sharp upward slope indicates that the model identifies its own errors via the uncertainty measure.
• Figure 3: Ablation studies of REPVLM.Left: Comparison of the proposed Riemannian formulation against Euclidean variants using uniform and Gaussian base distributions. Center: Impact of proxy training data size, varying from 50K to 1M image-caption samples. Right: Convergence of accuracy-rejection performance relative to the number of ODE integration steps.
• Figure 4: Out-of-Distribution Detection.Left: Distributions of uncertainty scores $U_\text{ep}(x) = -\log p(z|c)$ for ImageNet-1K (ID), ImageNet-R (Near-OOD), and EuroSAT (Far-OOD). Center & Right: Mean Receiver Operating Characteristic (ROC) and Mean Precision-Recall (PR) curves.
• Figure 5: Data Curation via Epistemic Uncertainty.Left: The density distribution of uncertainty scores, where the tail corresponds to high-uncertainty samples. Right: Qualitative examples of samples with the highest epistemic uncertainty.

Theorems & Definitions (1)

• Definition 3.1: Epistemic Uncertainty