Similarity-as-Evidence: Calibrating Overconfident VLMs for Interpretable and Label-Efficient Medical Active Learning

TL;DR

This work tackles miscalibration and cold-start challenges in Vision-Language Model (VLM)–driven medical active learning by recasting text-image similarities as Dirichlet evidence. The proposed Similarity-as-Evidence (SaE) framework introduces a Similarity Evidence Head (SEH) that maps similarities to a positive evidence strength $\lambda$, enabling a decomposed uncertainty representation into vacuity and dissonance via Subjective Logic. A dual-factor acquisition strategy leverages high-vacuity samples early to broaden coverage and high-dissonance samples later to refine decision boundaries, with PubMed-augmented prompts enriching semantic representations. Across ten public medical imaging datasets with a 20% labeling budget, SaE achieves state-of-the-art macro accuracy (82.57%) and excellent calibration (NLL 0.425 on BTMRI), demonstrating interpretable, label-efficient active learning suitable for clinical deployment. The results show that calibrating VLM uncertainties and guiding acquisition with interpretable evidence substantially improves both performance and trustworthiness in medical image analysis.

Abstract

Active Learning (AL) reduces annotation costs in medical imaging by selecting only the most informative samples for labeling, but suffers from cold-start when labeled data are scarce. Vision-Language Models (VLMs) address the cold-start problem via zero-shot predictions, yet their temperature-scaled softmax outputs treat text-image similarities as deterministic scores while ignoring inherent uncertainty, leading to overconfidence. This overconfidence misleads sample selection, wasting annotation budgets on uninformative cases. To overcome these limitations, the Similarity-as-Evidence (SaE) framework calibrates text-image similarities by introducing a Similarity Evidence Head (SEH), which reinterprets the similarity vector as evidence and parameterizes a Dirichlet distribution over labels. In contrast to a standard softmax that enforces confident predictions even under weak signals, the Dirichlet formulation explicitly quantifies lack of evidence (vacuity) and conflicting evidence (dissonance), thereby mitigating overconfidence caused by rigid softmax normalization. Building on this, SaE employs a dual-factor acquisition strategy: high-vacuity samples (e.g., rare diseases) are prioritized in early rounds to ensure coverage, while high-dissonance samples (e.g., ambiguous diagnoses) are prioritized later to refine boundaries, providing clinically interpretable selection rationales. Experiments on ten public medical imaging datasets with a 20% label budget show that SaE attains state-of-the-art macro-averaged accuracy of 82.57%. On the representative BTMRI dataset, SaE also achieves superior calibration, with a negative log-likelihood (NLL) of 0.425.

Figures (7)

• Figure 1: SaE addresses overconfidence in VLM-based AL by reframing text-image similarities as Dirichlet evidence. Left: Traditional VLM-based methods rely on temperature-scaled softmax, producing overconfident probabilities that mislead sample selection. Right: SaE calibrates similarities into evidence, enabling decomposition into vacuity (knowledge gaps) and dissonance (decision conflicts) for interpretable sample acquisition.
• Figure 2: Overview of the proposed SaE framework. A frozen VLM encodes images and PubMed-augmented class prompts to produce text-image similarities, which are mapped by a trainable SEH into Dirichlet evidence. We decompose this evidence into vacuity and dissonance, and use them to score unlabeled samples for AL. The inset shows one acquisition round: high-vacuity cases are prioritized early to cover unseen phenotypes, and high-dissonance boundary disputes are prioritized later for expert annotation.
• Figure 3: SaE achieves the lowest and most stable training loss across all AL rounds. Training trajectories on BTMRI ($\rho=0.2$) are compared. SaE avoids the high initial loss of BADGE and the volatility of BiomedCLIP+BADGE. Dashed red lines mark round transitions.
• Figure 4: SaE achieves superior calibration and mitigates VLM overconfidence. Reliability diagrams on BTMRI at a 20% label budget visualize empirical accuracy versus predicted confidence across 15 bins. Solid curves show accuracy, the dashed diagonal indicates perfect calibration, and blue bars represent the confidence distribution. SaE follows the diagonal most closely, BADGE BADGE is mildly underconfident, and PCB PCB remains strongly overconfident.
• Figure 5: SaE provides superior sample efficiency, achieving higher accuracy at all tested budget levels. Mean test accuracy on CHMNIST (5 seeds, 95% CI). SaE is shown at multiple budgets; baselines are at 20% for comparison.