Calibrating Uncertainty Quantification of Multi-Modal LLMs using Grounding

TL;DR

The paper addresses the misalignment between confidence and accuracy in multi-modal LLMs that rely on self-consistency for uncertainty quantification. It introduces a calibration framework that fuses self-consistency with cross-modal grounding, applying temperature-scaled grounding signals to refine the model’s reported confidence via $Conf = Conf_{baseline} \times Conf_{GM}^{(1/T)} + C$. Using Slake and VQA datasets with LLaVA variants and diverse grounding models, the approach yields significantly lower Expected Calibration Error (ECE) than baselines and calibrated baselines alone, demonstrating robustness across domains. This work improves trustworthy deployment of multi-modal LLMs by providing calibrated, evidence-backed confidence estimates and suggests avenues for extending grounding-based calibration to additional modalities.

Abstract

We introduce a novel approach for calibrating uncertainty quantification (UQ) tailored for multi-modal large language models (LLMs). Existing state-of-the-art UQ methods rely on consistency among multiple responses generated by the LLM on an input query under diverse settings. However, these approaches often report higher confidence in scenarios where the LLM is consistently incorrect. This leads to a poorly calibrated confidence with respect to accuracy. To address this, we leverage cross-modal consistency in addition to self-consistency to improve the calibration of the multi-modal models. Specifically, we ground the textual responses to the visual inputs. The confidence from the grounding model is used to calibrate the overall confidence. Given that using a grounding model adds its own uncertainty in the pipeline, we apply temperature scaling - a widely accepted parametric calibration technique - to calibrate the grounding model's confidence in the accuracy of generated responses. We evaluate the proposed approach across multiple multi-modal tasks, such as medical question answering (Slake) and visual question answering (VQAv2), considering multi-modal models such as LLaVA-Med and LLaVA. The experiments demonstrate that the proposed framework achieves significantly improved calibration on both tasks.

Figures (8)

• Figure 1: Consistently incorrect responses generated by LLaVA-Med-v1.5-Mistral-7B li2023llava on MRI image of the brain from the Slake Medical dataset (left). BiomedParse biomedparse, a grounding model for medical images, is not able to locate 'lung & spinal cord' on the MRI image of brain, and therefore labels the entire image as lung with zero confidence. It is, however, able to generate a bounding box for lung on the chest X-ray with $100\%$ confidence (right).
• Figure 2: Reliability diagrams with the expected accuracy of Llama-2-13B on COQA and TriviaQA datasets plotted as a function of the model's confidence predicted by self-consistency-based UQ approaches: 'Lexical Similarity' lexical_sim on the left and 'Semantic Entropy' kuhn on the right. A perfect calibration between the model's accuracy and the predicted confidence would have resulted in red points (average accuracy for each confidence bin) on the $x=y$ axis with a low variance (length of red lines).
• Figure 3: $Conf_{baseline}$: confidence from a self-consistency UQ baseline such as LexSim lexical_sim, PredEnt pred_entropy, NumSets kuhngen_with_confidence, SemEnt kuhn, etc. on a multi-modal LLM such as LLaVA-Med. $Conf_{GM}^{1/T}$: temperature-scaled calibrated confidence of grounding model on the accuracy of the generated responses. A grounding model can be as simple as the CLIP-based model, that provides its confidence in terms of a similarity score between embeddings of the generated response and the input image (e.g. BiomedCLIP), a detection model for response on the input image reporting its confidence on the detected bounding box (e.g. BiomedParse), or a foundation model that provides its verdict -- a confidence score in $[0,1]$ -- on the relevance of the generated response to the input image (e.g. LLaMA 3.2V, Qwen VL, etc.). $Conf$\ref{['our_proposal']} is the proposed calibrated confidence score for UQ of multi-modal LLMs resulting in substantially lower ECE than the baseline.
• Figure 4: (a) Histogram on the frequency of LLaVA's accuracy on VQA. Reliability Diagrams for UQ of LLaVA on VQA by (b) LexSim, and (c) SemEnt baseline. Each diagram shows plots and the respective ECE for the confidence reported by the baseline $Conf_{baseline}$, its calibrated version $Conf_{baseline}^{(1/T)}$, and the proposed approach \ref{['our_proposal']} for calibration with different grounding models.
• Figure 5: (a) Histogram on the frequency of LLaVA-Med's accuracy on Slake. Reliability Diagrams for UQ of LLaVA-Med on Slake by (b) LexSim, and (c) SemEnt. Each diagram shows plots and the respective ECE for the confidence reported by the baseline $Conf_{baseline}$, its calibrated version $Conf_{baseline}^{(1/T)}$, and the proposed approach \ref{['our_proposal']} for calibration with different grounding models.