Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Statistical Factuality Guarantee for Large Vision-Language Models

Zhuohang Li, Chao Yan, Nicholas J. Jackson, Wendi Cui, Bo Li, Jiaxin Zhang, Bradley A. Malin

TL;DR

ConfLVLM tackles LVLM hallucinations by casting generated text as a sequence of verifiable claims and applying a conformal-prediction-based filtering pipeline to guarantee factuality with finite-sample, distribution-free guarantees. The framework decomposes responses, ranks claims using internal and external conformity scores, and calibrates a threshold to meet a user-specified error tolerance $\lambda$ and confidence level $1-\alpha$, yielding a final response $Y^*$ with controlled risk. Validated across three diverse domains—general scene understanding, medical radiology report generation, and document understanding—ConfLVLM demonstrates precise coverage control, effective claim filtering, and flexible tradeoffs between coverage and utility for eight LVLMs and over $8.1\times 10^4$ claims. The work highlights the practical impact of statistical factuality guarantees for safe deployment of LVLMs in safety-critical settings, and points to a rich space of future improvements via discriminative versus generative critics and conditional validity extensions.

Abstract

Advancements in Large Vision-Language Models (LVLMs) have demonstrated promising performance in a variety of vision-language tasks involving image-conditioned free-form text generation. However, growing concerns about hallucinations in LVLMs, where the generated text is inconsistent with the visual context, are becoming a major impediment to deploying these models in applications that demand guaranteed reliability. In this paper, we introduce a framework to address this challenge, ConfLVLM, which is grounded on conformal prediction to achieve finite-sample distribution-free statistical guarantees on the factuality of LVLM output. This framework treats an LVLM as a hypothesis generator, where each generated text detail (or claim) is considered an individual hypothesis. It then applies a statistical hypothesis testing procedure to verify each claim using efficient heuristic uncertainty measures to filter out unreliable claims before returning any responses to users. We conduct extensive experiments covering three representative application domains, including general scene understanding, medical radiology report generation, and document understanding. Remarkably, ConfLVLM reduces the error rate of claims generated by LLaVa-1.5 for scene descriptions from 87.8\% to 10.0\% by filtering out erroneous claims with a 95.3\% true positive rate. Our results further demonstrate that ConfLVLM is highly flexible, and can be applied to any black-box LVLMs paired with any uncertainty measure for any image-conditioned free-form text generation task while providing a rigorous guarantee on controlling the risk of hallucination.

Towards Statistical Factuality Guarantee for Large Vision-Language Models

TL;DR

ConfLVLM tackles LVLM hallucinations by casting generated text as a sequence of verifiable claims and applying a conformal-prediction-based filtering pipeline to guarantee factuality with finite-sample, distribution-free guarantees. The framework decomposes responses, ranks claims using internal and external conformity scores, and calibrates a threshold to meet a user-specified error tolerance and confidence level , yielding a final response with controlled risk. Validated across three diverse domains—general scene understanding, medical radiology report generation, and document understanding—ConfLVLM demonstrates precise coverage control, effective claim filtering, and flexible tradeoffs between coverage and utility for eight LVLMs and over claims. The work highlights the practical impact of statistical factuality guarantees for safe deployment of LVLMs in safety-critical settings, and points to a rich space of future improvements via discriminative versus generative critics and conditional validity extensions.

Abstract

Advancements in Large Vision-Language Models (LVLMs) have demonstrated promising performance in a variety of vision-language tasks involving image-conditioned free-form text generation. However, growing concerns about hallucinations in LVLMs, where the generated text is inconsistent with the visual context, are becoming a major impediment to deploying these models in applications that demand guaranteed reliability. In this paper, we introduce a framework to address this challenge, ConfLVLM, which is grounded on conformal prediction to achieve finite-sample distribution-free statistical guarantees on the factuality of LVLM output. This framework treats an LVLM as a hypothesis generator, where each generated text detail (or claim) is considered an individual hypothesis. It then applies a statistical hypothesis testing procedure to verify each claim using efficient heuristic uncertainty measures to filter out unreliable claims before returning any responses to users. We conduct extensive experiments covering three representative application domains, including general scene understanding, medical radiology report generation, and document understanding. Remarkably, ConfLVLM reduces the error rate of claims generated by LLaVa-1.5 for scene descriptions from 87.8\% to 10.0\% by filtering out erroneous claims with a 95.3\% true positive rate. Our results further demonstrate that ConfLVLM is highly flexible, and can be applied to any black-box LVLMs paired with any uncertainty measure for any image-conditioned free-form text generation task while providing a rigorous guarantee on controlling the risk of hallucination.

Paper Structure

This paper contains 60 sections, 1 theorem, 4 equations, 28 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work.
  3. Our Contribution.
  4. Preliminaries
  5. Large Vision-Language Models
  6. Hallucination of LVLMs.
  7. Split Conformal Prediction
  8. Coverage Guarantee.
  9. Conformal Calibration.
  10. Ensuring Factuality for LVLMs
  11. Problem Formulation
  12. Error Control in LVLMs
  13. Error Control Procedure.
  14. Calibration.
  15. Deriving Conformity Scores
  16. ...and 45 more sections

Key Result

Theorem 3.1

Define the error scores ${\bm{E}}_{i} \coloneq \{{\mathcal{L}} (C_i^j, I_i): C_i^j \in {\bm{C}}_i\}$. Let $\{(X_i, I_i,{\bm{C}}_i, {\bm{E}}_i)\}_{i=1}^{n+1}$ be exchangeable, then the following lower bound holds for any $\alpha \in (\frac{1}{n+1}, 1)$: If the loss function is monotonic, meaning that ${\mathcal{L}}(\hat{F}_1({\bm{C}}_i, I_i)) \leq {\mathcal{L}}(\hat{F}_2({\bm{C}}_i, I_i))$ for any

Figures (28)

  • Figure 1: Overview: given user-specified error tolerance $\lambda$, error rate $\alpha$, and a calibration dataset, ConfLVLM returns a more reliable response for any new image and prompt at inference time through sampling, decomposing$D$, filtering$F$, and merging$M$, to ensure that the risk of the final response $Y^*$ is controlled with high probability. Illustrative examples, one for each application domain, are provided for outcome demonstration, where claims are highlighted to indicate ConfLVLM's confidence using specific conformity score and error tolerance level. Unhighlighted claims correspond to low confidence in factuality check.
  • Figure 2: Alignment between empirical and desired (theoretical) coverage in scene understanding (with $\lambda=0$). Vanilla LVLM (red dashed line) refers to the base setting where the LVLM-generated responses are returned to users without using ConfLVLM.
  • Figure 3: Average ratio of claims filtered with varying coverage using different scoring functions in scene understanding (with $\lambda=0$). Standard errors are marked.
  • Figure 4: Abstention rate with varying coverage using different scoring functions in scene understanding (with $\lambda=0$).
  • Figure 5: Comparison of Llama-3.2-11B-Vision's response with different error tolerances ($\lambda$) while fixing $\alpha=0.1$ in scene understanding.
  • ...and 23 more figures

Theorems & Definitions (2)

  • Theorem 3.1: SCP Coverage Guarantee shafer2008tutorialmohrilanguage
  • proof