Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verification

TL;DR

Pelican is a novel framework designed to detect and mitigate hallucinations through claim verification through a chain of sub-claims based on first-order predicates, which improves over prior work by introducing intermediate variables for precise grounding of object instances and shared computation for answering the sub-question to enable adaptive corrections and inconsistency identification.

Abstract

Large Visual Language Models (LVLMs) struggle with hallucinations in visual instruction following task(s), limiting their trustworthiness and real-world applicability. We propose Pelican -- a novel framework designed to detect and mitigate hallucinations through claim verification. Pelican first decomposes the visual claim into a chain of sub-claims based on first-order predicates. These sub-claims consist of (predicate, question) pairs and can be conceptualized as nodes of a computational graph. We then use Program-of-Thought prompting to generate Python code for answering these questions through flexible composition of external tools. Pelican improves over prior work by introducing (1) intermediate variables for precise grounding of object instances, and (2) shared computation for answering the sub-question to enable adaptive corrections and inconsistency identification. We finally use reasoning abilities of LLMs to verify the correctness of the claim by considering the consistency and confidence of the (question, answer) pairs from each sub-claim. Our experiments reveal a drop in hallucination rate by ~ 8% - 32% across various baseline LVLMs and a 27% drop compared to approaches proposed for hallucination mitigation on MMHal-Bench. Results on two other benchmarks further corroborate our results.

Figures (13)

• Figure 1: Overview of Pelican. Given an image ($I$) and a question ($q$), LVLM ($V_L$) outputs an answer ($a$). We transform the pair ($q$, $a$) to our claim ($c$). Our pipeline: Step 1: Visual Table constructs a tabular representation of the image, identifying the locations of visual objects using detection tools. For example, a row is created for the detected "motorcycle" with its bounding box. Step 2: Claim Decomposition generates a list of granular sub-claims and follow-up questions. This example also shows an intermediate variable, $person_riding, that is used for referencing a specific object. Step 3: Program of Thought translates these questions into Python code using the POT approach. Final Step: Integrated Verification Synthesis performs comprehensive reasoning assessments to validate the original claims using the answers from Step 3.
• Figure 2: Computational graph representation of the generated sub-claims with predicates as the node and edges defined by their dependencies.
• Figure 3: An illustration of hallucination in LVLMs. Three examples showcasing different types of question-answering styles, where both LLaVA-v1.5 and LLaVA-v1.6 hallucinates to the question. Pelican refines the answer from these models exhibiting significantly reduced hallucinations.
• Figure 4: Prompt template to generate a claim using both the question and answer.
• Figure 5: Template for prompting LLM to perform key concept extraction.