ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinations

TL;DR

This work tackles relation hallucinations in open LVLMs by proposing ChainMPQ, a training-free framework that enforces stepwise relational reasoning through a subject/object decomposition, multi-perspective prompts, and an interleaved chain of textual and visual memories. The approach enhances subject/object localization with cross-attention, generates five targeted sub-questions, and propagates textual and visual context across steps to ground reasoning in visual evidence. Empirical results on LLaVA-1.5-7B and InstructBLIP-7B across MMRel and R-Bench show consistent improvements in accuracy, precision, and F1, with ablations validating the necessity of each component. ChainMPQ offers a practical, model-agnostic strategy to mitigate relation hallucinations and improve the reliability of multimodal reasoning in LVLMs.

Abstract

While Large Vision-Language Models (LVLMs) achieve strong performance in multimodal tasks, hallucinations continue to hinder their reliability. Among the three categories of hallucinations, which include object, attribute, and relation, relation hallucinations account for the largest proportion but have received the least attention. To address this issue, we propose ChainMPQ (Multi-Perspective Questions guided Interleaved Chain of Image and Text), a training-free method that improves relational inference in LVLMs by utilizing accumulated textual and visual memories. ChainMPQ first extracts subject and object keywords from the question to enhance the corresponding image regions. It then constructs multi-perspective questions that focus on the three core components of a relationship: the subject, the object, and the relation that links them. These questions are sequentially input to the model, with textual and visual memories from earlier steps providing supporting context for subsequent ones, thereby forming an interleaved chain of images and text that guides progressive relational reasoning. Experiments on multiple LVLMs and benchmarks show that ChainMPQ substantially reduces relation hallucinations, while ablation studies further validate the effectiveness of its three core modules.

Figures (6)

• Figure 1: This figure illustrates a case in which an LVLM exhibits a relation hallucination, misrecognizing the true relation "riding" as "standing". With ChainMPQ, the model is guided to reason step by step and infer the correct relation.
• Figure 2: Overview of our proposed ChainMPQ. It comprises three modules. Text-guided Attention Enhancement: extracts subject, object, and relation, using cross-attention to emphasize relation-relevant visual regions; Multi-Perspective Aware Text Prompt: constructs five new questions based on these elements from different perspectives; Interleaved Text-image Reasoning Chain: sequentially inputs the questions, using each answer $A_i$ and its top-K active visual tokens to form mask $M_i$ as a bias when calculating subsequent attention maps. The original question is then answered to produce the final output and evaluation metrics.
• Figure 3: Ablation Results on MMRel using LLaVA-1.5: (a) Ablation study; (b) Sensitivity analysis.
• Figure 4: The full ChainMPQ answering process vs. directly output
• Figure 5: Comparison of attention maps between directly answering the original question and answering it using ChainMPQ.