Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts

TL;DR

The paper addresses the fragility of reasoning in large multimodal models when confronted with cross-modal conflicts, focusing on textual inertia where erroneous textual priors dominate over visual evidence. It introduces the LogicGraph Perturbation Protocol to structurally perturb reasoning graphs and study self-reflection across diverse LMMs, revealing weak grounding in visual data and a strong propensity to cling to text. To mitigate this, the authors propose Active Visual-Context Refinement (AVCR), a training-free inference paradigm that interleaves visual re-grounding and context folding to denoise reasoning history and enforce grounding in visuals. Across open-ended video reasoning benchmarks, AVCR substantially improves explicit self-correction and overall task accuracy, offering a practical, scalable approach to enhance robustness without permanent model parameter changes.

Abstract

Large Multimodal Models (LMMs) have demonstrated impressive capabilities in video reasoning via Chain-of-Thought (CoT). However, the robustness of their reasoning chains remains questionable. In this paper, we identify a critical failure mode termed textual inertia, where once a textual hallucination occurs in the thinking process, models tend to blindly adhere to the erroneous text while neglecting conflicting visual evidence. To systematically investigate this, we propose the LogicGraph Perturbation Protocol that structurally injects perturbations into the reasoning chains of diverse LMMs spanning both native reasoning architectures and prompt-driven paradigms to evaluate their self-reflection capabilities. The results reveal that models successfully self-correct in less than 10% of cases and predominantly succumb to blind textual error propagation. To mitigate this, we introduce Active Visual-Context Refinement, a training-free inference paradigm which orchestrates an active visual re-grounding mechanism to enforce fine-grained verification coupled with an adaptive context refinement strategy to summarize and denoise the reasoning history. Experiments demonstrate that our approach significantly stifles hallucination propagation and enhances reasoning robustness.

Figures (7)

• Figure 1: Illustration of visual blindness induced by erroneous textual context. While normal reasoning (a) grounds answers in visual evidence, perturbated reasoning (b) demonstrates that injecting a factual error causes the model to ignore conflicting visual signals. The model prioritizes consistency with the false history over visual reality, leading to incorrect justifications.
• Figure 2: Overview of the LogicGraph Perturbation Protocol. The framework systematically evaluates text inertia by structuring reasoning chains into semantic graphs and injecting probability-weighted counterfactual perturbations. This process creates a conflict between textual priors and visual reality to determine whether the model succumbs to contextual contamination or achieves explicit reflection through visual evidence.
• Figure 3: Statistics of the curated dataset derived from STAR, showing the distribution of task types and video properties across 100 high-quality samples.
• Figure 4: Overview of the Active Visual Context Refinement framework. It orchestrates an agentic loop to retrieve visual evidence upon uncertainty and folds erroneous history for robust reasoning.
• Figure 5: The prompt template used for Semantic Graph Structuring.