Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning

TL;DR

This work formalizes these failures under a unified notion of knowledge conflict, distinguishing input-level objective conflict from process-level effective conflict and provides a mechanism-level view of multimodal reasoning under knowledge conflict and enable principled diagnosis and control of long-CoT failures.

Abstract

Multimodal large language models (MLLMs) in long chain-of-thought reasoning often fail when different knowledge sources provide conflicting signals. We formalize these failures under a unified notion of knowledge conflict, distinguishing input-level objective conflict from process-level effective conflict. Through probing internal representations, we reveal that: (I) Linear Separability: different conflict types are explicitly encoded as linearly separable features rather than entangled; (II) Depth Localization: conflict signals concentrate in mid-to-late layers, indicating a distinct processing stage for conflict encoding; (III) Hierarchical Consistency: aggregating noisy token-level signals along trajectories robustly recovers input-level conflict types; and (IV) Directional Asymmetry: reinforcing the model's implicit source preference under conflict is far easier than enforcing the opposite source. Our findings provide a mechanism-level view of multimodal reasoning under knowledge conflict and enable principled diagnosis and control of long-CoT failures.

Figures (5)

• Figure 1: Overview of Knowledge Sources and Conflict Types. We categorize knowledge into Visual ($\mathcal{K}_{\text{vision}}$), Textual ($\mathcal{K}_{\text{text}}$), and Parametric Prior ($\mathcal{K}_{\text{prior}}$). Knowledge conflicts arise when factual statements from different sources act as incompatible signals. We define three primary conflict types: Vision-Text ($\mathcal{C}_{\mathrm{VT}}$), Vision-Prior ($\mathcal{C}_{\mathrm{VP}}$), and Prior-Text ($\mathcal{C}_{\mathrm{PT}}$).
• Figure 2: Token-level separability of effective conflict $\mathcal{C}^e_{i,j}(t \mid x)$. The left panel shows the confusion matrix over token-level conflict predictions. The right panels decompose performance into binary detection of conflict versus no-conflict, and fine-grained attribution among conflict types. Values denote row-normalized recall.
• Figure 3: Sample-level separability of conflict types. We visualize the t-SNE projection of hidden states at layer 20 ( R1-Onevision) and layer 39 ( Llama-3.2V). The three conflict categories are colored according to their Objective Conflict labels, pre-defined during dataset construction. The top-right confusion matrices illustrate the sample-level attribution performance.
• Figure 4: Cross-layer distribution of conflict signals. Top row: attention-head activation ratio on conflict tokens vs. no-conflict tokens (lines), and their difference (bars), computed using effective conflict labels. Middle/bottom rows: layer-wise probe performance (one-vs-rest AUC and Recall@0.1) for $\mathcal{C}_{\mathrm{VP}},\mathcal{C}_{\mathrm{PT}},\mathcal{C}_{\mathrm{VT}}$ across three MLLM backbones.
• Figure 5: Semantic performance of targeted source control. We evaluate three conflict subsets ($\mathcal{C}^{o}_{\mathrm{VP}}, \mathcal{C}^{o}_{\mathrm{VT}}, \mathcal{C}^{o}_{\mathrm{PT}}$) using judge-based metrics: ASR (Anchor Support Rate, $\uparrow$), ARR (Anchor Rejection Rate, $\downarrow$), and OER (Obvious Error Rate, $\downarrow$). Forward/Reverse denote intervening toward the truth-anchored (benchmark-reliable) vs. conflicting source. Arrows indicate relative changes against the baseline. Note that VCD is inapplicable to the non-visual $\mathcal{C}^{o}_{\mathrm{PT}}$ subset.