CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution

TL;DR

< CrossCheck-Bench addresses the problem of evaluating and diagnosing cross-modal contradictions in multimodal inputs. It introduces a three-level hierarchy (L1-L3) and seven atomic capabilities to diagnose where vision-language models fail when cues conflict, supported by a 15k QA dataset built from real-world artifacts. The study finds that while perception tasks scale with model size, integration and especially reasoning over multi-attribute contradictions remain bottlenecks; conventional prompting yields limited gains, whereas iterative, interleaved grounding and symbolic reasoning (MM-CoT) yields more robust improvements. These findings provide a concrete diagnostic framework and practical directions for developing robust cross-modal verification in open-world settings.>

Abstract

Multimodal Large Language Models are primarily trained and evaluated on aligned image-text pairs, which leaves their ability to detect and resolve real-world inconsistencies largely unexplored. In open-domain applications visual and textual cues often conflict, requiring models to perform structured reasoning beyond surface-level alignment. We introduce CrossCheck-Bench, a diagnostic benchmark for evaluating contradiction detection in multimodal inputs. The benchmark adopts a hierarchical task framework covering three levels of reasoning complexity and defines seven atomic capabilities essential for resolving cross-modal inconsistencies. CrossCheck-Bench includes 15k question-answer pairs sourced from real-world artifacts with synthetically injected contradictions. The dataset is constructed through a multi-stage annotation pipeline involving more than 450 expert hours to ensure semantic validity and calibrated difficulty across perception, integration, and reasoning. We evaluate 13 state-of-the-art vision-language models and observe a consistent performance drop as tasks shift from perceptual matching to logical contradiction detection. Most models perform well on isolated entity recognition but fail when multiple clues must be synthesized for conflict reasoning. Capability-level analysis further reveals uneven skill acquisition, especially in tasks requiring multi-step inference or rule-based validation. Additional probing shows that conventional prompting strategies such as Chain-of-Thought and Set-of-Mark yield only marginal gains. By contrast, methods that interleave symbolic reasoning with grounded visual processing achieve more stable improvements. These results highlight a persistent bottleneck in multimodal reasoning and suggest new directions for building models capable of robust cross-modal verification.

Figures (8)

• Figure 1: CrossCheck-Bench cascade case: the model answers the Level-1 perception query correctly, yet fails the dependent Level-2 integration and Level-3 conflict-reasoning tasks.
• Figure 2: Overview of CrossCheck-Bench. The benchmark spans three cognitive tiers—L1 Perception, L2 Integration, and L3 Reasoning—grounded in seven atomic capabilities (A1–A7) and eight representative tasks. It offers thousands of product samples, 14,690 adversarial QA pairs, and 22.8 K multimodal clue graphs, curated through 450+ expert hours. The right-hand bar chart contrasts average model accuracy with the human upper bound.
• Figure 3: Dataset–construction pipeline for CrossCheck-Bench. Stage 1 aggregates multimodal product data (30 + categories, five languages, three modalities) and encodes them as multimodal clue graphs that bind entities to their validated attributes. Stage 2 samples 1– $n$ clues from each graph to programmatically compose hierarchical question–answer pairs that probe the seven atomic capabilities across the three cognitive tiers (L1–L3). Stage 3 applies a three-step quality-control loop: expert review, ensemble-model filtering, and tier-wise difficulty balancing, to ensure answer correctness and uniform task difficulty.
• Figure 4: Dataset statistics for CrossCheck-Bench. The benchmark contains $\sim\!15{,}000$ question–answer pairs distributed over 15 subtasks and three cognitive levels (left). Six subtasks probe a single atomic capability (A1–A6), while the remaining nine require compositions capabilities (right). Subtask names followed by “*” take multi-frame input.
• Figure 5: Model-wise Performance on Atomic Capabilities. Each cell indicates accuracy on one capability, with * denoting the best-performing model per capability.