PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies

TL;DR

PRISMM-Bench introduces a real-world benchmark for evaluating large multimodal models on peer-review flagged inconsistencies in scientific papers. By grounding tasks in authentic reviewer feedback and employing a three-task MCQ suite (Ident, Remedy, Match) with JSON-based debiasing to minimize linguistic shortcuts, it reveals that even state-of-the-art models struggle with cross-modal reasoning and long-context grounding. The study demonstrates that proprietary models outperform open-weight counterparts on average, but overall performance remains far from reliable levels for automated scientific assistance. The findings underscore the need for advances in reasoning architectures, long-context grounding, and debiasing strategies to build trustworthy multimodal scientific assistants with robust cross-modal verification capabilities.

Abstract

Large Multimodal Models (LMMs) are increasingly applied to scientific research, yet it remains unclear whether they can reliably understand and reason over the multimodal complexity of papers. A central challenge lies in detecting and resolving inconsistencies across text, figures, tables, and equations, issues that are often subtle, domain-specific, and ultimately undermine clarity, reproducibility, and trust. Existing benchmarks overlook this issue, either isolating single modalities or relying on synthetic errors that fail to capture real-world complexity. We introduce PRISMM-Bench (Peer-Review-sourced Inconsistency Set for Multimodal Models), the first benchmark grounded in real reviewer-flagged inconsistencies in scientific papers. Through a multi-stage pipeline of review mining, LLM-assisted filtering and human verification, we curate 262 inconsistencies from 242 papers. Based on this set, we design three tasks, namely inconsistency identification, remedy and pair matching, which assess a model's capacity to detect, correct, and reason over inconsistencies across different modalities. Furthermore, to address the notorious problem of choice-only shortcuts in multiple-choice evaluation, where models exploit answer patterns without truly understanding the question, we further introduce structured JSON-based answer representations that minimize linguistic biases by reducing reliance on superficial stylistic cues. We benchmark 21 leading LMMs, including large open-weight models (GLM-4.5V 106B, InternVL3 78B) and proprietary models (Gemini 2.5 Pro, GPT-5 with high reasoning). Results reveal strikingly low performance (26.1-54.2%), underscoring the challenge of multimodal scientific reasoning and motivating progress towards trustworthy scientific assistants.

Figures (21)

• Figure 2: Pipeline of PRISMM-Bench. The top row illustrates the six main steps: (1) review sourcing, (2) LLM-based review filtering, and (3) manual annotation of metadata for reviewer-flagged inconsistencies (Sec. \ref{['sec:collection_reviewer_flagged_inconsistencies']}), (4) LMM-based task generation, (5) manual verification to construct benchmark tasks (Sec. \ref{['sec:generation_benchmark_tasks']}), and (6) LLM-based debiasing to mitigate language biases (Sec. \ref{['sec:alleviation_language_bias']}). The bottom row shows representative outputs at each stage: filtered reviews after step 2, inconsistency annotation after step 3, an example multiple-choice question in natural language after step 4, and its debiased JSON-format counterpart after step 6.
• Figure 3: A qualitative example of a text–table inconsistency and its corresponding evaluation tasks of Ident, Remedy and Match.
• Figure 4: A qualitative example of a figure-equation inconsistency and its corresponding evaluation tasks of Ident, Remedy and Match.
• Figure 5: Inconsistency example for case study. Right: Visual context. Left: Question and answer options for Ident task. Natural language options used for ease-of-comprehension, LMM was tasked using JSON.
• Figure 6: Raw reasoning output by InternVL3.5 38B on 3MDmM0rMPQ.