SIN-Bench: Tracing Native Evidence Chains in Long-Context Multimodal Scientific Interleaved Literature

TL;DR

The paper tackles the challenge of evaluating true understanding of long-form, multimodal scientific literature by introducing the Fish-in-the-Ocean paradigm and SIN-Bench, a four-task, evidence-grounded benchmark that preserves native document interleaving of text and figures. It replaces traditional answer-only metrics with a No Evidence, No Score framework and a multi-dimensional evidence-quality metric (Matching, Relevance, Logic) to diagnose grounding ability. Through SIN-Data and a semi-automated, human-in-the-loop benchmark construction pipeline, the authors demonstrate that grounding is the main bottleneck for current MLLMs, with notable differences between models such as Gemini-3-pro and GPT-5 in grounding versus answer accuracy. The work advocates for evaluating traceable reasoning over parametric knowledge, provides open resources, and highlights practical implications for robust, explainable scientific reasoning in long-context multimodal settings.

Abstract

Evaluating whether multimodal large language models truly understand long-form scientific papers remains challenging: answer-only metrics and synthetic "Needle-In-A-Haystack" tests often reward answer matching without requiring a causal, evidence-linked reasoning trace in the document. We propose the "Fish-in-the-Ocean" (FITO) paradigm, which requires models to construct explicit cross-modal evidence chains within native scientific documents. To operationalize FITO, we build SIN-Data, a scientific interleaved corpus that preserves the native interleaving of text and figures. On top of it, we construct SIN-Bench with four progressive tasks covering evidence discovery (SIN-Find), hypothesis verification (SIN-Verify), grounded QA (SIN-QA), and evidence-anchored synthesis (SIN-Summary). We further introduce "No Evidence, No Score", scoring predictions when grounded to verifiable anchors and diagnosing evidence quality via matching, relevance, and logic. Experiments on eight MLLMs show that grounding is the primary bottleneck: Gemini-3-pro achieves the best average overall score (0.573), while GPT-5 attains the highest SIN-QA answer accuracy (0.767) but underperforms on evidence-aligned overall scores, exposing a gap between correctness and traceable support.

Figures (14)

• Figure 1: Comparison of long-context multimodal evaluation paradigms. (a) Current NIAH approaches embed artificial "needles" into irrelevant noise, focusing on surface-level retrieval. (b) The proposed FITO paradigm evaluates deep comprehension within the native document ecosystem ("ocean"). It requires the model to aggregate interconnected knowledge units ("fish") across sections to form an evidence chain for reasoning.
• Figure 2: The framework of SIN-Bench. (a) SIN-Data Infrastructure: We parse raw source packages into a unified Scientific Interleaved format using a semantic-first strategy. (c) Construction Pipeline: Based on the data, we employ an iterative Multi-MLLM synthesis loop with cross-validation and human auditing to generate high-quality samples. (b) Task & Metrics: The benchmark features four hierarchical tasks evaluated under the "No Evidence, No Score" protocol. We assess evidence chains across Matching, Relevance, and Logic dimensions.
• Figure 3: Task-level overall performance heatmap across models in SIN-Bench. Darker cells imply higher scores.
• Figure 4: Qualitative examples of reasoning failures by Gemini-3-pro in the SIN-Find and SIN-QA tasks.
• Figure 5: Impact of interleaved input and modality encodings on SIN-QA and SIN-Summary (Gemini-3-pro).