SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers

TL;DR

SPIQA presents the first large-scale, open-ended QA benchmark that requires interpreting complex figures and tables within scientific CS papers, paired with full-text context. It introduces three interleaved tasks to probe direct and retrieval-based reasoning, plus a novel LLM-based evaluation metric (L3Score) to assess free-form answers. The dataset comprises 25,859 papers (2018–2023) and roughly 270K QA triplets, with evaluation splits including test-A (LLM-generated), test-B (QASA-derived), and test-C (QASPER-derived) to stress grounding and chain-of-thought capabilities. Experimental results show strong performance from最新 multimodal LLMs, significant gains from fine-tuning, and the potential of long-context reasoning for scientific QA, supported by comprehensive ablations and human evaluation.

Abstract

Seeking answers to questions within long scientific research articles is a crucial area of study that aids readers in quickly addressing their inquiries. However, existing question-answering (QA) datasets based on scientific papers are limited in scale and focus solely on textual content. We introduce SPIQA (Scientific Paper Image Question Answering), the first large-scale QA dataset specifically designed to interpret complex figures and tables within the context of scientific research articles across various domains of computer science. Leveraging the breadth of expertise and ability of multimodal large language models (MLLMs) to understand figures, we employ automatic and manual curation to create the dataset. We craft an information-seeking task on interleaved images and text that involves multiple images covering plots, charts, tables, schematic diagrams, and result visualizations. SPIQA comprises 270K questions divided into training, validation, and three different evaluation splits. Through extensive experiments with 12 prominent foundational models, we evaluate the ability of current multimodal systems to comprehend the nuanced aspects of research articles. Additionally, we propose a Chain-of-Thought (CoT) evaluation strategy with in-context retrieval that allows fine-grained, step-by-step assessment and improves model performance. We further explore the upper bounds of performance enhancement with additional textual information, highlighting its promising potential for future research and the dataset's impact on revolutionizing how we interact with scientific literature.

Figures (16)

• Figure 1: Illustration of the SPIQA tasks. Given a question anchored in figures from a research paper, we evaluate the capabilities of multimodal LLMs in comprehending and integrating information across multiple figures, tables and paper text.
• Figure 2: Statistics of the collected research papers and generated questions.
• Figure 3: Ablation on the importance of captions in the QA task. All Gemini and GPT variants suffer when captions are omitted. All numbers are for direct QA with figures and tables.
• Figure 4: Example questions, ground-truth answers, and responses by different baseline models. Both QAs belong to testA. Metrics colored in fooourgreen green denote correct evaluations, while those in fooourred red indicate incorrect scoring. R-L: ROUGE-L, BERT: BERTScore, L3S: L3Score.
• Figure B.1: Computation of L3Score using different LLMs. While the absolute values of L3Score vary depending on the choice of LLM, the relative changes in between different models remain consistent. All numbers are for direct QA with figures and tables on test-B.