Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models

Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, Arman Cohan

TL;DR

M3SciQA, a multi-modal, multi-document scientific question answering benchmark designed for a more comprehensive evaluation of foundation models, indicates that current foundation models still significantly underperform compared to human experts in multi-modal information retrieval and in reasoning across multiple scientific documents.

Abstract

Existing benchmarks for evaluating foundation models mainly focus on single-document, text-only tasks. However, they often fail to fully capture the complexity of research workflows, which typically involve interpreting non-textual data and gathering information across multiple documents. To address this gap, we introduce M3SciQA, a multi-modal, multi-document scientific question answering benchmark designed for a more comprehensive evaluation of foundation models. M3SciQA consists of 1,452 expert-annotated questions spanning 70 natural language processing paper clusters, where each cluster represents a primary paper along with all its cited documents, mirroring the workflow of comprehending a single paper by requiring multi-modal and multi-document data. With M3SciQA, we conduct a comprehensive evaluation of 18 foundation models. Our results indicate that current foundation models still significantly underperform compared to human experts in multi-modal information retrieval and in reasoning across multiple scientific documents. Additionally, we explore the implications of these findings for the future advancement of applying foundation models in multi-modal scientific literature analysis.

M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models

TL;DR

M3SciQA, a multi-modal, multi-document scientific question answering benchmark designed for a more comprehensive evaluation of foundation models, indicates that current foundation models still significantly underperform compared to human experts in multi-modal information retrieval and in reasoning across multiple scientific documents.

Abstract

Existing benchmarks for evaluating foundation models mainly focus on single-document, text-only tasks. However, they often fail to fully capture the complexity of research workflows, which typically involve interpreting non-textual data and gathering information across multiple documents. To address this gap, we introduce M3SciQA, a multi-modal, multi-document scientific question answering benchmark designed for a more comprehensive evaluation of foundation models. M3SciQA consists of 1,452 expert-annotated questions spanning 70 natural language processing paper clusters, where each cluster represents a primary paper along with all its cited documents, mirroring the workflow of comprehending a single paper by requiring multi-modal and multi-document data. With M3SciQA, we conduct a comprehensive evaluation of 18 foundation models. Our results indicate that current foundation models still significantly underperform compared to human experts in multi-modal information retrieval and in reasoning across multiple scientific documents. Additionally, we explore the implications of these findings for the future advancement of applying foundation models in multi-modal scientific literature analysis.

Paper Structure

This paper contains 58 sections, 3 equations, 14 figures, 21 tables.

Table of Contents

  1. Introduction
  2. The M3SciQA Benchmark
  3. Overview of M3SciQA
  4. Benchmark Construction Principles
  5. Benchmark Construction
  6. Expert Annotators.
  7. Anchor Papers.
  8. Visual Context QAs from Anchor Papers.
  9. Reference-Based QAs from Reference Papers.
  10. Combined Questions.
  11. Experiments
  12. Visual Context Evaluation
  13. Task Formulation.
  14. Evaluation Protocol.
  15. Experiment Setup.
  16. ...and 43 more sections

Figures (14)

  • Figure 1: (Top) The common workflow of comparative analysis in scientific research, particularly when a result, such as a figure/table in the Information Value paper (anchor paper)giulianelli2023information, prompts further examination of related research, such as details from DialoGPT (reference paper)zhang2020dialogpt. (Bottom) A demonstration of the workflow for constructing a visual context question, reference-based question, and combined question.
  • Figure 2: An overview of M3SciQA question construction pipeline.
  • Figure 3: Distribution of reasoning types of visual context and reference-based questions in M3SciQA.
  • Figure 4: Three examples from GPT-4o in answering visual context questions.
  • Figure 5: Performance scores of Mistral, Llama 3 70B, GPT-3.5, and GPT-4 in different retrieval settings.
  • ...and 9 more figures