Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MS2: Multi-Document Summarization of Medical Studies

Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, Lucy Lu Wang

TL;DR

MS2 tackles the lack of large-scale multi-document summarization resources in biomedicine by constructing 20K systematic reviews and 470K cited studies with both free-text and structured representations (PICO+Evidence Inference). It introduces two modeling formulations (texts-to-text and table-to-table) and a novel ΔEI evaluation to capture agreement on evidence directions, reporting fluent summaries yet only partial alignment with gold conclusions. The work highlights significant challenges in PICO tagging, co-reference resolution, and handling contradictory evidence across studies, pointing to avenues for future improvements. By releasing this dataset and baseline methodologies, MS2 aims to accelerate automated literature-review support and evidence synthesis in fast-evolving biomedical domains.

Abstract

To assess the effectiveness of any medical intervention, researchers must conduct a time-intensive and highly manual literature review. NLP systems can help to automate or assist in parts of this expensive process. In support of this goal, we release MS^2 (Multi-Document Summarization of Medical Studies), a dataset of over 470k documents and 20k summaries derived from the scientific literature. This dataset facilitates the development of systems that can assess and aggregate contradictory evidence across multiple studies, and is the first large-scale, publicly available multi-document summarization dataset in the biomedical domain. We experiment with a summarization system based on BART, with promising early results. We formulate our summarization inputs and targets in both free text and structured forms and modify a recently proposed metric to assess the quality of our system's generated summaries. Data and models are available at https://github.com/allenai/ms2

MS2: Multi-Document Summarization of Medical Studies

TL;DR

MS2 tackles the lack of large-scale multi-document summarization resources in biomedicine by constructing 20K systematic reviews and 470K cited studies with both free-text and structured representations (PICO+Evidence Inference). It introduces two modeling formulations (texts-to-text and table-to-table) and a novel ΔEI evaluation to capture agreement on evidence directions, reporting fluent summaries yet only partial alignment with gold conclusions. The work highlights significant challenges in PICO tagging, co-reference resolution, and handling contradictory evidence across studies, pointing to avenues for future improvements. By releasing this dataset and baseline methodologies, MS2 aims to accelerate automated literature-review support and evidence synthesis in fast-evolving biomedical domains.

Abstract

To assess the effectiveness of any medical intervention, researchers must conduct a time-intensive and highly manual literature review. NLP systems can help to automate or assist in parts of this expensive process. In support of this goal, we release MS^2 (Multi-Document Summarization of Medical Studies), a dataset of over 470k documents and 20k summaries derived from the scientific literature. This dataset facilitates the development of systems that can assess and aggregate contradictory evidence across multiple studies, and is the first large-scale, publicly available multi-document summarization dataset in the biomedical domain. We experiment with a summarization system based on BART, with promising early results. We formulate our summarization inputs and targets in both free text and structured forms and modify a recently proposed metric to assess the quality of our system's generated summaries. Data and models are available at https://github.com/allenai/ms2

Paper Structure

This paper contains 36 sections, 2 equations, 4 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Dataset
  4. Identifying suitable reviews and studies
  5. Background and target identification
  6. Structured form
  7. Adding PICO tags
  8. Adding Evidence Inference
  9. Clustering and train / test split
  10. Dataset statistics
  11. Experiments
  12. Texts-to-text task
  13. Table-to-table task
  14. $\Delta$EI metric
  15. Human evaluation & error analysis
  16. ...and 21 more sections

Figures (4)

  • Figure 1: Our primary formulation (texts-to-text) is a seq2seq MDS task. Given study abstracts and a background statement, generate the target summary.
  • Figure 2: The distributions of review and study publication years in MS2 show a clear temporal lag. Dashed lines mark the median year of publication.
  • Figure 3: Two input encoding configurations. Above: LongformerEncoderDecoder (LED), where all input studies are appended to the background and encoded together. Below: In the BART configuration, each input study is encoded independently with the review background. These are concatenated to form the input encoding.
  • Figure 4: Five fold cross-validation results from training a binary SciBERT classifier on the annotations. Precisions increase following a logistic curve over threshold choices; recalls decrease.