Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CARE: Extracting Experimental Findings From Clinical Literature

Aakanksha Naik, Bailey Kuehl, Erin Bransom, Doug Downey, Tom Hope

TL;DR

CARE introduces an expressive annotation schema and the CARE dataset to extract fine-grained clinical findings from biomedical abstracts, unifying discontinuous spans, nested relations, variable arity, and numeric results. The dataset covers 700 abstracts from clinical trials and case reports, with demonstrated generalization to computer science and materials science domains. Benchmarking across extractive IE baselines and generative LLMs shows that even state-of-the-art models struggle on this task, particularly for relation extraction, highlighting the dataset's difficulty and value for advancing literature-focused IE. The work also discusses evaluation nuances, cross-domain applicability, and provides resources to the community to foster research in result-oriented scientific information extraction and aggregation.

Abstract

Extracting fine-grained experimental findings from literature can provide dramatic utility for scientific applications. Prior work has developed annotation schemas and datasets for limited aspects of this problem, failing to capture the real-world complexity and nuance required. Focusing on biomedicine, this work presents CARE -- a new IE dataset for the task of extracting clinical findings. We develop a new annotation schema capturing fine-grained findings as n-ary relations between entities and attributes, which unifies phenomena challenging for current IE systems such as discontinuous entity spans, nested relations, variable arity n-ary relations and numeric results in a single schema. We collect extensive annotations for 700 abstracts from two sources: clinical trials and case reports. We also demonstrate the generalizability of our schema to the computer science and materials science domains. We benchmark state-of-the-art IE systems on CARE, showing that even models such as GPT4 struggle. We release our resources to advance research on extracting and aggregating literature findings.

CARE: Extracting Experimental Findings From Clinical Literature

TL;DR

CARE introduces an expressive annotation schema and the CARE dataset to extract fine-grained clinical findings from biomedical abstracts, unifying discontinuous spans, nested relations, variable arity, and numeric results. The dataset covers 700 abstracts from clinical trials and case reports, with demonstrated generalization to computer science and materials science domains. Benchmarking across extractive IE baselines and generative LLMs shows that even state-of-the-art models struggle on this task, particularly for relation extraction, highlighting the dataset's difficulty and value for advancing literature-focused IE. The work also discusses evaluation nuances, cross-domain applicability, and provides resources to the community to foster research in result-oriented scientific information extraction and aggregation.

Abstract

Extracting fine-grained experimental findings from literature can provide dramatic utility for scientific applications. Prior work has developed annotation schemas and datasets for limited aspects of this problem, failing to capture the real-world complexity and nuance required. Focusing on biomedicine, this work presents CARE -- a new IE dataset for the task of extracting clinical findings. We develop a new annotation schema capturing fine-grained findings as n-ary relations between entities and attributes, which unifies phenomena challenging for current IE systems such as discontinuous entity spans, nested relations, variable arity n-ary relations and numeric results in a single schema. We collect extensive annotations for 700 abstracts from two sources: clinical trials and case reports. We also demonstrate the generalizability of our schema to the computer science and materials science domains. We benchmark state-of-the-art IE systems on CARE, showing that even models such as GPT4 struggle. We release our resources to advance research on extracting and aggregating literature findings.
Paper Structure (30 sections, 8 figures, 12 tables)

This paper contains 30 sections, 8 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Information Extraction from Scientific Literature
  4. Extracting Numeric Information
  5. Annotation Schema
  6. Comparison to Existing Clinical Schemas
  7. Annotation Complexity
  8. Dataset Collection
  9. Annotation Pilots
  10. Full-Scale Annotation
  11. Dataset Statistics
  12. Benchmarking IE Models
  13. Extractive IE Baselines:
  14. Generative IE Baselines:
  15. End-to-End Evaluation:
  16. ...and 15 more sections

Figures (8)

  • Figure 1: A partial example of entity, attribute and relation annotation using our schema for a clinical trial.
  • Figure 2: A partial example of entity, attribute and relation annotation using our generalized schema for a materials science abstract.
  • Figure 3: Example prompt used to evaluate the performance of finetuned LLMs on entity extraction. Such prompts are generated for all seven entity types in our dataset.
  • Figure 4: Example prompt used to evaluate the performance of finetuned LLMs on attribute extraction with gold entities provided. Such prompts are generated for all nine attribute types in our dataset.
  • Figure 5: Example prompt used to evaluate the performance of finetuned LLMs on relation extraction with gold entities and attributes provided. Such prompts are generated for all four relation types in our dataset.
  • ...and 3 more figures