Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NSF-SciFy: Mining the NSF Awards Database for Scientific Claims

Delip Rao, Weiqiu You, Eric Wong, Chris Callison-Burch

TL;DR

NSF-SciFy introduces the largest open dataset for scientific claim extraction by mining NSF award abstracts, addressing the gap of relying on published literature. A zero-shot prompting pipeline extracts verifiable claims and investigation proposals, with subsequent fine-tuning of 7B-LMs to substantially boost extraction performance. The authors release NSF-SciFy and NSF-SciFy-MatSci along with novel LLM-based evaluation metrics, demonstrating strong gains and enabling longitudinal, cross-disciplinary analyses of research funding. This resource enables scalable claim verification, meta-science studies, and science-communication applications by providing millions of claims across five decades and all NSF domains.

Abstract

We present NSF-SciFy, a large-scale dataset for scientific claim extraction derived from the National Science Foundation (NSF) awards database, comprising over 400K grant abstracts spanning five decades. While previous datasets relied on published literature, we leverage grant abstracts which offer a unique advantage: they capture claims at an earlier stage in the research lifecycle before publication takes effect. We also introduce a new task to distinguish between existing scientific claims and aspirational research intentions in proposals. Using zero-shot prompting with frontier large language models, we jointly extract 114K scientific claims and 145K investigation proposals from 16K grant abstracts in the materials science domain to create a focused subset called NSF-SciFy-MatSci. We use this dataset to evaluate 3 three key tasks: (1) technical to non-technical abstract generation, where models achieve high BERTScore (0.85+ F1); (2) scientific claim extraction, where fine-tuned models outperform base models by 100% relative improvement; and (3) investigation proposal extraction, showing 90%+ improvement with fine-tuning. We introduce novel LLM-based evaluation metrics for robust assessment of claim/proposal extraction quality. As the largest scientific claim dataset to date -- with an estimated 2.8 million claims across all STEM disciplines funded by the NSF -- NSF-SciFy enables new opportunities for claim verification and meta-scientific research. We publicly release all datasets, trained models, and evaluation code to facilitate further research.

NSF-SciFy: Mining the NSF Awards Database for Scientific Claims

TL;DR

NSF-SciFy introduces the largest open dataset for scientific claim extraction by mining NSF award abstracts, addressing the gap of relying on published literature. A zero-shot prompting pipeline extracts verifiable claims and investigation proposals, with subsequent fine-tuning of 7B-LMs to substantially boost extraction performance. The authors release NSF-SciFy and NSF-SciFy-MatSci along with novel LLM-based evaluation metrics, demonstrating strong gains and enabling longitudinal, cross-disciplinary analyses of research funding. This resource enables scalable claim verification, meta-science studies, and science-communication applications by providing millions of claims across five decades and all NSF domains.

Abstract

We present NSF-SciFy, a large-scale dataset for scientific claim extraction derived from the National Science Foundation (NSF) awards database, comprising over 400K grant abstracts spanning five decades. While previous datasets relied on published literature, we leverage grant abstracts which offer a unique advantage: they capture claims at an earlier stage in the research lifecycle before publication takes effect. We also introduce a new task to distinguish between existing scientific claims and aspirational research intentions in proposals. Using zero-shot prompting with frontier large language models, we jointly extract 114K scientific claims and 145K investigation proposals from 16K grant abstracts in the materials science domain to create a focused subset called NSF-SciFy-MatSci. We use this dataset to evaluate 3 three key tasks: (1) technical to non-technical abstract generation, where models achieve high BERTScore (0.85+ F1); (2) scientific claim extraction, where fine-tuned models outperform base models by 100% relative improvement; and (3) investigation proposal extraction, showing 90%+ improvement with fine-tuning. We introduce novel LLM-based evaluation metrics for robust assessment of claim/proposal extraction quality. As the largest scientific claim dataset to date -- with an estimated 2.8 million claims across all STEM disciplines funded by the NSF -- NSF-SciFy enables new opportunities for claim verification and meta-scientific research. We publicly release all datasets, trained models, and evaluation code to facilitate further research.

Paper Structure

This paper contains 25 sections, 1 equation, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Scientific Claim Datasets
  4. Meta Science and Social Science
  5. Building NSF-SciFy
  6. Data Collection
  7. Data Processing
  8. Claim and Investigation Proposal Extraction
  9. Dataset Analysis
  10. NSF-SciFy
  11. NSF-SciFy-MatSci
  12. Technical vs. Non-Technical Abstracts
  13. Tasks, Metrics, and Experiments
  14. Data Preparation
  15. Models and Training
  16. ...and 10 more sections

Figures (3)

  • Figure 1: A sample record from our NSF-SciFy dataset. Each record contains 1) Award ID and title, 2) NSF Directorate, 3) Technical and non-technical abstracts, 4) Scientific Claims, 5) Investigation Proposals, and 6) Associated publications, when present.
  • Figure 2: Distribution of awards areas as represented by the National Science Foundation directorates in NSF-SciFy, illustrating the breadth and comprehensiveness of scientific claims in our dataset. The NSF-SciFy-MatSci subset spanning all of materials science awards represents 3.9% of the entire dataset.
  • Figure 3: The t-SNE plot of comparing content embeddings from SPECTER cohan-etal-2020-specter and style embeddings from STEL patel2025 for technical and non-technical abstracts in NSF-SciFy-MatSci. The somewhat clear separation between technical and non-technical abstracts when using style embeddings indicate marked stylistic differences between the two kinds abstracts.