Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning

Run-Ze Fan, Zengzhi Wang, Pengfei Liu

TL;DR

MegaScience addresses the scarcity and quality issues of open-source science reasoning data by introducing TextbookReasoning and MegaScience, a 1.25-million-instance data mix, plus an open evaluation framework spanning 15 benchmarks. It details end-to-end data curation, decontamination, and annotation pipelines that produce concise, efficient training targets and robust step-by-step solutions. Empirical results show TextbookReasoning and MegaScience yield state-of-the-art or near state-of-the-art performance across multiple base models, with larger models benefiting more from MegaScience. By releasing the data, pipelines, evaluation tools, and trained models, the work aims to accelerate scientific reasoning research in the open-source community.

Abstract

Scientific reasoning is critical for developing AI scientists and supporting human researchers in advancing the frontiers of natural science discovery. However, the open-source community has primarily focused on mathematics and coding while neglecting the scientific domain, largely due to the absence of open, large-scale, high-quality, verifiable scientific reasoning datasets. To bridge this gap, we first present TextbookReasoning, an open dataset featuring truthful reference answers extracted from 12k university-level scientific textbooks, comprising 650k reasoning questions spanning 7 scientific disciplines. We further introduce MegaScience, a large-scale mixture of high-quality open-source datasets totaling 1.25 million instances, developed through systematic ablation studies that evaluate various data selection methodologies to identify the optimal subset for each publicly available scientific dataset. Meanwhile, we build a comprehensive evaluation system covering diverse subjects and question types across 15 benchmarks, incorporating comprehensive answer extraction strategies to ensure accurate evaluation metrics. Our experiments demonstrate that our datasets achieve superior performance and training efficiency with more concise response lengths compared to existing open-source scientific datasets. Furthermore, we train Llama3.1, Qwen2.5, and Qwen3 series base models on MegaScience, which significantly outperform the corresponding official instruct models in average performance. In addition, MegaScience exhibits greater effectiveness for larger and stronger models, suggesting a scaling benefit for scientific tuning. We release our data curation pipeline, evaluation system, datasets, and seven trained models to the community to advance scientific reasoning research.

MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning

TL;DR

MegaScience addresses the scarcity and quality issues of open-source science reasoning data by introducing TextbookReasoning and MegaScience, a 1.25-million-instance data mix, plus an open evaluation framework spanning 15 benchmarks. It details end-to-end data curation, decontamination, and annotation pipelines that produce concise, efficient training targets and robust step-by-step solutions. Empirical results show TextbookReasoning and MegaScience yield state-of-the-art or near state-of-the-art performance across multiple base models, with larger models benefiting more from MegaScience. By releasing the data, pipelines, evaluation tools, and trained models, the work aims to accelerate scientific reasoning research in the open-source community.

Abstract

Scientific reasoning is critical for developing AI scientists and supporting human researchers in advancing the frontiers of natural science discovery. However, the open-source community has primarily focused on mathematics and coding while neglecting the scientific domain, largely due to the absence of open, large-scale, high-quality, verifiable scientific reasoning datasets. To bridge this gap, we first present TextbookReasoning, an open dataset featuring truthful reference answers extracted from 12k university-level scientific textbooks, comprising 650k reasoning questions spanning 7 scientific disciplines. We further introduce MegaScience, a large-scale mixture of high-quality open-source datasets totaling 1.25 million instances, developed through systematic ablation studies that evaluate various data selection methodologies to identify the optimal subset for each publicly available scientific dataset. Meanwhile, we build a comprehensive evaluation system covering diverse subjects and question types across 15 benchmarks, incorporating comprehensive answer extraction strategies to ensure accurate evaluation metrics. Our experiments demonstrate that our datasets achieve superior performance and training efficiency with more concise response lengths compared to existing open-source scientific datasets. Furthermore, we train Llama3.1, Qwen2.5, and Qwen3 series base models on MegaScience, which significantly outperform the corresponding official instruct models in average performance. In addition, MegaScience exhibits greater effectiveness for larger and stronger models, suggesting a scaling benefit for scientific tuning. We release our data curation pipeline, evaluation system, datasets, and seven trained models to the community to advance scientific reasoning research.

Paper Structure

This paper contains 44 sections, 27 figures, 15 tables.

Table of Contents

  1. Introduction
  2. TextbookReasoning Data Curation
  3. Textbooks Collection and Digitization
  4. Dual Q-A Pairs Extraction
  5. Question Deduplication
  6. Q-A pair Refinement
  7. LLM-based Question Decontamination
  8. MegaScience Data Curation
  9. Sourcing from Public Datasets
  10. Question Deduplication and Decontamination
  11. Data Selection
  12. Solution Annotation
  13. MegaScience Evaluation Framework
  14. Language Model Open Science Evaluation
  15. MegaScience Evaluation Suite
  16. ...and 29 more sections

Figures (27)

  • Figure 1: Trade-off between model performance and inference efficiency (average response length) on Qwen2.5-7B.
  • Figure 2: Comparison of base models trained on MegaScience vs. official instruct models (non-thinking).
  • Figure 3: The overall of MegaScience datasets.
  • Figure 4: The pipeline of TextbookReasoning data curation.
  • Figure 5: The overall of MegaScience data recipe.
  • ...and 22 more figures