Sci-Reasoning: A Dataset Decoding AI Innovation Patterns

TL;DR

Sci-Reasoning introduces a dataset that captures structured intellectual synthesis behind high-quality AI research, linking target papers to multiple key predecessors with explicit roles and synthesis narratives. It uses community signals (Oral/Spotlight) and an LLM-accelerated, human-verified pipeline to trace reasoning across NeurIPS, ICML, and ICLR (2023–2025). It identifies 15 thinking patterns, with Gap-Driven Reframing, Cross-Domain Synthesis, and Representation Shift accounting for 52.7% of cases, and shows powerful 'recipes' formed by pattern combinations. The dataset enables quantitative studies of scientific progress and training of AI research agents.

Abstract

While AI innovation accelerates rapidly, the intellectual process behind breakthroughs -- how researchers identify gaps, synthesize prior work, and generate insights -- remains poorly understood. The lack of structured data on scientific reasoning hinders systematic analysis and development of AI research agents. We introduce Sci-Reasoning, the first dataset capturing the intellectual synthesis behind high-quality AI research. Using community-validated quality signals and an LLM-accelerated, human-verified pipeline, we trace Oral and Spotlight papers across NeurIPS, ICML, and ICLR (2023-2025) to its key predecessors, articulating specific reasoning links in a structured format. Our analysis identifies 15 distinct thinking patterns, with three dominant strategies accounting for 52.7%: Gap-Driven Reframing (24.2%), Cross-Domain Synthesis (18.0%), and Representation Shift (10.5%). The most powerful innovation recipes combine multiple patterns: Gap-Driven Reframing + Representation Shift, Cross-Domain Synthesis + Representation Shift, and Gap-Driven Reframing + Cross-Domain Synthesis. This dataset enables quantitative studies of scientific progress and provides structured reasoning trajectories for training the next generation AI research agents.

Figures (9)

• Figure 1: Overview of the Sci-Reasoning dataset construction pipeline. Our methodology consists of four main stages: (1) identifying high-quality papers using community-validated signals, (2) tracing intellectual lineage to key predecessors via LLM analysis, (3) generating structured reasoning trajectories with lineage links that capture roles, relationships, and intellectual moves
• Figure 2: One complete dataset entry in Sci-Reasoning. For a target paper, our LLM pipeline identifies key predecessors (§\ref{['sec:lineage-tracing']}) and generates structured metadata (predecessor role, relationship type) and rich synthesis narratives (§\ref{['sec:reasoning-link']}). See Appendix \ref{['app:example']} for the complete JSON structure.
• Figure 3: Distribution of papers in the Sci-Reasoning dataset across conferences, years, and presentation types (Oral and Spotlight).
• Figure 4: Distribution of the 15 identified thinking patterns across 3,819 papers. The top three patterns---Gap-Driven Reframing, Cross-Domain Synthesis, and Representation Shift---account for 52.7% of all papers.
• Figure 5: Temporal evolution of the top 5 thinking patterns from 2023 to 2025. While Gap-Driven Reframing remains stable, Representation Shift peaked in 2024, Formal-Experimental approaches are declining, and Data/Evaluation engineering is rising.