ScholarEval: Research Idea Evaluation Grounded in Literature

TL;DR

ScholarEval addresses the need for rigorous, literature-grounded evaluation of AI-generated research ideas before execution. It introduces a retrieval-augmented framework that assesses ideas along two axes—soundness (empirical validity of proposed methods) and contribution (novelty across multiple dimensions)—grounded in the ScholarIdeas expert-annotated dataset spanning AI, neuroscience, biochemistry, and ecology. The system uses a multi-stage pipeline to extract methods, discover supporting literature, and generate dense, actionable feedback with citations, outperforming strong baselines in rubric coverage and providing high-quality evaluations across disciplines. An expert user study corroborates its real-world usefulness, and the authors release code, data, and tools to the community, highlighting ScholarEval’s potential to augment ideation workflows in science.

Abstract

As AI tools become increasingly common for research ideation, robust evaluation is critical to ensure the validity and usefulness of generated ideas. We introduce ScholarEval, a retrieval augmented evaluation framework that assesses research ideas based on two fundamental criteria: soundness - the empirical validity of proposed methods based on existing literature, and contribution - the degree of advancement made by the idea across different dimensions relative to prior research. To evaluate ScholarEval, we introduce ScholarIdeas, the first expert-annotated dataset of multi-domain research ideas and reviews, comprised of 117 ideas across four disciplines: artificial intelligence, neuroscience, biochemistry, and ecology. Our evaluation shows that ScholarEval achieves significantly higher coverage of points mentioned in the human expert annotated rubrics in ScholarIdeas compared to all baselines. Furthermore, ScholarEval is consistently preferred over our strongest baseline o4-mini-deep-research, a reasoning and search-enabled agentic system by OpenAI, in terms of evaluation actionability, depth, and evidence support. Our large-scale user study also shows that ScholarEval significantly outperforms deep research in literature engagement, idea refinement, and usefulness. We openly release our code, dataset, and ScholarEval tool for the community to use and build on.

Figures (7)

• Figure 1: Left: Given a research idea ScholarEval generates a literature-grounded evaluation based on soundness and contribution. Right: To evaluate ScholarEval, we measure the degree of coverage of expert-annotated review rubrics in ScholarIdeas. The final coverage score is the average over all rubrics.
• Figure 2: Overview of ScholarEval and its two modules. Top: The soundness module extracts the methods proposed in the research idea and conducts a thorough literature search for similar applications of each method to determine its potential effectiveness. Bottom: The contribution module identifies the dimensions along which the research idea is making contributions and conducts detailed comparisons with related papers along each dimension to identify areas of novelty or lack thereof.
• Figure 3: Overview of ScholarIdeas. Left:ScholarIdeas includes 117 research ideas across four disciplines. Right: Example of a neuroscience research idea and review in ScholarIdeas. Each review is composed of multiple rubrics, for a total of 1076 rubrics across the dataset. This idea-review pair is adapted from Bloem2025Dynamic.
• Figure 4: Win rate based on Depth, Act, and Evidence between ScholarEval$_{\text{Claude}}$ and o4-mini-deep-research using LLM judge.
• Figure 5: ScholarEval user interface.