Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learning

Andreas Sauter, Yuyue Zhao, Jacopo Urbani, Wenxiang Hu, Zaiqiao Meng, Lun Zhou, Xiaohui Yan, Yougang Lyu

Abstract

Scientific idea generation is a cornerstone of autonomous knowledge discovery, yet the iterative evolution required to transform initial concepts into high-quality research proposals remains a formidable challenge for Large Language Models (LLMs). Existing Reinforcement Learning (RL) paradigms often rely on rubric-based scalar rewards that provide global quality scores but lack actionable granularity. Conversely, language-based refinement methods are typically confined to inference-time prompting, targeting models that are not explicitly optimized to internalize such critiques. To bridge this gap, we propose \textbf{EvoIdeator}, a framework that facilitates the evolution of scientific ideas by aligning the RL training objective with \textbf{checklist-grounded feedback}. EvoIdeator leverages a structured judge model to generate two synergistic signals: (1) \emph{lexicographic rewards} for multi-dimensional optimization, and (2) \emph{fine-grained language feedback} that offers span-level critiques regarding grounding, feasibility, and methodological rigor. By integrating these signals into the RL loop, we condition the policy to systematically utilize precise feedback during both optimization and inference. Extensive experiments demonstrate that EvoIdeator, built on Qwen3-4B, significantly outperforms much larger frontier models across key scientific metrics. Crucially, the learned policy exhibits strong generalization to diverse external feedback sources without further fine-tuning, offering a scalable and rigorous path toward self-refining autonomous ideation.

EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learning

Abstract

Scientific idea generation is a cornerstone of autonomous knowledge discovery, yet the iterative evolution required to transform initial concepts into high-quality research proposals remains a formidable challenge for Large Language Models (LLMs). Existing Reinforcement Learning (RL) paradigms often rely on rubric-based scalar rewards that provide global quality scores but lack actionable granularity. Conversely, language-based refinement methods are typically confined to inference-time prompting, targeting models that are not explicitly optimized to internalize such critiques. To bridge this gap, we propose \textbf{EvoIdeator}, a framework that facilitates the evolution of scientific ideas by aligning the RL training objective with \textbf{checklist-grounded feedback}. EvoIdeator leverages a structured judge model to generate two synergistic signals: (1) \emph{lexicographic rewards} for multi-dimensional optimization, and (2) \emph{fine-grained language feedback} that offers span-level critiques regarding grounding, feasibility, and methodological rigor. By integrating these signals into the RL loop, we condition the policy to systematically utilize precise feedback during both optimization and inference. Extensive experiments demonstrate that EvoIdeator, built on Qwen3-4B, significantly outperforms much larger frontier models across key scientific metrics. Crucially, the learned policy exhibits strong generalization to diverse external feedback sources without further fine-tuning, offering a scalable and rigorous path toward self-refining autonomous ideation.
Paper Structure (34 sections, 7 equations, 3 figures, 2 tables)

This paper contains 34 sections, 7 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Method
  4. Checklist-Grounded Judge
  5. Lexicographic Rewards
  6. Actionable Language Feedback
  7. Training
  8. Experiments
  9. Research Questions
  10. Dataset Construction
  11. Evaluation Settings
  12. Baselines
  13. Implementation Details
  14. Experimental Results and Analysis
  15. Main Results (RQ1)
  16. ...and 19 more sections

Figures (3)

  • Figure 1: Overview of EvoIdeator. Given a research query and a relevant literature review, the EvoIdeator policy generates an initial candidate idea. The idea is then evaluated by a checklist-grounded judge, which produces two complementary signals: lexicographic rewards for multi-dimensional optimization and actionable language feedback that specifies how the idea should be revised. The language feedback, together with the current idea, is fed into the policy revision module to produce an improved idea. In parallel, the lexicographic rewards are used in an RL loop to update the EvoIdeator policy, aligning train-time optimization with inference-time refinement.
  • Figure 2: Performance trajectories across two generation steps. We compare the Informed model (Green) against the Uninformed model (Orange) and their respective untrained checkpoints (Blue/Gray). Step 0 represents the initial zero-shot generation; Step 1 represents the output after one round of refinement (via language feedback for Informed/Base, or self-correction for Non-informed). The lines represent the average over the summed scores. Shaded regions denote 95% confidence intervals.
  • Figure 3: Score trajectories for each checklist criteria. Informed (blue) is our model that has been trained with textual gradients and receives them during training; Non-Informed (orange) is our model that has been trained without textual gradients and is not receiving them during inference; Base (green) is the base model (Qwen3-4B-Thinking-2507) that receives textual gradients during inference; Non-Informed base is the base model that does not receive tetual gradients during inference.