Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Ideation: Designing LLM Agents to Generate Novel Research Ideas on Scientific Concept Network

Keyu Zhao, Weiquan Lin, Qirui Zheng, Fengli Xu, Yong Li

TL;DR

The paper introduces Deep Ideation, an LLM-driven framework that grounds research idea generation in a large-scale scientific concept network constructed from keyword co-occurrence in ~100k AI papers. Through an explore-expand-evolve workflow, an Idea Stack, and a critic model trained on reviewer data, the system iteratively refines ideas by expanding the keyword set and grounding proposals in literature-derived relationships. Empirical results across four AI domains show substantial gains in novelty and feasibility over baselines, with human evaluators confirming practical value. The work also provides a public dataset and prompts, advancing ideas on how to tightly couple ideation with dynamic knowledge graphs to improve scientific discovery.

Abstract

Novel research ideas play a critical role in advancing scientific inquiries. Recent advancements in Large Language Models (LLMs) have demonstrated their potential to generate novel research ideas by leveraging large-scale scientific literature. However, previous work in research ideation has primarily relied on simplistic methods, such as keyword co-occurrence or semantic similarity. These approaches focus on identifying statistical associations in the literature but overlook the complex, contextual relationships between scientific concepts, which are essential to effectively leverage knowledge embedded in human literature. For instance, papers that simultaneously mention "keyword A" and "keyword B" often present research ideas that integrate both concepts. Additionally, some LLM-driven methods propose and refine research ideas using the model's internal knowledge, but they fail to effectively utilize the scientific concept network, limiting the grounding of ideas in established research. To address these challenges, we propose the Deep Ideation framework to address these challenges, integrating a scientific network that captures keyword co-occurrence and contextual relationships, enriching LLM-driven ideation. The framework introduces an explore-expand-evolve workflow to iteratively refine research ideas, using an Idea Stack to track progress. A critic engine, trained on real-world reviewer feedback, guides the process by providing continuous feedback on the novelty and feasibility of ideas. Our experiments show that our approach improves the quality of generated ideas by 10.67% compared to other methods, with ideas surpassing top conference acceptance levels. Human evaluation highlights their practical value in scientific research, and ablation studies confirm the effectiveness of each component in the workflow. Code repo is available at https://github.com/kyZhao-1/Deep-Ideation.

Deep Ideation: Designing LLM Agents to Generate Novel Research Ideas on Scientific Concept Network

TL;DR

The paper introduces Deep Ideation, an LLM-driven framework that grounds research idea generation in a large-scale scientific concept network constructed from keyword co-occurrence in ~100k AI papers. Through an explore-expand-evolve workflow, an Idea Stack, and a critic model trained on reviewer data, the system iteratively refines ideas by expanding the keyword set and grounding proposals in literature-derived relationships. Empirical results across four AI domains show substantial gains in novelty and feasibility over baselines, with human evaluators confirming practical value. The work also provides a public dataset and prompts, advancing ideas on how to tightly couple ideation with dynamic knowledge graphs to improve scientific discovery.

Abstract

Novel research ideas play a critical role in advancing scientific inquiries. Recent advancements in Large Language Models (LLMs) have demonstrated their potential to generate novel research ideas by leveraging large-scale scientific literature. However, previous work in research ideation has primarily relied on simplistic methods, such as keyword co-occurrence or semantic similarity. These approaches focus on identifying statistical associations in the literature but overlook the complex, contextual relationships between scientific concepts, which are essential to effectively leverage knowledge embedded in human literature. For instance, papers that simultaneously mention "keyword A" and "keyword B" often present research ideas that integrate both concepts. Additionally, some LLM-driven methods propose and refine research ideas using the model's internal knowledge, but they fail to effectively utilize the scientific concept network, limiting the grounding of ideas in established research. To address these challenges, we propose the Deep Ideation framework to address these challenges, integrating a scientific network that captures keyword co-occurrence and contextual relationships, enriching LLM-driven ideation. The framework introduces an explore-expand-evolve workflow to iteratively refine research ideas, using an Idea Stack to track progress. A critic engine, trained on real-world reviewer feedback, guides the process by providing continuous feedback on the novelty and feasibility of ideas. Our experiments show that our approach improves the quality of generated ideas by 10.67% compared to other methods, with ideas surpassing top conference acceptance levels. Human evaluation highlights their practical value in scientific research, and ablation studies confirm the effectiveness of each component in the workflow. Code repo is available at https://github.com/kyZhao-1/Deep-Ideation.

Paper Structure

This paper contains 40 sections, 9 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Definition of Scientific Network
  4. Formulating Research Ideation Problem
  5. Methods
  6. Overview
  7. Key Components of Deep Ideation Framework
  8. Deep Ideation Framework
  9. Explore
  10. Expand
  11. Evolve
  12. critic model
  13. Experiments
  14. Experimental Setup
  15. Experimental Results
  16. ...and 25 more sections

Figures (5)

  • Figure 1: Illustration of the construction of the scientific network and the Deep Ideation process
  • Figure 2: Overview of our Deep Ideation framework. In this figure, we set the maximum size of the keyword set to 4.
  • Figure 3: The construction process of the training data for the Review Model
  • Figure 4: A case study of idea proposal generated by Deep Ideation.
  • Figure A.5: Effect of max neighborhood size and max keyword set size, where performance is the sum of novelty and feasibility.