Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spark: A System for Scientifically Creative Idea Generation

Aishik Sanyal, Samuel Schapiro, Sumuk Shashidhar, Royce Moon, Lav R. Varshney, Dilek Hakkani-Tur

TL;DR

Spark addresses the challenge of generating scientifically useful ideas by grounding large language model creativity in retrieval-augmented generation and a specialized evaluator trained on OpenReview reviews. It combines Xplor literature retrieval, an idea generator, and a Judge-based filter to simulate peer review and select promising ideas, releasing the OpenReview-derived dataset for reproducibility. The approach demonstrates modular integration of retrieval, generation, and evaluation and reports scalability to thousands of ideas. The work contributes to the computational creativity community by providing a concrete framework and data to study the generation and evaluation of scientific ideas.

Abstract

Recently, large language models (LLMs) have shown promising abilities to generate novel research ideas in science, a direction which coincides with many foundational principles in computational creativity (CC). In light of these developments, we present an idea generation system named Spark that couples retrieval-augmented idea generation using LLMs with a reviewer model named Judge trained on 600K scientific reviews from OpenReview. Our work is both a system demonstration and intended to inspire other CC researchers to explore grounding the generation and evaluation of scientific ideas within foundational CC principles. To this end, we release the annotated dataset used to train Judge, inviting other researchers to explore the use of LLMs for idea generation and creative evaluations.

Spark: A System for Scientifically Creative Idea Generation

TL;DR

Spark addresses the challenge of generating scientifically useful ideas by grounding large language model creativity in retrieval-augmented generation and a specialized evaluator trained on OpenReview reviews. It combines Xplor literature retrieval, an idea generator, and a Judge-based filter to simulate peer review and select promising ideas, releasing the OpenReview-derived dataset for reproducibility. The approach demonstrates modular integration of retrieval, generation, and evaluation and reports scalability to thousands of ideas. The work contributes to the computational creativity community by providing a concrete framework and data to study the generation and evaluation of scientific ideas.

Abstract

Recently, large language models (LLMs) have shown promising abilities to generate novel research ideas in science, a direction which coincides with many foundational principles in computational creativity (CC). In light of these developments, we present an idea generation system named Spark that couples retrieval-augmented idea generation using LLMs with a reviewer model named Judge trained on 600K scientific reviews from OpenReview. Our work is both a system demonstration and intended to inspire other CC researchers to explore grounding the generation and evaluation of scientific ideas within foundational CC principles. To this end, we release the annotated dataset used to train Judge, inviting other researchers to explore the use of LLMs for idea generation and creative evaluations.
Paper Structure (18 sections, 4 figures)

This paper contains 18 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Literature Retrieval via Xplor
  5. Text Embedding and Retrieval:
  6. Relevance Scoring and Maximal Marginal Relevance:
  7. Recursive Retrieval and Evidence Gathering Loop:
  8. Idea Generation with Spark Idea Generator
  9. Idea Evaluation with Spark Filter
  10. Evaluator Training: The Judge Model
  11. Data Acquisition and Preprocessing
  12. Multi-Task Training Framework
  13. Evaluation and Model Scaling
  14. Discussion and Limitations
  15. Looking Ahead: Computational Creativity and Scientific Idea Generation
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Spark’s end‑to‑end pipeline: Xplor performs embedding‑based, recursive literature retrieval with MMR re‑ranking; the Spark Idea Generator extracts key concepts and uses chain‑of‑thought LLM prompting to synthesize seed research proposals; and the Spark Filter applies our supervised fine-tuned judge model and a decision agent for automated peer‑style critique and ranking.
  • Figure 2: Spark Idea Generation pipeline: (A) Inputs, (B) Idea Generation Prompt, (C) LLM Response, (D) Generated Idea.
  • Figure 3: Spark Filter pipeline: (A) Input, (B) Filter Prompt, (C) LLM Response, (D) Generated Decision.
  • Figure 4: A research idea proposed by Spark, using context provided by Xplor about the use of LLMs for scientific creativity.