Scideator: Human-LLM Scientific Idea Generation Grounded in Research-Paper Facet Recombination

TL;DR

Scideator introduces a first-of-its-kind human-LLM interface for facet-based scientific ideation, grounding idea generation in recombination of purposes, mechanisms, and evaluations extracted from input and analogous papers. It combines three LLM-powered retrieval-augmented generation modules—Analogous Paper Facet Finder, Faceted Idea Generator, and Idea Novelty Checker—to support divergent idea exploration and convergent novelty evaluation. A within-subject user study (N=22 computer-science researchers) demonstrates significantly higher creativity support with Scideator than a strong baseline, especially for exploration and expressiveness, while a formative evaluation shows the novelty checker improves alignment with human novelty judgments. The results underscore the value of facet-based representations in enabling transparent, controllable, and novel scientific ideation, and point to avenues for improving novelty assessments and cross-domain applicability.

Abstract

The scientific ideation process often involves blending salient aspects of existing papers to create new ideas -- a framework known as facet-based ideation. To see how large language models (LLMs) might assist in this process, we contribute Scideator, the first human-LLM interface for facet-based scientific ideation. Starting from a user-provided set of scientific papers, Scideator extracts key facets -- purposes, mechanisms, and evaluations -- from these and related papers, allowing users to explore the idea space by interactively recombining facets to synthesize inventive ideas. Scideator also helps users gauge idea originality by searching the literature for overlaps, assessing idea novelty based on an explicit facet-based definition. To support these tasks, Scideator introduces three LLM-powered retrieval-augmented generation (RAG) modules: Analogous Paper Facet Finder, Faceted Idea Generator, and Idea Novelty Checker. In a within-subjects user study (N=22) with computer-science researchers comparing Scideator to a strong baseline, our tool provided significantly more creativity support, particularly with respect to exploration and expressiveness.

Figures (15)

• Figure 1: The Sci-deator workflow. 1) The user starts by providing an ideation topic and set of input papers as a starting point for ideation. 2) Sci-deator responds by retrieving analogous papers to the input papers and extracting facets (purpose, mechanism, and evaluation) from the input and analogous papers. (The evaluation facets are omitted in the figure for clarity, as it is not part of the main logic.) 3) The user then selects paper facets as well as adds their own facets for which they want to generate ideas. 4) Sci-deator recombines these selected facets into ideas with one purpose and one mechanism. If a purpose or mechanism facet is unspecified, the tool selects one. 5) The user selects an idea to assess for novelty. 6) Sci-deator classifies the idea as "novel" or "not novel" and provides a short rationale. 7) The user reviews the novelty classification and adjusts it if they disagree. 8) If the idea is deemed "not novel," Sci-deator suggests more novel ideas with one of the initial idea's facets replaced.
• Figure 2: Sci-deator's cold start. Above, the user selects or adds facets to generate ideas. They can also generate more facets to consider, and add custom instructions for the idea generation. Below, the user peruses their ideas and evaluates an idea for novelty by clicking the search icon to its left. The ideation topic here is human-AI collaboration in art.
• Figure 3: The Analogous Paper Facet Finder module. For a set of input papers, Sci-deator uses Semantic Scholar's API to retrieve similar papers (very near). It uses the input and very-near papers to create a summary of relevant works. Next, the tool extracts key facets from the input papers and determines the input papers' overarching purpose and mechanism, which it uses to come up with three queries for papers with an analogous purpose and mechanism. The queries are for analogous papers with varying distances from the input paper: same topic (near), same subarea (far), and different subarea (very far). Those queries are fed to the Semantic Scholar API to retrieve analogous papers. Finally, the facets of all the analogous papers are extracted by the LLM.
• Figure 4: Sci-deator's novelty assessment modal for one idea, which presents the idea (a) as well as its facets (b), related papers (c), adjustable novelty classification (d), and adjustable classification reason (e). When the idea is classified as "not novel," the system provides a set of three suggestions for more novel ideas (f), each of which replace one of the idea's original facets. The ideation topic here is human-AI collaboration in art.
• Figure 5: The Idea Novelty Checker module follows a retrieve-then-re-rank approach for novelty evaluation. In Step 1, it gathers a comprehensive set of papers relevant to an idea. This includes papers originally used to generate the idea, related papers, and additional papers retrieved through keyword and title searches extracted directly from the idea, as well as snippet searches using the entire idea as input. In Step 2, a two-stage re-ranking process is applied, where an embedding-based ranking strategy filters the large collection to top-$N$ papers, followed by a facet-based LLM re-ranker to identify the top-$k$ most relevant papers. In Step 3, these top-$k$ papers are used to assess the idea's novelty, guided by in-context examples that evaluate novelty with grounded reasoning. In Step 4, if an idea is classified as "not novel" by the tool or user, the LLM generates three idea suggestions, each replacing a different facet in the original idea in order to make the idea more novel compared to the relevant papers.