Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas

TL;DR

RINoBench is introduced, the first comprehensive benchmark for large-scale evaluation of research idea novelty judgments, which reveals that while LLM-generated reasoning closely mirrors human rationales, this alignment does not reliably translate into accurate novelty judgments, which diverge significantly from human gold standard judgments - even among leading reasoning-capable models.

Abstract

Judging the novelty of research ideas is crucial for advancing science, enabling the identification of unexplored directions, and ensuring contributions meaningfully extend existing knowledge rather than reiterate minor variations. However, given the exponential growth of scientific literature, manually judging the novelty of research ideas through literature reviews is labor-intensive, subjective, and infeasible at scale. Therefore, recent efforts have proposed automated approaches for research idea novelty judgment. Yet, evaluation of these approaches remains largely inconsistent and is typically based on non-standardized human evaluations, hindering large-scale, comparable evaluations. To address this, we introduce RINoBench, the first comprehensive benchmark for large-scale evaluation of research idea novelty judgments. It comprises 1,381 research ideas derived from and judged by human experts as well as nine automated evaluation metrics designed to assess both rubric-based novelty scores and textual justifications of novelty judgments. Using this benchmark, we evaluate several state-of-the-art large language models (LLMs) on their ability to judge the novelty of research ideas. Our findings reveal that while LLM-generated reasoning closely mirrors human rationales, this alignment does not reliably translate into accurate novelty judgments, which diverge significantly from human gold standard judgments - even among leading reasoning-capable models. Data and code available at: https://github.com/TimSchopf/RINoBench.

Figures (6)

• Figure 1: The task setup of rino. Given a research idea and its related works, a model must judge the novelty of the idea according to a five-point rubric. In addition, the model must provide a textual justification for its judgment, grounded in a comparison between the proposed research idea and the related works.
• Figure 2: Evaluation of justification alignment for novelty judgments using the G-Eval framework, which produces textual reasoning and a numerical score. We use only the numerical score for evaluation.
• Figure 3: Example illustrating known and novelty aspects in novelty judgment justifications. Known aspects refer to elements in a justification that highlight already established concepts or findings from previous work in a research idea. Novelty aspects denote elements in a justification that highlight new contributions of a research idea, which do not exist in prior work.
• Figure 4: Example of Known Aspects Recall and Novelty Aspects Recall for evaluation of novelty judgment justifications.
• Figure 5: Example evaluation of a model-generated novelty judgment justification using Additional Ratio and Hallucination Rate for known aspects and novelty aspects respectively.