Automatic Evaluation Metrics for Artificially Generated Scientific Research

TL;DR

This work investigates automatic evaluation metrics for AI-generated scientific content by predicting citation counts and review scores. It builds a unified OpenReview-ACL-OCL dataset with augmented metadata, including hypotheses, and evaluates content- and context-based predictors using SPECTER2 embeddings, comparing with LLM reviewers and human judgments. Citation-count prediction emerges as more reliable than review-score prediction, and a simple title+abstract representation can outperform LLM-based reviewers, though human consistency remains superior. The findings highlight the potential and limitations of automated metrics for assessing scientific quality and emphasize the need for standardized cross-venue evaluation pipelines and larger, richer datasets. Overall, automatic metrics offer scalable proxies for scientific impact but cannot yet fully replace human expertise in quality assessment.

Abstract

Foundation models are increasingly used in scientific research, but evaluating AI-generated scientific work remains challenging. While expert reviews are costly, large language models (LLMs) as proxy reviewers have proven to be unreliable. To address this, we investigate two automatic evaluation metrics, specifically citation count prediction and review score prediction. We parse all papers of OpenReview and augment each submission with its citation count, reference, and research hypothesis. Our findings reveal that citation count prediction is more viable than review score prediction, and predicting scores is more difficult purely from the research hypothesis than from the full paper. Furthermore, we show that a simple prediction model based solely on title and abstract outperforms LLM-based reviewers, though it still falls short of human-level consistency.

Figures (14)

• Figure 1: Architecture of the context model in case of the full paper representation. The paper representation is green (TA=title abstract) and the context representation is blue (RW=Related Work, M=Methodology, E&R=Experiments and Results, C=Conclusion).
• Figure 2: Pearson correlation heat map for the different dimensions of our unified review data model.
• Figure 3: Spearman correlations for context-based models applied to both the regression and pairwise comparison tasks. Results are averaged over five seeds, with error bars representing the standard deviations.
• Figure 4: Scatter plots of predicted review scores and groundtruth review scores on a subset of the test set of ICLR-2024 for the Sakana reviewer, the review score prediction model and human reviews. For human reviews, we randomly select a review as the predicted score and average over the rest.
• Figure 5: Schematic overview of the scientific Paper object. The Field of Study is a list of keywords that are part of the OpenReview submission, where the potential values depend on the venue. Number of citations and influential citations are retrieved from Semantic Scholar.