When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research

TL;DR

SPOT introduces a challenging multimodal benchmark for automated verification of scientific manuscripts, compiling 83 modern papers with 91 author-validated errors across 10 domains and long, figure-rich contexts. The benchmark uses a rigorous, multi-stage data-curation and normalization pipeline to ensure credible ground truth, and evaluates a mix of proprietary and open LLMs on precision, recall, and pass@K metrics. Across eight trials, even the best models achieve only 6.1% precision and 21.1% recall (pass@1/4: 18.4%/37.8%), with confidence estimates that are poorly calibrated, highlighting a substantial reliability gap for AI-assisted academic verification. Case studies in mathematics and materials science corroborate these findings, showing category-specific weaknesses and student-like errors, and collectively argue for more robust, domain-aware verification methods in AI-driven scientific workflows.

Abstract

Recent advances in large language models (LLMs) have fueled the vision of automated scientific discovery, often called AI Co-Scientists. To date, prior work casts these systems as generative co-authors responsible for crafting hypotheses, synthesizing code, or drafting manuscripts. In this work, we explore a complementary application: using LLMs as verifiers to automate the \textbf{academic verification of scientific manuscripts}. To that end, we introduce SPOT, a dataset of 83 published papers paired with 91 errors significant enough to prompt errata or retraction, cross-validated with actual authors and human annotators. Evaluating state-of-the-art LLMs on SPOT, we find that none surpasses 21.1\% recall or 6.1\% precision (o3 achieves the best scores, with all others near zero). Furthermore, confidence estimates are uniformly low, and across eight independent runs, models rarely rediscover the same errors, undermining their reliability. Finally, qualitative analysis with domain experts reveals that even the strongest models make mistakes resembling student-level misconceptions derived from misunderstandings. These findings highlight the substantial gap between current LLM capabilities and the requirements for dependable AI-assisted academic verification.

Figures (21)

• Figure 1: Overview of Spot. Green indicates benchmark construction process, from seed collection through validation to normalization; blue indicates evaluation, where LLM outputs are compared to ground-truth errors and classified as true positives, false positives, or false negatives.
• Figure 2: Distribution of annotated errors by research domain and error type.
• Figure 3: Performance of o3 and Llama-4-Maverick across six challenging STEM benchmarks. The short red horizontal lines mark the gap $\Delta = \text{o3} - \text{Llama-4-Maverick}$ for each benchmark.
• Figure 4: Category-specific performance and calibration of six LLMs on Spot.Left: Kernel density estimates of each model’s reported confidence; all six models predominantly express very low confidence. Right: Scatter plot of mean reported confidence (see Appendix \ref{['app:confidence_for_passk']} for further details) versus $\mathrm{pass}@4$ for each model (color), broken down by error type (shape). The dashed diagonal marks perfect calibration.
• Figure 5: o3’s feedback on petersen2024.