LEGOBench: Scientific Leaderboard Generation Benchmark

TL;DR

LEGOBench addresses the information overload in scientific publishing by proposing an automatic leaderboard-generation benchmark built from two large-scale sources: the PwC leaderboards (PwC-LDB) and the arXiv Papers' Collection (APC). It defines three leaderboard-generation tasks—RPG, RPLM, and LGPLM—across six configurations to evaluate graph-based and language-model-based systems using a diverse set of datasets and networks. The authors evaluate encoder-only scientific LMs and decoder LLMs, revealing sizable performance gaps and highlighting challenges in candidate retrieval, full-text utilization, and robust extraction of leaderboard entries. The benchmark and datasets enable systematic assessment of automatic leaderboard generation and related information-extraction tasks, offering a foundation for future improvements in foundation-model capabilities and practical applications in citation analysis and manuscript evaluation.

Abstract

The ever-increasing volume of paper submissions makes it difficult to stay informed about the latest state-of-the-art research. To address this challenge, we introduce LEGOBench, a benchmark for evaluating systems that generate scientific leaderboards. LEGOBench is curated from 22 years of preprint submission data on arXiv and more than 11k machine learning leaderboards on the PapersWithCode portal. We present four graph-based and two language model-based leaderboard generation task configurations. We evaluate popular encoder-only scientific language models as well as decoder-only large language models across these task configurations. State-of-the-art models showcase significant performance gaps in automatic leaderboard generation on LEGOBench. The code is available on GitHub ( https://github.com/lingo-iitgn/LEGOBench ) and the dataset is hosted on OSF ( https://osf.io/9v2py/?view_only=6f91b0b510df498ba01595f8f278f94c ).

Figures (9)

• Figure 1: Organization of leaderboards in PwC. A leaderboard is constructed for a <dataset, task, metric> tuple. Leaderboards can contain additional metadata, such as the code repository link and model description tags.
• Figure 2: Pipeline for constructing the APC datasets. Blue boxes denote various datasets in the APC collection and the PwC-LDB dataset.
• Figure 3: The graph illustrates the exponential growth in the number of papers published monthly on arXiv from 1995 to 2022. This trend showcases the continuous expansion of research and knowledge in the academic community.
• Figure 4: A snapshot of the leaderboards from PwC showcasing top-performing models for Image Clustering on MNIST Dataset and ranked based on NMI (Normalized Mutual Information) metric. Image clustering in the MNIST dataset is the process of grouping similar handwritten digit images and the NMI metric measures how well the clusters align with the actual categories.
• Figure 5: Pipeline for ranking papers with content and graph for leaderboard generation (RPG).