League: Leaderboard Generation on Demand

TL;DR

The paper tackles the challenge of keeping research leaderboards up-to-date in rapidly evolving AI fields by introducing League, a four-stage framework that automatically collects papers, extracts and classifies experiment tables, unpacks results and settings into structured quintuples, and generates refined leaderboards evaluated for topic relevance and content quality. By leveraging LLMs for classification, extraction, and refinement, League ensures fair comparisons through settings-aware data and enables dynamic updates. Empirical results show League produces high-quality leaderboards with strong recall and precision, near-human content quality, and substantial time savings (minutes instead of hours), with robust correlations to human judgments. This approach offers a scalable, reproducible, and timely tool for researchers to monitor progress and compare methods across datasets and tasks, addressing core challenges in automatic leaderboard construction.

Abstract

This paper introduces Leaderboard Auto Generation (LAG), a novel and well-organized framework for automatic generation of leaderboards on a given research topic in rapidly evolving fields like Artificial Intelligence (AI). Faced with a large number of AI papers updated daily, it becomes difficult for researchers to track every paper's proposed methods, experimental results, and settings, prompting the need for efficient automatic leaderboard construction. While large language models (LLMs) offer promise in automating this process, challenges such as multi-document summarization, leaderboard generation, and experiment fair comparison still remain under exploration. LAG solves these challenges through a systematic approach that involves the paper collection, experiment results extraction and integration, leaderboard generation, and quality evaluation. Our contributions include a comprehensive solution to the leaderboard construction problem, a reliable evaluation method, and experimental results showing the high quality of leaderboards.

Figures (8)

• Figure 1: Left: Growth trend of paper and leaderboard submissions on LLMs from 2022 to 2025-09. The leaderboard statistics are collected from Paper with Code. Right: Leaderboard of Multi-hop QA, the latest method is still stuck in 2023.
• Figure 2: The League framework for leaderboard automatic generation. In Stage 1, we automatically crawl scientific papers from arXiv. In Stage 2, we retrieve, extract, and classify tables from the LaTeX code. In Stage 3, we select the main results tables and extract datasets, metrics, results, and experiment settings from the main results table. In Stage 4, we generate Leaderboards from the selected results and evaluate the quality.
• Figure 3: The example leaderboard generated by League. Comparing with the Leaderboard of Multi-hop QA method in the right part of Figure \ref{['paper_submission_growth']}, our method could help summarize the experiment results from the top conference papers with experiment settings for fair comparison.
• Figure 4: Left: Impact of Iteration on League Performance. Right: Pearson Correlation Coefficient values given by four LLMs and human experts. Note that the Pearson Correlation Coefficient is between -1 and 1, the larger value indicates more positive correlations.
• Figure 5: A leaderboard (20 lines) of semi-supervised medical image segmentation on the LA dataset, using GPT-4o for table extraction and Qwen2.5-14B for leaderboard construction & refinement.