Can LLMs Help Uncover Insights about LLMs? A Large-Scale, Evolving Literature Analysis of Frontier LLMs

TL;DR

The paper tackles the rapid growth of LLM evaluations by introducing LLMEvalDB, a semi-automated, LLM-assisted pipeline for extracting experimental results from arXiv papers and organizing them into a dynamic, updatable dataset. It demonstrates substantial efficiency gains (over 93% reduction in manual effort) and validates the approach by reproducing key Chain-of-Thought findings while uncovering new insights, such as in-context learning benefits for coding and multimodal tasks and the nuanced interplay between CoT and demonstrations. The resulting LLMEvalDB enables fine-grained analyses of prompting behaviors across frontier LLMs, supports ongoing literature synthesis, and informs practical prompting decisions, albeit with acknowledged limitations in attribute descriptiveness and dataset canonicalization. Overall, the work contributes a scalable methodology and dataset that can drive continuous, evidence-based understanding of LLM behavior in a fast-moving research landscape.

Abstract

The surge of LLM studies makes synthesizing their findings challenging. Analysis of experimental results from literature can uncover important trends across studies, but the time-consuming nature of manual data extraction limits its use. Our study presents a semi-automated approach for literature analysis that accelerates data extraction using LLMs. It automatically identifies relevant arXiv papers, extracts experimental results and related attributes, and organizes them into a structured dataset, LLMEvalDB. We then conduct an automated literature analysis of frontier LLMs, reducing the effort of paper surveying and data extraction by more than 93% compared to manual approaches. We validate LLMEvalDB by showing that it reproduces key findings from a recent manual analysis of Chain-of-Thought (CoT) reasoning and also uncovers new insights that go beyond it, showing, for example, that in-context examples benefit coding & multimodal tasks but offer limited gains in math reasoning tasks compared to zero-shot CoT. Our automatically updatable dataset enables continuous tracking of target models by extracting evaluation studies as new data becomes available. Through LLMEvalDB and empirical analysis, we provide insights into LLMs while facilitating ongoing literature analyses of their behavior.

Figures (9)

• Figure 1: The diagram of our semi-automated literature analysis process and a key finding derived from data automatically extracted from the arXiv database.
• Figure 2: Data extraction pipeline overview with an extracted example from arXiv paper cheng2023batch. Target attributes are identified and extracted from the table, augmented with paper content.
• Figure 3: Dataset description generation pipeline overview using the example of "SVAMP" dataset patel2021nlp. Since LLM lacked knowledge of the given dataset, the model references the original dataset's arXiv source retrieved through extracted BibTex. If confident, however, the LLM's generated descriptions will be used.
• Figure 4: Log-scale frequency trends show a rapid increase of evaluation studies over successive quarters (Q). Reasoning remains the most popular evaluation category, while Multimodality exhibits recent rapid growth.
• Figure 5: CoT shows significant performance improvement over direct prompting in mathematical tasks, whereas its impact on reasoning tasks is less distinct due to their complexity and diversity. Grey dots indicate individual deltas (improvements), blue dots represent the mean delta per paper, and a purple star marks the mean delta for each category.