LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content

TL;DR

LiveXiv introduces a contamination-free, live benchmark for evaluating multi-modal models using continuously harvested ArXiv content. It automates VQA generation from figures and tables, applies filtering to ensure true multimodal grounding, and employs an efficient IRT-based evaluation to estimate model performance across evolving versions while re-evaluating only a small subset of models. The first version demonstrates a challenging dataset with 16k questions across diverse domains, revealing strong performance by Claude-Sonnet and newer models, and highlighting domain sensitivity and bias considerations. The framework feasibly scales to ongoing updates and other scientific archives, offering a practical path to robust, contamination-free benchmarking of LMMs. The work contributes a scalable data pipeline, an efficient evaluation protocol, and empirical insights into model behavior on contemporary scientific content.

Abstract

The large-scale training of multi-modal models on data scraped from the web has shown outstanding utility in infusing these models with the required world knowledge to perform effectively on multiple downstream tasks. However, one downside of scraping data from the web can be the potential sacrifice of the benchmarks on which the abilities of these models are often evaluated. To safeguard against test data contamination and to truly test the abilities of these foundation models we propose LiveXiv: A scalable evolving live benchmark based on scientific ArXiv papers. LiveXiv accesses domain-specific manuscripts at any given timestamp and proposes to automatically generate visual question-answer pairs (VQA). This is done without any human-in-the-loop, using the multi-modal content in the manuscripts, like graphs, charts, and tables. Moreover, we introduce an efficient evaluation approach that estimates the performance of all models on the evolving benchmark using evaluations of only a subset of models. This significantly reduces the overall evaluation cost. We benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the first version of our benchmark, showing its challenging nature and exposing the models true abilities, avoiding contamination. Lastly, in our commitment to high quality, we have collected and evaluated a manually verified subset. By comparing its overall results to our automatic annotations, we have found that the performance variance is indeed minimal (<2.5%). Our dataset is available online on HuggingFace, and our code will be available here.

Figures (21)

• Figure 1: Static benchmark contamination. As training data increases, the risk for test set contamination grows and static benchmarks becomes saturated, reflecting falsely improved capabilities.
• Figure 2: We propose LiveXiv, a new method for generating Live multi-modal dataset for Visual Question-Answering based on ArXiv content. Our pipeline automatically generates scalable and reliable questions along with an efficient evaluation method to reduce the computational and logistic overheads required for continually evaluating past and present models on new versions of the dataset.
• Figure 3: Our live dataset generation consists of several stages. We first extract the images and their corresponding metadata ( i.e. captions and table contents), then we classify the figures into categories using meta-prompting. All the extracted data is then fed to GPT4o to generate multiple questions-answer pairs per image. Since generative models are prone to errors, we apply several filtering steps, using an LLM and LMM to ensure that our dataset is truly multi-modal and reliable.
• Figure 4: Average prediction errors and rank correlations for overall performance. We report MAE ($\pm$ mean absolute deviation) for non-re-evaluated models and Spearman’s rank correlation across 19 LMMs on different LiveXiv versions. Re-evaluating just 3–5 models is generally sufficient for accurate performance prediction.
• Figure 5: Average prediction errors and rank correlations when we predict performance for each domain separately. We report average MAE ($\pm$ average mean absolute deviation) for non-re-evaluated models and average Spearman’s rank correlation across 19 LMMs on different LiveXiv versions. A worse performance LiveXiv v3 can be explained by a smaller number of evaluations.