DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models

TL;DR

DATE-LM addresses the lack of standardized, LLM-centric evaluation for data attribution by introducing a unified, application-driven benchmark with a modular pipeline and public leaderboard. It evaluates attribution methods across three real-world tasks—training data selection, toxicity/bias filtering, and factual attribution—under both pre-training and fine-tuning scenarios, using carefully designed setups to mitigate biases and confounds. The large-scale study reveals that no single method dominates across all tasks and that simple non-attribution baselines can match or exceed attribution performance in some settings, while evaluation design significantly shapes outcomes. These findings highlight trade-offs between accuracy and cost, the importance of robust, task-aware evaluation, and the value of DATE-LM as a foundation for ongoing benchmarking and community engagement in data attribution research for LLMs.

Abstract

Data attribution methods quantify the influence of training data on model outputs and are becoming increasingly relevant for a wide range of LLM research and applications, including dataset curation, model interpretability, data valuation. However, there remain critical gaps in systematic LLM-centric evaluation of data attribution methods. To this end, we introduce DATE-LM (Data Attribution Evaluation in Language Models), a unified benchmark for evaluating data attribution methods through real-world LLM applications. DATE-LM measures attribution quality through three key tasks -- training data selection, toxicity/bias filtering, and factual attribution. Our benchmark is designed for ease of use, enabling researchers to configure and run large-scale evaluations across diverse tasks and LLM architectures. Furthermore, we use DATE-LM to conduct a large-scale evaluation of existing data attribution methods. Our findings show that no single method dominates across all tasks, data attribution methods have trade-offs with simpler baselines, and method performance is sensitive to task-specific evaluation design. Finally, we release a public leaderboard for quick comparison of methods and to facilitate community engagement, with the motivation that DATE-LM can serve as a foundation for future data attribution research in LLMs.

Figures (3)

• Figure 1: The unified DATE-LM evaluation framework and pipeline. Users select a data attribution method and LLM to evaluate on training data selection, toxcity/bias filtering, or factual attribution tasks. Results can be uploaded to the DATE-LM leaderboard.
• Figure 2: Averaged evaluation accuracy of pre-train data selection methods, each at increasing gumbel-top-k temperatures [0.1, 0.5, 1.0, 2.0] when applicable. 1B model size; two training stages (10k steps as early stage, 30k steps as mid-stage); two reference datasets (LAMBADA and FLAN)
• Figure 3: Top 100 filtered data in Heterogeneous setting for Llama3.2-1B trained models