Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language model developers should report train-test overlap

Andy K Zhang, Kevin Klyman, Yifan Mai, Yoav Levine, Yian Zhang, Rishi Bommasani, Percy Liang

TL;DR

This paper argues that train-test overlap undermines the interpretability of benchmark results for language models and advocates for transparency through public reporting of overlap statistics or release of training data. It surveys current practices across 30 flagship developers, finding only 9 with adequate overlap disclosure, and reviews existing strategies (black-box estimation, private sets, novel data, canary strings) while highlighting their limitations. The authors propose a public protocol and standardization effort to improve comparability of overlap statistics, aiming to increase trust in evaluation results. The work highlights the practical impact of transparent reporting on accountability and progress in the AI evaluation ecosystem.

Abstract

Language models are extensively evaluated, but correctly interpreting evaluation results requires knowledge of train-test overlap which refers to the extent to which the language model is trained on the very data it is being tested on. The public currently lacks adequate information about train-test overlap: most models have no public train-test overlap statistics, and third parties cannot directly measure train-test overlap since they do not have access to the training data. To make this clear, we document the practices of 30 model developers, finding that just 9 developers report train-test overlap: 4 developers release training data under open-source licenses, enabling the community to directly measure train-test overlap, and 5 developers publish their train-test overlap methodology and statistics. By engaging with language model developers, we provide novel information about train-test overlap for three additional developers. Overall, we take the position that language model developers should publish train-test overlap statistics and/or training data whenever they report evaluation results on public test sets. We hope our work increases transparency into train-test overlap to increase the community-wide trust in model evaluations.

Language model developers should report train-test overlap

TL;DR

This paper argues that train-test overlap undermines the interpretability of benchmark results for language models and advocates for transparency through public reporting of overlap statistics or release of training data. It surveys current practices across 30 flagship developers, finding only 9 with adequate overlap disclosure, and reviews existing strategies (black-box estimation, private sets, novel data, canary strings) while highlighting their limitations. The authors propose a public protocol and standardization effort to improve comparability of overlap statistics, aiming to increase trust in evaluation results. The work highlights the practical impact of transparent reporting on accountability and progress in the AI evaluation ecosystem.

Abstract

Language models are extensively evaluated, but correctly interpreting evaluation results requires knowledge of train-test overlap which refers to the extent to which the language model is trained on the very data it is being tested on. The public currently lacks adequate information about train-test overlap: most models have no public train-test overlap statistics, and third parties cannot directly measure train-test overlap since they do not have access to the training data. To make this clear, we document the practices of 30 model developers, finding that just 9 developers report train-test overlap: 4 developers release training data under open-source licenses, enabling the community to directly measure train-test overlap, and 5 developers publish their train-test overlap methodology and statistics. By engaging with language model developers, we provide novel information about train-test overlap for three additional developers. Overall, we take the position that language model developers should publish train-test overlap statistics and/or training data whenever they report evaluation results on public test sets. We hope our work increases transparency into train-test overlap to increase the community-wide trust in model evaluations.

Paper Structure

This paper contains 31 sections, 3 equations, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Alternative Views
  3. Black-box methods
  4. Private test sets
  5. Novel test sets
  6. Canary strings
  7. Our position:
  8. Consequences of Not Reporting Train-Test Overlap
  9. Language Models
  10. Results
  11. Documenting Current Practices
  12. Scoring Criteria
  13. Scores
  14. Discussion
  15. Future Work: Standardization of Train-Test Overlap
  16. ...and 16 more sections

Figures (1)

  • Figure 1: Protocol for computing train-test overlap. The first party takes private training data and public test data to compute overlap statistics at the instance level. This is sent to the third party, which aggregates statistics to the dataset level and publishes the results publicly.