Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data Contamination Report from the 2024 CONDA Shared Task

Oscar Sainz, Iker García-Ferrero, Alon Jacovi, Jon Ander Campos, Yanai Elazar, Eneko Agirre, Yoav Goldberg, Wei-Lin Chen, Jenny Chim, Leshem Choshen, Luca D'Amico-Wong, Melissa Dell, Run-Ze Fan, Shahriar Golchin, Yucheng Li, Pengfei Liu, Bhavish Pahwa, Ameya Prabhu, Suryansh Sharma, Emily Silcock, Kateryna Solonko, David Stap, Mihai Surdeanu, Yu-Min Tseng, Vishaal Udandarao, Zengzhi Wang, Ruijie Xu, Jinglin Yang

TL;DR

The paper addresses data contamination in NLP, where evaluation data leaks into pre-training data, by presenting the CONDA 2024 Shared Task and the Data Contamination Database. It is built on an open, pull-request-driven methodology that collects 566 contamination entries from 23 contributors across 42 contaminated sources and 91 datasets, distinguishing data-based and model-based detection approaches. Key findings show contamination is prevalent in popular benchmarks and datasets, with detailed breakdowns for contaminated corpora and models, and temporal patterns that link newer models with newer data. The work provides a living, centralized resource to help researchers avoid citing contaminated resources and to understand the current landscape of data contamination in NLP.

Abstract

The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform. The platform continues to be online, open to contributions from the community.

Data Contamination Report from the 2024 CONDA Shared Task

TL;DR

The paper addresses data contamination in NLP, where evaluation data leaks into pre-training data, by presenting the CONDA 2024 Shared Task and the Data Contamination Database. It is built on an open, pull-request-driven methodology that collects 566 contamination entries from 23 contributors across 42 contaminated sources and 91 datasets, distinguishing data-based and model-based detection approaches. Key findings show contamination is prevalent in popular benchmarks and datasets, with detailed breakdowns for contaminated corpora and models, and temporal patterns that link newer models with newer data. The work provides a living, centralized resource to help researchers avoid citing contaminated resources and to understand the current landscape of data contamination in NLP.

Abstract

The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform. The platform continues to be online, open to contributions from the community.
Paper Structure (11 sections, 7 figures, 2 tables)

This paper contains 11 sections, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology and Previous Work
  3. Data-based approaches.
  4. Model-based approaches.
  5. Compilation of Evidence
  6. Contaminated corpora.
  7. Contaminated models.
  8. Analysis of the Reported Data
  9. Reported tasks.
  10. Year of publication of the reported data.
  11. Conclusions

Figures (7)

  • Figure 1: Taxonomy of papers that report contamination evidence. Including LLM's papers and technical reports, papers about methods for detecting contamination, and papers about corpus analysis.
  • Figure 2: Number of test sets reported for each corpus often used in pre-training.
  • Figure 3: Number of test sets reported for each pre-trained model.
  • Figure 4: Percentage of contaminated report per task
  • Figure 5: Number of downloads in the HuggingFace hub of the datasets in the report.
  • ...and 2 more figures