Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Contamination Resistant Benchmarks

Rahmatullah Musawi, Sheng Lu

TL;DR

The paper tackles data contamination as a critical flaw in evaluating large language models, proposing contamination resistance as a formal benchmark criterion. It introduces a dynamic Caesar cipher benchmark that tests logical deduction, arithmetic reasoning, and generalization across shifts $[3,6,9,12]$, with both natural and random plain texts to probe memorization and generalization. Experimental results show that most models struggle when contamination is controlled, with GPT-4o achieving limited exact-match success and strong evidence of contamination effects and lack of generalization in other models. The work argues for contamination-resistant benchmarks as a more reliable, updateable framework for evaluating true model capabilities, with implications for safety and responsible deployment. It also highlights the need for further exploration of dynamic benchmarks, prompt design, and advanced reasoning techniques in a contamination-aware setting.

Abstract

The rapid development of large language models (LLMs) has transformed the landscape of natural language processing. Evaluating LLMs properly is crucial for understanding their potential and addressing concerns such as safety. However, LLM evaluation is confronted by various factors, among which contamination stands out as a key issue that undermines the reliability of evaluations. In this work, we introduce the concept of contamination resistance to address this challenge. We propose a benchmark based on Caesar ciphers (e.g., "ab" to "bc" when the shift is 1), which, despite its simplicity, is an excellent example of a contamination resistant benchmark. We test this benchmark on widely used LLMs under various settings, and we find that these models struggle with this benchmark when contamination is controlled. Our findings reveal issues in current LLMs and raise important questions regarding their true capabilities. Our work contributes to the development of contamination resistant benchmarks, enabling more rigorous LLM evaluation and offering insights into the true capabilities and limitations of LLMs.

Towards Contamination Resistant Benchmarks

TL;DR

The paper tackles data contamination as a critical flaw in evaluating large language models, proposing contamination resistance as a formal benchmark criterion. It introduces a dynamic Caesar cipher benchmark that tests logical deduction, arithmetic reasoning, and generalization across shifts , with both natural and random plain texts to probe memorization and generalization. Experimental results show that most models struggle when contamination is controlled, with GPT-4o achieving limited exact-match success and strong evidence of contamination effects and lack of generalization in other models. The work argues for contamination-resistant benchmarks as a more reliable, updateable framework for evaluating true model capabilities, with implications for safety and responsible deployment. It also highlights the need for further exploration of dynamic benchmarks, prompt design, and advanced reasoning techniques in a contamination-aware setting.

Abstract

The rapid development of large language models (LLMs) has transformed the landscape of natural language processing. Evaluating LLMs properly is crucial for understanding their potential and addressing concerns such as safety. However, LLM evaluation is confronted by various factors, among which contamination stands out as a key issue that undermines the reliability of evaluations. In this work, we introduce the concept of contamination resistance to address this challenge. We propose a benchmark based on Caesar ciphers (e.g., "ab" to "bc" when the shift is 1), which, despite its simplicity, is an excellent example of a contamination resistant benchmark. We test this benchmark on widely used LLMs under various settings, and we find that these models struggle with this benchmark when contamination is controlled. Our findings reveal issues in current LLMs and raise important questions regarding their true capabilities. Our work contributes to the development of contamination resistant benchmarks, enabling more rigorous LLM evaluation and offering insights into the true capabilities and limitations of LLMs.
Paper Structure (19 sections, 2 figures, 15 tables)

This paper contains 19 sections, 2 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Model capabilities
  4. Model evaluation
  5. A contamination resistant benchmark based on Caesar ciphers
  6. Experimental setup
  7. Results
  8. The effect of shift
  9. The effect of plain text
  10. The effect of prompt
  11. The effect of output position
  12. The effect of shot
  13. Advanced reasoning models
  14. Integrity check
  15. Conclusion
  16. ...and 4 more sections

Figures (2)

  • Figure 1: We propose the concept of contamination resistance and establish a contamination resistant benchmark based on Caesar ciphers, which meets the following criteria: (a) this benchmark tests several capabilities, including logical deduction, arithmetic reasoning, and generalization; (b) this benchmark is dynamic, evolving over time so that an LLM can not memorize all possible queries, and the level of task complexity is consistent; (c) it is effortless to curate new test instances, making it more update-able than static benchmarks.
  • Figure 2: Exact match accuracy (EMA); GPT-4o; the zero-shot setting.