Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

C$^2$LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation

Yanyang Li, Tin Long Wong, Cheung To Hung, Jianqiao Zhao, Duo Zheng, Ka Wai Liu, Michael R. Lyu, Liwei Wang

TL;DR

C$^2$LEVA tackles the challenge of trustworthy LLM evaluation in the face of data contamination by introducing a comprehensive bilingual benchmark with 22 tasks that cover practical applications and core abilities. It advances contamination prevention through a combination of passive data renewal and a novel active protection strategy, including data licensing and watermarking, to reduce test-data leakage. The benchmark is validated on 15 models across English and Chinese, demonstrating strong alignment with ground-truth quality metrics while revealing limitations in current data protection approaches and cross-lingual transfer dynamics. The work offers a scalable, contamination-aware evaluation framework with a continuously maintained leaderboard, aiming to improve the reliability and interpretability of LLM assessments in real-world settings.

Abstract

Recent advances in large language models (LLMs) have shown significant promise, yet their evaluation raises concerns, particularly regarding data contamination due to the lack of access to proprietary training data. To address this issue, we present C$^2$LEVA, a comprehensive bilingual benchmark featuring systematic contamination prevention. C$^2$LEVA firstly offers a holistic evaluation encompassing 22 tasks, each targeting a specific application or ability of LLMs, and secondly a trustworthy assessment due to our contamination-free tasks, ensured by a systematic contamination prevention strategy that fully automates test data renewal and enforces data protection during benchmark data release. Our large-scale evaluation of 15 open-source and proprietary models demonstrates the effectiveness of C$^2$LEVA.

C$^2$LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation

TL;DR

CLEVA tackles the challenge of trustworthy LLM evaluation in the face of data contamination by introducing a comprehensive bilingual benchmark with 22 tasks that cover practical applications and core abilities. It advances contamination prevention through a combination of passive data renewal and a novel active protection strategy, including data licensing and watermarking, to reduce test-data leakage. The benchmark is validated on 15 models across English and Chinese, demonstrating strong alignment with ground-truth quality metrics while revealing limitations in current data protection approaches and cross-lingual transfer dynamics. The work offers a scalable, contamination-aware evaluation framework with a continuously maintained leaderboard, aiming to improve the reliability and interpretability of LLM assessments in real-world settings.

Abstract

Recent advances in large language models (LLMs) have shown significant promise, yet their evaluation raises concerns, particularly regarding data contamination due to the lack of access to proprietary training data. To address this issue, we present CLEVA, a comprehensive bilingual benchmark featuring systematic contamination prevention. CLEVA firstly offers a holistic evaluation encompassing 22 tasks, each targeting a specific application or ability of LLMs, and secondly a trustworthy assessment due to our contamination-free tasks, ensured by a systematic contamination prevention strategy that fully automates test data renewal and enforces data protection during benchmark data release. Our large-scale evaluation of 15 open-source and proprietary models demonstrates the effectiveness of CLEVA.

Paper Structure

This paper contains 50 sections, 9 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Missing a comprehensive task taxonomy.
  3. Overlooking the contamination risk.
  4. Related Work
  5. Reactive Contamination Detection.
  6. Preventative Contamination Mitigation.
  7. C$^2$LEVA
  8. Contamination Prevention Overview
  9. Discussion.
  10. C$^2$LEVA Task Taxonomy
  11. The Solution to Contamination Prevention
  12. Passive Prevention.
  13. Active Prevention.
  14. The Framework.
  15. Experimental Results
  16. ...and 35 more sections

Figures (9)

  • Figure 1: Contamination prevention overview. Solid lines indicate how data flows within a machine learning model development pipeline. The dotted line indicates where the test data leaks into the training data.
  • Figure 2: The task taxonomy of C$^2$LEVA.
  • Figure 3: The framework of C$^2$LEVA for contamination prevention.
  • Figure 4: The mean win rate of 15 models in 22 tasks of C$^2$LEVA.
  • Figure 5: Mean win rate comparison among models in different task groups. We choose the top-5 best-performing models in each language for visualization.
  • ...and 4 more figures