The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models

TL;DR

The Heap tackles data contamination in code-model evaluation by providing a large multilingual code dataset deduplicated against standard training corpora and restricted to non-permissively licensed code. It combines explicit license filtering with rigorous exact and near deduplication using SHA-256 hashing and MinHash LSH, respectively, to minimize leakage from training data. The dataset spans 57 languages, is GitHub-derived with quality and metadata provisions, and is organized to support contamination-free, reproducible evaluations. This resource enables fair, scalable assessment of code LLMs and lays out a clear path for future enhancements in data curation, annotation, and language coverage.

Abstract

The recent rise in the popularity of large language models has spurred the development of extensive code datasets needed to train them. This has left limited code available for collection and use in the downstream investigation of specific behaviors, or evaluation of large language models without suffering from data contamination. To address this problem, we release The Heap, a large multilingual dataset covering 57 programming languages that has been deduplicated with respect to other open datasets of code, enabling researchers to conduct fair evaluations of large language models without significant data cleaning overhead.

Figures (1)

• Figure 1: Example of final dataset structure for one entry