AICD Bench: A Challenging Benchmark for AI-Generated Code Detection

TL;DR

AICD Bench tackles the gap in AI-generated code detection by delivering a large-scale, multi-dimensional benchmark that evaluates detectors across languages, generator families, and adversarial/hybrid code. It introduces three tasks—robust binary classification under distribution shifts, model-family attribution, and fine-grained human–machine classification—anchored by a dataset of approximately $2M$ samples from $77$ generators spanning $9$ languages. Experiments with encoder-based and classical baselines reveal persistent generalization gaps, especially under domain shifts and for hybrid/adversarial code, underscoring the need for more robust detection approaches. The work also provides a standardized, extensible evaluation framework and outlines future directions toward adversarial and domain-adaptive training, meta-models, and ongoing benchmark expansion.

Abstract

Large language models (LLMs) are increasingly capable of generating functional source code, raising concerns about authorship, accountability, and security. While detecting AI-generated code is critical, existing datasets and benchmarks are narrow, typically limited to binary human-machine classification under in-distribution settings. To bridge this gap, we introduce $\emph{AICD Bench}$, the most comprehensive benchmark for AI-generated code detection. It spans $\emph{2M examples}$, $\emph{77 models}$ across $\emph{11 families}$, and $\emph{9 programming languages}$, including recent reasoning models. Beyond scale, AICD Bench introduces three realistic detection tasks: ($\emph{i}$)~$\emph{Robust Binary Classification}$ under distribution shifts in language and domain, ($\emph{ii}$)~$\emph{Model Family Attribution}$, grouping generators by architectural lineage, and ($\emph{iii}$)~$\emph{Fine-Grained Human-Machine Classification}$ across human, machine, hybrid, and adversarial code. Extensive evaluation on neural and classical detectors shows that performance remains far below practical usability, particularly under distribution shift and for hybrid or adversarial code. We release AICD Bench as a $\emph{unified, challenging evaluation suite}$ to drive the next generation of robust approaches for AI-generated code detection. The data and the code are available at https://huggingface.co/AICD-bench}.

Figures (11)

• Figure 1: Task 2 (Model Family Attribution): ModernBERT confusion matrix. The values are row-normalized percentages, showing the proportion of each true class assigned to each predicted class.
• Figure 2: Task 3 (Fine-Grained Human-Machine Classification): ModernBERT confusion matrix. The values are row-normalized percentages, showing the proportion of each true class assigned to each predicted class.
• Figure 3: Distribution of code properties.
• Figure 4: Task 1 (Robust Binary Classification): detectors' language-wise performance.
• Figure 5: Task 1 (Robust Binary Classification): performance evaluation of the detectors.