Automatic Detection of LLM-Generated Code: A Comparative Case Study of Contemporary Models Across Function and Class Granularities

TL;DR

This study systematically compares LLM-generated Python code across four contemporary models (GPT-3.5, GPT-OSS, Claude 3 Haiku, Claude Haiku 4.5) and two code granularities (function and class) using interpretable structural metrics. By generating parallel function- and class-level samples from CodeSearchNet and applying an intersection-based dataset curation, the authors show that detection signals are highly context- and model-dependent, with granularity exerting stronger influence than model architecture. They demonstrate that universal detectors are unlikely since differences flip signs across configurations, though a universal discriminator like the Comment-to-Code Ratio exists in principle—its strength varies by model. Their CatBoost detectors outperform a commercial baseline (GPTZero) on most configurations and reveal that detection performance degrades sharply on unseen future data, underscoring the need for continual, granularity-aware calibration. Practically, the work advocates for model- and granularity-specific detectors grounded in interpretable metrics rather than generic text-based signals, offering actionable guidance for software provenance and security in evolving coding ecosystems.

Abstract

The adoption of Large Language Models (LLMs) for code generation risks incorporating vulnerable code into software systems. Existing detectors face two critical limitations: a lack of systematic cross-model validation and opaque "black box" operation. We address this through a comparative study of code generated by four distinct LLMs: GPT-3.5, Claude 3 Haiku, Claude Haiku 4.5, and GPT-OSS. Analyzing 14,485 Python functions and 11,913 classes from the CodeSearchNet dataset, we generated corresponding code with all four LLMs. Using interpretable software metrics, we trained CatBoost classifiers for each configuration. Our analysis reveals that granularity effects dominate model differences by a factor of 8.6, with negligible feature overlap, indicating that function-level and class-level detection rely on fundamentally disjoint structural signatures. We discover critical granularity-dependent inversions: while modern models (Claude, GPT-OSS) are more detectable at the class level, GPT-3.5 is an anomaly that uniquely excels at the function level. SHAP analysis identifies the Comment-to-Code Ratio as the sole universal discriminator. However, its predictive magnitude varies drastically across models, explaining why detectors trained on specific LLMs fail to generalize. Our findings demonstrate that GPT-3.5's exceptional detectability (AUC-ROC 0.96) is unrepresentative of contemporary models (AUC-ROC approximately between 0.68 and 0.80). Robust detection requires moving beyond single-model studies to account for substantial diversity in structural fingerprints across architectures and granularities.

Figures (4)

• Figure 1: Feature frequency across eight configurations. Only Comment-to-Code Ratio (red) appears universally. Most features show model or granularity specificity.
• Figure 2: SHAP beeswarm plots for function-level detection across all four LLMs. Each point is a test instance; the x-axis position shows impact on prediction (left pushes toward "Human", right toward "LLM").
• Figure 3: SHAP beeswarm plots for class-level detection across all four LLMs. Compared to the function-level plots in Figure \ref{['fig:rq3_shap_function']}, feature importance is more distributed.
• Figure 4: Ranking based on the ScottKnott ESD tests of all trained models for each LLM-granularity configuration.