Whitespaces Don't Lie: Feature-Driven and Embedding-Based Approaches for Detecting Machine-Generated Code

TL;DR

This work addresses the problem of distinguishing human-written from machine-generated code by comparing two detection paradigms: lightweight handcrafted feature-based detectors and embedding-based detectors using pretrained code encoders. Using a large benchmark, feature-based models achieve strong recall and F1 (ROC-AUC and PR-AUC around $0.995$, F1 ≈ $0.971$), while CodeBERT embeddings yield nearly identical ROC-AUC/PR-AUC with slightly lower F1 but higher precision. The analysis reveals whitespace- and indentation-driven cues as highly discriminative, with embeddings capturing deeper semantic patterns and offering robustness to distribution shifts. The findings advocate for hybrid detection strategies that balance interpretability, efficiency, and robustness for academic integrity and responsible AI deployment.

Abstract

Large language models (LLMs) have made it remarkably easy to synthesize plausible source code from natural language prompts. While this accelerates software development and supports learning, it also raises new risks for academic integrity, authorship attribution, and responsible AI use. This paper investigates the problem of distinguishing human-written from machine-generated code by comparing two complementary approaches: feature-based detectors built from lightweight, interpretable stylometric and structural properties of code, and embedding-based detectors leveraging pretrained code encoders. Using a recent large-scale benchmark dataset of 600k human-written and AI-generated code samples, we find that feature-based models achieve strong performance (ROC-AUC 0.995, PR-AUC 0.995, F1 0.971), while embedding-based models with CodeBERT embeddings are also very competitive (ROC-AUC 0.994, PR-AUC 0.994, F1 0.965). Analysis shows that features tied to indentation and whitespace provide particularly discriminative cues, whereas embeddings capture deeper semantic patterns and yield slightly higher precision. These findings underscore the trade-offs between interpretability and generalization, offering practical guidance for deploying robust code-origin detection in academic and industrial contexts.

Figures (2)

• Figure 1: Radar plot comparing the best feature-based model (Random Forest) and embedding-based model (Logistic Regression) on the test set. PR--AUC and ROC--AUC are nearly identical, but the feature-based model achieves higher accuracy, F1 and recall, while the embedding-based model achieves slightly better precision. The plot is zoomed into the range 0.95--1.00 to better highlight subtle differences between the models.
• Figure 2: Feature importance ranking from the Random Forest classifier trained on handcrafted metrics. The most discriminative features include average leading spaces, average leading tabs, and blank-line ratio, followed by AST depth. These results highlight that stylistic and structural cues such as indentation habits and whitespace patterns carry strong signals for distinguishing human-written from machine-generated code.