Evaluating Software Plagiarism Detection in the Age of AI: Automated Obfuscation and Lessons for Academic Integrity

TL;DR

The paper evaluates how software plagiarism detectors fare against automated obfuscation attacks, including AI-driven transformations and AI-generated code. By implementing two attack-agnostic defenses—Token Sequence Normalization and Subsequence Match Merging—and testing their combination across six real-world datasets, the study demonstrates substantial gains in obfuscation resilience and even improved detection of AI-generated programs. It uses a robust methodology that avoids threshold bias and employs rigorous statistics to quantify practical significance, showing that defense-in-depth can markedly improve plagiarism detection without inflating false positives. The findings have practical implications for academic integrity in CS education, highlighting both the effectiveness of current defenses and the need for ongoing vigilance as AI-enabled obfuscation evolves, while stressing the continued importance of human review in adjudicating suspected cases.

Abstract

Plagiarism in programming assignments is a persistent issue in computer science education, increasingly complicated by the emergence of automated obfuscation attacks. While software plagiarism detectors are widely used to identify suspicious similarities at scale and are resilient to simple obfuscation techniques, they are vulnerable to advanced obfuscation based on structural modification of program code that preserves the original program behavior. While different defense mechanisms have been proposed to increase resilience against these attacks, their current evaluation is limited to the scope of attacks used and lacks a comprehensive investigation regarding AI-based obfuscation. In this paper, we investigate the resilience of these defense mechanisms against a broad range of automated obfuscation attacks, including both algorithmic and AI-generated methods, and for a wide variety of real-world datasets. We evaluate the improvements of two defense mechanisms over the plagiarism detector JPlag across over four million pairwise program comparisons. Our results show significant improvements in detecting obfuscated plagiarism instances, and we observe an improved detection of AI-generated programs, even though the defense mechanisms are not designed for this use case. Based on our findings, we provide an in-depth discussion of their broader implications for academic integrity and the role of AI in education.

Figures (9)

• Figure 1: Categorization of obfuscation attacks corresponding to the clone types by Karnalim2016 based on Faidhi1987, illustrating the effectiveness and applicability corresponding to the obfuscation complexity. Based on Saglam2025.
• Figure 2: Difference metrics comparing the separation between plagiarism instances and original student solutions based on the median and interquartile range (IQR) distances measured in percentage points.
• Figure 3: Visualization of alternative hypotheses ($H1$) for one-sided significance tests between plagiarism and original pairs. For plagiarism, the shift should be significantly greater ($H1_{Plag}$), while for originals, it should not ($H1_{Orig}$).
• Figure 4: Obfuscation vulnerability illustrated for the three token-based approaches JPlag, Dolos, and MOSS with the dataset PROGpedia-19 paiva2023 and plagiarism instances automatically obfuscated via statement insertion.
• Figure 5: Similarity scores for original program pairs and insertion-based plagiarism pairs. Ideally, plagiarism pairs exhibit high similarity, while original pairs should exhibit low similarity.