Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting

Tong Ye, Yangkai Du, Tengfei Ma, Lingfei Wu, Xuhong Zhang, Shouling Ji, Wenhai Wang

TL;DR

This work tackles the challenge of detecting LLM-generated code in zero-shot, black-box settings where token-probability signals are unreliable due to code's low-entropy tokens. It proposes a novel detector that rewrites code with an LLM and measures similarity between the original and rewritten versions using a self-supervised code-similarity model trained via SimCSE on code, yielding a robust detection score. The approach achieves substantial AUROC improvements over strong baselines on APPS and MBPP and generalizes across multiple languages and generation tools, with ablations confirming the critical roles of code rewriting and similarity measurement. The results highlight the practical potential of rewriting-based, language-aware detection for safeguarding educational and industrial code use from synthetic content, while remaining effective under various prompts, decodings, and even revised synthetic code scenarios.

Abstract

Large Language Models (LLMs) have demonstrated remarkable proficiency in generating code. However, the misuse of LLM-generated (synthetic) code has raised concerns in both educational and industrial contexts, underscoring the urgent need for synthetic code detectors. Existing methods for detecting synthetic content are primarily designed for general text and struggle with code due to the unique grammatical structure of programming languages and the presence of numerous ''low-entropy'' tokens. Building on this, our work proposes a novel zero-shot synthetic code detector based on the similarity between the original code and its LLM-rewritten variants. Our method is based on the observation that differences between LLM-rewritten and original code tend to be smaller when the original code is synthetic. We utilize self-supervised contrastive learning to train a code similarity model and evaluate our approach on two synthetic code detection benchmarks. Our results demonstrate a significant improvement over existing SOTA synthetic content detectors, with AUROC scores increasing by 20.5% on the APPS benchmark and 29.1% on the MBPP benchmark.

Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting

TL;DR

This work tackles the challenge of detecting LLM-generated code in zero-shot, black-box settings where token-probability signals are unreliable due to code's low-entropy tokens. It proposes a novel detector that rewrites code with an LLM and measures similarity between the original and rewritten versions using a self-supervised code-similarity model trained via SimCSE on code, yielding a robust detection score. The approach achieves substantial AUROC improvements over strong baselines on APPS and MBPP and generalizes across multiple languages and generation tools, with ablations confirming the critical roles of code rewriting and similarity measurement. The results highlight the practical potential of rewriting-based, language-aware detection for safeguarding educational and industrial code use from synthetic content, while remaining effective under various prompts, decodings, and even revised synthetic code scenarios.

Abstract

Large Language Models (LLMs) have demonstrated remarkable proficiency in generating code. However, the misuse of LLM-generated (synthetic) code has raised concerns in both educational and industrial contexts, underscoring the urgent need for synthetic code detectors. Existing methods for detecting synthetic content are primarily designed for general text and struggle with code due to the unique grammatical structure of programming languages and the presence of numerous ''low-entropy'' tokens. Building on this, our work proposes a novel zero-shot synthetic code detector based on the similarity between the original code and its LLM-rewritten variants. Our method is based on the observation that differences between LLM-rewritten and original code tend to be smaller when the original code is synthetic. We utilize self-supervised contrastive learning to train a code similarity model and evaluate our approach on two synthetic code detection benchmarks. Our results demonstrate a significant improvement over existing SOTA synthetic content detectors, with AUROC scores increasing by 20.5% on the APPS benchmark and 29.1% on the MBPP benchmark.
Paper Structure (43 sections, 4 equations, 10 figures, 6 tables, 1 algorithm)

This paper contains 43 sections, 4 equations, 10 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Challenges in Applying Existing Detection Methods to Code.
  3. Observations and Hypothesis.
  4. Related Works
  5. LLMs for Code Generation.
  6. Detection of Synthetic Text.
  7. Methodology
  8. Problem Definition.
  9. Intuition and Hypothesis.
  10. Code Rewriting.
  11. Similarity Measurement.
  12. Experiment Setting
  13. Benchmarks.
  14. Choice of Generation Tools.
  15. Constructing APPS benchmark.
  16. ...and 28 more sections

Figures (10)

  • Figure 1: Token Entropy Distribution in General Texts vs. Code, Estimated by Llama-13B touvron2023llamaopenefficientfoundation.
  • Figure 2: Rewriting Original Code (synthetic & human-writting) with GPT-3.5-Turbo.
  • Figure 3: Prompt for Code Rewriting.
  • Figure 4: Prompt templates for synthetic code generation. We use chain-of-thought prompting for reporting main results. We use direct prompting as part of model analysis for the impact of generation prompts.
  • Figure 5: Impact of generation prompts.
  • ...and 5 more figures