Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncovering Pretraining Code in LLMs: A Syntax-Aware Attribution Approach

Yuanheng Li, Zhuoyang Chen, Xiaoyun Liu, Yuhao Wang, Mingwei Liu, Yang Shi, Kaifeng Huang, Shengjie Zhao

TL;DR

The paper addresses copyright and licensing concerns around code in large language model pretraining and the need for auditing memorization of code. It introduces SynPrune, a syntax-aware membership inference method that prunes tokens determined by Python grammar using 47 conventions and AST validation, and defines the SPP metric to quantify membership. The authors construct an authentic benchmark with 1,000 member and 1,000 non-member Python functions and demonstrate that SynPrune consistently outperforms state-of-the-art baselines across multiple models and function lengths, achieving an average AUROC improvement of ~15.4%. The work advances transparency and accountability for code pretraining and suggests that syntax aware techniques can generalize to other languages for copyright auditing and compliance.

Abstract

As large language models (LLMs) become increasingly capable, concerns over the unauthorized use of copyrighted and licensed content in their training data have grown, especially in the context of code. Open-source code, often protected by open source licenses (e.g, GPL), poses legal and ethical challenges when used in pretraining. Detecting whether specific code samples were included in LLM training data is thus critical for transparency, accountability, and copyright compliance. We propose SynPrune, a syntax-pruned membership inference attack method tailored for code. Unlike prior MIA approaches that treat code as plain text, SynPrune leverages the structured and rule-governed nature of programming languages. Specifically, it identifies and excludes consequent tokens that are syntactically required and not reflective of authorship, from attribution when computing membership scores. Experimental results show that SynPrune consistently outperforms the state-of-the-arts. Our method is also robust across varying function lengths and syntax categories.

Uncovering Pretraining Code in LLMs: A Syntax-Aware Attribution Approach

TL;DR

The paper addresses copyright and licensing concerns around code in large language model pretraining and the need for auditing memorization of code. It introduces SynPrune, a syntax-aware membership inference method that prunes tokens determined by Python grammar using 47 conventions and AST validation, and defines the SPP metric to quantify membership. The authors construct an authentic benchmark with 1,000 member and 1,000 non-member Python functions and demonstrate that SynPrune consistently outperforms state-of-the-art baselines across multiple models and function lengths, achieving an average AUROC improvement of ~15.4%. The work advances transparency and accountability for code pretraining and suggests that syntax aware techniques can generalize to other languages for copyright auditing and compliance.

Abstract

As large language models (LLMs) become increasingly capable, concerns over the unauthorized use of copyrighted and licensed content in their training data have grown, especially in the context of code. Open-source code, often protected by open source licenses (e.g, GPL), poses legal and ethical challenges when used in pretraining. Detecting whether specific code samples were included in LLM training data is thus critical for transparency, accountability, and copyright compliance. We propose SynPrune, a syntax-pruned membership inference attack method tailored for code. Unlike prior MIA approaches that treat code as plain text, SynPrune leverages the structured and rule-governed nature of programming languages. Specifically, it identifies and excludes consequent tokens that are syntactically required and not reflective of authorship, from attribution when computing membership scores. Experimental results show that SynPrune consistently outperforms the state-of-the-arts. Our method is also robust across varying function lengths and syntax categories.

Paper Structure

This paper contains 15 sections, 5 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Preliminary
  5. Syntax Conventions
  6. Syntax-Pruned MIA
  7. Benchmark Construction
  8. Experiments
  9. Setup
  10. Results
  11. Robustness under Different Function Lengths
  12. Ablation Study
  13. Threshold Analysis
  14. Limitations
  15. Conclusions

Figures (2)

  • Figure 1: Illustration of our Approach Motivation and Proposed Solution
  • Figure 2: F1-Score of SynPrune Across Varying Thresholds