Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AST-PAC: AST-guided Membership Inference for Code

Roham Koohestani, Ali Al-Kaswan, Jonathan Katzy, Maliheh Izadi

TL;DR

The paper tackles data provenance auditing for code language models by evaluating membership inference attacks in a grey-box setting. It systematically compares Loss Attack and Polarized Augment Calibration (PAC) on 3B–7B code models and introduces AST-PAC, a syntax-aware variant using AST-guided perturbations to preserve code syntax during calibration. Key findings show PAC generally outperforms Loss on code-specialized models, but its effectiveness degrades for very large or highly complex files; AST-PAC improves robustness in those regimes while underperforming on small or alphanumeric-rich code. The work suggests that syntax-aware, size-adaptive calibration is crucial for reliable provenance auditing of code LLMs, with implications for governance and copyright compliance in code data usage.

Abstract

Code Large Language Models are frequently trained on massive datasets containing restrictively licensed source code. This creates urgent data governance and copyright challenges. Membership Inference Attacks (MIAs) can serve as an auditing mechanism to detect unauthorized data usage in models. While attacks like the Loss Attack provide a baseline, more involved methods like Polarized Augment Calibration (PAC) remain underexplored in the code domain. This paper presents an exploratory study evaluating these methods on 3B--7B parameter code models. We find that while PAC generally outperforms the Loss baseline, its effectiveness relies on augmentation strategies that disregard the rigid syntax of code, leading to performance degradation on larger, complex files. To address this, we introduce AST-PAC, a domain-specific adaptation that utilizes Abstract Syntax Tree (AST) based perturbations to generate syntactically valid calibration samples. Preliminary results indicate that AST-PAC improves as syntactic size grows, where PAC degrades, but under-mutates small files and underperforms on alphanumeric-rich code. Overall, the findings motivate future work on syntax-aware and size-adaptive calibration as a prerequisite for reliable provenance auditing of code language models.

AST-PAC: AST-guided Membership Inference for Code

TL;DR

The paper tackles data provenance auditing for code language models by evaluating membership inference attacks in a grey-box setting. It systematically compares Loss Attack and Polarized Augment Calibration (PAC) on 3B–7B code models and introduces AST-PAC, a syntax-aware variant using AST-guided perturbations to preserve code syntax during calibration. Key findings show PAC generally outperforms Loss on code-specialized models, but its effectiveness degrades for very large or highly complex files; AST-PAC improves robustness in those regimes while underperforming on small or alphanumeric-rich code. The work suggests that syntax-aware, size-adaptive calibration is crucial for reliable provenance auditing of code LLMs, with implications for governance and copyright compliance in code data usage.

Abstract

Code Large Language Models are frequently trained on massive datasets containing restrictively licensed source code. This creates urgent data governance and copyright challenges. Membership Inference Attacks (MIAs) can serve as an auditing mechanism to detect unauthorized data usage in models. While attacks like the Loss Attack provide a baseline, more involved methods like Polarized Augment Calibration (PAC) remain underexplored in the code domain. This paper presents an exploratory study evaluating these methods on 3B--7B parameter code models. We find that while PAC generally outperforms the Loss baseline, its effectiveness relies on augmentation strategies that disregard the rigid syntax of code, leading to performance degradation on larger, complex files. To address this, we introduce AST-PAC, a domain-specific adaptation that utilizes Abstract Syntax Tree (AST) based perturbations to generate syntactically valid calibration samples. Preliminary results indicate that AST-PAC improves as syntactic size grows, where PAC degrades, but under-mutates small files and underperforms on alphanumeric-rich code. Overall, the findings motivate future work on syntax-aware and size-adaptive calibration as a prerequisite for reliable provenance auditing of code language models.
Paper Structure (12 sections, 1 equation, 3 figures, 2 tables, 1 algorithm)

This paper contains 12 sections, 1 equation, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Methodology
  3. Attack Signals
  4. Experimental Setup
  5. RQ1: Comparative Performance
  6. RQ2: Impact of Data Characteristics
  7. RQ3: Code-Specific Adaptation
  8. Method: AST-PAC
  9. Results
  10. Discussion
  11. Threats to Validity
  12. Conclusion

Figures (3)

  • Figure 1: Experimental results for models and the loss and pac attacks grouped into categories based on different stratification of code attributes. Results for ROC-AUC (top row), and PR-AUC (bottom row).
  • Figure 2: Experimental results for models and the pac and pac_ast attacks grouped into categories based on different stratification of code attributes. Results for ROC-AUC (top row), and PR-AUC (bottom row).
  • Figure 3: Effective change ratio by syntactic size.