Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judge

Wenhan Mu, Ling Xu, Shuren Pei, Le Mi, Huichi Zhou

TL;DR

This work addresses the vulnerability of code language models to adversarial identifier substitutions by proposing EP-Shield, a naturalness-aware, two-stage defense that evaluates and purifies adversarial code using a lightweight LLM fine-tuned on human-aligned NES evaluations. It demonstrates that adversarial examples commonly exhibit unnatural patterns and that purifying these perturbations restores correct model predictions across clone detection, vulnerability detection, and code summarization tasks, achieving defense rates near GPT-4 while maintaining efficiency. The approach introduces EP-Metric as a robust evaluation proxy for human judgment and shows strong generalization to unseen attack patterns, with significant improvements over adversarial fine-tuning. The practical impact lies in providing a cost-effective, privacy-preserving defense against real-world identifier-substitution attacks in CLMs without sacrificing code quality or performance.

Abstract

The widespread adoption of code language models in software engineering tasks has exposed vulnerabilities to adversarial attacks, especially the identifier substitution attacks. Although existing identifier substitution attackers demonstrate high success rates, they often produce adversarial examples with unnatural code patterns. In this paper, we systematically assess the quality of adversarial examples using LLM-as-a-Judge. Our analysis reveals that over 80% of adversarial examples generated by state-of-the-art identifier substitution attackers (e.g., ALERT) are actually detectable. Based on this insight, we propose EP-Shield, a unified framework for evaluating and purifying identifier substitution attacks via naturalness-aware reasoning. Specifically, we first evaluate the naturalness of code and identify the perturbed adversarial code, then purify it so that the victim model can restore correct prediction. Extensive experiments demonstrate the superiority of EP-Shield over adversarial fine-tuning (up to 83.36% improvement) and its lightweight design 7B parameters) with GPT-4-level performance.

Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judge

TL;DR

This work addresses the vulnerability of code language models to adversarial identifier substitutions by proposing EP-Shield, a naturalness-aware, two-stage defense that evaluates and purifies adversarial code using a lightweight LLM fine-tuned on human-aligned NES evaluations. It demonstrates that adversarial examples commonly exhibit unnatural patterns and that purifying these perturbations restores correct model predictions across clone detection, vulnerability detection, and code summarization tasks, achieving defense rates near GPT-4 while maintaining efficiency. The approach introduces EP-Metric as a robust evaluation proxy for human judgment and shows strong generalization to unseen attack patterns, with significant improvements over adversarial fine-tuning. The practical impact lies in providing a cost-effective, privacy-preserving defense against real-world identifier-substitution attacks in CLMs without sacrificing code quality or performance.

Abstract

The widespread adoption of code language models in software engineering tasks has exposed vulnerabilities to adversarial attacks, especially the identifier substitution attacks. Although existing identifier substitution attackers demonstrate high success rates, they often produce adversarial examples with unnatural code patterns. In this paper, we systematically assess the quality of adversarial examples using LLM-as-a-Judge. Our analysis reveals that over 80% of adversarial examples generated by state-of-the-art identifier substitution attackers (e.g., ALERT) are actually detectable. Based on this insight, we propose EP-Shield, a unified framework for evaluating and purifying identifier substitution attacks via naturalness-aware reasoning. Specifically, we first evaluate the naturalness of code and identify the perturbed adversarial code, then purify it so that the victim model can restore correct prediction. Extensive experiments demonstrate the superiority of EP-Shield over adversarial fine-tuning (up to 83.36% improvement) and its lightweight design 7B parameters) with GPT-4-level performance.
Paper Structure (29 sections, 2 equations, 6 figures, 12 tables)

This paper contains 29 sections, 2 equations, 6 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Adversarial Attack on CLMs.
  4. Code Naturalness.
  5. Evaluation of Adversarial Examples.
  6. A Preliminary Study
  7. EP-Shield: Our Approach
  8. Identifier Substitution Attack
  9. Naturalness Evaluation via LLMs
  10. Code Purification via LLMs
  11. LLM Fine-Tuning
  12. EP-Shield as an Evaluation Metric
  13. EP-Shield as a Purifier
  14. Experiments and Analysis
  15. Experimental Setup
  16. ...and 14 more sections

Figures (6)

  • Figure 1: A motivating example of an identifier substitution attack, accompanied by an analysis of GPT-4 from the perspective of naturalness using LLM-as-a-Judge.
  • Figure 2: The density distribution of NES annotated by GPT-4 for adversarial examples generated by ALERT on CodeBERT across three code-related tasks.
  • Figure 3: An overview of our proposed EP-Shield.
  • Figure 4: The prompt template for LLMs to obtain the NES and analysis.
  • Figure 5: The prompt template for purification task.
  • ...and 1 more figures