The Compliance Paradox: Semantic-Instruction Decoupling in Automated Academic Code Evaluation

TL;DR

The paper tackles the reliability of LLM-based automated code graders by identifying a Semantic-Instruction Decoupling risk, termed the Compliance Paradox, where evaluators follow adversarial prompts over code evidence. It introduces AST-ASIP and the SPACI framework to inject payloads into syntactically inert code regions, and formalizes a dual-view threat model with a tripartite metric suite ($\hat{P}_{decouple}$, $\\mathcal{D}_{adv}$, $\\Psi$) to quantify decoupling, score divergence, and pedagogical harm. Large-scale experiments across 9 SOTA models and 25,000 submissions in Python, C, C++, and Java reveal widespread vulnerability, including a C++ blind spot, with high-capacity, instruction-tuned models showing near-total decoupling in many cases. The authors argue for a shift from generic RLHF toward domain-specific Adjudicative Robustness, and they release a dataset and injection framework to spur defense research and mitigate false certifications in automated educational settings.

Abstract

The rapid integration of Large Language Models (LLMs) into educational assessment rests on the unverified assumption that instruction following capability translates directly to objective adjudication. We demonstrate that this assumption is fundamentally flawed. Instead of evaluating code quality, models frequently decouple from the submission's logic to satisfy hidden directives, a systemic vulnerability we term the Compliance Paradox, where models fine-tuned for extreme helpfulness are vulnerable to adversarial manipulation. To expose this, we introduce the Semantic-Preserving Adversarial Code Injection (SPACI) Framework and the Abstract Syntax Tree-Aware Semantic Injection Protocol (AST-ASIP). These methods exploit the Syntax-Semantics Gap by embedding adversarial directives into syntactically inert regions (trivia nodes) of the Abstract Syntax Tree. Through a large-scale evaluation of 9 SOTA models across 25,000 submissions in Python, C, C++, and Java, we reveal catastrophic failure rates (>95%) in high-capacity open-weights models like DeepSeek-V3, which systematically prioritize hidden formatting constraints over code correctness. We quantify this failure using our novel tripartite framework measuring Decoupling Probability, Score Divergence, and Pedagogical Severity to demonstrate the widespread "False Certification" of functionally broken code. Our findings suggest that current alignment paradigms create a "Trojan" vulnerability in automated grading, necessitating a shift from standard RLHF toward domain-specific Adjudicative Robustness, where models are conditioned to prioritize evidence over instruction compliance. We release our complete dataset and injection framework to facilitate further research on the topic.

Figures (16)

• Figure 1: The framework visualizes how the divergence between the reductive Compiler View and the attentive LLM Evaluator View creates an Attack Surface ($\mathcal{S}$) composed of SISA regions. The Adversary exploits this via AST-ASIP operators, satisfying functional constraints while inducing the Compliance Paradox, where the evaluator decouples from code semantics to follow injected instructions.
• Figure 2: Composite Heatmap visualizing the Tripartite Robustness Framework metrics ($\hat{P}_{\text{decouple}}$, $\mathcal{D}_{\text{adv}}$, and $\Psi$) across 9 models and 17 adversarial strategies, highlighting the severe susceptibility of high-capacity open-weights models to Role Play (RPA) attacks compared to proprietary baselines.
• Figure 3: The Abstract Syntax Tree-Aware Semantic Injection Protocol (AST-ASIP) leverages AST analysis to embed adversarial payloads into syntactically inert but semantically active regions of code, avoiding compilation errors while influencing LLM behaviour.
• Figure 4: Operator A Visualization. The adversarial payload is encapsulated within the function's docstring. While functionally identical to the compiler, the adversarial version forces the LLM to decouple from the actual code logic (which returns None) and follow the injected persona instructions.
• Figure 5: Operator B Visualization. The transformation $x_{adv} = \text{Refactor}(x, \psi(V))$ demonstrates how semantic directives are injected directly into the Abstract Syntax Tree (AST) structure without breaking the program's execution logic.