Auto-SPT: Automating Semantic Preserving Transformations for Code
Ashish Hooda, Mihai Christodorescu, Chuangang Ren, Aaron Wilson, Kassem Fawaz, Somesh Jha
TL;DR
Auto-SPT introduces an automated, LLM-driven framework to design, implement, and compose semantic-preserving transformations for code. By formalizing strength and diversity of transformation sets, it demonstrates that diverse, strong SPTs degrade state-of-the-art code clone detectors and can improve robustness through data augmentation. Empirical results across multiple detectors and substantial CodeContests-based data show Auto-SPT yields far stronger perturbations than prior approaches and that diversity is key to effective composition. The work provides both theoretical and empirical contributions toward evaluating and enhancing robustness of code-understanding models in realistic, transformed code scenarios.
Abstract
Machine learning (ML) models for code clone detection determine whether two pieces of code are semantically equivalent, which in turn is a key building block for software-engineering tasks like refactoring and security tasks like vulnerability and malware detection. While these models are predominantly trained on clean, structured code datasets, real-world code often undergoes a variety of semantic-preserving transformations, including refactoring, minification, automated formatting, and compiler optimizations. To address this critical gap between training and test data, we propose Auto-SPT, a novel framework to automatically construct synthetic-data generators for code. Auto-SPT is designed to produce Semantic Preserving Transformations (SPTs) that alter a program's syntactic structure while preserving its functionality and is instantiated on top of Large Language Models (LLMs). In particular, we use LLMs to craft a diverse set of SPTs, generate strong implementations for these SPTs, and compose them to result into strong transformations. Our formal analysis shows that the diversity of SPTs impacts the strength of their composition. We then empirically demonstrate that Auto-SPT generates more diverse SPTs than existing approaches and these SPTs significantly drop the performance of state-of-the-art code clone detectors. Further experiments show Auto-SPT can be used to enhance code datasets for training, to produce code-clone detection models that are robust to real-world, adversarial code transformations.