Unprecedented Code Change Automation: The Fusion of LLMs and Transformation by Example
Malinda Dilhara, Abhiram Bellur, Timofey Bryksin, Danny Dig
TL;DR
This work tackles the bottleneck of automating code changes when unseen variants arise, by integrating Large Language Models (LLMs) with Transformation by Example (TBE) into a tool named PyCraft. PyCraft mines CPATs, prompts LLMs to generate unseen VT1/VT2 variations for the LHS of CPATs, and then applies rigorous static/dynamic validation plus rule synthesis to infer transformation rules that generalize to new target code. It demonstrates strong empirical results, achieving a high F-measure in identifying correct variants ($\text{F-measure}=\$96.6$\%$), expanding input coverage by about $58\times$, and enabling up to $39\times$ more automated transformations than prior baselines, with an 83\% patch acceptance rate across real-world projects. The work includes best-practice guidance for LLM-driven variant generation, open-sourcing PyCraft and its dataset, and a discussion of practical impacts and limitations, including the need to mitigate LLM hallucinations for robust software engineering tooling.
Abstract
Software developers often repeat code changes, known as "code change patterns" (CPATs), within and across projects. Automating these CPATs accelerates development, but current Transformation by Example (TBE) techniques are limited by the input examples' quality and quantity, missing variations with different syntax or flow yet semantically similar. Large Language Models (LLMs), trained on vast code datasets, can overcome these limitations by generating semantically equivalent, unseen CPAT variants, enhancing TBE effectiveness. We identified best practices for using LLMs to generate code variants meeting criteria of correctness, usefulness, and applicability. Implementing these in PyCraft, combining static and dynamic analysis with LLMs, we achieved an F-measure of 96.6% in identifying correct variants, expanding inputs by 58x on average, and automating changes to increase target codes by up to 39x. Patches from PyCraft were submitted to projects like microsoft/DeepSpeed and IBM/inFairness, with an 83% acceptance rate, validating our approach's usefulness.