Out of style: Misadventures with LLMs and code style transfer

TL;DR

This work tackles automated code style transfer, a task that requires precise identification and modification of code snippets without altering semantics. It introduces the Code Style Benchmark (CSB) across Python and Java with five transformation tasks and a DiffCorrect metric to jointly assess edits and functional correctness, leveraging CodeNet and BigQuery data. Experiments show state-of-the-art code LLMs struggle to perform these transforms reliably, particularly in Java, though fine-tuning small models yields some gains on select tasks. The paper provides public datasets and a robust evaluation framework to drive progress toward IDE-ready style transformations and more reliable code-generation models.

Abstract

Like text, programs have styles, and certain programming styles are more desirable than others for program readability, maintainability, and performance. Code style transfer, however, is difficult to automate except for trivial style guidelines such as limits on line length. Inspired by the success of using language models for text style transfer, we investigate if code language models can perform code style transfer. Code style transfer, unlike text transfer, has rigorous requirements: the system needs to identify lines of code to change, change them correctly, and leave the rest of the program untouched. We designed CSB (Code Style Benchmark), a benchmark suite of code style transfer tasks across five categories including converting for-loops to list comprehensions, eliminating duplication in code, adding decorators to methods, etc. We then used these tests to see if large pre-trained code language models or fine-tuned models perform style transfer correctly, based on rigorous metrics to test that the transfer did occur, and the code still passes functional tests. Surprisingly, language models failed to perform all of the tasks, suggesting that they perform poorly on tasks that require code understanding. We will make available the large-scale corpora to help the community build better code models.

Figures (7)

• Figure 1: An example of the code encapsulation task.
• Figure 2: Original program and the list comprehension transform
• Figure 3: Original program and its decorator transform
• Figure 4: Original program, and its casing transform
• Figure 5: Original program and the expected transformation for the reuse task