Revisiting Unnaturalness for Automated Program Repair in the Era of Large Language Models

TL;DR

This work revisits the notion of code naturalness by leveraging Large Language Model (LLM) entropy to enhance Automated Program Repair (APR) across fault localization, patch generation, and patch ranking. It introduces entropy-delta, a patch-naturalness metric, and shows that combining entropy signals with traditional APR tools yields substantial gains in Top-N fault localization, reduces the number of patches tested by template-based repair, and outperforms state-of-the-art patch disambiguation on plausible patch datasets. By modifying the TBar framework to incorporate entropy-based ordering and by validating on Defects4J, the authors demonstrate practical improvements while mitigating training-data leakage concerns. The results suggest that LLM-derived entropy is a valuable, low-overhead complement to existing APR components and can help reduce test-suite overfitting and data leakage risks in repair pipelines.

Abstract

Language models have improved by orders of magnitude with the recent emergence of Transformer-based Large Language Models (LLMs). LLMs have demonstrated their ability to generate natural code that is highly similar to code written by professional developers. One intermediate value an LLM can emit is entropy, which measures the naturalness of a token of code. We hypothesize that entropy can be used to improve the performance of Automated Program Repair (APR) tasks. While much progress has been made in Automated Program Repair (APR), fault localization techniques suffer from a lack of diversity in ranking scores, patch generation tools tend to be inefficient as all tests need to run before determining if a patch is likely to be correct, and patch ranking often suffers from the test-suite over-fitting problem. However, using an LLM directly for APR introduces concerns for training data leakage. In this work, we introduce a novel way of using the entropy of LLMs in combination with prior APR tools to improve all stages of APR. We show that entropy is highly complementary with prior fault localization tools. Our proposed re-ranking method achieves a 50% Top-5 score improvement over SBFL. We propose a patch-naturalness measurement, entropy-delta, to improve the efficiency of template-based repair techniques by ranking plausible patches before undergoing testing. When using entropy-delta for patch ranking and classification, our proposed method can rank correct patches more effectively than state-of-the-art machine learning tools with an 49% improvement in Top-1. Our work suggests that LLMs can be an effective addition to compliment prior APR tasks while minimizing both the test-suite overfitting problem and the LLM data leakage problem.

Figures (6)

• Figure 1: Chart bug 4 from Defects4J with its developer-written fix, a test-failing patch generated by TBar, and a test-passing patch generated by TBar. We show the InCoder-produced entropy of code in each patch.
• Figure 2: Fault localization pipeline using entropy. (1) We take a prior-FL suspicious score list, (2) query each code-line for entropy values, and (3) re-rank the list using LLM entropy scores.
• Figure 3: Example entropy-delta queries from an LLM. The MASK tokens enable models to learn the contextual relationships between tokens and make entropy predictions for missing, new, or replacement tokens. The EOM token is a special token that indicates the end of a mask.
• Figure 4: Entropy-delta and TBar ranking (lower is better) of test-passing patches on 72 Defects4J bugs.
• Figure 5: Median number of patches tested (lower is better) per project before succesful patch using TBar original ranking and entropy-delta re-ranking of test-passing patches on 100 Defects4J bugs.