Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Test-based Patch Clustering for Automatically-Generated Patches Assessment

Matias Martinez, Maria Kechagia, Anjana Perera, Justyna Petke, Federica Sarro, Aldeida Aleti

TL;DR

This work tackles the patch-overfitting problem in Automated Program Repair by introducing xTestCluster, a light-weight, test-based clustering method that groups plausible patches from multiple APR tools according to their dynamic behavior as exposed by newly generated tests. It operates in three steps—test-case generation, cross test-case execution, and clustering by failure patterns—to reduce the number of patches developers must review and to provide test inputs that highlight behavioral differences. Evaluated on 902 patches across 129 Defects4J bugs from 21 repair tools, xTestCluster achieves a median reduction of $50\%$ in patches to review and produces clusters with meaningful semantic separation in many cases, while also offering complementary information to state-of-the-art patch assessment techniques. The results indicate that higher-quality generated tests (more tests, greater LOC, and higher coverage) improve clustering effectiveness, suggesting practical impact for integration into patch review workflows and repair pipelines.

Abstract

Previous studies have shown that Automated Program Repair (APR) techniques suffer from the overfitting problem. Overfitting happens when a patch is run and the test suite does not reveal any error, but the patch actually does not fix the underlying bug or it introduces a new defect that is not covered by the test suite. Therefore, the patches generated by apr tools need to be validated by human programmers, which can be very costly, and prevents apr tool adoption in practice. Our work aims to minimize the number of plausible patches that programmers have to review, thereby reducing the time required to find a correct patch. We introduce a novel light-weight test-based patch clustering approach called xTestCluster, which clusters patches based on their dynamic behavior. xTestCluster is applied after the patch generation phase in order to analyze the generated patches from one or more repair tools and to provide more information about those patches for facilitating patch assessment. The novelty of xTestCluster lies in using information from execution of newly generated test cases to cluster patches generated by multiple APR approaches. A cluster is formed of patches that fail on the same generated test cases. The output from xTestCluster gives developers a) a way of reducing the number of patches to analyze, as they can focus on analyzing a sample of patches from each cluster, b) additional information attached to each patch. After analyzing 902 plausible patches from 21 Java APR tools, our results show that xTestCluster is able to reduce the number of patches to review and analyze with a median of 50%. xTestCluster can save a significant amount of time for developers that have to review the multitude of patches generated by apr tools, and provides them with new test cases that expose the differences in behavior between generated patches.

Test-based Patch Clustering for Automatically-Generated Patches Assessment

TL;DR

This work tackles the patch-overfitting problem in Automated Program Repair by introducing xTestCluster, a light-weight, test-based clustering method that groups plausible patches from multiple APR tools according to their dynamic behavior as exposed by newly generated tests. It operates in three steps—test-case generation, cross test-case execution, and clustering by failure patterns—to reduce the number of patches developers must review and to provide test inputs that highlight behavioral differences. Evaluated on 902 patches across 129 Defects4J bugs from 21 repair tools, xTestCluster achieves a median reduction of in patches to review and produces clusters with meaningful semantic separation in many cases, while also offering complementary information to state-of-the-art patch assessment techniques. The results indicate that higher-quality generated tests (more tests, greater LOC, and higher coverage) improve clustering effectiveness, suggesting practical impact for integration into patch review workflows and repair pipelines.

Abstract

Previous studies have shown that Automated Program Repair (APR) techniques suffer from the overfitting problem. Overfitting happens when a patch is run and the test suite does not reveal any error, but the patch actually does not fix the underlying bug or it introduces a new defect that is not covered by the test suite. Therefore, the patches generated by apr tools need to be validated by human programmers, which can be very costly, and prevents apr tool adoption in practice. Our work aims to minimize the number of plausible patches that programmers have to review, thereby reducing the time required to find a correct patch. We introduce a novel light-weight test-based patch clustering approach called xTestCluster, which clusters patches based on their dynamic behavior. xTestCluster is applied after the patch generation phase in order to analyze the generated patches from one or more repair tools and to provide more information about those patches for facilitating patch assessment. The novelty of xTestCluster lies in using information from execution of newly generated test cases to cluster patches generated by multiple APR approaches. A cluster is formed of patches that fail on the same generated test cases. The output from xTestCluster gives developers a) a way of reducing the number of patches to analyze, as they can focus on analyzing a sample of patches from each cluster, b) additional information attached to each patch. After analyzing 902 plausible patches from 21 Java APR tools, our results show that xTestCluster is able to reduce the number of patches to review and analyze with a median of 50%. xTestCluster can save a significant amount of time for developers that have to review the multitude of patches generated by apr tools, and provides them with new test cases that expose the differences in behavior between generated patches.
Paper Structure (30 sections, 1 equation, 5 figures, 6 tables, 4 algorithms)

This paper contains 30 sections, 1 equation, 5 figures, 6 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Our approach
  3. Test-case Generation
  4. Test-case Execution
  5. Clustering
  6. Research Questions
  7. Methodology
  8. Dataset
  9. RQ1: Hypothesis Validation
  10. RQ2: Patch Reduction Effectiveness
  11. RQ3: Clustering Effectiveness
  12. RQ4: Complementing State-of-the art patch assessment techniques
  13. RQ5: Quality of generated test cases and Effectiveness of xTestCluster
  14. Results
  15. RQ1: Hypothesis Validation
  16. ...and 15 more sections

Figures (5)

  • Figure 1: The steps executed by xTestCluster.
  • Figure 2: RQ1. Distribution of the number of clusters per bug. Bugs with a single patch (in total 51) are discarded.
  • Figure 3: RQ2. Distribution of patches to analyze per bug with and without our approach.
  • Figure 4: RQ2. Distribution of the percentage reduction of the number of patches to review. Reduction for a bug B is computed as follows: ((#patches for B - #clusters for B ) / #patches for B) $\times$ 100.
  • Figure 5: Metrics computed from newly-generated tests.