Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Characterizing Multi-Hunk Patches: Divergence, Proximity, and LLM Repair Challenges

Noor Nashid, Daniel Ding, Keheliya Gallaba, Ahmed E. Hassan, Ali Mesbah

TL;DR

This work tackles the challenge of multi-hunk patches, where fixes span non-contiguous regions and semantic interdependencies complicate automated repair. It introduces two metrics—hunk divergence and spatial proximity—and a real-world benchmark, Hunk4J, derived from Defects4J, to quantify patch heterogeneity and dispersion. The Birch framework enables reproducible, prompt-driven evaluation of six LLMs across retrieval strategies and contextual scopes, revealing that repair success declines with higher divergence and dispersion, with no model solving the most dispersed Fragment class. The findings motivate divergence-aware repair strategies and demonstrate the need for retrieval-augmented, context-sensitive approaches to tackle complex, distributed code edits in practice.

Abstract

Multi-hunk bugs, where fixes span disjoint regions of code, are common in practice, yet remain underrepresented in automated repair. Existing techniques and benchmarks pre-dominantly target single-hunk scenarios, overlooking the added complexity of coordinating semantically related changes across the codebase. In this work, we characterize HUNK4J, a dataset of multi-hunk patches derived from 372 real-world defects. We propose hunk divergence, a metric that quantifies the variation among edits in a patch by capturing lexical, structural, and file-level differences, while incorporating the number of hunks involved. We further define spatial proximity, a classification that models how hunks are spatially distributed across the program hierarchy. Our empirical study spanning six LLMs reveals that model success rates decline with increased divergence and spatial dispersion. Notably, when using the LLM alone, no model succeeds in the most dispersed Fragment class. These findings highlight a critical gap in LLM capabilities and motivate divergence-aware repair strategies.

Characterizing Multi-Hunk Patches: Divergence, Proximity, and LLM Repair Challenges

TL;DR

This work tackles the challenge of multi-hunk patches, where fixes span non-contiguous regions and semantic interdependencies complicate automated repair. It introduces two metrics—hunk divergence and spatial proximity—and a real-world benchmark, Hunk4J, derived from Defects4J, to quantify patch heterogeneity and dispersion. The Birch framework enables reproducible, prompt-driven evaluation of six LLMs across retrieval strategies and contextual scopes, revealing that repair success declines with higher divergence and dispersion, with no model solving the most dispersed Fragment class. The findings motivate divergence-aware repair strategies and demonstrate the need for retrieval-augmented, context-sensitive approaches to tackle complex, distributed code edits in practice.

Abstract

Multi-hunk bugs, where fixes span disjoint regions of code, are common in practice, yet remain underrepresented in automated repair. Existing techniques and benchmarks pre-dominantly target single-hunk scenarios, overlooking the added complexity of coordinating semantically related changes across the codebase. In this work, we characterize HUNK4J, a dataset of multi-hunk patches derived from 372 real-world defects. We propose hunk divergence, a metric that quantifies the variation among edits in a patch by capturing lexical, structural, and file-level differences, while incorporating the number of hunks involved. We further define spatial proximity, a classification that models how hunks are spatially distributed across the program hierarchy. Our empirical study spanning six LLMs reveals that model success rates decline with increased divergence and spatial dispersion. Notably, when using the LLM alone, no model succeeds in the most dispersed Fragment class. These findings highlight a critical gap in LLM capabilities and motivate divergence-aware repair strategies.

Paper Structure

This paper contains 17 sections, 9 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Characterization of Multi-hunk Patches
  3. Hunk Divergence
  4. Spatial Proximity
  5. Hunk4J: A Dataset of Multi-Hunk Bugs
  6. Hunk Characterization
  7. Empirical Evaluation
  8. Benchmarking Framework: Birch
  9. Models
  10. Evaluation Metrics
  11. Effectiveness of LLMs on Multi-Hunk Repair (RQ1)
  12. Impact of Augmented Prompting (RQ2)
  13. Repair Outcomes by Hunk Divergence and Proximity (RQ3)
  14. Divergence, Proximity, and Repair Difficulty (RQ4)
  15. Threats to Validity
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Distribution of different divergence components and the overall hunk divergence.
  • Figure 2: Overlap of fixes across model families
  • Figure 3: Upset-style plot showing bugs correctly repaired by the selected models, grouped by spatial proximity class. Bars indicate intersections among models; color segments denote proximity categories.
  • Figure 4: Faceted violin plots of average hunk divergence, grouped by model and split by outcome.

Theorems & Definitions (5)

  • Definition 1: Hunk
  • Definition 2: Multi-Hunk Patch
  • Definition 3: Pairwise Hunk Divergence
  • Definition 4: Hunk Divergence
  • Definition 5: Spatial Proximity Class