Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating the Impact of Data Cleaning on the Quality of Generated Pull Request Descriptions

Kutay Tire, Berk Çakar, Eray Tüzün

TL;DR

This study tackles the problem of noisy pull request (PR) descriptions by proposing four cleaning heuristics to filter noise from a large PR dataset, resulting in a cleaned benchmark of over 169K PRs drawn from 513 GitHub repositories. It evaluates four abstractive summarization models (BART, T5, PRSummarizer, iTAPE) trained on both raw and cleaned data, using ROUGE metrics and a two-stage manual evaluation to assess description quality. Automatic results show average F1 gains around 8.5–8.7% across ROUGE metrics after cleaning, with BART on cleaned data delivering the strongest performance; manual evaluation confirms improvements in relevance, descriptiveness, and clarity. The work provides a high-quality PR description generation benchmark and practical guidance for practitioners and researchers, highlighting the value of data quality for AI-assisted software engineering tasks and potentially informing PR-writing guidelines.

Abstract

Pull Requests (PRs) are central to collaborative coding, summarizing code changes for reviewers. However, many PR descriptions are incomplete, uninformative, or have out-of-context content, compromising developer workflows and hindering AI-based generation models trained on commit messages and original descriptions as "ground truth." This study examines the prevalence of "noisy" PRs and evaluates their impact on state-of-the-art description generation models. To do so, we propose four cleaning heuristics to filter noise from an initial dataset of 169K+ PRs drawn from 513 GitHub repositories. We train four models-BART, T5, PRSummarizer, and iTAPE-on both raw and cleaned datasets. Performance is measured via ROUGE-1, ROUGE-2, and ROUGE-L metrics, alongside a manual evaluation to assess description quality improvements from a human perspective. Cleaning the dataset yields significant gains: average F1 improvements of 8.6% (ROUGE-1), 8.7% (ROUGE-2), and 8.5% (ROUGE-L). Manual assessment confirms higher readability and relevance in descriptions generated by the best-performing model, BART when trained on cleaned data. Dataset refinement markedly enhances PR description generation, offering a foundation for more accurate AI-driven tools and guidelines to assist developers in crafting high-quality PR descriptions.

Evaluating the Impact of Data Cleaning on the Quality of Generated Pull Request Descriptions

TL;DR

This study tackles the problem of noisy pull request (PR) descriptions by proposing four cleaning heuristics to filter noise from a large PR dataset, resulting in a cleaned benchmark of over 169K PRs drawn from 513 GitHub repositories. It evaluates four abstractive summarization models (BART, T5, PRSummarizer, iTAPE) trained on both raw and cleaned data, using ROUGE metrics and a two-stage manual evaluation to assess description quality. Automatic results show average F1 gains around 8.5–8.7% across ROUGE metrics after cleaning, with BART on cleaned data delivering the strongest performance; manual evaluation confirms improvements in relevance, descriptiveness, and clarity. The work provides a high-quality PR description generation benchmark and practical guidance for practitioners and researchers, highlighting the value of data quality for AI-assisted software engineering tasks and potentially informing PR-writing guidelines.

Abstract

Pull Requests (PRs) are central to collaborative coding, summarizing code changes for reviewers. However, many PR descriptions are incomplete, uninformative, or have out-of-context content, compromising developer workflows and hindering AI-based generation models trained on commit messages and original descriptions as "ground truth." This study examines the prevalence of "noisy" PRs and evaluates their impact on state-of-the-art description generation models. To do so, we propose four cleaning heuristics to filter noise from an initial dataset of 169K+ PRs drawn from 513 GitHub repositories. We train four models-BART, T5, PRSummarizer, and iTAPE-on both raw and cleaned datasets. Performance is measured via ROUGE-1, ROUGE-2, and ROUGE-L metrics, alongside a manual evaluation to assess description quality improvements from a human perspective. Cleaning the dataset yields significant gains: average F1 improvements of 8.6% (ROUGE-1), 8.7% (ROUGE-2), and 8.5% (ROUGE-L). Manual assessment confirms higher readability and relevance in descriptions generated by the best-performing model, BART when trained on cleaned data. Dataset refinement markedly enhances PR description generation, offering a foundation for more accurate AI-driven tools and guidelines to assist developers in crafting high-quality PR descriptions.
Paper Structure (24 sections, 3 equations, 5 figures, 7 tables)

This paper contains 24 sections, 3 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Methodology
  4. Proposed Cleaning Heuristics
  5. Summarization Models
  6. Evaluation
  7. Automatic Evaluation
  8. Manual Evaluation
  9. Experimental Setup
  10. Data Collection
  11. Data Preprocessing
  12. Cleaning of the Dataset
  13. Results
  14. RQ1: Automatic Evaluation of Cleaning Heuristics
  15. RQ2: Manual Evaluation of Cleaning Heuristics
  16. ...and 9 more sections

Figures (5)

  • Figure 1: Overview of the First Stage of Manual Evaluation
  • Figure 2: Overview of the Second Stage of Manual Evaluation
  • Figure 3: Number of PRs Affected by Each Heuristic
  • Figure 4: F1 Scores of All Four Models for ROUGE-1, ROUGE-2, and ROUGE-L Metrics
  • Figure 5: Score Distributions of Generated Descriptions for Each Category