Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Update Natural Language Comments Based on Code Changes

Sheena Panthaplackel, Pengyu Nie, Milos Gligoric, Junyi Jessy Li, Raymond J. Mooney

TL;DR

This work tackles automatic updating of natural language code comments when the underlying code changes. It introduces a cross-modal edit model that represents code changes and comment edits as sequences, encoded by a dual-encoder architecture and decoded into a comment-edit sequence with a reranking mechanism. Experiments on a GitHub Java dataset show the edit-based approach outperforms baselines on edit-centric metrics and is favorably evaluated by humans, though some cases still require more context or even generation from scratch. The study provides a practical framework for maintaining synchronized documentation and releases accompanying open-source software.

Abstract

We formulate the novel task of automatically updating an existing natural language comment based on changes in the body of code it accompanies. We propose an approach that learns to correlate changes across two distinct language representations, to generate a sequence of edits that are applied to the existing comment to reflect the source code modifications. We train and evaluate our model using a dataset that we collected from commit histories of open-source software projects, with each example consisting of a concurrent update to a method and its corresponding comment. We compare our approach against multiple baselines using both automatic metrics and human evaluation. Results reflect the challenge of this task and that our model outperforms baselines with respect to making edits.

Learning to Update Natural Language Comments Based on Code Changes

TL;DR

This work tackles automatic updating of natural language code comments when the underlying code changes. It introduces a cross-modal edit model that represents code changes and comment edits as sequences, encoded by a dual-encoder architecture and decoded into a comment-edit sequence with a reranking mechanism. Experiments on a GitHub Java dataset show the edit-based approach outperforms baselines on edit-centric metrics and is favorably evaluated by humans, though some cases still require more context or even generation from scratch. The study provides a practical framework for maintaining synchronized documentation and releases accompanying open-source software.

Abstract

We formulate the novel task of automatically updating an existing natural language comment based on changes in the body of code it accompanies. We propose an approach that learns to correlate changes across two distinct language representations, to generate a sequence of edits that are applied to the existing comment to reflect the source code modifications. We train and evaluate our model using a dataset that we collected from commit histories of open-source software projects, with each example consisting of a concurrent update to a method and its corresponding comment. We compare our approach against multiple baselines using both automatic metrics and human evaluation. Results reflect the challenge of this task and that our model outperforms baselines with respect to making edits.

Paper Structure

This paper contains 41 sections, 17 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Task
  3. Edit Model Overview
  4. Representing Edits
  5. Edit Lexicon
  6. Code Edits
  7. Comment Edits
  8. Parsing Edit Sequences
  9. Features
  10. Reranking
  11. Data
  12. Experimental Method
  13. Baselines
  14. Generation Model
  15. Reranked Generation Model
  16. ...and 26 more sections

Figures (3)

  • Figure 1: Changes in the getRotX method and its corresponding @return comment between two subsequent commits of the rajawali-rajawali project, available on GitHub.
  • Figure 2: High-level overview of our system.
  • Figure 3: Changes in the getComplex method and its corresponding @return comment between two subsequent commits of the eclipse-january project, available on GitHub.