A Transformer-based Approach for Source Code Summarization

TL;DR

This work investigates transformer-based source code summarization, showing that modeling pairwise token relationships through relative position representations plus a copy mechanism delivers substantial gains over prior methods. It demonstrates that absolute position encodings are detrimental for code representation and that a simple, well-regularized Transformer with relative encodings serves as a strong baseline. Extensive ablations reveal the importance of directional relative positions, token copying, and tokenization strategies, with AST-based structure offering limited benefit. The findings provide a strong, practical baseline for future research in code summarization and related sequence generation tasks in software engineering.

Abstract

Generating a readable summary that describes the functionality of a program is known as source code summarization. In this task, learning code representation by modeling the pairwise relationship between code tokens to capture their long-range dependencies is crucial. To learn code representation for summarization, we explore the Transformer model that uses a self-attention mechanism and has shown to be effective in capturing long-range dependencies. In this work, we show that despite the approach is simple, it outperforms the state-of-the-art techniques by a significant margin. We perform extensive analysis and ablation studies that reveal several important findings, e.g., the absolute encoding of source code tokens' position hinders, while relative encoding significantly improves the summarization performance. We have made our code publicly available to facilitate future research.