Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LongCoder: A Long-Range Pre-trained Language Model for Code Completion

Daya Guo, Canwen Xu, Nan Duan, Jian Yin, Julian McAuley

TL;DR

LongCoder addresses the challenge of code completion with long input contexts by introducing a sparse Transformer that combines window-based local attention with bridge and memory tokens to maintain global access while preserving linear-time efficiency. It constructs the Long Code Completion (LCC) benchmark to emphasize long-context code across Python, Java, and C#, demonstrating that LongCoder outperforms prior models both on long-context tasks and standard benchmarks like CodeXGLUE, with comparable inference costs. Key contributions include the three attention mechanisms, a dedicated long-context dataset, and a pretrained LongCoder model. The work advances practical long-range code understanding and sets the stage for cross-file and project-wide code completion technologies.

Abstract

In this paper, we introduce a new task for code completion that focuses on handling long code input and propose a sparse Transformer model, called LongCoder, to address this task. LongCoder employs a sliding window mechanism for self-attention and introduces two types of globally accessible tokens - bridge tokens and memory tokens - to improve performance and efficiency. Bridge tokens are inserted throughout the input sequence to aggregate local information and facilitate global interaction, while memory tokens are included to highlight important statements that may be invoked later and need to be memorized, such as package imports and definitions of classes, functions, or structures. We conduct experiments on a newly constructed dataset that contains longer code context and the publicly available CodeXGLUE benchmark. Experimental results demonstrate that LongCoder achieves superior performance on code completion tasks compared to previous models while maintaining comparable efficiency in terms of computational resources during inference. All the codes and data are available at https://github.com/microsoft/CodeBERT.

LongCoder: A Long-Range Pre-trained Language Model for Code Completion

TL;DR

LongCoder addresses the challenge of code completion with long input contexts by introducing a sparse Transformer that combines window-based local attention with bridge and memory tokens to maintain global access while preserving linear-time efficiency. It constructs the Long Code Completion (LCC) benchmark to emphasize long-context code across Python, Java, and C#, demonstrating that LongCoder outperforms prior models both on long-context tasks and standard benchmarks like CodeXGLUE, with comparable inference costs. Key contributions include the three attention mechanisms, a dedicated long-context dataset, and a pretrained LongCoder model. The work advances practical long-range code understanding and sets the stage for cross-file and project-wide code completion technologies.

Abstract

In this paper, we introduce a new task for code completion that focuses on handling long code input and propose a sparse Transformer model, called LongCoder, to address this task. LongCoder employs a sliding window mechanism for self-attention and introduces two types of globally accessible tokens - bridge tokens and memory tokens - to improve performance and efficiency. Bridge tokens are inserted throughout the input sequence to aggregate local information and facilitate global interaction, while memory tokens are included to highlight important statements that may be invoked later and need to be memorized, such as package imports and definitions of classes, functions, or structures. We conduct experiments on a newly constructed dataset that contains longer code context and the publicly available CodeXGLUE benchmark. Experimental results demonstrate that LongCoder achieves superior performance on code completion tasks compared to previous models while maintaining comparable efficiency in terms of computational resources during inference. All the codes and data are available at https://github.com/microsoft/CodeBERT.
Paper Structure (21 sections, 5 equations, 4 figures, 6 tables)

This paper contains 21 sections, 5 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Code Completion
  4. Long-Range Transformer Models
  5. Long Code Completion
  6. LongCoder
  7. Window Attention
  8. Bridge Attention
  9. Global Attention
  10. Experiments
  11. Experimental Settings
  12. Baselines
  13. Benchmarks
  14. Evaluation Metrics
  15. Training Details
  16. ...and 6 more sections

Figures (4)

  • Figure 1: (Left) An example of how LongCoder facilitates completion with longer context. The memory tokens save potentially useful information (including package imports, class and function definitions) for global access despite whether they are within the sliding window. The bridge tokens aggregate local information by attending to a fixed length of tokens. The information flow within the window is omitted for clarity. (Right) Attention patterns used in BigBird bigbird, Longformer beltagy2020longformer and LongCoder. Best viewed in color.
  • Figure 2: Distribution of average attention scores between two tokens within different distances in CodeGPT lu2021codexglue. The attention score is an average of 100 Python code examples across all Transformer layers.
  • Figure 3: Visualization of the attention matrix (partially shown for clarity) in CodeGPT lu2021codexglue. The attention matrix is averaged across all Transformer layers.
  • Figure 4: Two LCC examples of Python (top) and Java (bottom) code and predictions of different models. Codex-2048 refers to the original Codex model with the maximum context length of 2,048 while Codex-512 is the same model with a maximum context length set to 512. Key information is highlighted with arrows.