Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Pretraining for Project-Level Code Completion

Maksim Sapronov, Evgeniy Glukhov

TL;DR

This paper investigates how repository-processing strategies and extended context windows affect project-level code completion in an OpenCoder 1.5B model. It trains on about 1B curated repository tokens to extend the context to 16K and compares multiple context composers, finding that RoPE ABF-based scaling is the dominant factor for long-context gains, while the choice of composer offers only marginal benefits. The results achieve state-of-the-art performance on the Long Code Arena LCA-large benchmark with relatively little data and simpler training, suggesting a more accessible path for repository-level code completion research. The study also shows that a straightforward file-level training approach at the original sequence length remains effective, enabling use with constrained data and compute. Overall, the work highlights practical design choices for leveraging repository-level information in code LLMs and points toward future generalization to more models and improved retrieval strategies.

Abstract

Repository-level pretraining is commonly used to enable large language models for code to leverage codebase-wide context. This enhances their ability to generate accurate and context-aware code completions. In this work, we investigate how different repository-processing strategies affect in-context learning in OpenCoder, a 1.5B-parameter model. We extend its context window from 4,096 to 16,384 tokens by training on additional 1B tokens of curated repository-level data. Despite relying on a smaller dataset than competing models (which often use hundreds of billions of tokens), our model achieves comparable performance on the Long Code Arena benchmark. We find that various repository-processing techniques yield similarly strong results, with the primary gain coming from adapting to a new rotary positional embedding (RoPE) scaling parameter. Finally, we show that a simpler file-level training approach at the original sequence length remains highly effective, opening up repository-level code completion research to settings with more constrained data and compute resources.

On Pretraining for Project-Level Code Completion

TL;DR

This paper investigates how repository-processing strategies and extended context windows affect project-level code completion in an OpenCoder 1.5B model. It trains on about 1B curated repository tokens to extend the context to 16K and compares multiple context composers, finding that RoPE ABF-based scaling is the dominant factor for long-context gains, while the choice of composer offers only marginal benefits. The results achieve state-of-the-art performance on the Long Code Arena LCA-large benchmark with relatively little data and simpler training, suggesting a more accessible path for repository-level code completion research. The study also shows that a straightforward file-level training approach at the original sequence length remains effective, enabling use with constrained data and compute. Overall, the work highlights practical design choices for leveraging repository-level information in code LLMs and points toward future generalization to more models and improved retrieval strategies.

Abstract

Repository-level pretraining is commonly used to enable large language models for code to leverage codebase-wide context. This enhances their ability to generate accurate and context-aware code completions. In this work, we investigate how different repository-processing strategies affect in-context learning in OpenCoder, a 1.5B-parameter model. We extend its context window from 4,096 to 16,384 tokens by training on additional 1B tokens of curated repository-level data. Despite relying on a smaller dataset than competing models (which often use hundreds of billions of tokens), our model achieves comparable performance on the Long Code Arena benchmark. We find that various repository-processing techniques yield similarly strong results, with the primary gain coming from adapting to a new rotary positional embedding (RoPE) scaling parameter. Finally, we show that a simpler file-level training approach at the original sequence length remains highly effective, opening up repository-level code completion research to settings with more constrained data and compute resources.
Paper Structure (22 sections, 2 figures, 4 tables)

This paper contains 22 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction and Motivation
  2. Experiment Design
  3. Training Data
  4. Training
  5. Evaluation
  6. Results
  7. Benchmarking Against State-of-the-Art
  8. Impact of Context Composer Choice
  9. Conclusion
  10. Context Composers
  11. Complete List
  12. Input Formatting
  13. Training Dataset
  14. Filtering
  15. Training Details
  16. ...and 7 more sections

Figures (2)

  • Figure 1: File Representation
  • Figure 2: Performance comparison of File-level and Path Distance .py approaches across different context sizes for OpenCoder 1.5B model. The plots show the Exact Match accuracy for both inproject (top) and infile (bottom) categories. The dashed vertical lines represent the context length used during repository-level pretraining.