Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy

TL;DR

The paper tackles the limited context window of transformers in large code files by introducing eWASH, a syntax-hierarchy–driven context compression that prioritizes high-level code elements. By applying this extended window approach to Python data with models XGPT-C and XPyMT5, the authors achieve state-of-the-art results in code completion, method completion, and code summarization, while also releasing new CodeXGLUE tasks focused on user experience. The results show that leveraging syntactic structure yields substantial gains, especially at shorter context lengths, and that the approach generalizes beyond code to natural language tasks. Overall, eWASH offers a general, architecture-agnostic method for long-range modeling of structured documents using hierarchy-aware context selection.

Abstract

Statistical language modeling and translation with transformers have found many successful applications in program understanding and generation tasks, setting high benchmarks for tools in modern software development environments. The finite context window of these neural models means, however, that they will be unable to leverage the entire relevant context of large files and packages for any given task. While there are many efforts to extend the context window, we introduce an architecture-independent approach for leveraging the syntactic hierarchies of source code for incorporating entire file-level context into a fixed-length window. Using concrete syntax trees of each source file we extract syntactic hierarchies and integrate them into context window by selectively removing from view more specific, less relevant scopes for a given task. We evaluate this approach on code generation tasks and joint translation of natural language and source code in Python programming language, achieving a new state-of-the-art in code completion and summarization for Python in the CodeXGLUE benchmark. We also introduce new CodeXGLUE benchmarks for user-experience-motivated tasks: code completion with normalized literals, method body completion/code summarization conditioned on file-level context.

Figures (4)

• Figure 1: An example scenario where both method completion (top right) and code completion (bottom right) are performed by our eWASH models XPyMT5 and XGPT-C, respectively. Method completion aims to predict a whole method body conditioned on a signature and docstring and other context, and code completion aims to predict any number of tokens to complete a member, line, or even a scope context conditioned on any incomplete code string. In this case XPyMT5 is tasked with predicting the whole body of pythonConvNet.forward. In this case it's clear that both the layers assigned in pythonConvNet.__init__ and the import statements above are important information. In another case XGPT-C aims to complete the assignment, and again the class attributes and import statements are important. Both models have a limit context window illustrated at left, and exclude important information.
• Figure 2: An illustration of the syntax hierarchy of the input context in the method completion example from Fig. \ref{['fig:ec-example']}. The eWASH (Extended Window Access by Syntax Hierarchy) method selectively fills the model context going down the file, in the order of priority level indicated, and stops when the token budget of the model context is filled. eWASH presupposes that names of entities at higher scopes are more relevant to the task at hand than entities at lower scopes.
• Figure 3: Comparing baseline GPT-C with XGPT-C in an offline evaluation of ExactMatch@1-5 code completion as a function of total token context length for the normalized literal scenario. Surprisingly, eWASH leads XGPT-C to benefit most over GPT-C at the shorter context lengths. XGPT-C also more exactly predicts tokens with longer context as well.
• Figure 4: Comparing baseline GPT-C with XGPT-C in an offline evaluation of ExactMatch@1-5 code completion as a function of local, same-line context length for the normalized literal scenario. XGPT-C is better than GPT-C by all measures, showing how leveraging longer range context can most help developers even when a line being edited has a few tokens.