Deep Learning-based Code Completion: On the Impact on Performance of Contextual Information

TL;DR

This work addresses how different contextual signals influence DL-based code completion by evaluating eight context families (coding, process, developer) across 18 T5 models on Java-method completion. The authors build eight context-augmented datasets from 1,072 compilable releases and 42,182 methods, masking realistic statements via git blame and enriching inputs with structured code, issue-tracker data, and developer activity. They find that coding-context information yields the largest gains, with combined contexts delivering up to an 11% relative improvement, and a confidence-based ensemble achieving a 37.43% accuracy, a 22.4% relative uplift over the baseline. The results demonstrate the value of context-aware inputs and model selection in practice, while also highlighting complementarity between models and the potential for confident, multi-model ensembles to enhance developer productivity.

Abstract

Code completion aims at speeding up code writing by recommending to developers the next tokens they are likely to type. Deep Learning (DL) models pushed the boundaries of code completion by redefining what these coding assistants can do: We moved from predicting few code tokens to automatically generating entire functions. One important factor impacting the performance of DL-based code completion techniques is the context provided as input. With "context" we refer to what the model knows about the code to complete. In a simple scenario, the DL model might be fed with a partially implemented function to complete. In this case, the context is represented by the incomplete function and, based on it, the model must generate a prediction. It is however possible to expand such a context to include additional information, like the whole source code file containing the function to complete, which could be useful to boost the prediction performance. In this work, we present an empirical study investigating how the performance of a DL-based code completion technique is affected by different contexts. We experiment with 8 types of contexts and their combinations. These contexts include: (i) coding contexts, featuring information extracted from the code base in which the code completion is invoked (e.g., code components structurally related to the one to "complete"); (ii) process context, with information aimed at depicting the current status of the project in which a code completion task is triggered (e.g., a textual representation of open issues relevant for the code to complete); and (iii) developer contexts, capturing information about the developer invoking the code completion (e.g., the APIs frequently used). Our results show that additional contextual information can benefit the performance of DL-based code completion, with relative improvements up to +22% in terms of correct predictions.

Figures (4)

• Figure 1: Experimented contexts: Examples from a real instance in our dataset.
• Figure 2: Method calls context helping the prediction.
• Figure 3: Percentage of correct predictions when varying the confidence of the model.
• Figure 4: Class signatures context helping the prediction of the confidence-based model.