Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Control Models for In-IDE Code Completion

Aral de Moor, Yana Hrynevich, Hleb Badzeika, Vladyslav Furda, Marko Kojic, Artem Savelev, Kostadin Cvejoski, Darya Rovdo, Ekaterina Garanina

TL;DR

This paper addresses misalignment between LLM-powered code completion and developer flow by introducing control models that gate and filter in-editor suggestions using in-IDE telemetry and code context. It evaluates two architectures—gradient-boosting on tabular data and transformer-based classifiers that incorporate code context—across offline datasets and live A/B tests in JetBrains IDEs, reporting around 0.20 of completions avoided without harming user experience and with improved acceptance and cancellation metrics. The work discusses the practical challenges of latency, privacy, and integration for transformer-based approaches, and demonstrates that lightweight boosting models can achieve meaningful gains while remaining deployable. It also outlines directions for long-term productivity metrics, user personalisation, and potential end-to-end reinforcement learning approaches. Overall, the results highlight the potential of control models to enable smarter in-IDE AI integration and point to concrete paths for improving efficiency and user satisfaction.

Abstract

We introduce control models for LLM-powered code completion in JetBrains IDEs: ML classifiers which trigger inference and filter the generated suggestions to better align them with users and reduce unnecessary requests. To this end, we evaluate boosting- and transformer-based architectures on an offline dataset of real code completions with n=98 users. We further evaluate the offline classification performance of our boosting-based approach on a range of syntactically diverse languages; and perform an A/B study in a production environment where they improve completion efficiency and quality metrics. With this study, we hope to demonstrate the potential in using auxiliary models for smarter in-IDE integration of LLM-driven features, highlight fruitful future directions, and open problems.

Control Models for In-IDE Code Completion

TL;DR

This paper addresses misalignment between LLM-powered code completion and developer flow by introducing control models that gate and filter in-editor suggestions using in-IDE telemetry and code context. It evaluates two architectures—gradient-boosting on tabular data and transformer-based classifiers that incorporate code context—across offline datasets and live A/B tests in JetBrains IDEs, reporting around 0.20 of completions avoided without harming user experience and with improved acceptance and cancellation metrics. The work discusses the practical challenges of latency, privacy, and integration for transformer-based approaches, and demonstrates that lightweight boosting models can achieve meaningful gains while remaining deployable. It also outlines directions for long-term productivity metrics, user personalisation, and potential end-to-end reinforcement learning approaches. Overall, the results highlight the potential of control models to enable smarter in-IDE AI integration and point to concrete paths for improving efficiency and user satisfaction.

Abstract

We introduce control models for LLM-powered code completion in JetBrains IDEs: ML classifiers which trigger inference and filter the generated suggestions to better align them with users and reduce unnecessary requests. To this end, we evaluate boosting- and transformer-based architectures on an offline dataset of real code completions with n=98 users. We further evaluate the offline classification performance of our boosting-based approach on a range of syntactically diverse languages; and perform an A/B study in a production environment where they improve completion efficiency and quality metrics. With this study, we hope to demonstrate the potential in using auxiliary models for smarter in-IDE integration of LLM-driven features, highlight fruitful future directions, and open problems.
Paper Structure (24 sections, 3 figures, 2 tables)

This paper contains 24 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Productive Human-AI Interaction
  4. Inline Code Completion
  5. Control Models for Code Completion
  6. Approach
  7. Model Architectures
  8. Gradient-Boosting Models for Tabular Data
  9. Transformer-based Models for Code Context
  10. Data Sources
  11. Tabular Telemetry Features
  12. Code Context
  13. Metrics
  14. Setup & Results
  15. Offline Classification Performance of Boosting and Transformer Control Models
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Control Models Problem Setting: Completion Flow with Positive (Accept) and Negative (Reject) Targets.
  • Figure 2: Impact of Boosting- and Transformer-based Control Models on Completion Quality Metrics, at Different Generation Filter Rates and Overall False-Negative Rates (FNR), Computed Offline. How to Read These Graphs: More control is handed over to the trigger model as % Generations Filtered increases.
  • Figure 3: Impact of Boosting-based Control Models on Completion Quality Metrics, at Different Generation Filter Rates and Overall 0.10 False-Negative Rate (FNR), for Different Languages, Computed Offline. How to Read These Graphs: More control is handed over to the trigger model as % Generations Filtered increases.