Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Achieving Productivity Gains with AI-based IDE features: A Journey at Google

Maxim Tabachnyk, Xu Shu, Alexander Frömmgen, Pavel Sychev, Vahid Meimand, Ilia Krets, Stanislav Pyatykh, Abner Araujo, Kristóf Molnár, Satish Chandra

TL;DR

This work details Google's experience building and deploying two internal AI-enabled IDE features—code completion and Transform Code—across UI, backend, and model layers to achieve measurable productivity gains in an enterprise setting. It presents data-driven iterations focusing on latency, discoverability, and suggestion quality, including adaptive caching, contextual snippet rendering, discoverable entry points, and curated data for supervised fine-tuning. The study combines online experiments and observational causal analysis to quantify productivity effects, reporting improvements in metrics such as the Fraction of Code Written by ML (FCML), Change List Throughput (CLT), and reductions in investigation time, while acknowledging limitations of observational data. Collectively, the work offers a pragmatic blueprint for refining AI-powered developer tools at scale, illustrating how cross-layer optimizations and robust evaluation can yield tangible productivity benefits in large organizations.

Abstract

We discuss Google's journey in developing and refining two internal AI-based IDE features: code completion and natural-language-driven code transformation (Transform Code). We address challenges in latency, user experience and suggestion quality, all backed by rigorous experimentation. The article serves as an example of how to refine AI developer tools across the user interface, backend, and model layers, to deliver tangible productivity improvements in an enterprise setting.

Achieving Productivity Gains with AI-based IDE features: A Journey at Google

TL;DR

This work details Google's experience building and deploying two internal AI-enabled IDE features—code completion and Transform Code—across UI, backend, and model layers to achieve measurable productivity gains in an enterprise setting. It presents data-driven iterations focusing on latency, discoverability, and suggestion quality, including adaptive caching, contextual snippet rendering, discoverable entry points, and curated data for supervised fine-tuning. The study combines online experiments and observational causal analysis to quantify productivity effects, reporting improvements in metrics such as the Fraction of Code Written by ML (FCML), Change List Throughput (CLT), and reductions in investigation time, while acknowledging limitations of observational data. Collectively, the work offers a pragmatic blueprint for refining AI-powered developer tools at scale, illustrating how cross-layer optimizations and robust evaluation can yield tangible productivity benefits in large organizations.

Abstract

We discuss Google's journey in developing and refining two internal AI-based IDE features: code completion and natural-language-driven code transformation (Transform Code). We address challenges in latency, user experience and suggestion quality, all backed by rigorous experimentation. The article serves as an example of how to refine AI developer tools across the user interface, backend, and model layers, to deliver tangible productivity improvements in an enterprise setting.
Paper Structure (21 sections, 5 figures)

This paper contains 21 sections, 5 figures.

Table of Contents

  1. Introduction
  2. AI-based code completion
  3. Basic model
  4. Challenges
  5. Adaptive Caching to mitigate latency
  6. Providing Useful Context
  7. Results
  8. Transform Code
  9. Basic model
  10. Challenges
  11. Discoverable entry points
  12. Improved edit rendering for accelerating suggestion review
  13. User rewrites to bridge the distribution gap
  14. Results
  15. Measuring Productivity
  16. ...and 6 more sections

Figures (5)

  • Figure 1: An opportunity funnel starting from the software engineer's (SWE) actions down to actual application of ML-based suggestions. Opportunities are lost if the model prediction is not confident enough, the model doesn’t respond or responds too late, the prediction is subpar, the user doesn’t notice the prediction, and so on.
  • Figure 2: Transform Code in action. The user selects from code (background), and can choose to enter an edit prompt. The foreground shows the AI's response to the user prompt.
  • Figure 3: Example of syntax-preserving file rendering for C++. Relevant snippets are marked in yellow. Pruned pieces are replaced with a single-token marker "…".
  • Figure 4: Examples of undesired edit suggestions that got resolved by training on high quality user rewrites. In this example, the model fixes an additional typo and removes in-progress code snippets.
  • Figure 5: Average treatment effect in CLT and MeanDurInvSess across all treatment cohorts before and after adoption