How much does AI impact development speed? An enterprise-based randomized controlled trial

TL;DR

An estimate of the size of the effect of three AI features on the time developers spent on a complex, enterprise-grade task found that AI significantly shortened the time developers spent on task.

Abstract

How much does AI assistance impact developer productivity? To date, the software engineering literature has provided a range of answers, targeting a diversity of outcomes: from perceived productivity to speed on task and developer throughput. Our randomized controlled trial with 96 full-time Google software engineers contributes to this literature by sharing an estimate of the impact of three AI features on the time developers spent on a complex, enterprise-grade task. We found that AI significantly shortened the time developers spent on task. Our best estimate of the size of this effect, controlling for factors known to influence developer time on task, stands at about 21\%, although our confidence interval is large. We also found an interesting effect whereby developers who spend more hours on code-related activities per day were faster with AI. Product and future research considerations are discussed. In particular, we invite further research that explores the impact of AI at the ecosystem level and across multiple suites of AI-enhanced tools, since we cannot assume that the effect size obtained in our lab study will necessarily apply more broadly, or that the effect of AI found using internal Google tooling in the summer of 2024 will translate across tools and over time.

Figures (7)

• Figure 1: AI Code Completion in Cider V. When a user starts typing code, the feature auto-completes the code block in light-grey font based on the context provided. After typing the first line of a new function to be evaluated at compile time, the user starts to type the return logic and the AI Code Completion makes a suggestion based on the entered parameters in light-grey text. Pressing TAB accepts the suggestion.
• Figure 2: Smart Paste feature in Cider V. When a user pastes code, Smart Paste provides an automatic adjustment to the code, then shows the inline diff highlights the removal of tryfromenv (strikethrough) and the insertion of flagfile (italic and lower opacity). The user can accept the adjustment using the established TAB shortcut.
• Figure 3: Natural language to code feature in Cider V. When users need help to write code, they can move their cursor to where the code can be inserted, and trigger the feature so that a pop-up window hovers above the code, and they can enter their query. The prompt here is "implement also for days", and the feature suggests the code that would match the prompt. The user can then review the suggestion and click a button to add the code to their file.
• Figure 4: Study design: Activities and randomization
• Figure 5: Theoretical framework: Factors predicting time on task