Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond the Commit: Developer Perspectives on Productivity with AI Coding Assistants

Valerie Chen, Jasmyn He, Behnjamin Williams, Jason Valentino, Ameet Talwalkar

TL;DR

This paper tackles how to measure developer productivity in the era of AI coding assistants by conducting a sequential mixed-methods study at BNY Mellon, combining a large-scale DX survey with in-depth interviews. It demonstrates that productivity is multi-dimensional, identifying six factors that span development, deployment, and long-term career impact, including self-sufficiency, cognitive load, task throughput, peer review, technical expertise, and ownership. The work shows that high tool satisfaction does not guarantee substantial time savings and argues that existing frameworks like SPACE and DORA miss long-term considerations, proposing a holistic, context-aware evaluation approach. The findings offer operational guidance for both industry and academia to measure AI-assisted productivity more comprehensively and responsibly, with implications for onboarding, skill development, and workflow design.

Abstract

Measuring developer productivity is a topic that has attracted attention from both academic research and industrial practice. In the age of AI coding assistants, it has become even more important for both academia and industry to understand how to measure their impact on developer productivity, and to reconsider whether earlier measures and frameworks still apply. This study analyzes the validity of different approaches to evaluating the productivity impacts of AI coding assistants by leveraging mixed-method research. At BNY Mellon, we conduct a survey with 2989 developer responses and 11 in-depth interviews. Our findings demonstrate that a multifaceted approach is needed to measure AI productivity impacts: survey results expose conflicting perspectives on AI tool usefulness, while interviews elicit six distinct factors that capture both short-term and long-term dimensions of productivity. In contrast to prior work, our factors highlight the importance of long-term metrics like technical expertise and ownership of work. We hope this work encourages future research to incorporate a broader range of human-centered factors, and supports industry in adopting more holistic approaches to evaluating developer productivity.

Beyond the Commit: Developer Perspectives on Productivity with AI Coding Assistants

TL;DR

This paper tackles how to measure developer productivity in the era of AI coding assistants by conducting a sequential mixed-methods study at BNY Mellon, combining a large-scale DX survey with in-depth interviews. It demonstrates that productivity is multi-dimensional, identifying six factors that span development, deployment, and long-term career impact, including self-sufficiency, cognitive load, task throughput, peer review, technical expertise, and ownership. The work shows that high tool satisfaction does not guarantee substantial time savings and argues that existing frameworks like SPACE and DORA miss long-term considerations, proposing a holistic, context-aware evaluation approach. The findings offer operational guidance for both industry and academia to measure AI-assisted productivity more comprehensively and responsibly, with implications for onboarding, skill development, and workflow design.

Abstract

Measuring developer productivity is a topic that has attracted attention from both academic research and industrial practice. In the age of AI coding assistants, it has become even more important for both academia and industry to understand how to measure their impact on developer productivity, and to reconsider whether earlier measures and frameworks still apply. This study analyzes the validity of different approaches to evaluating the productivity impacts of AI coding assistants by leveraging mixed-method research. At BNY Mellon, we conduct a survey with 2989 developer responses and 11 in-depth interviews. Our findings demonstrate that a multifaceted approach is needed to measure AI productivity impacts: survey results expose conflicting perspectives on AI tool usefulness, while interviews elicit six distinct factors that capture both short-term and long-term dimensions of productivity. In contrast to prior work, our factors highlight the importance of long-term metrics like technical expertise and ownership of work. We hope this work encourages future research to incorporate a broader range of human-centered factors, and supports industry in adopting more holistic approaches to evaluating developer productivity.
Paper Structure (39 sections, 2 figures, 3 tables)

This paper contains 39 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background on productivity in software engineering.
  4. Measuring productivity with AI coding assistants.
  5. Research Design
  6. Survey
  7. Participants.
  8. Protocol.
  9. Analysis.
  10. Interview
  11. Participants.
  12. Protocol.
  13. Analysis.
  14. Results
  15. RQ1: Do AI productivity metrics align with each other?
  16. ...and 24 more sections

Figures (2)

  • Figure 1: Survey responses from 2989 engineers on productivity with GitHub Copilot. In the left and middle panels, we show two different metrics relating to general satisfaction and perceived time savings. In the right panel, we provide a fine-grained breakdown of the two metrics.
  • Figure 2: Mapping use cases of GitHub Copilot to productivity metrics, both in terms of positive and negative impact based on interviewee sentiment.