Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges

Jenny T. Liang, Chenyang Yang, Brad A. Myers

TL;DR

This study investigates the usability of AI programming assistants through a large-scale survey of 410 developers, revealing that users mainly value autocomplete, task speed, and syntax recall over brainstorming support. It shows widespread reliance on Copilot and organization-specific tools, with significant challenges in understanding what influences outputs and in controlling generations, especially when non-functional requirements are involved. The authors provide a detailed account of 10 successful use cases and 7 user-input strategies, and they offer concrete recommendations for tool design, such as better code-context understanding, configurable interactions, and explicit handling of non-functional constraints. Overall, the work highlights a practical gap between user needs and current AI code-generation capabilities, guiding future design toward low-cognitive-load, controllable, and context-aware interactions.

Abstract

The software engineering community recently has witnessed widespread deployment of AI programming assistants, such as GitHub Copilot. However, in practice, developers do not accept AI programming assistants' initial suggestions at a high frequency. This leaves a number of open questions related to the usability of these tools. To understand developers' practices while using these tools and the important usability challenges they face, we administered a survey to a large population of developers and received responses from a diverse set of 410 developers. Through a mix of qualitative and quantitative analyses, we found that developers are most motivated to use AI programming assistants because they help developers reduce key-strokes, finish programming tasks quickly, and recall syntax, but resonate less with using them to help brainstorm potential solutions. We also found the most important reasons why developers do not use these tools are because these tools do not output code that addresses certain functional or non-functional requirements and because developers have trouble controlling the tool to generate the desired output. Our findings have implications for both creators and users of AI programming assistants, such as designing minimal cognitive effort interactions with these tools to reduce distractions for users while they are programming.

A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges

TL;DR

This study investigates the usability of AI programming assistants through a large-scale survey of 410 developers, revealing that users mainly value autocomplete, task speed, and syntax recall over brainstorming support. It shows widespread reliance on Copilot and organization-specific tools, with significant challenges in understanding what influences outputs and in controlling generations, especially when non-functional requirements are involved. The authors provide a detailed account of 10 successful use cases and 7 user-input strategies, and they offer concrete recommendations for tool design, such as better code-context understanding, configurable interactions, and explicit handling of non-functional constraints. Overall, the work highlights a practical gap between user needs and current AI code-generation capabilities, guiding future design toward low-cognitive-load, controllable, and context-aware interactions.

Abstract

The software engineering community recently has witnessed widespread deployment of AI programming assistants, such as GitHub Copilot. However, in practice, developers do not accept AI programming assistants' initial suggestions at a high frequency. This leaves a number of open questions related to the usability of these tools. To understand developers' practices while using these tools and the important usability challenges they face, we administered a survey to a large population of developers and received responses from a diverse set of 410 developers. Through a mix of qualitative and quantitative analyses, we found that developers are most motivated to use AI programming assistants because they help developers reduce key-strokes, finish programming tasks quickly, and recall syntax, but resonate less with using them to help brainstorm potential solutions. We also found the most important reasons why developers do not use these tools are because these tools do not output code that addresses certain functional or non-functional requirements and because developers have trouble controlling the tool to generate the desired output. Our findings have implications for both creators and users of AI programming assistants, such as designing minimal cognitive effort interactions with these tools to reduce distractions for users while they are programming.
Paper Structure (50 sections, 2 figures, 4 tables)

This paper contains 50 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Participants
  5. Survey
  6. Analysis
  7. Usage Characteristics
  8. Usage patterns
  9. Motivation
  10. Successful use cases
  11. Repetitive code (78$\times$)
  12. Code with simple logic (68$\times$)
  13. Autocomplete (28$\times$)
  14. Quality assurance (21$\times$)
  15. Proof-of-concepts (20$\times$)
  16. ...and 35 more sections

Figures (2)

  • Figure 1: An overview of the topics covered in our usability study of AI programming assistants.
  • Figure 2: A subset of the actual survey questions about the usability of AI programming assistants. An open-ended question is indicated with a star (). The complete survey instrument is in the supplemental materials supplemental-materials.