Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cracking CodeWhisperer: Analyzing Developers' Interactions and Patterns During Programming Tasks

Jeena Javahar, Tanya Budhrani, Manaal Basha, Cleidson R. B. de Souza, Ivan Beschastnikh, Gema Rodriguez-Perez

TL;DR

This study investigates how developers interact with CodeWhisperer, an LLM-based code generator, by conducting two user studies and deploying CodeWatcher to capture low-level IDE events. It identifies four behavioral patterns—incremental code refinement, explicit natural language prompts, baseline structuring with model suggestions, and integrative use with external sources—and links these patterns to how often AI suggestions are retained in final code. Retention of AI-generated content increases with continued use, suggesting growing trust and reliance on CodeWhisperer over time. The findings offer design guidance for code-generation tools, including supporting partial acceptance, capturing guiding NL comments, and integrating external validation to improve productivity and user confidence.

Abstract

The use of AI code-generation tools is becoming increasingly common, making it important to understand how software developers are adopting these tools. In this study, we investigate how developers engage with Amazon's CodeWhisperer, an LLM-based code-generation tool. We conducted two user studies with two groups of 10 participants each, interacting with CodeWhisperer - the first to understand which interactions were critical to capture and the second to collect low-level interaction data using a custom telemetry plugin. Our mixed-methods analysis identified four behavioral patterns: 1) incremental code refinement, 2) explicit instruction using natural language comments, 3) baseline structuring with model suggestions, and 4) integrative use with external sources. We provide a comprehensive analysis of these patterns .

Cracking CodeWhisperer: Analyzing Developers' Interactions and Patterns During Programming Tasks

TL;DR

This study investigates how developers interact with CodeWhisperer, an LLM-based code generator, by conducting two user studies and deploying CodeWatcher to capture low-level IDE events. It identifies four behavioral patterns—incremental code refinement, explicit natural language prompts, baseline structuring with model suggestions, and integrative use with external sources—and links these patterns to how often AI suggestions are retained in final code. Retention of AI-generated content increases with continued use, suggesting growing trust and reliance on CodeWhisperer over time. The findings offer design guidance for code-generation tools, including supporting partial acceptance, capturing guiding NL comments, and integrating external validation to improve productivity and user confidence.

Abstract

The use of AI code-generation tools is becoming increasingly common, making it important to understand how software developers are adopting these tools. In this study, we investigate how developers engage with Amazon's CodeWhisperer, an LLM-based code-generation tool. We conducted two user studies with two groups of 10 participants each, interacting with CodeWhisperer - the first to understand which interactions were critical to capture and the second to collect low-level interaction data using a custom telemetry plugin. Our mixed-methods analysis identified four behavioral patterns: 1) incremental code refinement, 2) explicit instruction using natural language comments, 3) baseline structuring with model suggestions, and 4) integrative use with external sources. We provide a comprehensive analysis of these patterns .

Paper Structure

This paper contains 13 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. The CodeWatcher Plugin
  5. Phase 2: Second User Study
  6. Phase 3: Post Processing and Data Analysis
  7. Results
  8. RQ1: How do users interact with and edit code using CodeWhisperer to solve programming tasks, and what detailed interaction patterns emerged from that?
  9. RQ2: What fraction of CodeWhisperer’s suggestions are retained by users?
  10. Discussion
  11. Conclusion and Future Work
  12. Additional Figures
  13. Additional Figures

Figures (2)

  • Figure 1: Study Phases
  • Figure 2: Comparison of participant interactions across two tasks. To simplify the Figure, we aggregated related interaction variables. For instance, CGI and PGI are presented only as "insertions".