Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE Sessions

Ningzhi Tang, Chaoran Chen, Zihan Fang, Gelei Xu, Maria Dhakal, Yiyu Shi, Collin McMillan, Yu Huang, Toby Jia-Jun Li

Abstract

IDE-integrated AI coding assistants, which operate conversationally within developers' working codebases with access to project context and multi-file editing, are rapidly reshaping software development. However, empirical investigation of this shift remains limited: existing studies largely rely on small-scale, controlled settings or analyze general-purpose chatbots rather than codebase-aware IDE workflows. We present, to the best of our knowledge, the first large-scale study of real-world conversational programming in IDE-native settings, analyzing 74,998 developer messages from 11,579 chat sessions across 1,300 repositories and 899 developers using Cursor and GitHub Copilot. These chats were committed to public repositories as part of routine development, capturing in-the-wild behavior. Our findings reveal three shifts in how programming work is organized: conversational programming operates as progressive specification, with developers iteratively refining outputs rather than specifying complete tasks upfront; developers redistribute cognitive work to AI, delegating diagnosis, comprehension, and validation rather than engaging with code and outputs directly; and developers actively manage the collaboration, externalizing plans into persistent artifacts, and negotiating AI autonomy through context injection and behavioral constraints. These results provide foundational empirical insights into AI-assisted development and offer implications for the design of future programming environments.

Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE Sessions

Abstract

IDE-integrated AI coding assistants, which operate conversationally within developers' working codebases with access to project context and multi-file editing, are rapidly reshaping software development. However, empirical investigation of this shift remains limited: existing studies largely rely on small-scale, controlled settings or analyze general-purpose chatbots rather than codebase-aware IDE workflows. We present, to the best of our knowledge, the first large-scale study of real-world conversational programming in IDE-native settings, analyzing 74,998 developer messages from 11,579 chat sessions across 1,300 repositories and 899 developers using Cursor and GitHub Copilot. These chats were committed to public repositories as part of routine development, capturing in-the-wild behavior. Our findings reveal three shifts in how programming work is organized: conversational programming operates as progressive specification, with developers iteratively refining outputs rather than specifying complete tasks upfront; developers redistribute cognitive work to AI, delegating diagnosis, comprehension, and validation rather than engaging with code and outputs directly; and developers actively manage the collaboration, externalizing plans into persistent artifacts, and negotiating AI autonomy through context injection and behavioral constraints. These results provide foundational empirical insights into AI-assisted development and offer implications for the design of future programming environments.

Paper Structure

This paper contains 36 sections, 4 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Empirical Studies of AI Programming in IDE
  4. Large-Scale Studies of Coding with LLMs
  5. Methodology
  6. Data Collection and Preparation
  7. Data Cleaning and Exclusion.
  8. Dataset Characteristics.
  9. Behavioral Intent Characterization
  10. Iterative Codebook Development
  11. LLM-Based Classification
  12. Inductive Qualitative Coding
  13. Session Archetypes and Dynamics Analysis
  14. Session Clustering
  15. Session Representation.
  16. ...and 21 more sections

Figures (6)

  • Figure 1: Illustrative conversational programming session annotated with behavioral intent labels. The interaction exemplifies the Failure-Driven Debugging archetype.
  • Figure 2: Co-occurrence patterns among behavioral intent subcategories. Cell values show the percentage of messages carrying the row label that also carry the column label; values below 1% are hidden. Diagonal cells are masked. Subcategory indices correspond to Table \ref{['tab:taxonomy']}.
  • Figure 3: Cumulative distribution of user messages per session ($n = 11{,}579$). Dotted lines mark selected percentiles.
  • Figure 4: Session archetype structure visualized as a t-SNE projection van2008visualizing of 4,864 sessions. Stars denote medoids.
  • Figure 5: Behavioral intent composition of each session archetype (% of messages per main category).
  • ...and 1 more figures