Developer Interaction Patterns with Proactive AI: A Five-Day Field Study

TL;DR

The paper addresses the challenge of prompting-heavy in-IDE AI tools by introducing ProAIDE, a proactive code-quality assistant embedded in Fleet. It reports a five-day field study with 15 professional developers, analyzing 5,732 interaction points and 229 proactive interventions to understand when proactive suggestions are most receptive. Key findings show that timing matters: post-commit interventions achieve about 52% engagement, while mid-task prompts are often dismissed, and proactive suggestions are interpreted faster than reactive ones, indicating improved workflow efficiency. The study provides actionable design implications for timing, context-awareness, explainability, and user control in production IDEs, and offers replication materials to guide future development of proactive AI in software engineering.

Abstract

Current in-IDE AI coding tools typically rely on time-consuming manual prompting and context management, whereas proactive alternatives that anticipate developer needs without explicit invocation remain underexplored. Understanding when humans are receptive to such proactive AI assistance during their daily work remains an open question in human-AI interaction research. We address this gap through a field study of proactive AI assistance in professional developer workflows. We present a five-day in-the-wild study with 15 developers who interacted with a proactive feature of an AI assistant integrated into a production-grade IDE that offers code quality suggestions based on in-IDE developer activity. We examined 229 AI interventions across 5,732 interaction points to understand how proactive suggestions are received across workflow stages, how developers experience them, and their perceived impact. Our findings reveal systematic patterns in human receptivity to proactive suggestions: interventions at workflow boundaries (e.g., post-commit) achieved 52% engagement rates, while mid-task interventions (e.g., on declined edit) were dismissed 62% of the time. Notably, well-timed proactive suggestions required significantly less interpretation time than reactive suggestions (45.4s versus 101.4s, W = 109.00, r = 0.533, p = 0.0016), indicating enhanced cognitive alignment. This study provides actionable implications for designing proactive coding assistants, including how to time interventions, align them with developer context, and strike a balance between AI agency and user control in production IDEs.

Figures (5)

• Figure 1: User interface for proactive intervention at an ambiguous prompt. (a) Lightweight in-editor cue indicating AI activity. (b) Opened chat panel with: (b1) suggestion title, (b2) rationale, and (b3) code snippet. The UI for declined AI edit followed the same structure. For post-commit, the interaction interface remained, but the visual cue (a) appeared as a popup in the upper-right corner after a successful commit.
• Figure 2: Main stages of the code iteration workflow and corresponding points where ProAIDE triggers proactive suggestions.
• Figure 3: Rates of interaction with proactive AI interventions across various development stages: formulating needs (Ambiguous Prompt), executing the idea (Declined Edit) and finishing writing code (Commit Changes). Users had the option to Engage, Dismiss, or Ignore AI interventions. Active participation decreased from 18 to 8 users between Day 1 and Day 5, yet interaction patterns remained stable across intervention types. Commit Changes interventions achieve the highest engagement (52%), while Declined Edit interventions show consistently low engagement (31%). In total, we recorded 229 proactive AI interventions over the 5-day study period.
• Figure 4: System Usability Scale and user experience responses from 15 participants who completed the entire study period. Results show above-average usability ($SUS = 72.8$).
• Figure 5: Interpretation time comparison between proactively triggered chat sessions ($\mu=45.4s$, $N=33$) and reactively triggered ones ($\mu=101.4s$, $N=50$). Proactive suggestions demonstrate significantly faster interpretation times ($p=0.0016$, Wilcoxon signed-rank test), suggesting improved efficiency when AI assistance is tailored to the user's workflow context.