Need Help? Designing Proactive AI Assistants for Programming

TL;DR

Proactive chat-based AI assistants can extend beyond reactive prompts to anticipate developer needs in programming tasks. The authors design and implement an IDE-integrated proactive assistant powered by LLMs, including a preview-diff mechanism and context-aware suggestion generation, and evaluate it through a randomized study. Results show productivity gains and nuanced user experiences dependent on timing, interaction mode, and feature set, with cross-cutting insights about design considerations and user behavior. The work offers practical guidance for building future proactive coding assistants that enhance workflow while maintaining control, trust, and minimal disruption.

Abstract

While current chat-based AI assistants primarily operate reactively, responding only when prompted by users, there is significant potential for these systems to proactively assist in tasks without explicit invocation, enabling a mixed-initiative interaction. This work explores the design and implementation of proactive AI assistants powered by large language models. We first outline the key design considerations for building effective proactive assistants. As a case study, we propose a proactive chat-based programming assistant that automatically provides suggestions and facilitates their integration into the programmer's code. The programming context provides a shared workspace enabling the assistant to offer more relevant suggestions. We conducted a randomized experimental study examining the impact of various design elements of the proactive assistant on programmer productivity and user experience. Our findings reveal significant benefits of incorporating proactive chat assistants into coding environments and uncover important nuances that influence their usage and effectiveness.

Figures (6)

• Figure 1: A walkthrough of the proactive assistant interface for coding. (A) Overview of suggestions, where the assistant provides a short description, and users can choose to expand a suggestion for more details (this can involve implementation and a brief explanation); (B) Buttons that allow participants to preview the implementation in their code, accept a suggestion to ask follow-up questions on, or delete the suggestion; (C) Integrating a suggestion into the editor via a diff format, where users can decide if they want to accept or hide the changes; (D) Invoking suggestions, where users can also request suggestions from the assistant (for example, if they did not like the original suggestion or wanted a suggestion before the assistant provided one).
• Figure 2: Prompting the assistant for suggestions. An overview of how different inputs (e.g., prior messages, current code, and terminal outputs) are incorporated in the generation of standard and debugging suggestions, the former which is shown generally and the latter which is only triggered when code is run. Full prompts for both types of suggestions are provided in the Supplementary Material.
• Figure 3: Percentage of sub-tasks completed correctly. Comparing baseline chat to proactive assistants across the four tasks, where Task 1 and Task 2 are system-building questions and Task 3 and 4 are ones where participants work with new packages and functionality. We report average performance and standard error. While performance varied by task, we observed that all variants of proactive assistants tended to increase the number of test cases passed compared to the baseline chat assistant across the board.
• Figure 4: Comparing participant perception of proactive versus baseline. User experience is important to the adoption of proactive assistants. We compare how often participants preferred a proactive assistant to the baseline variant in each condition and find that participants generally preferred the proactive variant only in the Suggest and Suggest and Preview conditions.
• Figure 5: Comparing interactions with different proactive conditions. Visualizing sampled participant trajectories from coding with different proactive assistants. We denote "expanding" a suggestion with E, "accepting" a suggestion with A, and "copying" a suggestion with C---all of which are available in every condition, and "previewing" a suggestion with P, which is only available in the Suggest and Preview condition. In the Persistent Suggestion condition, participants tend to expand suggestions without utilizing them further. In contrast, participants tend to copy expanded suggestions in the Suggest condition while participants in the Suggest and Preview condition tend to preview suggestions.