In-IDE Human-AI Experience in the Era of Large Language Models; A Literature Review

TL;DR

This paper surveys the in-IDE Human-AI Experience (HAX) literature, focusing on LLM-driven AI assistants embedded in IDEs. It analyzes 36 studies published between 2020 and 2024 from major digital libraries to synthesize three core research branches: Design (UI principles and integration), Impact (workflow, productivity, and trust), and Quality (correctness, understandability, and security). The review highlights how UI design shapes usefulness, how AI assistance reshapes developers' workflows with trade-offs between speed and validation, and how model quality and safety influence adoption. It also outlines future directions, including task-specific interfaces, trust-enhancing mechanisms, and readability-focused model tuning to improve practical impact in software development.

Abstract

Integrated Development Environments (IDEs) have become central to modern software development, especially with the integration of Artificial Intelligence (AI) to enhance programming efficiency and decision-making. The study of in-IDE Human-AI Experience is critical in understanding how these AI tools are transforming the software development process, impacting programmer productivity, and influencing code quality. We conducted a literature review to study the current state of in-IDE Human-AI Experience research, bridging a gap in understanding the nuanced interactions between programmers and AI assistants within IDEs. By analyzing 36 selected papers, our study illustrates three primary research branches: Design, Impact, and Quality of Interaction. The trends, challenges, and opportunities identified in this paper emphasize the evolving landscape of software development and inform future directions for research and development in this dynamic field. Specifically, we invite the community to investigate three aspects of these interactions: designing task-specific user interface, building trust, and improving readability.