Understanding and supporting how developers prompt for LLM-powered code editing in practice

TL;DR

This paper investigates how developers prompt for LLM-powered code editing in practice by analyzing Transform Code usage at Google, constructing a dataset of unsatisfactory prompts, and evaluating automatic prompt enhancement. Through a three-phase study, it reveals common gaps in prompts (specifications, operationalization, scope, context, intent) and demonstrates that an auto-prompting tool, AutoPrompter, can improve edit correctness by about 27%. The work combines telemetry analysis, qualitative coding, and LL-driven augmentation to reduce cognitive load and improve communication with code-editing LLMs, while acknowledging limitations such as data scope and model limits. Overall, it highlights actionable directions for improving enterprise AI developer tools through targeted prompt engineering and model-alignment strategies.

Abstract

Large Language Models (LLMs) are rapidly transforming software engineering, with coding assistants embedded in an IDE becoming increasingly prevalent. While research has focused on improving the tools and understanding developer perceptions, a critical gap exists in understanding how developers actually use these tools in their daily workflows, and, crucially, where they struggle. This paper addresses part of this gap through a multi-phased investigation of developer interactions with an LLM-powered code editing feature, Transform Code, in an IDE widely used at Google. First, we analyze telemetry logs of the feature usage, revealing that frequent re-prompting can be an indicator of developer struggles with using Transform Code. Second, we conduct a qualitative analysis of unsatisfactory requests, identifying five key categories of information often missing from developer prompts. Finally, based on these findings, we propose and evaluate a tool, AutoPrompter, for automatically improving prompts by inferring missing information from the surrounding code context, leading to a 27% improvement in edit correctness on our test set.

Figures (4)

• Figure 1: Transform Code, an LLM-based code editing tool
• Figure 2: In-IDE developer action transition graphs. Each node displays an IDE action, and each edge indicates the proportion of transitions between the connected actions. Blue arrows show the first action taken right after a Transform Code request, and gray arrows indicate the second action following the first. For space and readability reasons, edges representing transition frequencies lower than 3% (for the first action) and 10% (for the second action) have been omitted.
• Figure 3: Distribution of the normalized edit distances between two consecutive prompts of the entire data, including prompts written for different tasks (top), and distributions of the normalized edit distances (middle) and the edit distances (bottom) between two consecutive prompts for the same task.
• Figure 4: Overview of the developer prompt error analysis (purple) and automatic prompt enhancement (green). For error analysis, the LLM was given (1) the original user prompt, (2) the original Transform Code edit, (3) the code context, and (4) the desired code edit, to identify missing context from the original user prompt. For automatic user prompt enhancement, AutoPrompter was instructed with the learned missing context from the error analysis, and given (1) the original user prompt, (2) the original Transform Code edit, and (3) the code context. Then, Transform Code generates the code edit using the improved prompt.