Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using an LLM to Help With Code Understanding

Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, Brad Myers

TL;DR

The paper investigates an in-IDE LLM-based information support tool, GILT, designed to help programmers understand unfamiliar code without requiring explicit prompts. Through a 32-participant within-subject user study across data-visualization and 3D-rendering tasks, GILT improves task progress compared with web search, though it does not significantly change completion time or code understanding. Findings reveal that benefits vary by user type, with professionals gaining more from GILT, and reveal rich insights into how users interact with prompt-based versus prompt-less features. The work highlights the promise of prompt-less, context-aware LLM assistance integrated directly into the IDE, while underscoring the need for better UI, broader contexts, and real-world evaluation to realize practical impact.

Abstract

Understanding code is challenging, especially when working in new and complex development environments. Code comments and documentation can help, but are typically scarce or hard to navigate. Large language models (LLMs) are revolutionizing the process of writing code. Can they do the same for helping understand it? In this study, we provide a first investigation of an LLM-based conversational UI built directly in the IDE that is geared towards code understanding. Our IDE plugin queries OpenAI's GPT-3.5-turbo model with four high-level requests without the user having to write explicit prompts: to explain a highlighted section of code, provide details of API calls used in the code, explain key domain-specific terms, and provide usage examples for an API. The plugin also allows for open-ended prompts, which are automatically contextualized to the LLM with the program being edited. We evaluate this system in a user study with 32 participants, which confirms that using our plugin can aid task completion more than web search. We additionally provide a thorough analysis of the ways developers use, and perceive the usefulness of, our system, among others finding that the usage and benefits differ between students and professionals. We conclude that in-IDE prompt-less interaction with LLMs is a promising future direction for tool builders.

Using an LLM to Help With Code Understanding

TL;DR

The paper investigates an in-IDE LLM-based information support tool, GILT, designed to help programmers understand unfamiliar code without requiring explicit prompts. Through a 32-participant within-subject user study across data-visualization and 3D-rendering tasks, GILT improves task progress compared with web search, though it does not significantly change completion time or code understanding. Findings reveal that benefits vary by user type, with professionals gaining more from GILT, and reveal rich insights into how users interact with prompt-based versus prompt-less features. The work highlights the promise of prompt-less, context-aware LLM assistance integrated directly into the IDE, while underscoring the need for better UI, broader contexts, and real-world evaluation to realize practical impact.

Abstract

Understanding code is challenging, especially when working in new and complex development environments. Code comments and documentation can help, but are typically scarce or hard to navigate. Large language models (LLMs) are revolutionizing the process of writing code. Can they do the same for helping understand it? In this study, we provide a first investigation of an LLM-based conversational UI built directly in the IDE that is geared towards code understanding. Our IDE plugin queries OpenAI's GPT-3.5-turbo model with four high-level requests without the user having to write explicit prompts: to explain a highlighted section of code, provide details of API calls used in the code, explain key domain-specific terms, and provide usage examples for an API. The plugin also allows for open-ended prompts, which are automatically contextualized to the LLM with the program being edited. We evaluate this system in a user study with 32 participants, which confirms that using our plugin can aid task completion more than web search. We additionally provide a thorough analysis of the ways developers use, and perceive the usefulness of, our system, among others finding that the usage and benefits differ between students and professionals. We conclude that in-IDE prompt-less interaction with LLMs is a promising future direction for tool builders.
Paper Structure (24 sections, 5 figures, 3 tables)

This paper contains 24 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Studies on Developers Information Seeking
  4. Studies on LLM-based Developer Tools
  5. The GILT Prototype Tool
  6. Interacting with GILT
  7. Our Design Process and Decisions
  8. Human Study Design
  9. RQ1: Effects of GILT
  10. Data Collection
  11. Methodology
  12. Results
  13. Additional Analysis
  14. RQ2: GILT Usage
  15. Usage of Features
  16. ...and 9 more sections

Figures (5)

  • Figure 1: Overview of our prototype. (1) A trigger button; (2) code used as context when prompting LLM; (3) code summary (no-prompt trigger); (4) buttons for further details; (5) an input box for user prompts; (6) options to embed information to code (Embed) and a hide/view button; (7) options to clear the panel (Clear all) and an abort LLM button; (8) a refresh button.
  • Figure 2: A 3D-rendering example sub-task (open3d-3). With these start and goal outputs, we asked the participants to "Make the bunny sit upright on the chair." See Figure \ref{['fig:tool']} for the corresponding starter code and the tool output.
  • Figure 3: The sequences of feature usage in GILT. Each row corresponds to an individual participant, and the color cells are arranged chronologically, from left to right.
  • Figure 4: Participants' report on the importance of GILT features.
  • Figure 5: Transition Graphs for User Interaction. Each node displays the number of times users interacted with respective features, and each edge indicates the counted number of transitions between the connected features. For space and readability reasons, Prompt, Prompt-Context, and Prompt-Followup are merged into prompt, and API, Concept, and Usage are merged into buttons. Counts lower than 5 are omitted except for the edges connected to the 'Success' and 'Fail' nodes.