Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bug Detective and Quality Coach: Developers' Mental Models of AI-Assisted IDE Tools

Paolo Buono, Mary Cerullo, Stefano Cirillo, Giuseppe Desolda, Francesco Greco, Emanuela Guglielmi, Grazia Margarella, Giuseppe Polese, Simone Scalabrino, Cesare Tucci

TL;DR

This study investigates how developers conceptually and practically interact with AI-assisted IDEs for bug detection and code readability. Using six co-design workshops with 58 participants, the authors identify two core mental models—Bug Detective and Quality Coach—and derive seven design principles for Human-Centered AI in IDEs that balance disruption with support, conciseness with depth, and automation with human agency. The findings reveal nuanced requirements for explanations, timing, placement, and personalization to foster trust and effective collaboration between humans and AI, with concrete prototype concepts illustrating how these principles can be realized. The work advances the design of AI-assisted development tools by foregrounding user-centered explanations, adaptive timing, and configurable controls, with implications for broader AI-enabled evaluative systems beyond software engineering.

Abstract

AI-assisted tools support developers in performing cognitively demanding tasks such as bug detection and code readability assessment. Despite the advancements in the technical characteristics of these tools, little is known about how developers mentally model them and how mismatches affect trust, control, and adoption. We conducted six co-design workshops with 58 developers to elicit their mental models about AI-assisted bug detection and readability features. It emerged that developers conceive bug detection tools as \textit{bug detectives}, which warn users only in case of critical issues, guaranteeing transparency, actionable feedback, and confidence cues. Readability assessment tools, on the other hand, are envisioned as \textit{quality coaches}, which provide contextual, personalized, and progressive guidance. Trust, in both tasks, depends on the clarity of explanations, timing, and user control. A set of design principles for Human-Centered AI in IDEs has been distilled, aiming to balance disruption with support, conciseness with depth, and automation with human agency.

Bug Detective and Quality Coach: Developers' Mental Models of AI-Assisted IDE Tools

TL;DR

This study investigates how developers conceptually and practically interact with AI-assisted IDEs for bug detection and code readability. Using six co-design workshops with 58 participants, the authors identify two core mental models—Bug Detective and Quality Coach—and derive seven design principles for Human-Centered AI in IDEs that balance disruption with support, conciseness with depth, and automation with human agency. The findings reveal nuanced requirements for explanations, timing, placement, and personalization to foster trust and effective collaboration between humans and AI, with concrete prototype concepts illustrating how these principles can be realized. The work advances the design of AI-assisted development tools by foregrounding user-centered explanations, adaptive timing, and configurable controls, with implications for broader AI-enabled evaluative systems beyond software engineering.

Abstract

AI-assisted tools support developers in performing cognitively demanding tasks such as bug detection and code readability assessment. Despite the advancements in the technical characteristics of these tools, little is known about how developers mentally model them and how mismatches affect trust, control, and adoption. We conducted six co-design workshops with 58 developers to elicit their mental models about AI-assisted bug detection and readability features. It emerged that developers conceive bug detection tools as \textit{bug detectives}, which warn users only in case of critical issues, guaranteeing transparency, actionable feedback, and confidence cues. Readability assessment tools, on the other hand, are envisioned as \textit{quality coaches}, which provide contextual, personalized, and progressive guidance. Trust, in both tasks, depends on the clarity of explanations, timing, and user control. A set of design principles for Human-Centered AI in IDEs has been distilled, aiming to balance disruption with support, conciseness with depth, and automation with human agency.

Paper Structure

This paper contains 61 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Mental Models
  4. Bug Detection in HCI
  5. Code Readability Assessment in HCI
  6. Methodology
  7. Study Design
  8. Participants
  9. Procedure
  10. Data Collection and Analysis
  11. Results
  12. Interaction with Bug Detection Alerts (Task A, Scenario 1)
  13. What information should alerts report?
  14. How should alerts be displayed?
  15. Where should alerts be displayed?
  16. ...and 46 more sections

Figures (5)

  • Figure 1: Metaphorical Representation of AI-Assisted IDE Mental Models. It illustrates the two primary mental models identified for AI-assisted tools: the "Bug Detective" (left) and the "Quality Coach" (right). It highlights how these "partners" collaborate with the developer to offer actionable feedback, critical warnings, and personalized guidance, emphasizing the crucial role of "Trust" fostered through explanation clarity, user control, and adaptive timing.
  • Figure 3: Themes emerged by analyzing the developers' answers to the questions on the bug detection task.
  • Figure 4: Themes emerged by analyzing the developers' answers to the questions on the code readability assessment task.
  • Figure 5: Full interface with the Alerts tab selected. The first alert is expanded in place, showing rationale, evidence, confidence, and a preview patch.
  • Figure 6: Full interface with the Readability tab selected. Hovering the Formatting factor highlights relevant (or representative) lines.