An LLM's Attempts to Adapt to Diverse Software Engineers' Problem-Solving Styles: More Inclusive & Equitable?

TL;DR

The paper investigates whether a code-explanation LLM can adapt to diverse software engineers problem-solving styles using GenderMag and evaluates inclusivity and equity outcomes. Using COBOL programs and a between-subjects design across five style treatments, the study finds that unadapted explanations are often inequitable, while matching adaptations can enhance both inclusivity and equity, though not universally and sometimes at the cost of other groups. Mismatches generally reduce equity and can undermine inclusivity, highlighting complex interactions among multiple style dimensions. The work advances inclusive UX in AI-assisted programming by experimentally examining the tradeoffs and proposing an optimization framing for multi-style adaptations.

Abstract

Software engineers use code-fluent large language models (LLMs) to help explain unfamiliar code, yet LLM explanations are not adapted to engineers' diverse problem-solving needs. We prompted an LLM to adapt to five problem-solving style types from an inclusive design method, the Gender Inclusiveness Magnifier (GenderMag). We ran a user study with software engineers to examine the impact of explanation adaptations on software engineers' perceptions, both for explanations which matched and mismatched engineers' problem-solving styles. We found that explanations were more frequently beneficial when they matched problem-solving style, but not every matching adaptation was equally beneficial; in some instances, diverse engineers found as much (or more) benefit from mismatched adaptations. Through an equity and inclusivity lens, our work highlights the benefits of having an LLM adapt its explanations to match engineers' diverse problem-solving style values, the potential harms when matched adaptations were not perceived well by engineers, and a comparison of how matching and mismatching LLM adaptations impacted diverse engineers.

Figures (13)

• Figure 1: Portion of the LLM prompt used for the "Tim"-like attitude towards risk. Each adaptation's prompt was composed using the same four sections (appended together into a single prompt), with the problem-solving style description and details changed to suit each "Tim"-like or "Abi"-like problem-solving styles. The '...' indicate text that has been removed for this figure for clarity.
• Figure 2: What participants saw in the information processing style treatment for llama-3's Boreas adaptation to the Infinite Loop COBOL program (Table \ref{['tab:05_cobol_programs']}). We used medium fidelity prototypes for this study because of prototypes' comparable success to interactive systems in the literature and also to maintain participants' focus on what was helpful/problematic about these outputs, rather than losing the participants to endless prompting or fixations on style/color/etc.
• Figure 3: Unadapted LLM Inequities: The number of inequitable ratings for each problem-solving style type (centered at the black axis). The LLMs were inequitable for all five problem-solving types, but the Orange (left of axis) "Abi"-like problem-solvers were more frequently disadvantaged than the Blue (right of axis) "Tim"-like problem-solvers.
• Figure 4: Inclusivity increases ($\rightarrow$) and decreases ($\leftarrow$) for process-oriented and tinkering-oriented learners' ratings. LLM-Learn-Process increased inclusivity in 12 of the process-oriented learners' ratings. LLM-Learn-Tinker increased inclusivity in 11 of the tinkering-oriented learners' ratings.
• Figure 5: Rating inequities (boxes) between process-oriented and tinkering-oriented learners for the unadapted LLM (left) and the same-value learning-style adapted LLMs (right). A majority of the unadapted LLM disadvantages went against the process-oriented learners. However, the inclusivity benefits (Figure \ref{['fig:01_learn_inclusion_for_me']}) improved equity, eliminating inequities in Cognitive Load and Utility.