"Over-the-Hood" AI Inclusivity Bugs and How 3 AI Product Teams Found and Fixed Them

TL;DR

The paper defines and investigates 'over-the-hood' AI inclusivity bugs—user-facing issues in AI products that disproportionately affect users with specific problem-solving approaches. Through a field study with three AI product teams (Game, Weather, Farm) using GenderMag, it identifies six AI inclusivity bug types across 83 instances and demonstrates fixes for 47 of them, while introducing GenderMag-for-AI variants to address AI-specific challenges. An external study corroborates that post-GenderMag fixes improve users' mental models and gender equity, highlighting practical benefits for real-world AI deployment. The work shows that while the original GenderMag method helps uncover AI inclusivity bugs, AI-specific adaptations—especially a Pre-Action Fork variant—better capture user doubt about AI outputs and guide more effective fixes. Collectively, the findings offer actionable guidance for AI teams to design more inclusive user-facing AI systems by aligning explanations and interactions with diverse problem-solving styles and trust dynamics.

Abstract

While much research has shown the presence of AI's "under-the-hood" biases (e.g., algorithmic, training data, etc.), what about "over-the-hood" inclusivity biases: barriers in user-facing AI products that disproportionately exclude users with certain problem-solving approaches? Recent research has begun to report the existence of such biases -- but what do they look like, how prevalent are they, and how can developers find and fix them? To find out, we conducted a field study with 3 AI product teams, to investigate what kinds of AI inclusivity bugs exist uniquely in user-facing AI products, and whether/how AI product teams might harness an existing (non-AI-oriented) inclusive design method to find and fix them. The teams' work resulted in identifying 6 types of AI inclusivity bugs arising 83 times, fixes covering 47 of these bug instances, and a new variation of the GenderMag inclusive design method, GenderMag-for-AI, that is especially effective at detecting certain kinds of AI inclusivity bugs.

Figures (22)

• Figure 1: An AI-powered agricultural tool for predicting when crops are at risk of dying from cold temperatures (y-axis) over time (x-axis). The risky events are where the AI-predictions for this crop (yellow line) intersect with the forecast's low temperature (blue line).
• Figure 2: Number of "Abi" problem-solving style values reported by the women (left bar in each pair, bright orange) and men (right bars, dark blue) in Anderson et al. anderson2024measuring. X-axis: number of Abi problem-solving style values (Table \ref{['tab:Background-Persona-Table']}). Y-axis: number of participants having this number of Abi problem-solving style values. Statistically, men skewed significantly more toward "Tim" problem-solving style values than women did.
• Figure 3: (Left): The GenderMag walkthrough graph. Each node represents a step in the walkthrough and is repeated in a loop until the overall goal is complete. (Right): The GenderMag walkthrough questions at each step.
• Figure 4: Team Game’s eXplainable AI (XAI) interface. On the left is Team Game's gameboard, on the top-right is the Scores Best-to-Worst explanation, on the middle-right is the Scores Through-Time explanation, and on the bottom-right is the Scores On-the-Board explanation. Call-outs with enlarged portions have been superimposed for readability.
• Figure 5: Team Weather AI's $LTE_{10}$ cold-hardiness predictions (yellow line) for different days (x-axis). The $LTE_{10}$ predictions were compared to minimum forecasted temperature (blue line), and when $LTE_{10}$$>$ the minimum, the interface marked these events with a red diamond to help agriculturalists decide whether and how to deploy frost-mitigation methods.