Seamful XAI: Operationalizing Seamful Design in Explainable AI

TL;DR

This paper introduces Seamful XAI, a design approach that leverages sociotechnical seams—mismatches between AI design assumptions and deployment reality—to enhance explainability and user agency. It operationalizes seams into a three-step design process (envisioning breakdowns, anticipating and crafting seams, designing with seams) and validates it through a scenario-based study with 43 practitioners and end-users in lending, healthcare, and other domains. Findings show the process helps identify, craft, and design with seams, and demonstrates tangible benefits: improved explainability via peripheral context and why-not reasoning, and augmented agency through actionability, contestability, and appropriation, all while supporting proactive harm mitigation. The work expands the XAI design space beyond algorithmic transparency, offers practical methods for Responsible AI, and presents a teachable, collaborative framework for cross-disciplinary teams to anticipate failures and empower end users to act when AI falls short.

Abstract

Mistakes in AI systems are inevitable, arising from both technical limitations and sociotechnical gaps. While black-boxing AI systems can make the user experience seamless, hiding the seams risks disempowering users to mitigate fallouts from AI mistakes. Instead of hiding these AI imperfections, can we leverage them to help the user? While Explainable AI (XAI) has predominantly tackled algorithmic opaqueness, we propose that seamful design can foster AI explainability by revealing and leveraging sociotechnical and infrastructural mismatches. We introduce the concept of Seamful XAI by (1) conceptually transferring "seams" to the AI context and (2) developing a design process that helps stakeholders anticipate and design with seams. We explore this process with 43 AI practitioners and real end-users, using a scenario-based co-design activity informed by real-world use cases. We found that the Seamful XAI design process helped users foresee AI harms, identify underlying reasons (seams), locate them in the AI's lifecycle, learn how to leverage seamful information to improve XAI and user agency. We share empirical insights, implications, and reflections on how this process can help practitioners anticipate and craft seams in AI, how seamfulness can improve explainability, empower end-users, and facilitate Responsible AI.

Figures (6)

• Figure 1: An overview of the Seamful XAI design process with key questions relevant to each step.
• Figure 2: A screenshot of the virtual whiteboard used for the seamful XAI design activity in the study, with zoomed-in examples. Area 1: Envisioning breakdown (Step 1). In the study, we provided sample breakdowns (A), which participants could either use directly or get inspiration for their own envisioning. Area 2: Anticipating & crafting seams (Step 2). We provided guiding prompts (B) for effectively crafting the seams. We also shared exemplary seams (C) for each stage of the AI lifecycle framework. Area 3: Designing with seams (Step 3). We asked participants to articulate their reasoning for choosing a seam and tag which user goals the selected seam (E) can support for augmenting user agency. (Appendix \ref{['sec:appendix_high_res_whiteboard']} has a non-annotated higher resolution version of this picture.)
• Figure 3: A visual presentation of the lending scenario used in our study showcasing the "backstory" of each persona. The different loan officer personas were used by participants to think of variations in the scenario to generate new breakdowns and seams.
• Figure 4: A bird's eye view of all seams from all participants for all breakdowns along all AI lifecycle. A: the dots above this breakdown provide the number of times it was chosen (e.g. 17 times). All the seams appearing below this column were crafted in connection to this particular breakdown. B: the dots showcase the number of seams crafted for a given AI lifecycle stage (6 seams were crafted for this stage). C: a zoomed-in view to provide a sample of seams crafted by our participants along with the justifications for how the seams enhances agency. (Appendix \ref{['sec:appendix_high_res_allseams']} has a non-annotated higher resolution version of this picture.)
• Figure 5: A higher resolution picture of Figure \ref{['fig:mural_design_process']} showing a screenshot of the virtual whiteboard used for the seamful XAI design activity in the study, with zoomed-in examples. Area 1: Envisioning breakdown (Step 1). In the study, we provided sample breakdowns, which participants could either use directly or get inspiration for their own envisioning. Area 2: Anticipating & crafting seams (Step 2). We provided guiding prompts for crafting the seams can procedurally guide the process of writing down the seam in an effective manner. We also shared exemplary seams for each stage of the AI lifecycle framework. Area 3: Designing with seams. We asked participants to articulate their reasoning for choosing a seam and tag which user goals the selected seam can support for augmenting user agency. If viewed on a PDF viewer like Adobe Acrobat, the reader can zoom in as necessary to read the text on this board.