Explainability for Embedding AI: Aspirations and Actuality

TL;DR

The paper investigates whether explainability techniques can assist software developers in building reliable AI-enabled software. It conducts three online surveys to assess demand for explanations, perceptions of XAI tools (notably LIME and SHAP), and the potential of explanations to help fault recognition. Results show strong overall demand for explanations across AI-powered and traditional apps, but user experiences with existing XAI tools are mixed and trust-related concerns persist, especially among less experienced developers; experienced developers show somewhat higher confidence in interpreting explanations. The authors argue for a developer-focused, human-centered approach to XAI research and tool design to improve debugging, fault detection, and the overall quality of AI-embedded software.

Abstract

With artificial intelligence (AI) embedded in many everyday software systems, effectively and reliably developing and maintaining AI systems becomes an essential skill for software developers. However, the complexity inherent to AI poses new challenges. Explainable AI (XAI) may allow developers to understand better the systems they build, which, in turn, can help with tasks like debugging. In this paper, we report insights from a series of surveys with software developers that highlight that there is indeed an increased need for explanatory tools to support developers in creating AI systems. However, the feedback also indicates that existing XAI systems still fall short of this aspiration. Thus, we see an unmet need to provide developers with adequate support mechanisms to cope with this complexity so they can embed AI into high-quality software in the future.