The Contribution of XAI for the Safe Development and Certification of AI: An Expert-Based Analysis

TL;DR

The paper investigates whether eXplainable AI (XAI) can support safe development and certification of AI systems in the context of rising regulation. Through fifteen expert interviews, it distinguishes XAI’s role as a debugging aid from its potential as a certification instrument, finding that XAI can reveal biases and errors and improve data understanding but struggles to provide robust, verifiable assurances required by certification. Practitioners also raise concerns about trust, reproducibility, and the double-black-box nature of explanations, suggesting that XAI alone cannot satisfy rigorous certification demands. The study highlights the need for new explanation paradigms, domain- and user-centric evaluation, and clearer regulatory metrics, guiding policymakers and practitioners toward more feasible integration of XAI into safety-critical AI governance.

Abstract

Developing and certifying safe - or so-called trustworthy - AI has become an increasingly salient issue, especially in light of upcoming regulation such as the EU AI Act. In this context, the black-box nature of machine learning models limits the use of conventional avenues of approach towards certifying complex technical systems. As a potential solution, methods to give insights into this black-box - devised in the field of eXplainable AI (XAI) - could be used. In this study, the potential and shortcomings of such methods for the purpose of safe AI development and certification are discussed in 15 qualitative interviews with experts out of the areas of (X)AI and certification. We find that XAI methods can be a helpful asset for safe AI development, as they can show biases and failures of ML-models, but since certification relies on comprehensive and correct information about technical systems, their impact is expected to be limited.