Position: Explain to Question not to Justify

TL;DR

The paper argues that the XAI field suffers from competing goals, presenting a two-culture framework: BLUE XAI (human-values oriented) and RED XAI (model validation oriented). It shows that RED XAI is under-explored and outlines concrete challenges, such as constructing complementary explanations, establishing benchmarks and tooling, adopting an explorer mindset, and leveraging Rashomon sets for debugging and discovery. It contends that focusing on RED XAI can advance model robustness and safety, complementing user-centric explanations. The discussion includes a Covid-19 risk example to illustrate how BLUE and RED perspectives can converge and where specialized RED tools are needed. The work advocates for methodological diversification, education shifts, and the development of standards to accelerate RED XAI research.

Abstract

Explainable Artificial Intelligence (XAI) is a young but very promising field of research. Unfortunately, the progress in this field is currently slowed down by divergent and incompatible goals. We separate various threads tangled within the area of XAI into two complementary cultures of human/value-oriented explanations (BLUE XAI) and model/validation-oriented explanations (RED XAI). This position paper argues that the area of RED XAI is currently under-explored, i.e., more methods for explainability are desperately needed to question models (e.g., extract knowledge from well-performing models as well as spotting and fixing bugs in faulty models), and the area of RED XAI hides great opportunities and potential for important research necessary to ensure the safety of AI systems. We conclude this paper by presenting promising challenges in this area.

Figures (4)

• Figure 1: Summary of the main claims in this paper. The field of explainable machine learning (XAI) is currently experiencing a crisis of identity. Misconceptions about the role and goals of the XAI method are partly responsible for this crisis, and in the first chapter, we discuss the most prominent misconceptions. Having diagnosed the problems, we find that there are two communities of researchers working in the XAI field having different goals. In the second chapter, we identify these cultures and disentangle the main goals and motivations behind each. In addition, we find that the current narrative in survey papers is dominated by a discussion of the goals of the BLUE XAI culture. However, it is the RED XAI culture that has very promising research challenges ahead. A key conclusion of this diagnosis is to identify new challenges and research areas that concern RED XAI culture. These are presented in the third part of this work.
• Figure 2: Schematic diagram of multifaceted explanations. Many explanation techniques present one aspect of how models work. Multifaceted explanations show several complementary aspects simultaneously. Figure adapted from baniecki2023grammar
• Figure 3: Partial Dependence profiles for five different predictive models (linear model, decision tree, random forest, boosting model) for Covid-19 data. Each color indicates a different model fitted to the same data. The profiles show the conditional response of a model predicting mortality conditional on two variables: age and the presence of concomitant diseases. Figure adapted from RML
• Figure 4: Illustration of questions and needs raised by stakeholders interested in Blue XAI and Red XAI techniques. Questions are framed using the example of a hypothetical covid death risk assessment model used to prioritize access to vaccination. For each of the seven example areas, a sample question is presented that can be answered using XAI techniques.