The Grammar of Interactive Explanatory Model Analysis

TL;DR

The paper tackles the problem that single-explanation methods are insufficient to understand opaque predictive models, due to the Rashomon effect. It introduces Interactive Explanatory Model Analysis (IEMA), a formal grammar and taxonomy for generating sequences of complementary explanations, enabling interactive human–model dialogues. Implemented in the open-source modelStudio framework, IEMA is validated through FIFA-20 and COVID-19 use-cases and a rigorously designed user study showing that sequential juxtaposition of explanations increases accuracy and confidence in human decision making. The work advocates a human-centered XIML paradigm with interactivity, customizability, and automation, and discusses practical challenges and responsibilities in model transparency and governance.

Abstract

The growing need for in-depth analysis of predictive models leads to a series of new methods for explaining their local and global properties. Which of these methods is the best? It turns out that this is an ill-posed question. One cannot sufficiently explain a black-box machine learning model using a single method that gives only one perspective. Isolated explanations are prone to misunderstanding, leading to wrong or simplistic reasoning. This problem is known as the Rashomon effect and refers to diverse, even contradictory, interpretations of the same phenomenon. Surprisingly, most methods developed for explainable and responsible machine learning focus on a single-aspect of the model behavior. In contrast, we showcase the problem of explainability as an interactive and sequential analysis of a model. This paper proposes how different Explanatory Model Analysis (EMA) methods complement each other and discusses why it is essential to juxtapose them. The introduced process of Interactive EMA (IEMA) derives from the algorithmic side of explainable machine learning and aims to embrace ideas developed in cognitive sciences. We formalize the grammar of IEMA to describe potential human-model dialogues. It is implemented in a widely used human-centered open-source software framework that adopts interactivity, customizability and automation as its main traits. We conduct a user study to evaluate the usefulness of IEMA, which indicates that an interactive sequential analysis of a model increases the performance and confidence of human decision making.

Figures (16)

• Figure 1: modelStudio automatically produces an HTML file - an interactive and customizable dashboard with model explanations and EDA visualizations. Here, we present a screenshot of its exemplary layout for the black-box model predicting a player's value on the FIFA-20 data, see https://iema.drwhy.ai.
• Figure 2: Increasing computing power and the availability of automated machine learning tools resulted in complex models that are effectively black-boxes. The first generation of model explanations aims at exploring individual aspects of model behavior. The second generation of model explanation aims to integrate individual aspects into a vibrant and multi-threaded customizable story about the black-box that addresses the needs of various stakeholders. We call this process Interactive Explanatory Model Analysis (IEMA).
• Figure 3: The concept of Interactive Explanatory Model Analysis shows how the various methods for model analysis enrich each other. Columns and rows span the taxonomy of explanations in IEMA, where names of well-known techniques are listed in cells. The graph's edges indicate complementary explanations.
• Figure 4: Left: SHAP Attributions to the model's prediction shows which variables are most important for a specific instance. Right: Ceteris Paribus shows the instance prediction profile for a specific variable.
• Figure 5: Left: Ceteris Paribus for the age variable shows the monotonicity of the instance prediction profile, for which values are large or small. Right: Histogram shows the distribution of the age variable's values.