ExplainReduce: Summarising local explanations via proxies

TL;DR

ExplainReduce tackles the instability and redundancy of local explanations by aggregating them into a compact proxy set that globally explains a closed-box model. It formalises the reduction as optimization with coverage and fidelity objectives and demonstrates that greedy reduction can yield proxy sets that rival or exceed the full set in fidelity, while offering greater interpretability. The approach is agnostic to the underlying closed-box model and the local explanation method, and it generalises from limited initial explanations, enabling scalable, globally interpretable AI explanations. This has practical impact for delivering succinct, stable, and faithful global explanations in diverse domains, including synthetic benchmarks and scientific datasets like particle jets.

Abstract

Most commonly used non-linear machine learning methods are closed-box models, uninterpretable to humans. The field of explainable artificial intelligence (XAI) aims to develop tools to examine the inner workings of these closed boxes. An often-used model-agnostic approach to XAI involves using simple models as local approximations to produce so-called local explanations; examples of this approach include LIME, SHAP, and SLISEMAP. This paper shows how a large set of local explanations can be reduced to a small "proxy set" of simple models, which can act as a generative global explanation. This reduction procedure, ExplainReduce, can be formulated as an optimisation problem and approximated efficiently using greedy heuristics.

Figures (12)

• Figure 1: A simple example of the idea behind ExplainReduce. A closed-box model (left) can have many local explanations (middle). We can reduce the size of the local explanation set to get a global explanation consisting of two simple models (right).
• Figure 2: Ground PCA of a synthetic test dataset (left) and the same dataset where colours correspond to reduced model indices, based on smoothgrad local models and reduced using a greedy coverage-maximising algorithm (right). We can see that the reduction is able to faithfully approximate the ground truth clustering.
• Figure 3: Radar plots showing ground truth local model parameters (blue) and corresponding proxy model parameters (red). The reduction method is able to find very similar surrogate models to the ground truth.
• Figure 4: Example of the ExplainReduce procedure on a particle jet classification task. The dataset consists of LHC proton-proton collision particle jets that are created by decaying gluons or quarks. We train a random forest classifier on the dataset and use slisemap to generate 500 explanations, which are then reduced to $k=4$ proxies using the const min loss reduction algorithm. The left panel shows a swarm plot of the 500 items sorted horizontally based on the predicted probability of corresponding to a gluon; the y-axis has no significance. The right panel shows the coefficients of the logistic regression proxy models, which match the underlying physical theory.
• Figure 5: Test set fidelity of explanations as a function of proxy set size $k$. The rows show different datasets, and the columns show different XAI generation methods. The different line colours and styles denote the reduction strategy, and the black horizontal line shows the performance of the full explanation set. The loss-minimising reduction methods consistently reach a fidelity comparable or even better than the full explanation set.