Finding Minimum-Cost Explanations for Predictions made by Tree Ensembles

TL;DR

This work tackles explainability for tree ensembles in safety-critical settings by developing a sound, complete oracle based on abstract interpretation to assess explanation validity. It introduces m-MARCO, a minimum-cost variant of MARCO, to efficiently extract a single least-redundant explanation for a given prediction, and demonstrates substantial speedups over existing approaches. The study reveals that large numbers of minimal explanations can exist for a single prediction, yet minimum explanations are notably less verbose, underscoring the practical value of minimizing explanations. Across experiments with non-trivial tree ensembles, the proposed framework achieves orders-of-magnitude improvements in runtime and enables thorough analysis of explanation characteristics, including their verbosity and variable usage. The results suggest significant potential for deploying compact, provably correct explanations in real-world decision-support systems and point toward extending these methods to broader model families.

Abstract

The ability to explain why a machine learning model arrives at a particular prediction is crucial when used as decision support by human operators of critical systems. The provided explanations must be provably correct, and preferably without redundant information, called minimal explanations. In this paper, we aim at finding explanations for predictions made by tree ensembles that are not only minimal, but also minimum with respect to a cost function. To this end, we first present a highly efficient oracle that can determine the correctness of explanations, surpassing the runtime performance of current state-of-the-art alternatives by several orders of magnitude when computing minimal explanations. Secondly, we adapt an algorithm called MARCO from related works (calling it m-MARCO) for the purpose of computing a single minimum explanation per prediction, and demonstrate an overall speedup factor of two compared to the MARCO algorithm which enumerates all minimal explanations. Finally, we study the obtained explanations from a range of use cases, leading to further insights of their characteristics. In particular, we observe that in several cases, there are more than 100,000 minimal explanations to choose from for a single prediction. In these cases, we see that only a small portion of the minimal explanations are also minimum, and that the minimum explanations are significantly less verbose, hence motivating the aim of this work.

Figures (12)

• Figure 1: A fictive bank-loan application processing system, where a positive outcome indicates that the applicant should be denied a loan.
• Figure 2: A decision tree (to the left) partition an input space into disjoint regions (to the right), and associates each region with an output value.
• Figure 3: A simple Android malware classifier trained on the DREBIN dataset.
• Figure 4: Shrink an unsatisfiable $S \subseteq C$ to a minimal unsatisfiable subset (MUS), or grow a satisfiable $S \subseteq C$ to a maximal satisfiable subset (MSS).
• Figure 5: A simple Android malware classifier realized by a tree ensemble with a single tree (to the left), and a Hasse diagram (to the right) of the power set lattice of elements that can be removed from a valid explanation to the prediction $p: f_{bin}(1, 1, 1; \{T_1\}) \mapsto 1$, where crossed-over elements yield non-valid explanations, and bold elements yield minimal explanations.

Theorems & Definitions (42)

• definition 1: Decision Tree Prediction Function
• definition 2: Variables Referenced by a Tree
• definition 3: Tree Ensemble Prediction Function
• definition 4: Variables Referenced by a Tree Ensemble
• definition 5: Sigmoid Function
• definition 6: Tree-based Binary Classifier
• definition 7: Valid Explanation
• definition 8: Minimal Explanation
• definition 9: Minimum Explanation
• definition 10: Minimal Unsatisfiable Subset