Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges

TL;DR

This survey outlines fundamental principles and ten grand challenges in interpretable ML, arguing for inherently interpretable models over posthoc explanations in high-stakes contexts. It systematically categorizes interpretable approaches—from sparse logical models and scoring systems to GAMs, case-based reasoning, and disentangled neural representations—and discusses optimization, scalability, and domain-specific constraints. It also introduces the Rashomon set as a framework to understand model multiplicity and highlights physics-informed learning, DR for visualization, and RL interpretability as crucial frontiers. Collectively, the work provides a roadmap for developing interpretable methods that maintain accuracy, enable troubleshooting, and support trustworthy deployment across domains.

Abstract

Interpretability in machine learning (ML) is crucial for high stakes decisions and troubleshooting. In this work, we provide fundamental principles for interpretable ML, and dispel common misunderstandings that dilute the importance of this crucial topic. We also identify 10 technical challenge areas in interpretable machine learning and provide history and background on each problem. Some of these problems are classically important, and some are recent problems that have arisen in the last few years. These problems are: (1) Optimizing sparse logical models such as decision trees; (2) Optimization of scoring systems; (3) Placing constraints into generalized additive models to encourage sparsity and better interpretability; (4) Modern case-based reasoning, including neural networks and matching for causal inference; (5) Complete supervised disentanglement of neural networks; (6) Complete or even partial unsupervised disentanglement of neural networks; (7) Dimensionality reduction for data visualization; (8) Machine learning models that can incorporate physics and other generative or causal constraints; (9) Characterization of the "Rashomon set" of good models; and (10) Interpretable reinforcement learning. This survey is suitable as a starting point for statisticians and computer scientists interested in working in interpretable machine learning.

Figures (18)

• Figure 1: Knowledge Discovery Process Fayyad96fromdata
• Figure 2: Predicting which individuals are arrested within two years of release by a decision tree (a), a decision list (b), and a decision set (c). The dataset used here is the ProPublica recidivism dataset propublica2016
• Figure 3: (a) 16-leaf decision tree learned by CART Breiman84 and (b) 9-leaf decision tree generated by GOSDT lin2020generalized for the classic Monk 2 dataset Dua:2019. The GOSDT tree is optimal with respect to a balance between accuracy and sparsity.
• Figure 4: Hierarchical relationships between GAMs, additive models, linear models, and scoring systems.
• Figure 5: Left: All component functions of a GAM model trained using the interpret package nori2019interpretml on a diabetes dataset Dua:2019; Right: zoom-in of component function for glucose concentration.