Data Science Principles for Interpretable and Explainable AI

TL;DR

The paper addresses the challenge of making AI systems interpretable and trustworthy across domains by clarifying a precise vocabulary (glass-box vs explainability; global vs local explanations) and outlining principled evaluation. It provides a tutorial-style synthesis that spans intrinsically interpretable models (sparse logistic regression and decision trees) and post-hoc XAI methods (embedding visualization, integrated gradients, and concept bottleneck models), illustrated with a microbiome-inspired longitudinal simulation. Through a detailed application to the simulation, including both traditional interpretable baselines and modern deep learning approaches (transformers and GPT-2), the work demonstrates how representation choice, task formulation, and domain concepts influence interpretability-accuracy trade-offs and explanatory content. The discussion emphasizes open challenges, worthy directions for interactive, audience-aware design, and scalable evaluation frameworks, offering code and benchmarks to drive reproducible work in interpretable and explainable AI.

Abstract

Society's capacity for algorithmic problem-solving has never been greater. Artificial Intelligence is now applied across more domains than ever, a consequence of powerful abstractions, abundant data, and accessible software. As capabilities have expanded, so have risks, with models often deployed without fully understanding their potential impacts. Interpretable and interactive machine learning aims to make complex models more transparent and controllable, enhancing user agency. This review synthesizes key principles from the growing literature in this field. We first introduce precise vocabulary for discussing interpretability, like the distinction between glass box and explainable models. We then explore connections to classical statistical and design principles, like parsimony and the gulfs of interaction. Basic explainability techniques -- including learned embeddings, integrated gradients, and concept bottlenecks -- are illustrated with a simple case study. We also review criteria for objectively evaluating interpretability approaches. Throughout, we underscore the importance of considering audience goals when designing interactive data-driven systems. Finally, we outline open challenges and discuss the potential role of data science in addressing them. Code to reproduce all examples can be found at https://go.wisc.edu/3k1ewe.

Figures (5)

• Figure 1: Example simulated trajectories. (a - b). Trajectories for two latent types. Each sample is a mixture of these types. Even for a fixed type, samples may have different variants, e.g., blooms at different times. (c) Observed trajectories are mixtures of these pure underlying trajectory types. The contributions of five latent trajectory patterns for each sample are shown in this heatmap. (d) Healthy and disease samples generated through this process for a subset of samples and taxa.
• Figure 2: Results from intrinsically interpretable models applied to the simulation study when $N = 500$. (a) Coefficient paths on the original, unfeaturized data. $\ell_{1}$-regularization strength decreases from left to right. Only those that are nonzero for at least one penalty $\lambda$ are displayed. (b) The corresponding coefficient paths for the featurized data. Though there are fewer input features overall, more are selected. (c - d) Selected decision trees on the original and the featurized data, respectively. With derived features, a very simple tree achieves surprisingly good performance (compare with Table \ref{['tab:tab1']}) (e - f) Stability of cross-validation optimized sparse logistic regression fits on the original and featurized data, respectively. The regression using manually derived features is more stable, which is consistent with the high temporal correlation in the original data (see Fig \ref{['fig:example_trajectories']}a).
• Figure 3: Global explanation of the simulation's transformer model via embeddings. (a) Healthy and disease classes generally overlap in the original data. Each point gives the projection of a subject onto the top two sparse PCA directions. (b) Transformer representations more clearly separate the two classes. (c) A sequence of points along the linear interpolation between samples 110 and 378 in embedding space restricted to features from species 21. (d) Trajectories associated with each labeled point in (c).
• Figure 4: Integrated gradients for the transformer model overlaid on a subset of samples from the simulation study. Each curve represents the trajectory for species 21, with symbols indicating the size and sign of the integrated gradient at that sample and time point. The subset of relevant time points can vary from sample to sample.
• Figure 5: Summary of XAI techniques. (a) Deep learning models transform input $\mathbf{x}_{i}$ into higher-level semantic representations. The final layer is used for prediction. (b) Global embeddings represent a layer of interest with lower-dimensional vectors $\mathbf{z}_{i}$, which can be interpreted alongside sample-level metadata, like point color in this example. (c) CBMs only use the black box to predict dense, concept-level annotations, not the final response. An interpretable model translates predicted concepts into the prediction (d) In IG, a sample is subject to a sequence of perturbations, and the impact on the final prediction is recorded to identify important coordinates.