FINCH: Locally Visualizing Higher-Order Feature Interactions in Black Box Models

TL;DR

FINCH advances explainable AI by enabling local visualization of higher-order feature interactions in black-box models through a subset-based, incremental visualization approach. It preserves realistic data by avoiding perturbations, builds interaction curves by conditioning on realistic subsets, and provides trust-calibration views (ground truth and uncertainty) to support expert judgment. Across Titanic, California Housing, and BRFSS case studies, FINCH demonstrates the ability to reveal interactions beyond pairwise relationships and to compare with SHAP and PDP baselines, supported by a user study that reports high usability (SUS ~82) and favorable understandability and usefulness. The work offers a practical, open-source tool implemented with Panel/Bokeh, extensible to various tabular datasets and models, with future work aimed at sparse data, automatic local interaction detection, and adaptation to non-tabular data types.

Abstract

In an era where black-box AI models are integral to decision-making across industries, robust methods for explaining these models are more critical than ever. While these models leverage complex feature interplay for accurate predictions, most explanation methods only assign relevance to individual features. There is a research gap in methods that effectively illustrate interactions between features, especially in visualizing higher-order interactions involving multiple features, which challenge conventional representation methods. To address this challenge in local explanations focused on individual instances, we employ a visual, subset-based approach to reveal relevant feature interactions. Our visual analytics tool FINCH uses coloring and highlighting techniques to create intuitive, human-centered visualizations, and provides additional views that enable users to calibrate their trust in the model and explanations. We demonstrate FINCH in multiple case studies, demonstrating its generalizability, and conducted an extensive human study with machine learning experts to highlight its helpfulness and usability. With this approach, FINCH allows users to visualize feature interactions involving any number of features locally.

Figures (17)

• Figure 1: Bike rentals based on different subsets of the data set
• Figure 2: Our algorithm incrementally adding features, by calculating and visualizing subsets for each added feature.
• Figure 3: The incremental visualization of the interaction of hour, weekends, and winter in the bike rental data set. Colored areas visualize the change in each step.
• Figure 4: Interaction effect visualization. This visualization separates the main and interaction effects of a newly added feature by showing the main effect through a blue dotted line and the interaction effect through its difference from the actual purple line.
• Figure 5: Distribution heatmap. Directly below the dependence plot are the density distributions for the first selected feature, which is visualized on the x-axis.