A Unified Approach to Extract Interpretable Rules from Tree Ensembles via Integer Programming

TL;DR

This work addresses the interpretability gap of tree ensembles by extracting a compact, faithful rule list using a set-partitioning Integer Programming formulation. It decouples preprocessing (stability and loss computation) from optimization, enabling flexible loss functions and applicability to regression, multi-class classification, and time-series data, with internal and external fidelity metrics to assess surrogate faithfulness. Empirical results on tabular and temporal tasks show competitive predictive performance and strong internal fidelity, producing rule lists that resemble tree-like structures rather than opaque black boxes. While preprocessing incurs notable cost, the approach provides a principled, customizable framework for faithful, interpretable explanations of ensemble predictions across diverse data types.

Abstract

Tree ensembles are very popular machine learning models, known for their effectiveness in supervised classification and regression tasks. Their performance derives from aggregating predictions of multiple decision trees, which are renowned for their interpretability properties. However, tree ensemble models do not reliably exhibit interpretable output. Our work aims to extract an optimized list of rules from a trained tree ensemble, providing the user with a condensed, interpretable model that retains most of the predictive power of the full model. Our approach consists of solving a set partitioning problem formulated through Integer Programming. The proposed method works with either tabular or time series data, for both classification and regression tasks, and its flexible formulation can include any arbitrary loss or regularization functions. Our extensive computational experiments offer statistically significant evidence that our method is competitive with other rule extraction methods in terms of predictive performance and fidelity towards the tree ensemble. Moreover, we empirically show that the proposed method effectively extracts interpretable rules from tree ensemble that are designed for time series data.

Figures (9)

• Figure 1: Example of a binary decision tree of depth 2 for the classification of tabular data.
• Figure 2: Example of shapelets-based decision tree built upon the ItalyPowerDemand dataset from the UCR repository UCRArchive2018. The upper subplots show the training time series in red (Class 0) and blue (Class 1). The right subplots show the two shapelets $\bm{s}_1, \bm{s}_2$ in dark green. The left subplot displays the scatter plot of the time series based on dist$(\bm{x}, \bm{s}_1)$ and dist$(\bm{x}, \bm{s}_2)$.
• Figure 3: Example of a shapelets-based decision tree of depth 2 for the classification of temporal data.
• Figure 4: Comparison between exact and heuristic values for the lower bound $\underline{\ell}$ over 36 benchmark datasets.
• Figure 5: Critical distance diagram obtained by running the Friedman test demvsar2006statistical combined with the signed-rank Wilcoxon-Holm post-hoc test wilcoxon2019 on the results about regression of tabular data of Table \ref{['tab:regression']}.