Efficiently Transforming Neural Networks into Decision Trees: A Path to Ground Truth Explanations with RENTT

TL;DR

The paper tackles the fundamental challenge of neural network interpretability by presenting RENTT, an exact algorithm that transforms FCN, CNN, and RNN architectures with piecewise linear activations into multivariate decision trees, preserving the original model behavior. It further introduces RENTT-FI to extract ground-truth feature importance from the transformed tree at global, regional, and local levels, addressing faithfulness in explanations. Theoretical development is backed by rigorous proofs and comprehensive experiments showing RENTT's runtime and memory efficiency, and that RENTT-FI provides a reliable baseline far outperforming conventional XAI methods like LIME and SHAP in terms of alignment with true model reasoning. The work demonstrates substantial practical impact by enabling faithful explanations for large networks and offering open-source tooling, with clear pathways to extend to CNNs and RNNs and to further refine explanation types. Overall, this approach bridges the gap between neural performance and interpretable, trustworthy decisions, supporting safer deployment in high-stakes domains.

Abstract

Although neural networks are a powerful tool, their widespread use is hindered by the opacity of their decisions and their black-box nature, which result in a lack of trustworthiness. To alleviate this problem, methods in the field of explainable Artificial Intelligence try to unveil how such automated decisions are made. But explainable AI methods are often plagued by missing faithfulness/correctness, meaning that they sometimes provide explanations that do not align with the neural network's decision and logic. Recently, transformations to decision trees have been proposed to overcome such problems. Unfortunately, they typically lack exactness, scalability, or interpretability as the size of the neural network grows. Thus, we generalize these previous results, especially by considering convolutional neural networks, recurrent neural networks, non-ReLU activation functions, and bias terms. Our findings are accompanied by rigorous proofs and we present a novel algorithm RENTT (Runtime Efficient Network to Tree Transformation) designed to compute an exact equivalent decision tree representation of neural networks in a manner that is both runtime and memory efficient. The resulting decision trees are multivariate and thus, possibly too complex to understand. To alleviate this problem, we also provide a method to calculate the ground truth feature importance for neural networks via the equivalent decision trees - for entire models (global), specific input regions (regional), or single decisions (local). All theoretical results are supported by detailed numerical experiments that emphasize two key aspects: the computational efficiency and scalability of our algorithm, and that only RENTT succeeds in uncovering ground truth explanations compared to conventional approximation methods like LIME and SHAP. All code is available at https://github.com/HelenaM23/RENTT .

Figures (41)

• Figure 1: fcnn with two hidden layers of size $[2,3]$ and relu activation. The bias is depicted as neurons with dashed lines and appended as a 1 to the input.
• Figure 2: dt for fcnn with hidden layers of size $[2,3]$ and relu activation. The colors are referring to Figure \ref{['fig:net']}, i.e., green depicts the first, blue the second and orange the output layer. The different shades of the colors denote the individual neurons in each layer. Index $m$ is written in binary system for better visualization of the activation state.
• Figure 3: Flattening process of $\tilde{\mathbf{x}}_i$ with $n^c=3$, $n^w=3$ and $n^h=2$ to obtain $\mathbf{x}_i$. In the image we neglect $\tilde{ }$ and index $i$ for better visualization.
• Figure 4: Flattening process of $\Psi_i$ with $k^c=2$, $k^w=2$ and $k^h=2$ when $n^w=3$ and $n^h=2$ to obtain $\left[\bar{\Psi}_{i}\right]_{c,h,k}$, where $c$ indicates the filter number, $h$ the filter height index and $k$ the channel number of $\left[\mathbf{x}_i\right]_k$. The image index $i$ is neglected for better visualization.
• Figure 5: Super convolutional matrix $\mathbf{C}_i$ of stacked $\left[\bar{\Psi}_{i}\right]_{c,h,k}$. Note in the image index $i$ is neglected for better visualization.

Theorems & Definitions (14)

• Definition 1: Fully Connected Neural Networks
• Remark 1: Bias
• Remark 2: Non-piecewise-linear Activation Function
• Example 1: Activation Function
• Remark 3: Product Operator
• Definition 2: Activation Pattern
• Remark 4: Multi-Way Tree
• Definition 3: Convolutional Mapping
• Definition 4: Max Pooling
• Remark 5: Number of Nodes