A Rectification-Based Approach for Distilling Boosted Trees into Decision Trees
Gilles Audemard, Sylvie Coste-Marquis, Pierre Marquis, Mehdi Sabiri, Nicolas Szczepanski
TL;DR
This work tackles the interpretability-accuracy trade-off by distilling a boosted tree $P$ into an interpretable decision tree $I$ using a rectification-based, incremental approach. At inference time, misclassified instances ${\bm x}$ by $I$ relative to $P$ trigger computation of an abductive explanation $t$ for $P$, from which a classification rule $R = t \Rightarrow y$ implied by $P$ is formed to rectify $I$; the updates are proven to be efficient and guarantee progress toward alignment with $P$. The framework specializes rectification to the case where $P$ is a boosted tree and $I$ is a decision tree, enabling an iterative, rule-by-rule distillation with logical guarantees and without requiring full retraining. Empirical results show that rectification-based distillation can outperform retraining in aligning $I$ with $P$ on several datasets, while also facilitating faster extraction of explanations (sufficient reasons) from the distilled $I$ than from $P$, albeit with potential growth in $I$ size. The work positions rectification as a practical, explanation-focused path to producing compact, faithful, and interpretable approximations of complex tree ensembles.
Abstract
We present a new approach for distilling boosted trees into decision trees, in the objective of generating an ML model offering an acceptable compromise in terms of predictive performance and interpretability. We explain how the correction approach called rectification can be used to implement such a distillation process. We show empirically that this approach provides interesting results, in comparison with an approach to distillation achieved by retraining the model.