Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interpretable classifiers for tabular data via discretization and feature selection

Reijo Jaakkola, Tomi Janhunen, Antti Kuusisto, Masood Feyzbakhsh Rankooh, Miikka Vilander

TL;DR

The paper tackles the challenge of producing immediately interpretable classifiers for tabular data by discretizing numerical features into Boolean predicates and selecting small feature subsets to construct short $DNF$-form classifiers. It introduces the empirical ideal classifier and a fixed-parameter linear algorithm to efficiently compute the best possible classifier within a chosen feature set, enabling global interpretability while maintaining competitive accuracy with state-of-the-art methods such as Random Forests and XGBoost. A nested cross-validation framework and a theoretical sample-size bound guarantee that, with sufficient data, the learned empirical classifier closely approximates the ideal classifier under the data distribution. The approach demonstrates strong interpretability across 12 datasets, often achieving accuracies on par with black-box models, and provides practical insights through concrete formulas and runtime efficiency, with future directions including improved discretization and extensions beyond Boolean predicates.

Abstract

We introduce a method for computing immediately human interpretable yet accurate classifiers from tabular data. The classifiers obtained are short Boolean formulas, computed via first discretizing the original data and then using feature selection coupled with a very fast algorithm for producing the best possible Boolean classifier for the setting. We demonstrate the approach via 12 experiments, obtaining results with accuracies comparable to ones obtained via random forests, XGBoost, and existing results for the same datasets in the literature. In most cases, the accuracy of our method is in fact similar to that of the reference methods, even though the main objective of our study is the immediate interpretability of our classifiers. We also prove a new result on the probability that the classifier we obtain from real-life data corresponds to the ideally best classifier with respect to the background distribution the data comes from.

Interpretable classifiers for tabular data via discretization and feature selection

TL;DR

The paper tackles the challenge of producing immediately interpretable classifiers for tabular data by discretizing numerical features into Boolean predicates and selecting small feature subsets to construct short -form classifiers. It introduces the empirical ideal classifier and a fixed-parameter linear algorithm to efficiently compute the best possible classifier within a chosen feature set, enabling global interpretability while maintaining competitive accuracy with state-of-the-art methods such as Random Forests and XGBoost. A nested cross-validation framework and a theoretical sample-size bound guarantee that, with sufficient data, the learned empirical classifier closely approximates the ideal classifier under the data distribution. The approach demonstrates strong interpretability across 12 datasets, often achieving accuracies on par with black-box models, and provides practical insights through concrete formulas and runtime efficiency, with future directions including improved discretization and extensions beyond Boolean predicates.

Abstract

We introduce a method for computing immediately human interpretable yet accurate classifiers from tabular data. The classifiers obtained are short Boolean formulas, computed via first discretizing the original data and then using feature selection coupled with a very fast algorithm for producing the best possible Boolean classifier for the setting. We demonstrate the approach via 12 experiments, obtaining results with accuracies comparable to ones obtained via random forests, XGBoost, and existing results for the same datasets in the literature. In most cases, the accuracy of our method is in fact similar to that of the reference methods, even though the main objective of our study is the immediate interpretability of our classifiers. We also prove a new result on the probability that the classifier we obtain from real-life data corresponds to the ideally best classifier with respect to the background distribution the data comes from.
Paper Structure (30 sections, 2 theorems, 39 equations, 1 figure, 1 table)

This paper contains 30 sections, 2 theorems, 39 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Overview of our method and contributions
  3. Further related work
  4. Preliminaries
  5. Feature selection and overfitting
  6. Bounds on sample size
  7. Experiments
  8. Experimental setup
  9. Results
  10. Examples of obtained formulas
  11. Conclusions
  12. Appendix.
  13. Proof of Theorem \ref{['thm:sample_size']}
  14. Hyperparameter spaces for Random Forest and XGBoost
  15. Random Forest
  16. ...and 15 more sections

Key Result

Theorem 3.1

Fix a vocabulary $\tau$, a proposition symbol $q\not\in \tau$ and a probability distribution $\mu : T_{\tau \cup \{q\}} \to [0,1]$. Let $\varepsilon, \delta > 0$ and Then with probability at least $1 - \delta$, the empirical ideal classifier with respect to a sample $M$ of size $n$ agrees with the ideal classifier with respect to $\mu$ on every $t \in T_\tau$ which is $\varepsilon$-separated by $

Figures (1)

  • Figure 1: The average test accuracies obtained with our method, random forests and XGBoost for all datasets. For all but the Colon dataset, also standard deviations are reported. For the UCI datasets we also include a comparison to the formula size method. When available, we have also included accuracies reported in literature, though these are not directly comparable due to different technical particularities in the experiments.

Theorems & Definitions (3)

  • Theorem 3.1
  • Corollary 3.2
  • proof