Rule Generation for Classification: Scalability, Interpretability, and Fairness

TL;DR

The paper tackles scalable, interpretable, and fair multi-class classification by formulating rule generation as a column-generation based linear program. The master problem produces a weighted ensemble of rules, with predictions given by $\hat{\bm{y}}_i(\bm{w}) = \sum_j a_{ij} \bm{R}_j(\bm{x}_i) w_j$ and hinge-loss optimization governed by $\mathcal{L}(\hat{\bm{y}}_i(\bm{w}), \bm{y}_i) = \max\{1 - \kappa \hat{\bm{y}}_i(\bm{w})^\top \bm{y}_i, 0\}$, where $\kappa = (K-1)/K$. Because the pricing subproblem is NP-hard, the authors introduce a fast proxy by training weighted decision trees to generate candidate rules, enabling scalable CG. The work extends fairness notions to multi-class and multi-group settings via Disparate Mistreatment per Class (DMC) and Overall Disparate Mistreatment (ODM) and embeds these as LP constraints, balancing accuracy with interpretability and fairness. Empirical results on diverse datasets, including a credit-risk case study, show that the proposed approach achieves strong accuracy with interpretable rule sets and favorable fairness behavior, while the proxy pricing strategy delivers substantial speedups with little sacrifice in performance.

Abstract

We introduce a new rule-based optimization method for classification with constraints. The proposed method leverages column generation for linear programming, and hence, is scalable to large datasets. The resulting pricing subproblem is shown to be NP-Hard. We recourse to a decision tree-based heuristic and solve a proxy pricing subproblem for acceleration. The method returns a set of rules along with their optimal weights indicating the importance of each rule for learning. We address interpretability and fairness by assigning cost coefficients to the rules and introducing additional constraints. In particular, we focus on local interpretability and generalize a separation criterion in fairness to multiple sensitive attributes and classes. We test the performance of the proposed methodology on a collection of datasets and present a case study to elaborate on its different aspects. The proposed rule-based learning method exhibits a good compromise between local interpretability and fairness on the one side, and accuracy on the other side.

Figures (4)

• Figure 1: Overview of models discussed in Section \ref{['sec:mat_mod']}.
• Figure 2: Proposed rule generation algorithm. The notation $\bm{e}$ stands for vector of ones.
• Figure 3: Comparison of metrics between RUG with proxy PSP and RUG with the exact solution of the PSP on the test set.
• Figure 4: Local interpretations for two samples (left and right). The rules used by RUG to classify the samples are given above the feature importance plots that are produced after applying SHAP to the trained LightGBM model.

Theorems & Definitions (6)

• Example 3.1
• Example 3.2
• Proposition 1
• Corollary 1
• Proposition 2
• Lemma 1