Scalable Rule Lists Learning with Sampling

TL;DR

This work tackles scalable learning of interpretable rule lists by introducing SamRuLe, a sampling-based framework that provides rigorous approximation guarantees for finding near-optimal rule lists on large datasets. The method hinges on VC-dimension bounds for the rule-list class to derive a sample size m_hat that guarantees an (ε, θ)-approximation with probability at least 1-δ, while keeping computation practical by solving the optimization on a small sample using an exact solver like CORELS. The authors establish tight upper and lower bounds on the VC-dimension of rule lists, derive the corresponding sample-complexity results, and demonstrate that SamRuLe achieves up to two orders of magnitude speedups over exact approaches without sacrificing accuracy, often outperforming state-of-the-art heuristics. The approach yields rule lists that closely resemble the optimal and provides a principled, provable trade-off between sample size, accuracy, and interpretability, making it suitable for large-scale, high-stakes decision settings. Future directions include extending sampling-based guarantees to other rule-based models and exploring data-dependent complexity measures to tighten the bounds further.

Abstract

Learning interpretable models has become a major focus of machine learning research, given the increasing prominence of machine learning in socially important decision-making. Among interpretable models, rule lists are among the best-known and easily interpretable ones. However, finding optimal rule lists is computationally challenging, and current approaches are impractical for large datasets. We present a novel and scalable approach to learn nearly optimal rule lists from large datasets. Our algorithm uses sampling to efficiently obtain an approximation of the optimal rule list with rigorous guarantees on the quality of the approximation. In particular, our algorithm guarantees to find a rule list with accuracy very close to the optimal rule list when a rule list with high accuracy exists. Our algorithm builds on the VC-dimension of rule lists, for which we prove novel upper and lower bounds. Our experimental evaluation on large datasets shows that our algorithm identifies nearly optimal rule lists with a speed-up up to two orders of magnitude over state-of-the-art exact approaches. Moreover, our algorithm is as fast as, and sometimes faster than, recent heuristic approaches, while reporting higher quality rule lists. In addition, the rules reported by our algorithm are more similar to the rules in the optimal rule list than the rules from heuristic approaches.

Figures (9)

• Figure 1: (a): example of a dataset $\mathcal{D}$ with $n=5$ instances and $d=4$ features. (b): example of a rule list $R$ with length $k=3$ with conjunctions with at most $z=2$ terms. The rule list $R$ perfectly classifies the instances of $\mathcal{D}$.
• Figure 2: Performance and accuracy comparison between SamRuLe and CORELS on adult and bank datasets, for different values of $\varepsilon$ and $\theta$. (a)-(b): running times of CORELS and SamRuLe. (c)-(d): average deviations $| \ell(\tilde{R} , \mathcal{D}) - \ell(R^\star , \mathcal{D}) |$ of the loss of the rule list $\tilde{R}$ found by SamRuLe with the optimal rule list $R^\star$ found by CORELS (purple horizontal line drawn at $y = \ell(R^\star , \mathcal{D})$). The deviation plots only show upper errors bars at $+$std to improve readability. See Figures \ref{['fig:runningtimes']} and \ref{['fig:avgdevs']} in the Appendix for the plots for all datasets and with $\pm$std bars.
• Figure 3: (a): optimal rule $R^\star$ computed by CORELS on the adult dataset ($\ell(R^\star,\mathcal{D})=0.176$) to predict high income (the label $1$ denotes $\text{\lq\lq}\geq 50K\text{\rq\rq}$). (b): set of rule lists computed by SamRuLe over $10$ runs. SamRuLe identified the optimal rule and slight variations $\tilde{R}_1$ and $\tilde{R}_2$ that differ in the second rule of the list: they predict a lower outcome using the age ($\tilde{R}_1$) and the per-week work hours features ($\tilde{R}_2$) with respective loss $\ell(\tilde{R}_1,\mathcal{D})=0.1763$ and $\ell(\tilde{R}_2,\mathcal{D})=0.1775$.
• Figure 4: Comparison in terms of running time (a) and accuracy (b) between SamRuLe, SBRL, and RIPPER. (c): rule $R$ computed by SBRL on adult with loss $\ell(R,\mathcal{D})=0.238$ over all $10$ runs.
• Figure 5: Number of samples $\hat{m}(\mathcal{R}_{k}^{z} , \mathcal{D})$ used by SamRuLe varying $\varepsilon$ and $\theta$ for all datasets. $k$ is set as in \ref{['tab:datasets']} and $z=1$.

Theorems & Definitions (11)

• definition 1
• theorem 1
• corollary 1
• corollary 2
• theorem 2
• theorem 3
• theorem 4
• theorem 5
• theorem 6
• lemma 1: Lemma 6 of littlestone1988learning