Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalized Optimal Classification Trees: A Mixed-Integer Programming Approach

Jiancheng Tu, Wenqi Fan, Zhibin Wu

TL;DR

This work presents Generalized Optimal Classification Trees, a Mixed-Integer Programming framework that directly optimizes nonlinear, imbalance-aware metrics (such as the $F_β$ score and MCC) for decision trees. It introduces Weighted Flow OCT (WFlowOCT) to reduce problem size via unique data instances and employs a tailored branch-and-cut strategy, instance reduction, and warm-starting to achieve scalability. By linearizing nonlinear metrics with binary expansions and McCormick envelopes, the approach unifies linear and nonlinear objectives and enables direct optimization of metrics beyond accuracy. Empirical results on 50 datasets show superior performance and significantly faster solution times compared with state-of-the-art OCT methods, especially for nonlinear objectives and imbalanced tasks. The framework advances interpretable learning by delivering globally optimal trees that optimize practically relevant metrics while maintaining tractable computation on real-world data.

Abstract

Global optimization of decision trees is a long-standing challenge in combinatorial optimization, yet such models play an important role in interpretable machine learning. Although the problem has been investigated for several decades, only recent advances in discrete optimization have enabled practical algorithms for solving optimal classification tree problems on real-world datasets. Mixed-integer programming (MIP) offers a high degree of modeling flexibility, and we therefore propose a MIP-based framework for learning optimal classification trees under nonlinear performance metrics, such as the F1-score, that explicitly addresses class imbalance. To improve scalability, we develop problem-specific acceleration techniques, including a tailored branch-and-cut algorithm, an instance-reduction scheme, and warm-start strategies. We evaluate the proposed approach on 50 benchmark datasets. The computational results show that the framework can efficiently optimize nonlinear metrics while achieving strong predictive performance and reduced solution times compared with existing methods.

Generalized Optimal Classification Trees: A Mixed-Integer Programming Approach

TL;DR

This work presents Generalized Optimal Classification Trees, a Mixed-Integer Programming framework that directly optimizes nonlinear, imbalance-aware metrics (such as the score and MCC) for decision trees. It introduces Weighted Flow OCT (WFlowOCT) to reduce problem size via unique data instances and employs a tailored branch-and-cut strategy, instance reduction, and warm-starting to achieve scalability. By linearizing nonlinear metrics with binary expansions and McCormick envelopes, the approach unifies linear and nonlinear objectives and enables direct optimization of metrics beyond accuracy. Empirical results on 50 datasets show superior performance and significantly faster solution times compared with state-of-the-art OCT methods, especially for nonlinear objectives and imbalanced tasks. The framework advances interpretable learning by delivering globally optimal trees that optimize practically relevant metrics while maintaining tractable computation on real-world data.

Abstract

Global optimization of decision trees is a long-standing challenge in combinatorial optimization, yet such models play an important role in interpretable machine learning. Although the problem has been investigated for several decades, only recent advances in discrete optimization have enabled practical algorithms for solving optimal classification tree problems on real-world datasets. Mixed-integer programming (MIP) offers a high degree of modeling flexibility, and we therefore propose a MIP-based framework for learning optimal classification trees under nonlinear performance metrics, such as the F1-score, that explicitly addresses class imbalance. To improve scalability, we develop problem-specific acceleration techniques, including a tailored branch-and-cut algorithm, an instance-reduction scheme, and warm-start strategies. We evaluate the proposed approach on 50 benchmark datasets. The computational results show that the framework can efficiently optimize nonlinear metrics while achieving strong predictive performance and reduced solution times compared with existing methods.
Paper Structure (51 sections, 5 theorems, 62 equations, 2 figures, 5 tables, 3 algorithms)

This paper contains 51 sections, 5 theorems, 62 equations, 2 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. MIP-based Approaches for Optimal Classification Trees
  3. Optimal Classification Trees for Imbalanced Data
  4. Contributions
  5. Weighted Flow-based Optimal Classification Trees
  6. Problem Formulation
  7. Imbalanced Datasets
  8. F-beta Score
  9. Matthews Correlation Coefficient (MCC)
  10. (i) Binary expansion of $U\cdot V$.
  11. (ii) Linearization of $\mathrm{MCC2}\cdot (U V)$.
  12. Combinations
  13. Acceleration Techniques
  14. Cutting Planes
  15. Valid inequalities
  16. ...and 36 more sections

Key Result

Proposition 1

WFlowOCT is equivalent to FlowOCT in the sense that there exists a bijection between optimal solutions: given any optimal FlowOCT solution on $\mathcal{I}$, there is an optimal WFlowOCT solution on $(\mathcal{U},w)$ with identical tree structure and objective value, and vice versa.

Figures (2)

  • Figure 1: A depth--2 decision tree (left) and its induced flow network (right). The network is constructed by augmenting the tree nodes with a source and a sink; arcs correspond to tree edges and terminal connections from leaves to the sink. Here, $\mathcal{B}=\{1,2,3\}$ and $\mathcal{L}=\{4,5,6,7\}$, while $\mathcal{V}=\{s,1,2,\ldots,7,t\}$ and $\mathcal{A} = \{(s,1),(1,2),\ldots,(7,t)\}$aghaei2025.
  • Figure 2: The results of MIP-based models displaying (a) instances reaching optimality over time and instances attaining a prescribed optimality-gap threshold at the time limit, (b) the average objective value by each approach across depths $d\in${2,3,4,5}.

Theorems & Definitions (11)

  • Definition 1: Unique Dataset
  • Proposition 1
  • Proposition 2
  • Remark 1
  • Proposition 3
  • Remark 2
  • Definition 2: Conflict subset
  • Theorem 1
  • Example 1
  • Proposition 4: Feature-activated conflict subset
  • ...and 1 more