Tight Mixed-Integer Optimization Formulations for Prescriptive Trees

TL;DR

This paper addresses prescriptive optimization with trained decision trees and tree ensembles by formulating the tree predictions as mixed-integer optimization problems. It introduces a tight union-of-polyhedra formulation based on a projected extended formulation that is ideal for a single tree, plus tightening strategies for binary feature encodings, including expset and elbow constraints that remove fractional relaxations in many settings. The authors prove ideality for the one-tree case and show significant solve-time improvements and tighter linear relaxations in numerical experiments, particularly when the ensemble is small or the feature dimension is low. The practical impact lies in enabling faster, more reliable optimization using tree-based predictions in applications like prescriptive analytics and constrained decision-making, with guidance on which formulation to use depending on forest size and feature dimensionality. The work also highlights regimes where existing formulations may still be preferable and lays out directions for extending the framework to more complex tree structures and encoding schemes.

Abstract

We focus on modeling the relationship between an input feature vector and the predicted outcome of a trained decision tree using mixed-integer optimization. This can be used in many practical applications where a decision tree or tree ensemble is incorporated into an optimization problem to model the predicted outcomes of a decision. We propose tighter mixed-integer optimization formulations than those previously introduced. Existing formulations can be shown to have linear relaxations that have fractional extreme points, even for the simple case of modeling a single decision tree. A formulation we propose, based on a projected union of polyhedra approach, is ideal for a single decision tree. While the formulation is generally not ideal for tree ensembles or if additional constraints are added, it generally has fewer extreme points, leading to a faster time to solve, particularly if the formulation has relatively few trees. However, previous work has shown that formulations based on a binary representation of the feature vector perform well computationally and hence are attractive for use in practical applications. We present multiple approaches to tighten existing formulations with binary vectors, and show that fractional extreme points are removed when there are multiple splits on the same feature. At an extreme, we prove that this results in ideal formulations for tree ensembles modeling a one-dimensional feature vector. Building on this result, we also show via numerical simulations that these additional constraints result in significantly tighter linear relaxations when the feature vector is low dimensional. We also present instances where the time to solve to optimality is significantly improved using these formulations.

Figures (14)

• Figure 1: Examples of decision tree with corresponding notation and partition of the feature space
• Figure 2: Examples of trees with fractional solutions and notation
• Figure 3: Tree ensemble formulation is not ideal or sharp. Extreme points of $Q^{proj}$ are shown with hollow circles, while the convex hull of the tree ensemble graph is shown in shaded purple.
• Figure 4: Trees for which different formulations are tighter, and an illustration of the notation used in the expset formulation, showing the sets $\textbf{left}(s)=\{3\}$ and $\textbf{below}(s)=\{1,3\}$.
• Figure 5: Example: cuts removing extreme point

Theorems & Definitions (22)

• Definition 1
• Definition 2
• Definition 3
• Example 1: veliborsPaper not ideal for a single tree with a single feature
• Example 2: biggs2018optimizing not ideal for a single tree with a single feature
• Theorem 1: Ideal formulation for a tree
• Definition 4
• Lemma 1
• Definition 5
• Example 3: Intersection of trees, using $Q^{proj}_t$ for each tree $t$, is not ideal or sharp