Adaptive Forests For Classification

TL;DR

Adaptive Forests introduces input-dependent weighting for an ensemble of CARTs by coupling an interpretable Optimal Predictive-Policy Tree with dynamic weight generation via Mixed Integer Optimization. By learning measurement-informed, leaf-specific weight vectors, AF systematically improves predictive performance over Random Forests, XGBoost, and other weighted RF variants on 20+ real-world datasets, while using fewer base learners. The approach combines global optimization for policy learning (OP2T/OPT) with iterative MIO-driven exploration of the weight space, yielding robust AUC gains in binary and multiclass settings and offering a scalable, interpretable alternative for ensemble classification. Its practical impact lies in delivering stronger accuracy with leaner ensembles and a flexible framework that can adapt to dataset-specific characteristics through configurable reward definitions and data selection for OPT training.

Abstract

Random Forests (RF) and Extreme Gradient Boosting (XGBoost) are two of the most widely used and highly performing classification and regression models. They aggregate equally weighted CART trees, generated randomly in RF or sequentially in XGBoost. In this paper, we propose Adaptive Forests (AF), a novel approach that adaptively selects the weights of the underlying CART models. AF combines (a) the Optimal Predictive-Policy Trees (OP2T) framework to prescribe tailored, input-dependent unequal weights to trees and (b) Mixed Integer Optimization (MIO) to refine weight candidates dynamically, enhancing overall performance. We demonstrate that AF consistently outperforms RF, XGBoost, and other weighted RF in binary and multi-class classification problems over 20+ real-world datasets.

Figures (1)

• Figure 1: The analogy between human decision making and Adaptive Forests: Consider a binary classification problem where doctors A and B provide opinions on whether a patient should undergo surgery, with doctor A being more specialized in disease a. The decision should weigh their opinions based on two features: (a) time spent consulting doctor A and (b) the probability that this patient has disease a. For example, if a patient has an 80% probability of disease a and has consulted doctor A for 8 hours, doctor A’s opinion should be prioritized. A policy tree derives this weighting scheme from input features, enhancing interpretability by showing how and when opinions are weighted adaptively. Similarly, for two CART models, A and B, we can learn an adaptive weighting scheme based on feature values to improve predictions. While this example illustrates the core idea behind Adaptive Forests, real-world implementations involve many CART models, a more complex policy tree, and a high-dimensional feature space that cannot be visualized in 2D.