Model Ensembling for Constrained Optimization

TL;DR

The paper tackles the problem of ensembling multiple multidimensional prediction models whose outputs form a linear objective in constrained downstream optimization. It introduces two multicalibration-based approaches: a white-box method that post-processes predictive models to achieve self-consistency and a black-box method that relies only on the induced policies, offering a swap-regret-like guarantee. The authors prove convergence guarantees for the update procedures, establish bounds relating self-assessed payoffs to actual payoffs, and demonstrate that the ensembles outperform their constituents in synthetic experiments, with the white-box variant often achieving near-optimal performance while the black-box variant offers faster training. This work advances prescriptive analytics by enabling robust, provably better-than-base-policy decisions in high-dimensional action spaces and under general feasibility constraints. The practical impact lies in producing superior decision policies in domains where actions are constrained and depend on high-dimensional predictions, with clear trade-offs between interpretability, access to models, and computational cost.

Abstract

There is a long history in machine learning of model ensembling, beginning with boosting and bagging and continuing to the present day. Much of this history has focused on combining models for classification and regression, but recently there is interest in more complex settings such as ensembling policies in reinforcement learning. Strong connections have also emerged between ensembling and multicalibration techniques. In this work, we further investigate these themes by considering a setting in which we wish to ensemble models for multidimensional output predictions that are in turn used for downstream optimization. More precisely, we imagine we are given a number of models mapping a state space to multidimensional real-valued predictions. These predictions form the coefficients of a linear objective that we would like to optimize under specified constraints. The fundamental question we address is how to improve and combine such models in a way that outperforms the best of them in the downstream optimization problem. We apply multicalibration techniques that lead to two provably efficient and convergent algorithms. The first of these (the white box approach) requires being given models that map states to output predictions, while the second (the \emph{black box} approach) requires only policies (mappings from states to solutions to the optimization problem). For both, we provide convergence and utility guarantees. We conclude by investigating the performance and behavior of the two algorithms in a controlled experimental setting.

Figures (6)

• Figure 1: Covariance-constrained policy. Here, $\boldsymbol{\pi}$ is the vector of decision variables (i.e. the induced policy), where each term is constrained to be a probability. In the final constraint, $\mathbf{C}$ is the covariance matrix of the true labels, and the quadratic constraint is bounded by a value $\alpha$ which was chosen appropriately to the scale of $\mathbf{C}$.
• Figure 2: Linearly-constrained policy. The constraints $\alpha$ and $\beta$ were chosen to be 0.5 and 0.6 respectively for the purposes of our experiments.
• Figure 3: Comparison of payoffs of white box and black box algorithms in each experiment. Dashed lines correspond to predicted payoff, while solid green and teal are realized payoff.
• Figure 4: Realized payoffs of the component policies of the white box ensemble over round of debiaising. Note that experiments A and C have 4 initial models and accompanying induced policies (one specializing in each coordinate of the prediction) while experiments B and D have 5 initial models (one specializing in each subgroup of the dataset).
• Figure 5: Heatmap of distribution of how constituent policies were chosen by the ensembling process across rounds of debiasing. For instance, in experiments A and B, no single policy is prioritized substantially more than the others, while in experiments C and D, the first policy ends up being prioritized.

Theorems & Definitions (26)

• Definition 1: Consistent Predictions
• Definition 2: Policy Level Sets
• Definition 3: Consistency to a Policy
• Lemma 1: Monotone Decrease of Squared Error. (roth2022uncertain)
• Lemma 2
• Lemma 3
• proof : Proof of Lemma \ref{['lem:consistent']}
• Corollary 1
• Lemma 4
• Definition 4: White Box Ensemble