Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decision-Focused Sequential Experimental Design: A Directional Uncertainty-Guided Approach

Beichen Wan, Mo Liu, Paul Grigas, Zuo-Jun Max Shen

TL;DR

This work proposes a directional-based metric to quantify predictive uncertainty and shows that the resulting sequential design criterion enjoys strong consistency and convergence guarantees and attains an earlier stopping time than decision-blind designs.

Abstract

We consider the sequential experimental design problem in the predict-then-optimize paradigm. In this paradigm, the outputs of the prediction model are used as coefficient vectors in a downstream linear optimization problem. Traditional sequential experimental design aims to control the input variables (features) so that the improvement in prediction accuracy from each experimental outcome (label) is maximized. However, in the predict-then-optimize setting, performance is ultimately evaluated based on the decision loss induced by the downstream optimization, rather than by prediction error. This mismatch between prediction accuracy and decision loss renders traditional decision-blind designs inefficient. To address this issue, we propose a directional-based metric to quantify predictive uncertainty. This metric does not require solving an optimization oracle and is therefore computationally tractable. We show that the resulting sequential design criterion enjoys strong consistency and convergence guarantees. Under a broad class of distributions, we demonstrate that our directional uncertainty-based design attains an earlier stopping time than decision-blind designs. This advantage is further supported by real-world experiments on an LLM job allocation problem.

Decision-Focused Sequential Experimental Design: A Directional Uncertainty-Guided Approach

TL;DR

This work proposes a directional-based metric to quantify predictive uncertainty and shows that the resulting sequential design criterion enjoys strong consistency and convergence guarantees and attains an earlier stopping time than decision-blind designs.

Abstract

We consider the sequential experimental design problem in the predict-then-optimize paradigm. In this paradigm, the outputs of the prediction model are used as coefficient vectors in a downstream linear optimization problem. Traditional sequential experimental design aims to control the input variables (features) so that the improvement in prediction accuracy from each experimental outcome (label) is maximized. However, in the predict-then-optimize setting, performance is ultimately evaluated based on the decision loss induced by the downstream optimization, rather than by prediction error. This mismatch between prediction accuracy and decision loss renders traditional decision-blind designs inefficient. To address this issue, we propose a directional-based metric to quantify predictive uncertainty. This metric does not require solving an optimization oracle and is therefore computationally tractable. We show that the resulting sequential design criterion enjoys strong consistency and convergence guarantees. Under a broad class of distributions, we demonstrate that our directional uncertainty-based design attains an earlier stopping time than decision-blind designs. This advantage is further supported by real-world experiments on an LLM job allocation problem.
Paper Structure (29 sections, 8 theorems, 55 equations, 16 figures, 4 tables, 3 algorithms)

This paper contains 29 sections, 8 theorems, 55 equations, 16 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Sequential Design for Predict-then-Optimize
  3. Application Example: Contextual Job Assignment
  4. Related Work
  5. Algorithm and Performance Guarantees
  6. Algorithm Motivation
  7. Importance Weighted Sequential Experimental Design
  8. Preliminaries for Theoretical Guarantees
  9. Risk Bound and Rates
  10. Comparison with Decision-blind Sequential Design
  11. Comparison Under Optimality Gap Assumption
  12. Justifications and Examples of Assumptions
  13. Justification of the Directional Margin Condition
  14. Examples of Optimality Gap
  15. Numerical Study
  16. ...and 14 more sections

Key Result

Lemma 1

Under Assumption assumption:tau_lower_bound, for any $h_1, h_2 \in \mathcal{H}$ and any $(x,c)\in \mathcal{X}\times \mathcal{ C }$, there exists a constant $L = \frac{\Delta(\mathcal{S})\rho(\mathcal{C})}{ 2\eta}$ such that $\ell_{\textnormal{SPO}}(.,c)$ is L-Lipchitz with respect to the directional

Figures (16)

  • Figure 1: Illustration of Sequential Experimental Design.
  • Figure 2: Cost Vector Prediction with Corresponding Optimizer
  • Figure 3: Cost Vector Prediction with $\ell_2$ Distance
  • Figure 4: $\ell_2$ Based Uncertainty
  • Figure 5: Direction-Based Uncertainty
  • ...and 11 more figures

Theorems & Definitions (23)

  • Example 1
  • Definition 1: Distance to Degeneracy, el2022generalization
  • Remark 1: Geometric Interpretation of Directional Margin Condition.
  • Lemma 1: "Lipschitz-like" property of $\ell_{\textnormal{SPO}}$ under directional margin condition
  • Theorem 1: Risk Bound
  • Theorem 2: Sample complexity
  • Proposition 1
  • Remark 2: Consistency of metrics for training and elimination
  • Theorem 3: Earlier stopping time than decision-blind design
  • Example 2: Directional uncertainty yields an earlier stopping time under squared loss
  • ...and 13 more