Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Decision-Focused Learning

Aymeric Capitaine, Maxime Haddouche, Eric Moulines, Michael I. Jordan, Etienne Boursier, Alain Durmus

TL;DR

This paper extends decision-focused learning from the batch setting to online, non-stationary environments where the objective evolves over time. It introduces two algorithms, DF-FTPL and DF-OGD, that regularize the inner optimization to obtain differentiability and leverage approximate oracles to cope with non-convexity, achieving provable static and dynamic regret guarantees. Theoretical results show sublinear regret bounds: a rate of roughly $\tilde{O}( m^{3/4}\sqrt{n}\,T^{-1/4} + \xi)$ for static regret and $\tilde{O}( \sqrt{n}(1+P_T)^{1/4}T^{-1/4} + \xi)$ for dynamic regret, with $P_T$ capturing environment variability. Empirical evaluation on a knapsack-inspired task demonstrates that the proposed methods outperform prediction-focused online learning and online SPO in terms of downstream decision costs, while tolerating higher prediction error, highlighting the practical value of online decision-focused learning in dynamic decision problems.

Abstract

Decision-focused learning (DFL) is an increasingly popular paradigm for training predictive models whose outputs are used in decision-making tasks. Instead of merely optimizing for predictive accuracy, DFL trains models to directly minimize the loss associated with downstream decisions. However, existing studies focus solely on scenarios where a fixed batch of data is available and the objective function does not change over time. We instead investigate DFL in dynamic environments where the objective function and data distribution evolve over time. This setting is challenging for online learning because the objective function has zero or undefined gradients -- which prevents the use of standard first-order optimization methods -- and is generally non-convex. To address these difficulties, we (i) regularize the objective to make it differentiable and (ii) use perturbation techniques along with a near-optimal oracle to overcome non-convexity. Combining those techniques yields two original online algorithms tailored for DFL, for which we establish respectively static and dynamic regret bounds. These are the first provable guarantees for the online decision-focused problem. Finally, we showcase the effectiveness of our algorithms on a knapsack experiment, where they outperform two standard benchmarks.

Online Decision-Focused Learning

TL;DR

This paper extends decision-focused learning from the batch setting to online, non-stationary environments where the objective evolves over time. It introduces two algorithms, DF-FTPL and DF-OGD, that regularize the inner optimization to obtain differentiability and leverage approximate oracles to cope with non-convexity, achieving provable static and dynamic regret guarantees. Theoretical results show sublinear regret bounds: a rate of roughly for static regret and for dynamic regret, with capturing environment variability. Empirical evaluation on a knapsack-inspired task demonstrates that the proposed methods outperform prediction-focused online learning and online SPO in terms of downstream decision costs, while tolerating higher prediction error, highlighting the practical value of online decision-focused learning in dynamic decision problems.

Abstract

Decision-focused learning (DFL) is an increasingly popular paradigm for training predictive models whose outputs are used in decision-making tasks. Instead of merely optimizing for predictive accuracy, DFL trains models to directly minimize the loss associated with downstream decisions. However, existing studies focus solely on scenarios where a fixed batch of data is available and the objective function does not change over time. We instead investigate DFL in dynamic environments where the objective function and data distribution evolve over time. This setting is challenging for online learning because the objective function has zero or undefined gradients -- which prevents the use of standard first-order optimization methods -- and is generally non-convex. To address these difficulties, we (i) regularize the objective to make it differentiable and (ii) use perturbation techniques along with a near-optimal oracle to overcome non-convexity. Combining those techniques yields two original online algorithms tailored for DFL, for which we establish respectively static and dynamic regret bounds. These are the first provable guarantees for the online decision-focused problem. Finally, we showcase the effectiveness of our algorithms on a knapsack experiment, where they outperform two standard benchmarks.
Paper Structure (42 sections, 16 theorems, 77 equations, 3 figures, 4 algorithms)

This paper contains 42 sections, 16 theorems, 77 equations, 3 figures, 4 algorithms.

Table of Contents

  1. Introduction.
  2. Contributions.
  3. Additional related work.
  4. Outline.
  5. Notation.
  6. Framework
  7. Sequential decision-making.
  8. Online decision-focused learning.
  9. Online algorithms for decision-focused learning .
  10. On the need of regularization to get differentiation.
  11. On the choice of regularization on a general polytope.
  12. Approximate oracles to handle non-convexity.
  13. Decision-Focused Follow The Perturbed Leader
  14. Decision-Focused Online Gradient Descent
  15. Experiments
  16. ...and 27 more sections

Key Result

Theorem 1

Assume assumption:regularity, assumption:margin and having access to an $\xi$-approximate oracle $\mathbf{O}_{\xi}$ adapted to $\{\sum_{i=1}^{t} \Tilde{f}_i-\mathopen{}\mathclose{\left\langle\sigma_t, \cdot\right\rangle \}_{t\in[T]}}$. Let $\{\theta_t\}_{t\in[T]}$ be the output of DF-FTPL (algorithm where $\mathbb{E}$ denotes the expectation on both data and the intrinsic randomness of DF-FTPL and

Figures (3)

  • Figure 1: Average cumulated cost (left) and prediction errors (right) of DF-OGD, PF-OGD, DF-FTPL and online SPO.
  • Figure 2: Figure 2 from mandi2024decision.
  • Figure 3: Average cumulated cost gap PFL - DFL as a function of $\gamma$.

Theorems & Definitions (29)

  • Remark 1
  • Definition 1
  • Theorem 1
  • Theorem 2
  • Proposition 1
  • proof
  • Proposition 2
  • Proposition 3
  • Remark 2
  • Proposition 4: Theorem 1 of suggala2020online
  • ...and 19 more