Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Imitation Learning for Combinatorial Optimisation under Uncertainty

Prakash Gawas, Antoine Legrain, Louis-Martin Rousseau

TL;DR

The paper addresses how the choice of expert impacts imitation learning for sequential decision problems under uncertainty. It offers a formal taxonomy of expert types along uncertainty treatment, optimality, and interaction, and proposes a generalized DAgger framework to compare configurations. Empirical results on dynamic physician-to-patient assignment show that stochastic experts generally outperform deterministic or full-information ones, and that interactive learning improves data efficiency and policy quality. The findings provide practical guidance for selecting expert models and interaction schemes to balance learning performance and computational cost in combinatorial optimisation tasks. The work also demonstrates the viability of integrating IL with optimisation solvers to achieve near-optimised decisions with fast inference.

Abstract

Imitation learning (IL) provides a data-driven framework for approximating policies for large-scale combinatorial optimisation problems formulated as sequential decision problems (SDPs), where exact solution methods are computationally intractable. A central but underexplored aspect of IL in this context is the role of the \emph{expert} that generates training demonstrations. Existing studies employ a wide range of expert constructions, yet lack a unifying framework to characterise their modelling assumptions, computational properties, and impact on learning performance. This paper introduces a systematic taxonomy of experts for IL in combinatorial optimisation under uncertainty. Experts are classified along three dimensions: (i) their treatment of uncertainty, including myopic, deterministic, full-information, two-stage stochastic, and multi-stage stochastic formulations; (ii) their level of optimality, distinguishing task-optimal and approximate experts; and (iii) their interaction mode with the learner, ranging from one-shot supervision to iterative, interactive schemes. Building on this taxonomy, we propose a generalised Dataset Aggregation (DAgger) algorithm that supports multiple expert queries, expert aggregation, and flexible interaction strategies. The proposed framework is evaluated on a dynamic physician-to-patient assignment problem with stochastic arrivals and capacity constraints. Computational experiments compare learning outcomes across expert types and interaction regimes. The results show that policies learned from stochastic experts consistently outperform those learned from deterministic or full-information experts, while interactive learning improves solution quality using fewer expert demonstrations. Aggregated deterministic experts provide an effective alternative when stochastic optimisation becomes computationally challenging.

Imitation Learning for Combinatorial Optimisation under Uncertainty

TL;DR

The paper addresses how the choice of expert impacts imitation learning for sequential decision problems under uncertainty. It offers a formal taxonomy of expert types along uncertainty treatment, optimality, and interaction, and proposes a generalized DAgger framework to compare configurations. Empirical results on dynamic physician-to-patient assignment show that stochastic experts generally outperform deterministic or full-information ones, and that interactive learning improves data efficiency and policy quality. The findings provide practical guidance for selecting expert models and interaction schemes to balance learning performance and computational cost in combinatorial optimisation tasks. The work also demonstrates the viability of integrating IL with optimisation solvers to achieve near-optimised decisions with fast inference.

Abstract

Imitation learning (IL) provides a data-driven framework for approximating policies for large-scale combinatorial optimisation problems formulated as sequential decision problems (SDPs), where exact solution methods are computationally intractable. A central but underexplored aspect of IL in this context is the role of the \emph{expert} that generates training demonstrations. Existing studies employ a wide range of expert constructions, yet lack a unifying framework to characterise their modelling assumptions, computational properties, and impact on learning performance. This paper introduces a systematic taxonomy of experts for IL in combinatorial optimisation under uncertainty. Experts are classified along three dimensions: (i) their treatment of uncertainty, including myopic, deterministic, full-information, two-stage stochastic, and multi-stage stochastic formulations; (ii) their level of optimality, distinguishing task-optimal and approximate experts; and (iii) their interaction mode with the learner, ranging from one-shot supervision to iterative, interactive schemes. Building on this taxonomy, we propose a generalised Dataset Aggregation (DAgger) algorithm that supports multiple expert queries, expert aggregation, and flexible interaction strategies. The proposed framework is evaluated on a dynamic physician-to-patient assignment problem with stochastic arrivals and capacity constraints. Computational experiments compare learning outcomes across expert types and interaction regimes. The results show that policies learned from stochastic experts consistently outperform those learned from deterministic or full-information experts, while interactive learning improves solution quality using fewer expert demonstrations. Aggregated deterministic experts provide an effective alternative when stochastic optimisation becomes computationally challenging.
Paper Structure (40 sections, 24 equations, 10 figures, 13 tables, 2 algorithms)

This paper contains 40 sections, 24 equations, 10 figures, 13 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Markov Decision Process
  4. Machine Learning
  5. Machine Learning for MDPs
  6. Imitation Learning: Models and Challenges
  7. Learning Paradigms
  8. Behavioural Cloning (BC)
  9. Comparing PPT and AIT
  10. Distribution Shift
  11. Inverse Reinforcement Learning (IRL)
  12. Learning from Noisy or Suboptimal Experts
  13. Applications of IL in SDPs
  14. ML-Augmented Optimisation
  15. End-to-End Learning
  16. ...and 25 more sections

Figures (10)

  • Figure 1: PPT or RMT : Loss = $C({x, a}, \boldsymbol{\xi}) - C({x, a^*}, \boldsymbol{\xi})$
  • Figure 2: AIT or LtO: Loss = $\ell({a} , {a^*})$
  • Figure 3: Classification of literature.
  • Figure 4: Test performance of learned models with different numbers of episodes $N$
  • Figure 5: Test performance of learned models with different numbers of episodes $N$
  • ...and 5 more figures