Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Markov Decision Process for Variable Selection in Branch & Bound

Paul Strang, Zacharie Alès, Côme Bissuel, Olivier Juan, Safia Kedad-Sidhoum, Emmanuel Rachelson

TL;DR

This work reframes variable selection in Branch & Bound MILP as a proper Markov decision process by introducing BBMDP, a vanilla MDP that preserves temporality and allows standard dynamic programming and RL methods. It contrasts BBMDP with prior TreeMDP formulations, demonstrating that TreeMDP can misalign with MDP principles and multi-step learning, while BBMDP supports $k$-step TD, TD($\lambda$), and MCTS. The authors propose an approximate dynamic programming approach that learns $\bar{Q}^*$ via a neural network using $k$-step updates and HL-Gauss loss, and they show that a DQN-BBMDP agent achieves state-of-the-art performance among RL methods on four Ecole MILP benchmarks, often outperforming SCIP and narrowing the gap to imitation-based strong branching. The results indicate that a principled MDP formulation unlocks the potential of RL in B&B, with practical impact on solving larger and more diverse MILPs, though generalization remains challenging and distribution shift is an area for future work.

Abstract

Mixed-Integer Linear Programming (MILP) is a powerful framework used to address a wide range of NP-hard combinatorial optimization problems, often solved by Branch and Bound (B&B). A key factor influencing the performance of B&B solvers is the variable selection heuristic governing branching decisions. Recent contributions have sought to adapt reinforcement learning (RL) algorithms to the B&B setting to learn optimal branching policies, through Markov Decision Processes (MDP) inspired formulations, and ad hoc convergence theorems and algorithms. In this work, we introduce BBMDP, a principled vanilla MDP formulation for variable selection in B&B, allowing to leverage a broad range of RL algorithms for the purpose of learning optimal B\&B heuristics. Computational experiments validate our model empirically, as our branching agent outperforms prior state-of-the-art RL agents on four standard MILP benchmarks.

A Markov Decision Process for Variable Selection in Branch & Bound

TL;DR

This work reframes variable selection in Branch & Bound MILP as a proper Markov decision process by introducing BBMDP, a vanilla MDP that preserves temporality and allows standard dynamic programming and RL methods. It contrasts BBMDP with prior TreeMDP formulations, demonstrating that TreeMDP can misalign with MDP principles and multi-step learning, while BBMDP supports -step TD, TD(), and MCTS. The authors propose an approximate dynamic programming approach that learns via a neural network using -step updates and HL-Gauss loss, and they show that a DQN-BBMDP agent achieves state-of-the-art performance among RL methods on four Ecole MILP benchmarks, often outperforming SCIP and narrowing the gap to imitation-based strong branching. The results indicate that a principled MDP formulation unlocks the potential of RL in B&B, with practical impact on solving larger and more diverse MILPs, though generalization remains challenging and distribution shift is an area for future work.

Abstract

Mixed-Integer Linear Programming (MILP) is a powerful framework used to address a wide range of NP-hard combinatorial optimization problems, often solved by Branch and Bound (B&B). A key factor influencing the performance of B&B solvers is the variable selection heuristic governing branching decisions. Recent contributions have sought to adapt reinforcement learning (RL) algorithms to the B&B setting to learn optimal branching policies, through Markov Decision Processes (MDP) inspired formulations, and ad hoc convergence theorems and algorithms. In this work, we introduce BBMDP, a principled vanilla MDP formulation for variable selection in B&B, allowing to leverage a broad range of RL algorithms for the purpose of learning optimal B\&B heuristics. Computational experiments validate our model empirically, as our branching agent outperforms prior state-of-the-art RL agents on four standard MILP benchmarks.
Paper Structure (34 sections, 1 theorem, 9 equations, 4 figures, 7 tables, 1 algorithm)

This paper contains 34 sections, 1 theorem, 9 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem statement
  3. Mixed-integer linear programming
  4. Reinforcement learning
  5. Related work
  6. Branch and bound Markov decision process
  7. Definition
  8. Misconceptions when learning to explore B&B trees
  9. Approximate dynamic programming
  10. BBMDP vs TreeMDP
  11. Experiments
  12. Experimental setup
  13. Results
  14. Ablation study
  15. Conclusion and perspectives
  16. ...and 19 more sections

Key Result

Proposition 3.1

In DFS-BBMDP, the Bellman equation $V^\pi(s) = \mathcal{R}(s, \pi(s)) + V^\pi(s')$ yields: with $o_1=\rho(s)$, $o'_1 = \rho(\mathcal{T}(s,\pi(s))$, and $o'_2$ is the sibling of $o'_1$ in the B&B tree.

Figures (4)

  • Figure 1: Normalized scores in log scale of IL, RL and random agents across the Ecole benchmark prouvost_ecole_2020.
  • Figure 2: Solving a MILP by B&B using variable selection policy $\pi$ and node selection policy $\rho$. Each node $v_i$ represents a MILP derived from the original problem, each edge represents the bound adjustment applied to derive child nodes from their parent. At each step, nodes $o_i \in \mathcal{O}$ are re-indexed according to $\rho$.
  • Figure 3: When minimizing 1-step temporal difference, TreeMDP and BBMDP yield equivalent results, see \ref{['fig:clean_one_step']}, \ref{['fig:tree_one_step']}. However, over $k$-step trajectories, the two methods diverge as shown in \ref{['fig:v_tree']}, \ref{['fig:w_tree']}.
  • Figure 4: Validation curves for DQN-BBMDP, DQN-Retro and DQN-tMDP agents, in log scale. Throughout training, agents are evaluated on 20 validation instances after each batch of 100 training instances solved. Note that on the multiple knapsack benchmark, none of the agents reach convergence.

Theorems & Definitions (2)

  • Proposition 3.1
  • proof