Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SORREL: Suboptimal-Demonstration-Guided Reinforcement Learning for Learning to Branch

Shengyu Feng, Yiming Yang

TL;DR

This work tackles the challenge of learning branching heuristics for MILP Branch-and-Bound by proposing SORREL, a two-stage RL framework that leverages suboptimal demonstrations. It combines offline RL on suboptimal data with online fine-tuning via self-imitation learning within a generalized tree MDP, incorporating BC regularization and PPO/SIL updates to improve efficiency and robustness. Empirical results show that SORREL outperforms neural baselines in solving time and tree size across multiple benchmarks and demonstrates strong transfer generalization, with notable gains on the MK problem where FSB is less informative. The approach offers a practical neural MILP solver with better training efficiency and resilience to demonstration quality, though it acknowledges limitations tied to SIL’s exploration conservativeness and highlights directions for future improvements in rewards and heuristic integration.

Abstract

Mixed Integer Linear Program (MILP) solvers are mostly built upon a Branch-and-Bound (B\&B) algorithm, where the efficiency of traditional solvers heavily depends on hand-crafted heuristics for branching. The past few years have witnessed the increasing popularity of data-driven approaches to automatically learn these heuristics. However, the success of these methods is highly dependent on the availability of high-quality demonstrations, which requires either the development of near-optimal heuristics or a time-consuming sampling process. This paper averts this challenge by proposing Suboptimal-Demonstration-Guided Reinforcement Learning (SORREL) for learning to branch. SORREL selectively learns from suboptimal demonstrations based on value estimation. It utilizes suboptimal demonstrations through both offline reinforcement learning on the demonstrations generated by suboptimal heuristics and self-imitation learning on past good experiences sampled by itself. Our experiments demonstrate its advanced performance in both branching quality and training efficiency over previous methods for various MILPs.

SORREL: Suboptimal-Demonstration-Guided Reinforcement Learning for Learning to Branch

TL;DR

This work tackles the challenge of learning branching heuristics for MILP Branch-and-Bound by proposing SORREL, a two-stage RL framework that leverages suboptimal demonstrations. It combines offline RL on suboptimal data with online fine-tuning via self-imitation learning within a generalized tree MDP, incorporating BC regularization and PPO/SIL updates to improve efficiency and robustness. Empirical results show that SORREL outperforms neural baselines in solving time and tree size across multiple benchmarks and demonstrates strong transfer generalization, with notable gains on the MK problem where FSB is less informative. The approach offers a practical neural MILP solver with better training efficiency and resilience to demonstration quality, though it acknowledges limitations tied to SIL’s exploration conservativeness and highlights directions for future improvements in rewards and heuristic integration.

Abstract

Mixed Integer Linear Program (MILP) solvers are mostly built upon a Branch-and-Bound (B\&B) algorithm, where the efficiency of traditional solvers heavily depends on hand-crafted heuristics for branching. The past few years have witnessed the increasing popularity of data-driven approaches to automatically learn these heuristics. However, the success of these methods is highly dependent on the availability of high-quality demonstrations, which requires either the development of near-optimal heuristics or a time-consuming sampling process. This paper averts this challenge by proposing Suboptimal-Demonstration-Guided Reinforcement Learning (SORREL) for learning to branch. SORREL selectively learns from suboptimal demonstrations based on value estimation. It utilizes suboptimal demonstrations through both offline reinforcement learning on the demonstrations generated by suboptimal heuristics and self-imitation learning on past good experiences sampled by itself. Our experiments demonstrate its advanced performance in both branching quality and training efficiency over previous methods for various MILPs.

Paper Structure

This paper contains 18 sections, 3 theorems, 9 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. The B&B Algorithm
  4. Reinforcement Learning
  5. Method
  6. Tree MDP for B&B
  7. Pretraining with Offline Reinforcement Learning
  8. Finetuning with Self-Imitation Learning
  9. Related Work
  10. Experiment
  11. Experimental Setup
  12. Baselines.
  13. Benchmarks.
  14. Metrics.
  15. Main Results
  16. ...and 3 more sections

Key Result

Proposition 1

Any B&B process following Definition def:rl can be formulated as a tree MDP.

Figures (4)

  • Figure 1: Overview of SORREL. SORREL first trains a branching agent via offline RL on demonstrations collected using existing heuristics, then finetunes the agent via online RL augmented by self-imitation learning from past good experiences.
  • Figure 2: Tree MDP & its associated rewards.
  • Figure 3: Effect of SIL on the sample efficiency. Training dynamics in tree sizes are shown. The $y$-axis is in log-scale.
  • Figure 4: Comparative results on the MIS datset under different levels of suboptimality (varying $p$ in VHB).

Theorems & Definitions (4)

  • Definition 1
  • Proposition 1
  • Proposition 2
  • Proposition 3