Unraveling the Rainbow: can value-based methods schedule?

TL;DR

The paper investigates whether value-based deep RL approaches can match or exceed policy-gradient methods for two classic scheduling problems, JSSP and FJSP. It presents an end-to-end graph-based DRL framework, implements Rainbow and its components, and compares them against policy-gradient baselines, including ablations and generalization tests. Across multiple instance sizes and distributions, value-based methods demonstrate lower training variance and strong cross-size/cross-distribution generalization, with results highly dependent on problem structure (FJSP favoring value-based, larger JSSP sometimes favoring policy-gradient). The findings suggest that value-based DRL deserves more attention in scheduling research and practice, and the authors release open-source code to enable reproducibility and further exploration.

Abstract

In this work, we conduct an extensive empirical study of several deep reinforcement learning algorithms on two challenging combinatorial optimization problems: the job-shop and flexible job-shop scheduling problems, both fundamental challenges with multiple industrial applications. Broadly, deep reinforcement learning algorithms fall into two categories: policy-gradient and value-based. While value-based algorithms have achieved notable success in domains such as the Arcade Learning Environment, the combinatorial optimization community has predominantly favored policy-gradient algorithms, often overlooking the potential of value-based alternatives. From our results, value-based algorithms demonstrated a lower variance and a more stable convergence profile compared to policy-gradient ones. Moreover, they achieved superior cross-size and cross-distribution generalization, that is, effectively solving instances that are substantially larger or structurally distinct from those seen during training. Finally, our analysis also suggests that the relative performance of each category of algorithms may be dependent on structural properties of the problem, such as problem flexibility and instance size. Overall, our findings challenge the prevailing assumption that policy-gradient algorithms are inherently superior for combinatorial optimization. We show instead that value-based algorithms can match or even surpass the performance of policy-gradient algorithms, suggesting that they deserve greater attention from the combinatorial optimization community. Our code is openly available at: https://github.com/AJ-Correa/Unraveling-the-Rainbow

Figures (8)

• Figure 1: JSSP and FJSP instances.
• Figure 2: Unraveling the Rainbow architecture.
• Figure 3: Average validation makespan throughout training for each value-based algorithm, shown separately for all instance sizes. Each subplot corresponds to a specific instance size. The top row displays results for the FJSP, and the bottom row for the JSSP.
• Figure 4: Average validation makespan throughout training for each policy-gradient algorithm, shown separately for all instance sizes. Each subplot corresponds to a specific instance size. The top row displays results for the FJSP, and the bottom row for the JSSP.
• Figure 5: Average validation makespan throughout training for each class of DRL algorithms, shown separately for all instance sizes. Each subplot corresponds to a specific instance size. The top row displays results for the FJSP, and the bottom row for the JSSP. Shaded regions represent one standard deviation above and below the mean.