Black-Box Combinatorial Optimization with Order-Invariant Reinforcement Learning
Olivier Goudet, Quentin Suire, Adrien Goëffon, Frédéric Saubion, Sylvain Lamprier
TL;DR
The paper tackles discrete black-box optimization by introducing an order-invariant reinforcement learning framework for Estimation-of-Distribution Algorithms (EDAs). By training neural autoregressive generators with randomly sampled generation orders and training orders, the method achieves robust exploration and mitigates reliance on a fixed dependency structure. A Proximal Policy Optimization–based backbone with scale-invariant GRPO advantages stabilizes updates, while a rank-based advantage supports monotone-transform invariant performance. Empirical results across QUBO, NK, and NK3 benchmarks show strong scalability and competitive performance, particularly on larger instances, indicating practical impact for high-dimensional discrete optimization tasks. Overall, the approach combines permutation-invariant modeling, structural regularization, and robust RL updates to advance sample-efficient, black-box combinatorial optimization.
Abstract
We introduce an order-invariant reinforcement learning framework for black-box combinatorial optimization. Classical estimation-of-distribution algorithms (EDAs) often rely on learning explicit variable dependency graphs, which can be costly and fail to capture complex interactions efficiently. In contrast, we parameterize a multivariate autoregressive generative model trained without a fixed variable ordering. By sampling random generation orders during training - a form of information-preserving dropout - the model is encouraged to be invariant to variable order, promoting search-space diversity and shaping the model to focus on the most relevant variable dependencies, improving sample efficiency. We adapt Generalized Reinforcement Policy Optimization (GRPO) to this setting, providing stable policy-gradient updates from scale-invariant advantages. Across a wide range of benchmark algorithms and problem instances of varying sizes, our method frequently achieves the best performance and consistently avoids catastrophic failures.