Quantum-Inspired Episode Selection for Monte Carlo Reinforcement Learning via QUBO Optimization

TL;DR

This work tackles the sample efficiency challenge of Monte Carlo reinforcement learning in sparse or large-state environments by reformulating episode selection as a Quadratic Unconstrained Binary Optimization (QUBO) problem and solving it with quantum-inspired samplers. The core idea is to select a small, diverse set of high-value episodes by optimizing a QUBO energy $F(x) = x^T Q x + q^T x$ that balances rewards and trajectory similarity, with solvers such as Simulated Quantum Annealing (SQA) and Simulated Bifurcation (SB) treated as black-box oracles. Empirically, MC+QUBO outperforms vanilla MC in converging to higher-quality policies on finite-horizon GridWorlds, with larger gains in bigger grids, while maintaining practical computation times via cloud-based quantum-inspired resources. This demonstrates the viability of integrating quantum-inspired optimization as a decision-making subroutine inside RL, potentially extending to continuous control and hierarchical or multi-agent settings in future work.

Abstract

Monte Carlo (MC) reinforcement learning suffers from high sample complexity, especially in environments with sparse rewards, large state spaces, and correlated trajectories. We address these limitations by reformulating episode selection as a Quadratic Unconstrained Binary Optimization (QUBO) problem and solving it with quantum-inspired samplers. Our method, MC+QUBO, integrates a combinatorial filtering step into standard MC policy evaluation: from each batch of trajectories, we select a subset that maximizes cumulative reward while promoting state-space coverage. This selection is encoded as a QUBO, where linear terms favor high-reward episodes and quadratic terms penalize redundancy. We explore both Simulated Quantum Annealing (SQA) and Simulated Bifurcation (SB) as black-box solvers within this framework. Experiments in a finite-horizon GridWorld demonstrate that MC+QUBO outperforms vanilla MC in convergence speed and final policy quality, highlighting the potential of quantum-inspired optimization as a decision-making subroutine in reinforcement learning.

Figures (2)

• Figure 1: Iterative Quantum-Assisted Reinforcement Learning — The policy generates multiple episodes, which are encoded as a QUBO and solved by a quantum black-box solver. The best sample guides the policy update, and the process repeats iteratively to converge toward the optimal policy.
• Figure 2: Rolling mean returns (with a window size of 6) over policy update steps for Monte Carlo (MC) and MC combined with Quadratic Unconstrained Binary Optimization (QUBO) methods in gridworld environments. The top row displays results for smaller grids (3×3, 5×5, 8×8) with an obstacle density of 0.22, while the bottom row shows larger grids (10×10, 15×15) with an obstacle density of 0.1 and (20×20) with an obstacle density of 0.01. Blue lines represent the MC method, orange lines represent the MC+QUBO method, and shaded regions indicate variability in the returns.