Surpassing legacy approaches to PWR core reload optimization with single-objective Reinforcement learning
Paul Seurin, Koroush Shirvan
TL;DR
The paper tackles the challenging problem of optimizing PWR core loading patterns to minimize fuel-cycle cost under multiple constraints. It compares a Proximal Policy Optimization (PPO)–based reinforcement learning approach against classical stochastic optimization methods (SA, GA/ES, TS, PSA, PESA) across classical and extended PWR designs with eighth and quarter symmetry. The results show PPO achieving superior statistical performance in most single-objective designs, demonstrating robust convergence, improved sample efficiency, and the ability to adapt its search from global exploration to local exploitation as the objective space curvature changes. The study underscores PPO’s practical potential for nuclear core reload optimization, while highlighting remaining challenges in the hardest 81-quarter designs and suggesting avenues for further research and resource scaling. Overall, the work provides a rigorous, data-driven demonstration that RL with learnable policy weights can surpass legacy heuristics for complex, constrained core loading problems, with significant implications for reactor operation economics and safety compliance.
Abstract
Optimizing the fuel cycle cost through the optimization of nuclear reactor core loading patterns involves multiple objectives and constraints, leading to a vast number of candidate solutions that cannot be explicitly solved. To advance the state-of-the-art in core reload patterns, we have developed methods based on Deep Reinforcement Learning (DRL) for both single- and multi-objective optimization. Our previous research has laid the groundwork for these approaches and demonstrated their ability to discover high-quality patterns within a reasonable time frame. On the other hand, stochastic optimization (SO) approaches are commonly used in the literature, but there is no rigorous explanation that shows which approach is better in which scenario. In this paper, we demonstrate the advantage of our RL-based approach, specifically using Proximal Policy Optimization (PPO), against the most commonly used SO-based methods: Genetic Algorithm (GA), Parallel Simulated Annealing (PSA) with mixing of states, and Tabu Search (TS), as well as an ensemble-based method, Prioritized Replay Evolutionary and Swarm Algorithm (PESA). We found that the LP scenarios derived in this paper are amenable to a global search to identify promising research directions rapidly, but then need to transition into a local search to exploit these directions efficiently and prevent getting stuck in local optima. PPO adapts its search capability via a policy with learnable weights, allowing it to function as both a global and local search method. Subsequently, we compared all algorithms against PPO in long runs, which exacerbated the differences seen in the shorter cases. Overall, the work demonstrates the statistical superiority of PPO compared to the other considered algorithms.