Multistep Criticality Search and Power Shaping in Microreactors with Reinforcement Learning

TL;DR

This work demonstrates an end-to-end RL framework for autonomous reactivity control in a microreactor by coupling high-fidelity Serpent data with neural-network surrogates and a Gym-based RL environment. Proximal Policy Optimization (PPO) consistently outperforms Advantage Actor-Critic (A2C) in learning a policy that drives the core to criticality while achieving symmetric hexant power distribution across burnup states, achieving HPTR ≈ 1.002 and $k_{eff}$ within roughly $\pm 10$ pcm. The approach leverages fast surrogate-informed decision making to enable near real-time control and highlights practical considerations for data efficiency, burnup-specific modeling, and potential digital-twin deployment, with avenues for extending to transient and load-following scenarios.

Abstract

Reducing operation and maintenance costs is a key objective for advanced reactors in general and microreactors in particular. To achieve this reduction, developing robust autonomous control algorithms is essential to ensure safe and autonomous reactor operation. Recently, artificial intelligence and machine learning algorithms, specifically reinforcement learning (RL) algorithms, have seen rapid increased application to control problems, such as plasma control in fusion tokamaks and building energy management. In this work, we introduce the use of RL for intelligent control in nuclear microreactors. The RL agent is trained using proximal policy optimization (PPO) and advantage actor-critic (A2C), cutting-edge deep RL techniques, based on a high-fidelity simulation of a microreactor design inspired by the Westinghouse eVinci\textsuperscript{TM} design. We utilized a Serpent model to generate data on drum positions, core criticality, and core power distribution for training a feedforward neural network surrogate model. This surrogate model was then used to guide a PPO and A2C control policies in determining the optimal drum position across various reactor burnup states, ensuring critical core conditions and symmetrical power distribution across all six core portions. The results demonstrate the excellent performance of PPO in identifying optimal drum positions, achieving a hextant power tilt ratio of approximately 1.002 (within the limit of $<$ 1.02) and maintaining criticality within a 10 pcm range. A2C did not provide as competitive of a performance as PPO in terms of performance metrics for all burnup steps considered in the cycle. Additionally, the results highlight the capability of well-trained RL control policies to quickly identify control actions, suggesting a promising approach for enabling real-time autonomous control through digital twins.

Figures (3)

• Figure 1: EMD with control drums oriented at 90$^\circ$, shown with the slice at the core midplane (left) and a 3D rendering with a diagonal cut (right). Hexant divisions are shown with dotted red lines. Control drums are not rendered in the 3D depiction.
• Figure 2: The RL control framework adopted in this work.
• Figure 3: Progression of PPO and A2C training as a function of the number of epochs. Each epoch comprises 30,000 time steps for which the statistics (mean, std, max, min) are derived. The top row shows the convergence of $k_{eff}$, HPTR, and the reward value as indicated in Eq.\ref{['eq:rwd']} for PPO while the bottom row shows these plots for A2C. All values shown are averaged over the three reactor states and 20 parallel agents. Also, note that the reward scale in this plot is slightly different from Figure 3 of radaideh2024demonstration (see the main text for explanation).