A Digital Twin Framework for Generation-IV Reactors with Reinforcement Learning-Enabled Health-Aware Supervisory Control

TL;DR

The paper tackles the challenge of safely and economically deploying Gen-IV reactors by introducing a digital twin for a Fluoride-salt-cooled High Temperature Reactor (FHR) that blends physics-based and data-driven surrogates. It combines a reinforcement-learning-based Operator (Soft Actor-Critic), a Reference Governor for constraint enforcement, and Ensemble Kalman Filtering for online state and parameter estimation, all connected through a Virtual Asset built from a gFHR Surrogate and Pump Health Surrogate. The framework is demonstrated via long-term maintenance planning, short-term high-frequency data assimilation, and system-shock capturing with surrogate recalibration, showing robust health-aware, constraint-compliant operation and maintenance scheduling. The approach emphasizes physics-informed, model-aware design to enhance reliability and decision support, with potential applicability to other advanced reactor concepts and complex engineering systems.

Abstract

Generation IV (Gen-IV) nuclear power plants are envisioned to replace the current reactor fleet, bringing improvements in performance, safety, reliability, and sustainability. However, large cost investments currently inhibit the deployment of these advanced reactor concepts. Digital twins bridge real-world systems with digital tools to reduce costs, enhance decision-making, and boost operational efficiency. In this work, a digital twin framework is designed to operate the Gen-IV Fluoride-salt-cooled High-temperature Reactor, utilizing data-enhanced methods to optimize operational and maintenance policies while adhering to system constraints. The closed-loop framework integrates surrogate modeling, reinforcement learning, and Bayesian inference to streamline end-to-end communication for online regulation and self-adjustment. Reinforcement learning is used to consider component health and degradation to drive the target power generations, with constraints enforced through a Reference Governor control algorithm that ensures compliance with pump flow rate and temperature limits. These input driving modules benefit from detailed online simulations that are assimilated to measurement data with Bayesian filtering. The digital twin is demonstrated in three case studies: a one-year long-term operational period showcasing maintenance planning capabilities, short-term accuracy refinement with high-frequency measurements, and system shock capturing that demonstrates real-time recalibration capabilities when change in boundary conditions. These demonstrations validate robustness for health-aware and constraint-informed nuclear plant operation, with general applicability to other advanced reactor concepts and complex engineering systems.

Figures (22)

• Figure 1: SAM gFHR reactor configuration. This schematic is from Lim:25 and is adapted with permission from Dave:2023.
• Figure 2: Digital Twin Framework. The digital twin framework is developed for the SAM gFHR reactor configuration, containing four modules: the Physical Asset, the Virtual Asset, the Virtual to Physical Module, and the Physical to Virtual Module.
• Figure 3: Surrogate Compression of the Virtual Asset to enable Reinforcement Learning training. The compressed model only outputs the power and pump degradation rates $K_i$ on an hourly timescale.
• Figure 4: Constraint Sub-Module Functionality. Using the Reference Governor (RG) algorithm, the Constraint submodule interacts with the Virtual Asset gFHR Surrogate to enforce system constraints.
• Figure 5: Demonstration of the Reference Governor (RG) in the Constraint submodule intervening in a load-following case.