Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Objective Reinforcement Learning-based Approach for Pressurized Water Reactor Optimization

Paul Seurin, Koroush Shirvan

TL;DR

The paper tackles the challenge of multi-objective optimization for Pressurized Water Reactor loading pattern design by introducing PEARL, a single-policy reinforcement learning framework that envelops the Pareto front. It develops unconstrained and constrained variants (PEARL-e, PEARL-ε, PEARL-NdS; and C-PEARL with Curriculum Learning) to adaptively balance multiple objectives and handle constraints via CMDP formulations and hierarchical learning. Thorough evaluation on classical MOO benchmarks and realistic PWR problems demonstrates that PEARL-NdS with CL often achieves superior Pareto front coverage (HV) and feasible solution counts, outperforming traditional SO approaches in constrained settings and offering competitive performance in unconstrained cases. The work highlights the practical potential of an RL-based, envelope-focused approach for engineering design, providing scalable, efficient exploration of trade-offs with real-world applicability and open-source availability through NEORL.

Abstract

A novel method, the Pareto Envelope Augmented with Reinforcement Learning (PEARL), has been developed to address the challenges posed by multi-objective problems, particularly in the field of engineering where the evaluation of candidate solutions can be time-consuming. PEARL distinguishes itself from traditional policy-based multi-objective Reinforcement Learning methods by learning a single policy, eliminating the need for multiple neural networks to independently solve simpler sub-problems. Several versions inspired from deep learning and evolutionary techniques have been crafted, catering to both unconstrained and constrained problem domains. Curriculum Learning is harnessed to effectively manage constraints in these versions. PEARL's performance is first evaluated on classical multi-objective benchmarks. Additionally, it is tested on two practical PWR core Loading Pattern optimization problems to showcase its real-world applicability. The first problem involves optimizing the Cycle length and the rod-integrated peaking factor as the primary objectives, while the second problem incorporates the mean average enrichment as an additional objective. Furthermore, PEARL addresses three types of constraints related to boron concentration, peak pin burnup, and peak pin power. The results are systematically compared against conventional approaches. Notably, PEARL, specifically the PEARL-NdS variant, efficiently uncovers a Pareto front without necessitating additional efforts from the algorithm designer, as opposed to a single optimization with scaled objectives. It also outperforms the classical approach across multiple performance metrics, including the Hyper-volume.

Multi-Objective Reinforcement Learning-based Approach for Pressurized Water Reactor Optimization

TL;DR

The paper tackles the challenge of multi-objective optimization for Pressurized Water Reactor loading pattern design by introducing PEARL, a single-policy reinforcement learning framework that envelops the Pareto front. It develops unconstrained and constrained variants (PEARL-e, PEARL-ε, PEARL-NdS; and C-PEARL with Curriculum Learning) to adaptively balance multiple objectives and handle constraints via CMDP formulations and hierarchical learning. Thorough evaluation on classical MOO benchmarks and realistic PWR problems demonstrates that PEARL-NdS with CL often achieves superior Pareto front coverage (HV) and feasible solution counts, outperforming traditional SO approaches in constrained settings and offering competitive performance in unconstrained cases. The work highlights the practical potential of an RL-based, envelope-focused approach for engineering design, providing scalable, efficient exploration of trade-offs with real-world applicability and open-source availability through NEORL.

Abstract

A novel method, the Pareto Envelope Augmented with Reinforcement Learning (PEARL), has been developed to address the challenges posed by multi-objective problems, particularly in the field of engineering where the evaluation of candidate solutions can be time-consuming. PEARL distinguishes itself from traditional policy-based multi-objective Reinforcement Learning methods by learning a single policy, eliminating the need for multiple neural networks to independently solve simpler sub-problems. Several versions inspired from deep learning and evolutionary techniques have been crafted, catering to both unconstrained and constrained problem domains. Curriculum Learning is harnessed to effectively manage constraints in these versions. PEARL's performance is first evaluated on classical multi-objective benchmarks. Additionally, it is tested on two practical PWR core Loading Pattern optimization problems to showcase its real-world applicability. The first problem involves optimizing the Cycle length and the rod-integrated peaking factor as the primary objectives, while the second problem incorporates the mean average enrichment as an additional objective. Furthermore, PEARL addresses three types of constraints related to boron concentration, peak pin burnup, and peak pin power. The results are systematically compared against conventional approaches. Notably, PEARL, specifically the PEARL-NdS variant, efficiently uncovers a Pareto front without necessitating additional efforts from the algorithm designer, as opposed to a single optimization with scaled objectives. It also outperforms the classical approach across multiple performance metrics, including the Hyper-volume.
Paper Structure (25 sections, 8 equations, 3 figures, 19 tables, 3 algorithms)

This paper contains 25 sections, 8 equations, 3 figures, 19 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Multi-Objective Reinforcement Learning (RL)
  3. Definition of Multi-Objective Optimization (MOO)
  4. Definition of RL within the MOO context
  5. Application and limitations of RL to MOO
  6. Derivation of the Pareto Envelope Augmented with Reinforcement Learning (PEARL)
  7. Unconstrained PEARL
  8. Constrained PEARL (C-PEARL)
  9. Constraints handling in Reinforcement Learning (RL)
  10. Curriculum Learning and hierarchical approach for Constrained-PEARL (C-PEARL)
  11. Design of experiments
  12. Tools for performance comparisons
  13. Test sets
  14. Classical benchmarks
  15. PWR multi-objective problems
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Two-obj ((a) to (f)) and Three-obj ((g) to (L))) solutions generated for all the algorithms utilized. The pareto front is a line for the Two-objective case, while it is drawn with scattered points for the Three-objective one.
  • Figure 2: Evolution of the target FOMs for the Two-objective ((a) and (b)) and the Three-objective ((c) to (e)) problem over 100 generations for PEARL. We added vertical grid lines in the Three-objective case to help with visibility and interpretation of the results: (a) and (c) $F_{\Delta h}$, (b) and (d) $L_{cy}$, and (e) Average feed enrichment. The lighter areas cover the average $\pm \sigma$.
  • Figure 3: Two-obj ((a) to (d)) and Three-obj ((e) to (i)) feasible solutions generated for all the algorithms utilized. The Pareto front is a line for the Two-obj case, while it is drawn with scattered points for the Three-objective one.