Adaptive Model-Based Reinforcement Learning for Orbit Feedback Control in NSLS-II Storage Ring

TL;DR

The paper tackles the challenge of stabilizing the NSLS-II storage-ring beam orbit under drift, noise, and nonlinear dynamics. It introduces a model-based reinforcement learning framework with two optimizations: trajectory optimization on a differentiable surrogate and online model optimization using real-time data to update the dynamics. Across simulation and NSLS-II experiments, the approach yields substantial improvements in beam stability, achieving RMS on the order of $\sim 1\,\mu\mathrm{m}$ and significantly outperforming the traditional SVD-based method. The key contributions are the dual-optimization RL scheme, differentiable modeling for policy learning, and real-world validation demonstrating robust, adaptive control in a high-dimensional MIMO system.

Abstract

The National Synchrotron Light Source II (NSLS-II) uses highly stable electron beam to produce high-quality X-ray beams with high brightness and low-emittance synchrotron radiation. The traditional algorithm to stabilize the beam applies singular value decomposition (SVD) on the orbit response matrix to remove noise and extract actions. Supervised learning has been studied on NSLS-II storage ring stabilization and other accelerator facilities recently. Several problems, for example, machine status drifting, environment noise, and non-linear accelerator dynamics, remain unresolved in the SVD-based and supervised learning algorithms. To address these problems, we propose an adaptive training framework based on model-based reinforcement learning. This framework consists of two types of optimizations: trajectory optimization attempts to minimize the expected total reward in a differentiable environment, and online model optimization learns non-linear machine dynamics through the agent-environment interaction. Through online training, this framework tracks the internal status drifting in the electron beam ring. Simulation and real in-facility experiments on NSLS-II reveal that our method stabilizes the beam position and minimizes the alignment error, defined as the root mean square (RMS) error between adjusted beam positions and the reference position, down to ~1$μ$m.

Figures (8)

• Figure 1: An illustration of a storage ring.
• Figure 2: A degenerated final state after a long run. The dimensions that lose control are marked in red.
• Figure 3: An illustration of the closed loop feedback system.
• Figure 4: The SVD-based PID control: $q_i$ and $z_i$ stand for each component of the spectrum space after the transformation of the current input.
• Figure 5: The data flow for model-based RL with online model optimization.