Self-Organizing Dual-Buffer Adaptive Clustering Experience Replay (SODASER) for Safe Reinforcement Learning in Optimal Control

TL;DR

Comparative evaluations against random and clustering-based experience replay methods demonstrate that SODACER achieves faster convergence, improved sample efficiency, and a superior bias-variance trade-off, while maintaining safe system trajectories, validated via the Friedman test.

Abstract

This paper proposes a novel reinforcement learning framework, named Self-Organizing Dual-buffer Adaptive Clustering Experience Replay (SODACER), designed to achieve safe and scalable optimal control of nonlinear systems. The proposed SODACER mechanism consisting of a Fast-Buffer for rapid adaptation to recent experiences and a Slow-Buffer equipped with a self-organizing adaptive clustering mechanism to maintain diverse and non-redundant historical experiences. The adaptive clustering mechanism dynamically prunes redundant samples, optimizing memory efficiency while retaining critical environmental patterns. The approach integrates SODASER with Control Barrier Functions (CBFs) to guarantee safety by enforcing state and input constraints throughout the learning process. To enhance convergence and stability, the framework is combined with the Sophia optimizer, enabling adaptive second-order gradient updates. The proposed SODACER-Sophia's architecture ensures reliable, effective, and robust learning in dynamic, safety-critical environments, offering a generalizable solution for applications in robotics, healthcare, and large-scale system optimization. The proposed approach is validated on a nonlinear Human Papillomavirus (HPV) transmission model with multiple control inputs and safety constraints. Comparative evaluations against random and clustering-based experience replay methods demonstrate that SODACER achieves faster convergence, improved sample efficiency, and a superior bias-variance trade-off, while maintaining safe system trajectories, validated via the Friedman test.

Figures (7)

• Figure 1: Gaussian membership functions illustrating a 95% overlap. The blue curve represents the first Gaussian function centered at center zero with a standard deviation one , while the red curve represents the second Gaussian function centered at 0.3 with the standard deviation 0.3. The red cluster is redundant and can be absorbed into the blue cluster for improved efficiency.
• Figure 2: Sequential Workflow of the Proposed Approach
• Figure 3: System states of the HPV model: top panel shows states without control over time, and bottom panel displays states with constant controls ($u_1 = 0.5$, $u_2 = 0.2$, $w_1 = 0.2$, $w_2 = 0.1$, $\alpha = 0.2$) over time.
• Figure 4: Spectral representation of HPV system states using the proposed approach, based on 200 simulation runs.
• Figure 5: Spectral representation of control signals using the proposed approach, based on 200 simulation runs.