Soft Switching Expert Policies for Controlling Systems with Uncertain Parameters

TL;DR

The paper tackles controlling systems with uncertain, time-varying parameters and the reality gap between simulators and real systems. It introduces a two-stage soft-switching expert policies (SSEP) framework: train expert policies for representative parameter samples in a simulator, then form an adaptive policy as a normalized weighted sum with online convex optimization-based weight updates using real observations. The approach is demonstrated on a nonlinear pendulum-like system, showing that the adaptive policy can smoothly switch among experts as parameters vary, offering robustness without relying on recurrent architectures. Key contributions include an OCO-based weight update (FTRL with negentropy) and a discounted variant for nonstationary parameters, providing a practical bridge between DRL in simulation and real-world control.

Abstract

This paper proposes a simulation-based reinforcement learning algorithm for controlling systems with uncertain and varying system parameters. While simulators are useful to safely synthesize control policies for physical systems using reinforcement learning, mitigating the reality gap remains a major challenge. To address the challenge, we propose a two-stage algorithm. In the first stage, multiple control policies are learned for systems with different parameters in a simulator. In the second stage, for a real system, the control policies learned in the first stage are smoothly switched using an online convex optimization algorithm based on observations. Our proposed algorithm is demonstrated through numerical experiments.

Figures (5)

• Figure 1: Illustration of our proposed two-stage algorithm, which is called the soft switching expert policies algorithm. We select representative points $\{\xi^{(j)}\}_{j=1}^{M}$ from $\Xi$ beforehand. In the first stage, an expert policy $\mu_{\theta,j}$ is learned for each simulated system $f(\cdot,\cdot;\xi^{(j)})$. In the second stage, we construct an adaptive policy as a normalized weighted sum of the expert policies and update its weights using an online convex optimization algorithm.
• Figure 2: $G(\mu|\xi)$ of the policies learned using DDPG with DR.
• Figure 3: $G(\mu|\xi)$ of three expert policies learned for representative simulated systems using DDPG.
• Figure 4: $G(\mu|\xi)$ of the adaptive policy with FTRL.
• Figure 5: Time response of the system controlled by the adaptive policy with discounted FTRL.