Online Adaptive Reinforcement Learning with Echo State Networks for Non-Stationary Dynamics

TL;DR

The paper tackles the problem of deploying RL policies in non-stationary real-world environments where dynamics can shift abruptly within an episode. It introduces ESN-OA, a lightweight online adaptation framework that couples an Echo State Network (fixed reservoir) with Recursive Least Squares to update only the readout weights, avoiding backpropagation and pretraining, and integrates this with Soft Actor-Critic (SAC) by augmenting the state with the ESN-predicted next state $\hat{s}_{t+1}$. The approach achieves zero-shot adaptation to changing dynamics and demonstrates robust performance on CartPole and HalfCheetah under severe disturbances, outperforming Domain Randomization and representative baselines while maintaining real-time efficiency (e.g., about $0.27$ ms per step on CartPole). The work offers a practical, gradient-free mechanism for online adaptation suitable for edge deployment in robotics, enabling rapid recovery from intra-episode dynamics changes without policy resets.

Abstract

Reinforcement learning (RL) policies trained in simulation often suffer from severe performance degradation when deployed in real-world environments due to non-stationary dynamics. While Domain Randomization (DR) and meta-RL have been proposed to address this issue, they typically rely on extensive pretraining, privileged information, or high computational cost, limiting their applicability to real-time and edge systems. In this paper, we propose a lightweight online adaptation framework for RL based on Reservoir Computing. Specifically, we integrate an Echo State Networks (ESNs) as an adaptation module that encodes recent observation histories into a latent context representation, and update its readout weights online using Recursive Least Squares (RLS). This design enables rapid adaptation without backpropagation, pretraining, or access to privileged information. We evaluate the proposed method on CartPole and HalfCheetah tasks with severe and abrupt environment changes, including periodic external disturbances and extreme friction variations. Experimental results demonstrate that the proposed approach significantly outperforms DR and representative adaptive baselines under out-of-distribution dynamics, achieving stable adaptation within a few control steps. Notably, the method successfully handles intra-episode environment changes without resetting the policy. Due to its computational efficiency and stability, the proposed framework provides a practical solution for online adaptation in non-stationary environments and is well suited for real-world robotic control and edge deployment.

Figures (6)

• Figure 1: Overview of the proposed ESN-based online adaptation system. The reservoir captures temporal context, and RLS updates the readout weights $W^{\mathrm{out}}$ in real-time based on prediction error $e_t$. The predicted next state $\hat{s}_{t+1}$ is concatenated with the current state $s_t$ and used as the input to the policy network of the SAC agent.
• Figure 2: Average reward in the CartPole environment under increasing periodic wind disturbances.
• Figure 3: Inference time per control step on CartPole.
• Figure 4: Average reward in the HalfCheetah environment under varying friction coefficients.
• Figure 5: Reward transition after an abrupt friction change from $1.0\times$ to $10.0\times$ during an episode.