DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delays

TL;DR

DEER addresses reinforcement learning under environmental delays by decoupling encoding from decision making. It trains a Seq2Seq encoder-decoder on delay-free offline data to map delayed information states and action histories into fixed-length context representations, which are then used by standard RL algorithms with no delay-specific modifications, while the encoder remains fixed during online learning. The framework generalizes across constant and random delays and is validated via SAC on Gym and MuJoCo tasks, showing competitive or superior performance to existing delay-aware RL methods. Key contributions include offline pretraining of a universal encoder, an interpretable two-stage architecture, and extensive analyses on representation dimension and dataset composition, highlighting practical guidelines for encoder data and performance. The approach offers improved robustness and interpretability, with potential extensions to visual RL and real-world robotic systems.

Abstract

Classic reinforcement learning (RL) frequently confronts challenges in tasks involving delays, which cause a mismatch between received observations and subsequent actions, thereby deviating from the Markov assumption. Existing methods usually tackle this issue with end-to-end solutions using state augmentation. However, these black-box approaches often involve incomprehensible processes and redundant information in the information states, causing instability and potentially undermining the overall performance. To alleviate the delay challenges in RL, we propose $\textbf{DEER (Delay-resilient Encoder-Enhanced RL)}$, a framework designed to effectively enhance the interpretability and address the random delay issues. DEER employs a pretrained encoder to map delayed states, along with their variable-length past action sequences resulting from different delays, into hidden states, which is trained on delay-free environment datasets. In a variety of delayed scenarios, the trained encoder can seamlessly integrate with standard RL algorithms without requiring additional modifications and enhance the delay-solving capability by simply adapting the input dimension of the original algorithms. We evaluate DEER through extensive experiments on Gym and Mujoco environments. The results confirm that DEER is superior to state-of-the-art RL algorithms in both constant and random delay settings.

Figures (8)

• Figure 1: Overview of DEER. The overall process consists of two main parts: pre-training an encoder using the offline dataset from undelayed environments to obtain a fixed-length feature representation for information states, and utilizing these context representations to guide decision-making during the agent's interaction with the delayed environments. Specifically, for policy learning in environments with a constant delay of $d$, the construction of the information is depicted by the thick solid line. In random environments, where the agent misses state $s_{t-d}$, the thin dashed line illustrates the process of constructing the information state using the preceding state and action sequences from the previous time step. The variable $\text{D}$ in the figure denotes the maximum delay the agent can tolerate ($\text{D} = d_I + d_M$). Subsequently, information states are mapped into fixed-length context representations, which the agent uses for decision-making.
• Figure 2: Process of model pretraining. Firstly, the information state dataset is created based on the original undelayed dataset. All state sequences are standardized to a uniform length $D$, where $D$ represents the maximum delay in the environment. Next, these datasets are fed into the Seq2Seq model and trained in a supervised manner.
• Figure 3: Comparison of algorithms under diverse constant delays.
• Figure 4: Comparison of algorithms under diverse random delays
• Figure 5: Comparison of DEER's performance with different number of expert trajectories in Walker2d.

Theorems & Definitions (2)

• Definition 3.1
• Definition A.1