Reinforcement Learning via Conservative Agent for Environments with Random Delays

TL;DR

This work addresses reinforcement learning under random delays that can break the Markov property by introducing a conservative agent that constructs a constant-delay surrogate using the maximum delay $\Delta_{\max}$. This enables any constant-delay RL method to be applied directly to random-delay environments without modeling the delay distribution, and it yields delay-agnostic performance as long as $\Delta_{\max}$ remains fixed. The authors provide theoretical bounds on the performance gap between policies in random-delay and constant-delay settings, and show that the conservative approach achieves near-identical performance to optimal constant-delay policies under bounded delays. Empirically, conservative-BPQL (and Conservative-VDPO) outperform state-of-the-art baselines on MuJoCo tasks with $\Delta_{\max} \in \{5,10,20\}$, demonstrating superior asymptotic performance and sample efficiency. Overall, the paper offers a practical, distribution-agnostic framework that unifies constant-delay and random-delay RL, with broad applicability to real-world delayed feedback scenarios.

Abstract

Real-world reinforcement learning applications are often hindered by delayed feedback from environments, which violates the Markov assumption and introduces significant challenges. Although numerous delay-compensating methods have been proposed for environments with constant delays, environments with random delays remain largely unexplored due to their inherent variability and unpredictability. In this study, we propose a simple yet robust agent for decision-making under random delays, termed the conservative agent, which reformulates the random-delay environment into its constant-delay equivalent. This transformation enables any state-of-the-art constant-delay method to be directly extended to the random-delay environments without modifying the algorithmic structure or sacrificing performance. We evaluate the conservative agent-based algorithm on continuous control tasks, and empirical results demonstrate that it significantly outperforms existing baseline algorithms in terms of asymptotic performance and sample efficiency.

Figures (8)

• Figure 1: A visual example illustrating the conservative decision-making process under random delays with $\Delta_\text{max}=3$, where the subscripts denote state-generation times and the superscripts indicate the corresponding delays. Despite simultaneous or out-of-order observations, each state is used for decision-making exactly $\Delta_\text{max}$ time steps after its generation.
• Figure 2: Normalized performance of the conservative agent and the normal agent in HalfCheetah-v3 task in MuJoCo benchmark under an upper-truncated Poisson delay distribution with rate parameter $\mu \in \{1, 3, 5, 7, 9\}$ shown on the left. The performance of each agent is averaged over 10 random seeds and normalized to its respective $\mu = 1$ baseline. While the normalized performance of the conservative agent is invariant to $\mu$, that of the normal agent deteriorates as $\mu$ increases.
• Figure 3: Normalized performance of the conservative agents, with (Conservative-BPQL) and without (Conservative-SAC) mitigation of the sample complexity issue, in MuJoCo environments under a uniform delay distribution with $\Delta_\text{max} = 5$. The results are averaged over five random seeds and normalized to the delay-free baseline (delay-free SAC).
• Figure 4: Runtime overheads of each algorithm measured over one million global time steps and averaged over five trials.
• Figure 5: Environments in the MuJoCo benchmark.

Theorems & Definitions (12)

• Definition 1
• Definition 2
• Proposition 3.1
• proof
• Lemma 3.2: Theorem 4.3.1 in DelaysInRL
• Theorem 3.3
• proof
• Theorem 3.4
• proof
• proof : Proof sketch