Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays

TL;DR

This work tackles reinforcement learning with constant observation delays by introducing Auxiliary-Delayed Reinforcement Learning (AD-RL), which learns an auxiliary value function for a shorter delay $\Delta^{\tau}$ and bootstraps the main delayed task. The method is instantiated for discrete and continuous control as AD-DQN and AD-SAC, and provides theoretical guarantees on sample efficiency, performance gaps, and convergence. Empirically, AD-RL achieves state-of-the-art performance on deterministic and stochastic benchmarks (e.g., Acrobot and MuJoCo) with notable gains in sample efficiency, while revealing a trade-off between $\Delta^{\tau}$ and robustness under stochastic delays. The work offers practical algorithms, convergence guarantees, and guidance on selecting auxiliary delays to balance learning speed and policy quality.

Abstract

Reinforcement learning (RL) is challenging in the common case of delays between events and their sensory perceptions. State-of-the-art (SOTA) state augmentation techniques either suffer from state space explosion or performance degeneration in stochastic environments. To address these challenges, we present a novel Auxiliary-Delayed Reinforcement Learning (AD-RL) method that leverages auxiliary tasks involving short delays to accelerate RL with long delays, without compromising performance in stochastic environments. Specifically, AD-RL learns a value function for short delays and uses bootstrapping and policy improvement techniques to adjust it for long delays. We theoretically show that this can greatly reduce the sample complexity. On deterministic and stochastic benchmarks, our method significantly outperforms the SOTAs in both sample efficiency and policy performance. Code is available at https://github.com/QingyuanWuNothing/AD-RL.

Figures (6)

• Figure 1: Our AD-RL method introduces an adjoint task with short delays, enhancing the original augmentation-based method (A-QL) in deterministic MDP (Fig. \ref{['fig:de_motivating_example']}) with delay $\Delta=10$, shown in Fig. \ref{['fig:de_aux_delay_impact']}. Whereas, in stochastic MDP (Fig. \ref{['fig:sto_motivating_example']}) with delay $\Delta=10$, a short auxiliary delays may lead to performance improvement (AD-QL(5)) or drop (AD-QL(0)) as shown in Fig. \ref{['fig:sto_aux_delay_impact']}. BPQL always uses a fixed $0$ auxiliary delays, equivalent to AD-QL(0) in these examples. Notably, the optimal auxiliary delays is irregular and task-specific, which we discussed in subsequent experiments in Section \ref{['boad_experiment']}.
• Figure 2: The overview of AD-RL framework. Compared with the conventional augmentation-based method, AD-RL additionally introduces the auxiliary-delayed task shown in the dashline box.
• Figure 3: Results of deterministic Acrobot for (a) learning curves with 10 delays and (b) final performance with varying delays (5-50). The shaded area is the standard deviation. Results of stochastic Acrobot for (c) the normalized return of $\Delta^\tau_{best}$ with different delays (10-50). Different colors stand for different best returns achieved by different optimal auxiliary delays $\Delta^\tau_{best}$.
• Figure 4: Results of MuJoCo tasks with 5 delays for learning curves. The shaded areas represented the standard deviation (10 seeds).
• Figure 5: Results of MuJoCo tasks with 25 delays for learning curves. The shaded areas represented the standard deviation (10 seeds).

Theorems & Definitions (33)

• Definition 3.1: Lipschitz Continuous MDP rl_lipschitz_continuous
• Definition 3.2: Lipschitz Continuous Policy rl_lipschitz_continuous
• Definition 3.3: Lipschitz Continuous Q-function rl_lipschitz_continuous
• Remark 4.1: Implicit Delayed Belief
• Remark 4.2
• Remark 5.1: Sample Inefficiency Issue
• Lemma 5.2: General Delayed Performance Difference, see Appendix \ref{['appendix_general_delayed_performance_diff']} for proof
• Theorem 5.3: Delayed Performance Difference Bound, proof in Appendix \ref{['appendix_delayed_performance_difference_bound']}
• Theorem 5.4: Delayed Q-value Difference Bound, proof in Appendix \ref{['appendix_delayed_q_value_difference_bound']}
• Remark 5.5: Deterministic MDP Case