Consolidated Adaptive T-soft Update for Deep Reinforcement Learning

TL;DR

This study develops a consolidated and adaptive T-soft (CAT-soft) update based on approximate maximum likelihood estimation of student-t distribution and an additional consolidation that outperformed the conventional methods in numerical simulations.

Abstract

Demand for deep reinforcement learning (DRL) is gradually increased to enable robots to perform complex tasks, while DRL is known to be unstable. As a technique to stabilize its learning, a target network that slowly and asymptotically matches a main network is widely employed to generate stable pseudo-supervised signals. Recently, T-soft update has been proposed as a noise-robust update rule for the target network and has contributed to improving the DRL performance. However, the noise robustness of T-soft update is specified by a hyperparameter, which should be tuned for each task, and is deteriorated by a simplified implementation. This study develops adaptive T-soft (AT-soft) update by utilizing the update rule in AdaTerm, which has been developed recently. In addition, the concern that the target network does not asymptotically match the main network is mitigated by a new consolidation for bringing the main network back to the target network. This so-called consolidated AT-soft (CAT-soft) update is verified through numerical simulations.

Figures (3)

• Figure 1: Rough sketches of the proposed consolidation: when $\theta_i$ is not far from $\mathcal{T}$ as in (a), almost no consolidation works; when parts of $\theta_i$ are far from $\mathcal{T}$ as in (b), only $\theta_i^c$, which leads to $w_1 \simeq 0$, is consolidated to the target network.
• Figure 2: Learning curves for benchmarks: the upper row shows mean of the deviation between the main and target networks, $|\theta - \bar{\theta}|$; the lower row plots mean of $1 - w_1 / \bar{w}_1$, which corresponds to the noise robustness (or the magnitude of consolidation); compared to T-soft update, AT-soft and CAT-soft updates have the larger deviation, but this was suppressed by the consolidation in the early stages of learning; as learning progresses, the cases, where updates of the target network were suppressed due to noise, were decreased, and the consolidation was relaxed, resulting in AT-soft and CAT-soft updates converging to roughly the same degree of deviation.
• Figure 3: Demonstration results: CAT-soft update showed the more stable learning curve than that of T-soft update and reached the success level of the task in the final performance.