VA-learning as a more efficient alternative to Q-learning

TL;DR

This work introduces VA-learning, which directly learns advantage function and value function using bootstrapping, without explicit reference to Q-functions, and improves the sample efficiency over Q-learning both in tabular implementations and deep RL agents on Atari-57 games.

Abstract

In reinforcement learning, the advantage function is critical for policy improvement, but is often extracted from a learned Q-function. A natural question is: Why not learn the advantage function directly? In this work, we introduce VA-learning, which directly learns advantage function and value function using bootstrapping, without explicit reference to Q-functions. VA-learning learns off-policy and enjoys similar theoretical guarantees as Q-learning. Thanks to the direct learning of advantage function and value function, VA-learning improves the sample efficiency over Q-learning both in tabular implementations and deep RL agents on Atari-57 games. We also identify a close connection between VA-learning and the dueling architecture, which partially explains why a simple architectural change to DQN agents tends to improve performance.

Figures (7)

• Figure 1: A simple scenario to illustrate the effectiveness of VA-learning over Q-learning. There are two states $x,y$ and from state $y$ there are two actions $a,b$. Assume there is a back-up target $\widehat{Q}(y,a)$, VA-learning will update the prediction for both $Q(y,a)$ and $Q(y,b)$ thanks to the shared value function $V(y)$. In contrary, Q-learning only updates the prediction $Q(y,a)$. The accelerated learning of $Q(y,b)$ helps accelerate learning $Q(x,\cdot)$ when bootstrapping from $Q(y,\cdot)$.
• Figure 2: Comparing VA-learning (Section \ref{['sec:VA-learning']}), Q-learning with behavior dueling (Section \ref{['sec:b-dueling']}), Q-learning with uniform dueling wang2016sample and regular Q-learning. We experiment on tabular MDPs with fixed behavior policy $\mu=\epsilon u + (1-\epsilon)\pi_\text{det}$ for some randomly sampled and fixed deterministic policy $\pi_\text{det}$, uniform policy $u$ and $\epsilon=0.8$. The performance evaluates the greedy policy with learned Q-function. New algorithmic variants significantly outperform prior methods.
• Figure 3: Comparing different algorithmic variants in tabular MDPs with fixed behavior policy $\mu=\epsilon u + (1-\epsilon)\pi_\text{det}$ for some randomly sampled and fixed deterministic policy $\pi_\text{det}$, uniform policy $u$ and varying degree of $\epsilon$ ($x$-axis). As $\epsilon\rightarrow 1$ and $\mu$ approaches uniform, Q-learning with dueling architecture catches up in performance with behavior dueling and VA-learning.
• Figure 4: Comparing algorithmic variants implemented with full Atari action set. VA-learning and behavior dueling are significantly better than the uniform dueling architecture, which further improves over the $n$-step Q-learning baseline. Compared to the standard Atari setup in Figure \ref{['fig:epsilon-dqn']}(b), the performance of VA-learning and behavior dueling does not degrade.
• Figure 5: (a) Comparing VA-learning (Section \ref{['sec:VA-learning']}) with Q-learning for tabular policy evaluation. We evaluate a target policy $\pi$ formed as a convex combination of a deterministic policy and uniform policy, using a fixed number of trajectories data collected under uniform policy. The $y$-axis shows the approximation error to the advantage $\left\lVert \widehat{A}_t-A^\pi\right\rVert_2$ at each iteration $k$. Given any data budget, VA-learning obtains more accurate approximations to the advantage function compared to Q-learning. (b) The same setup as before. The $y$-axis shows the approximation error to the Q-function $\left\lVert \widehat{Q}_t-Q^\pi\right\rVert_2$ at each iteration $k$. Given any data budget, VA-learning obtains a slightly faster rate of approximating the Q-function compared to Q-learning.

Theorems & Definitions (9)

• Theorem 1
• proof
• Theorem 1
• proof
• Theorem 2
• Lemma 2
• proof
• Lemma 2
• proof