A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning

TL;DR

A unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually biased, and conducts experiments on Iterated Prisoner's Dilemma and Atari games to show how other methods such as off-policy learning and low-bias estimator can help fix the gradient bias forGMRL algorithms in general.

Abstract

Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. In this paper, we develop a unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually \textbf{biased}. Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by the two-level problem structure, which has an upper bound of $\mathcal{O}\big(Kα^{K}\hatσ_{\text{In}}|τ|^{-0.5}\big)$ \emph{w.r.t.} inner-loop update step $K$, learning rate $α$, estimate variance $\hatσ^{2}_{\text{In}}$ and sample size $|τ|$, and 2) the multi-step Hessian estimation bias $\hatΔ_{H}$ due to the use of autodiff, which has a polynomial impact $\mathcal{O}\big((K-1)(\hatΔ_{H})^{K-1}\big)$ on the meta-gradient bias. We study tabular MDPs empirically and offer quantitative evidence that testifies our theoretical findings on existing stochastic meta-gradient estimators. Furthermore, we conduct experiments on Iterated Prisoner's Dilemma and Atari games to show how other methods such as off-policy learning and low-bias estimator can help fix the gradient bias for GMRL algorithms in general.

Figures (11)

• Figure 1: (a, b) Ablation study on sample size and estimators in MAML-RL. "S" is for stochastic estimation while "E" is for exact solution. AD refers to automatic differentiation. (c,d) Ablation study on sample size, steps and estimators in LIRPG.
• Figure 2: (a, b, c) Ablation study of meta-gradient bias due to the compositional bias in different estimators, step sizes, learning rates. Loaded-DiCE, LVC and AD achieve exactly the same compositional bias because they have the same first-order gradient, (d) Ablation study of meta-gradient bias due to the Hessian bias in different learning rates and Hessian bias coefficients.
• Figure 3: Experiment result of LOLA-DiCE over 10 seeds. The Inner$\_A\_$Outer$\_B$ legend means we use $A$ samples to estimate inner-loop gradient while $B$ samples to estimate outer-loop gradient. The 'exact' means we use analytical solution of policy gradient instead of estimation.
• Figure 4: Experimental results on Atari game over 5 random seeds.
• Figure 5: Ablation study on sample size and estimator in 1-step inner-loop setting. (1) Outer-loop policy gradient is important for estimation (2) Compositional bias correction helps increase the correlation (3) The LVC and Loaded-DiCE can achieve higher correlation compared with AD when the Hessian matrix is estimated.

Theorems & Definitions (24)

• Proposition 3.0: $K$-step Meta-Gradient
• Lemma 4.3: Compositional Bias
• proof
• Theorem 4.4: Upper bound for the bias and the variance
• proof
• Proposition E.0: $K$-step Meta-Gradient
• proof
• Lemma F.0: Compositional Bias
• proof
• Theorem G.1: Upper bound for the bias and the variance