Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TGRL: An Algorithm for Teacher Guided Reinforcement Learning

Idan Shenfeld, Zhang-Wei Hong, Aviv Tamar, Pulkit Agrawal

TL;DR

TGRL tackles the problem of when to trust a teacher during reinforcement learning by constraining a joint reward+imitation objective with a reward-only auxiliary policy. It uses a dual optimization with a Lagrange multiplier $λ$ to adaptively weight teacher guidance, implemented in an off-policy, two-critic, two-actor architecture to allow efficient learning and reuse of data. The method achieves strong results across diverse domains, including highly observable and severely partially observable tasks, and can surpass sub-optimal teachers without task-specific hyperparameter tuning. This yields a practical framework for leveraging teacher knowledge in challenging RL problems, particularly where teachers are imperfect or have privileged information.

Abstract

Learning from rewards (i.e., reinforcement learning or RL) and learning to imitate a teacher (i.e., teacher-student learning) are two established approaches for solving sequential decision-making problems. To combine the benefits of these different forms of learning, it is common to train a policy to maximize a combination of reinforcement and teacher-student learning objectives. However, without a principled method to balance these objectives, prior work used heuristics and problem-specific hyperparameter searches to balance the two objectives. We present a $\textit{principled}$ approach, along with an approximate implementation for $\textit{dynamically}$ and $\textit{automatically}$ balancing when to follow the teacher and when to use rewards. The main idea is to adjust the importance of teacher supervision by comparing the agent's performance to the counterfactual scenario of the agent learning without teacher supervision and only from rewards. If using teacher supervision improves performance, the importance of teacher supervision is increased and otherwise it is decreased. Our method, $\textit{Teacher Guided Reinforcement Learning}$ (TGRL), outperforms strong baselines across diverse domains without hyper-parameter tuning.

TGRL: An Algorithm for Teacher Guided Reinforcement Learning

TL;DR

TGRL tackles the problem of when to trust a teacher during reinforcement learning by constraining a joint reward+imitation objective with a reward-only auxiliary policy. It uses a dual optimization with a Lagrange multiplier to adaptively weight teacher guidance, implemented in an off-policy, two-critic, two-actor architecture to allow efficient learning and reuse of data. The method achieves strong results across diverse domains, including highly observable and severely partially observable tasks, and can surpass sub-optimal teachers without task-specific hyperparameter tuning. This yields a practical framework for leveraging teacher knowledge in challenging RL problems, particularly where teachers are imperfect or have privileged information.

Abstract

Learning from rewards (i.e., reinforcement learning or RL) and learning to imitate a teacher (i.e., teacher-student learning) are two established approaches for solving sequential decision-making problems. To combine the benefits of these different forms of learning, it is common to train a policy to maximize a combination of reinforcement and teacher-student learning objectives. However, without a principled method to balance these objectives, prior work used heuristics and problem-specific hyperparameter searches to balance the two objectives. We present a approach, along with an approximate implementation for and balancing when to follow the teacher and when to use rewards. The main idea is to adjust the importance of teacher supervision by comparing the agent's performance to the counterfactual scenario of the agent learning without teacher supervision and only from rewards. If using teacher supervision improves performance, the importance of teacher supervision is increased and otherwise it is decreased. Our method, (TGRL), outperforms strong baselines across diverse domains without hyper-parameter tuning.
Paper Structure (18 sections, 6 theorems, 39 equations, 11 figures, 1 table, 1 algorithm)

This paper contains 18 sections, 6 theorems, 39 equations, 11 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Method
  4. Teacher Guided Reinforcement Learning (TGRL)
  5. Implementation
  6. Experiments
  7. TGRL performs well, without a need for hyperparameter tuning
  8. TGRL can solve difficult environments with significant partial observability.
  9. TGRL can surpass the Teacher's peformance
  10. Ablations
  11. Discussion
  12. Derivations and Proofs
  13. Derivation of the Dual Problem
  14. Derivation of update rule for $\lambda$
  15. Duality Gap - Proof for Proposition \ref{['pro:dual_gap']}
  16. ...and 3 more sections

Key Result

Proposition 2.1

In the setting described above, denote $\pi^{TSL} = \mathop{\mathrm{arg\,max}}\limits_\pi {J_I(\pi)}$ and $f(o^T):\Omega_T\rightarrow\Omega$ as the function that maps the teacher's observations to the student's observations. Then, for any $o^T\in \Omega_T$ with $o=f(o^T)$, we have that $\pi^{TSL}(o)

Figures (11)

  • Figure 1: The Tiger Door environment. On the left is the teacher's observation, where the goal cell (in green) and the trap cell (in blue) are perceptible. On the right is the student's observation, where these cells are not visible, but there is a pink button; touching which reveals the other cells.
  • Figure 2: Success rate of a pen reorientation task by Shadow Hand robot, using tactile sensing only. While vanilla reinforcement learning takes a long time to converge, and Teacher-Student methods lead to a major drop in performance compared to the teacher, our algorithm is able to solve the task with reasonable sample efficiency.
  • Figure 3: Comparing TGRL (blue) against algorithms proposed in prior work. TGRL is the only algorithm that performs consistently well across all environments.
  • Figure 4: Adaptively balancing teacher guidance and rewards results in better asymptotic performance compared to fixing the balancing coefficient ($\lambda$). Experiment on the Shadow Hand environment. (Left) Dynamics of $\lambda$ during training: At the start of training the agent relies more on the teacher and gradually the coefficient decreases, indicating more reliance on rewards. (Right) Performance of TGRL (blue) and ablated versions of TGRL using a fixed balancing coefficient ($\lambda$).
  • Figure 5: TGRL performance in the Shadow Hand environment with a sub-optimal teacher. TGRL is able to surpass the teacher and achieves asymptotic performance similar to that of RL.
  • ...and 6 more figures

Theorems & Definitions (9)

  • Proposition 2.1
  • proof
  • Proposition 3.1
  • Proposition 3.2
  • Theorem 1.4
  • Proposition 1.5
  • proof
  • Proposition 1.6
  • proof