Reinforcement Teaching

TL;DR

Reinforcement Teaching presents a unifying framework that treats meta-learning as a reinforcement learning problem where a teaching policy adapts any student’s learning process. It introduces a parametric-behavior embedder to produce a compact, problem-agnostic state representation from a student’s inputs and outputs, enabling scalable policy learning for non-differentiable or deep learners. The authors further propose learning-progress–based reward shaping to improve credit assignment and accelerate policy learning. Through curriculum learning for RL agents and step-size adaptation for supervised learners, the approach demonstrates superior or comparable performance to baselines and shows transferability across architectures and datasets, highlighting its generality and practical impact for accelerating learning across domains.

Abstract

Machine learning algorithms learn to solve a task, but are unable to improve their ability to learn. Meta-learning methods learn about machine learning algorithms and improve them so that they learn more quickly. However, existing meta-learning methods are either hand-crafted to improve one specific component of an algorithm or only work with differentiable algorithms. We develop a unifying meta-learning framework, called Reinforcement Teaching, to improve the learning process of \emph{any} algorithm. Under Reinforcement Teaching, a teaching policy is learned, through reinforcement, to improve a student's learning algorithm. To learn an effective teaching policy, we introduce the parametric-behavior embedder that learns a representation of the student's learnable parameters from its input/output behavior. We further use learning progress to shape the teacher's reward, allowing it to more quickly maximize the student's performance. To demonstrate the generality of Reinforcement Teaching, we conduct experiments in which a teacher learns to significantly improve both reinforcement and supervised learning algorithms. Reinforcement Teaching outperforms previous work using heuristic reward functions and state representations, as well as other parameter representations.

Figures (27)

• Figure 1: The teacher takes actions $a \in \mathcal{A}$, which will influence an aspect of the teaching MDP, such as the student, $f_{\theta}$, learning algorithm, $\mathcal{A}lg$, or learning domain $\mathcal{D}$. The student will then update its parameters, $\theta$, and the teaching MDP will then output $r, s'$ based on the student's new parameters.
• Figure 2: The neural network architecture used for Reinforcement Teaching with the parametric-behavior embedding state representation. For a given student, $f_\theta$, the parametric-behavior embedder independently projects a mini-batch of student inputs, $\{x_i\}_{i = 1}^M$, and student outputs, $\{f_\theta(x_i)\}_{i=1}^M$, into a latent space before concatenation and pooling, providing a state representation of $\theta$.
• Figure 3: The beginning (left), middle (center), and ending (right) stages of the curriculum generated by the PE-Actions + LP method for the Maze environment. States outlined in white indicate possible teacher actions. The state outlined in blue indicates the target start state and the green state indicates the target goal state. Brighter colors (more yellow/white) indicate the start state was chosen more frequently by the teacher. Darker red/black indicates the start state was chosen less frequently by the teacher.
• Figure 4: The left plots are learning curves for the teacher. The y-axis is the number of episodes needed for the student to reach the performance threshold, $m^{*}$, with the teacher's current policy, as the teacher learns over episodes on the x-axis (lower is better). The right plots are the student's training curves while using the trained teacher's curriculum policy (higher is better).
• Figure 5: State ablation experiments. The left plots are learning curves for the teacher. The y-axis is the number of gradient steps needed for the student to reach the performance threshold with the teacher's current policy, as the teacher learns over episodes on the x-axis (lower is better). The right plots are the student's training curves while using the trained teacher's step-size policy (higher is better). Top: student's base optimizer is SGD. Bottom: student's base optimizer is Adam, classification task is harder.