Online Policy Distillation with Decision-Attention

TL;DR

This paper tackles the lack of a well-trained teacher in policy distillation by proposing Online Policy Distillation with Decision-Attention (OPD-DA), which enables multiple policies to learn collaboratively in the same environment. It combines RL signals with decision-based and feature-based supervision and introduces a cross-attention based Decision-Attention module to generate per-policy aggregation weights, mitigating homogenization. Empirical results on Atari show that OPD-DA improves performance for both PPO and DQN across several tasks, with ablations confirming the importance of decision loss, feature loss, and the attention mechanism. The approach provides a practical, teacher-free framework for online policy transfer, enhancing sample efficiency and reward accumulation in challenging environments.

Abstract

Policy Distillation (PD) has become an effective method to improve deep reinforcement learning tasks. The core idea of PD is to distill policy knowledge from a teacher agent to a student agent. However, the teacher-student framework requires a well-trained teacher model which is computationally expensive.In the light of online knowledge distillation, we study the knowledge transfer between different policies that can learn diverse knowledge from the same environment.In this work, we propose Online Policy Distillation (OPD) with Decision-Attention (DA), an online learning framework in which different policies operate in the same environment to learn different perspectives of the environment and transfer knowledge to each other to obtain better performance together. With the absence of a well-performance teacher policy, the group-derived targets play a key role in transferring group knowledge to each student policy. However, naive aggregation functions tend to cause student policies quickly homogenize. To address the challenge, we introduce the Decision-Attention module to the online policies distillation framework. The Decision-Attention module can generate a distinct set of weights for each policy to measure the importance of group members. We use the Atari platform for experiments with various reinforcement learning algorithms, including PPO and DQN. In different tasks, our method can perform better than an independent training policy on both PPO and DQN algorithms. This suggests that our OPD-DA can transfer knowledge between different policies well and help agents obtain more rewards.

Figures (9)

• Figure 1: Overview of our method. During policy distillation, there exist three policies that include feature extractor and linear layer. Our method uses attention values to get decision loss and feature loss. These policies are trained using RL loss, decision loss, and feature loss.
• Figure 2: Results of BeamRiderNoFrameskip on PPO.
• Figure 3: Results of BreakoutNoFrameskip on PPO.
• Figure 4: Results of SpaceInvaderNoFrameskip on DQN.
• Figure 5: Results of BreakoutNoFrameskip on DQN.