Distilling Realizable Students from Unrealizable Teachers

TL;DR

This work tackles policy distillation under privileged information by formalizing a Contextual MDP setting where a fully informed teacher guides a partially observable student. It introduces CritiQ, an imitation-learning method that restricts teacher queries to critical states detected by a discriminator, and ReTRy, a reinforcement-learning approach that gradually expands resets from teacher-visited to student-visited states to improve sample efficiency. The authors analyze limitations of standard DAgger in the presence of state aliasing and show how CritiQ mitigates unrealizability growth, while ReTRy leverages structured resets and the Performance Difference Lemma to tighten the performance gap. Empirical results in simulated and real-world robotics demonstrate that both methods outperform baselines, with ReTRy offering stronger robustness and transfer to the real world, albeit at higher training cost, underscoring practical gains in learning under information asymmetry.

Abstract

We study policy distillation under privileged information, where a student policy with only partial observations must learn from a teacher with full-state access. A key challenge is information asymmetry: the student cannot directly access the teacher's state space, leading to distributional shifts and policy degradation. Existing approaches either modify the teacher to produce realizable but sub-optimal demonstrations or rely on the student to explore missing information independently, both of which are inefficient. Our key insight is that the student should strategically interact with the teacher --querying only when necessary and resetting from recovery states --to stay on a recoverable path within its own observation space. We introduce two methods: (i) an imitation learning approach that adaptively determines when the student should query the teacher for corrections, and (ii) a reinforcement learning approach that selects where to initialize training for efficient exploration. We validate our methods in both simulated and real-world robotic tasks, demonstrating significant improvements over standard teacher-student baselines in training efficiency and final performance. The project website is available at : https://portal-cornell.github.io/CritiQ_ReTRy/

Figures (6)

• Figure 1: DAgger vs CritiQ. The goal is to find the box containing a coin, but the location is hidden from the student among three closed boxes. The student must search one-by-one. In the first iteration, the student selects a trajectory from three demonstrations but cannot proceed further due to insufficient demontration. DAgger queries the expert at random states, leading to a dataset that becomes increasingly unrealizable, causing policy divergence. CritiQ, instead, queries only at critical states, ensuring necessary information is collected while maintaining realizability and improving policy stability.
• Figure 2: ReTRy. The bidirectional interaction between the teacher and student provides a feasible reset distribution, enabling the RL agent to acquire policies more effectively. As iterations progress, the RL student’s exploration space becomes denser, forming a structured search area for policy learning. The dark blue region represents high-probability reset states, while lighter blue areas indicate lower probability, and white regions have zero probability.
• Figure 3: Simulation and real robot tasks. In each of these tasks, the teacher has access to the goal, while the student needs to explore the different possible goals.
• Figure 4: Results on Simulation Tasks We evaluate the success rate on 3 different robot tasks in simulation. We collected 100 episodes per method for all simulation tasks. Goals for each episode were sampled uniformly on a fixed seed.
• Figure 5: Exploration level in the Drawer Search Task over Training Time. If the student fails to explore any drawer, the exploration level is None. Once all three drawers are explored, the exploration level is considered High. The x-axis represents training time (minutes)