When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning?

TL;DR

The paper investigates when to prefer State-to-Visual DAgger over Visual RL for learning policies from visual inputs. It presents a rigorous, standardized comparison across 16 tasks from ManiSkill, DMControl, and Adroit, contrasting a two-stage State-to-Visual DAgger pipeline with a representative Visual RL baseline based on an asymmetric actor‑critic SAC. The key finding is that no single approach dominates all tasks: State-to-Visual DAgger excels on hard, complex tasks with more consistent convergence and yields faster wall-clock training, while Visual RL performs competitively on easier tasks. The study offers practical guidelines for practitioners, highlighting task difficulty and compute considerations as the main factors driving method choice, and contributes a standardized implementation to facilitate future research in visual policy learning.

Abstract

Learning policies from high-dimensional visual inputs, such as pixels and point clouds, is crucial in various applications. Visual reinforcement learning is a promising approach that directly trains policies from visual observations, although it faces challenges in sample efficiency and computational costs. This study conducts an empirical comparison of State-to-Visual DAgger, a two-stage framework that initially trains a state policy before adopting online imitation to learn a visual policy, and Visual RL across a diverse set of tasks. We evaluate both methods across 16 tasks from three benchmarks, focusing on their asymptotic performance, sample efficiency, and computational costs. Surprisingly, our findings reveal that State-to-Visual DAgger does not universally outperform Visual RL but shows significant advantages in challenging tasks, offering more consistent performance. In contrast, its benefits in sample efficiency are less pronounced, although it often reduces the overall wall-clock time required for training. Based on our findings, we provide recommendations for practitioners and hope that our results contribute valuable perspectives for future research in visual policy learning.

Figures (6)

• Figure 1: Overview of Methods. While Visual RL directly trains a visual policy using RL, State-to-Visual DAgger adopts a two-stage process: initially training a teacher policy with low-dimensional state observations, followed by teaching a visual policy via online imitation learning.
• Figure 2: Examples of Tasks. We consider control tasks spanning $\mathbf{3}$ benchmarks. The first row contains tasks from ManiSkill (stationary and mobile robot arm manipulation, dual-arm coordination). The first five tasks in the second row are from DMControl (various robot morphologies for locomotion and classical control tasks), and the remaining three tasks in the second row are from Adroit (dexterous hand manipulation).
• Figure 3: Performance Overview. The figure features histograms comparing average performance across different dimensions. On the left, three histograms present performance by benchmark (success rates for ManiSkill and Adroit, and returns for DMControl). In the center, two histograms categorize performance by task difficulty, utilizing normalized scores (success rate for ManiSkill and Adroit, return divided by 1000 for DMControl) to accommodate the varying metrics across benchmarks. The error bars represent the 95% CI over three seeds.
• Figure 4: Learning curves against environment steps. Success rate (ManiSkill and Adroit) and return (DMControl) in each task. Tasks are categorized as easy if state-based RL converges within 4M steps, while the others are considered hard. State-to-Visual DAgger (Stage 2) comparisons with visual RL should account for the cost of Stage 1. The curve for stage 1 serves as a reference but is not directly comparable to others due to its state-based policy nature. The shaded region represents the 95% CI across three seeds.
• Figure 5: Wall-clock Time. Similar to Fig. \ref{['fig:environment_steps']}, however, we use the wall-clock time as the x-axis instead of the environment steps. We find that State-to-Visual DAgger has better wall-clock time efficiency than visual RL on most tasks.