Learning Quadrotor Control From Visual Features Using Differentiable Simulation

TL;DR

The paper tackles the problem of sample-inefficient reinforcement learning for robotics, especially vision-based control, by leveraging differentiable simulation to backpropagate through the dynamics. It combines a fast surrogate backward model with optional state-representation pretraining to achieve rapid, low-variance policy gradients for a quadrotor, trained in simulation and transferred to real hardware via hardware-in-the-loop testing. Key contributions include (1) demonstrating substantially faster learning than model-free PPO for both state- and vision-based control, (2) showing that surrogate gradients can accelerate training without hurting performance, and (3) showing that pretraining on state representations improves convergence and final performance, with real-world stabilization from a manual throw demonstrated. This work suggests differentiable simulation can provide a practical, scalable alternative to traditional RL in robotics, enabling faster development of robust, vision-based flight controllers.

Abstract

The sample inefficiency of reinforcement learning (RL) remains a significant challenge in robotics. RL requires large-scale simulation and can still cause long training times, slowing research and innovation. This issue is particularly pronounced in vision-based control tasks where reliable state estimates are not accessible. Differentiable simulation offers an alternative by enabling gradient back-propagation through the dynamics model, providing low-variance analytical policy gradients and, hence, higher sample efficiency. However, its usage for real-world robotic tasks has yet been limited. This work demonstrates the great potential of differentiable simulation for learning quadrotor control. We show that training in differentiable simulation significantly outperforms model-free RL in terms of both sample efficiency and training time, allowing a policy to learn to recover a quadrotor in seconds when providing vehicle states and in minutes when relying solely on visual features. The key to our success is two-fold. First, the use of a simple surrogate model for gradient computation greatly accelerates training without sacrificing control performance. Second, combining state representation learning with policy learning enhances convergence speed in tasks where only visual features are observable. These findings highlight the potential of differentiable simulation for real-world robotics and offer a compelling alternative to conventional RL approaches.

Figures (9)

• Figure 1: Learning quadrotor control from visual features.Top: Stabilizing a quadrotor from a hand throw. Bottom Left: Backpropagation through time (BPTT) via differentiable simulation outperforms PPO, enabling state-based control in seconds and vision-based control in minutes. Bottom Right: A visualization of observed visual features using hardware-in-the-loop simulation.
• Figure 2: Overview of policy training using differentiable simulation. A neural network policy takes actions based on visual feature observations. The simulation state is then updated using differentiable quadrotor dynamics. Based on the next state, new observations are computed with a differentiable camera model. For each state and action pair, the actor receives a reward. The differentiability of the whole pipeline allows back-propagating the gradients from the rewards to the policy parameters.
• Figure 3: Vision-based control. The actor observers only a history of pixel coordinates of seven points on the ground and the last three actions taken.
• Figure 4: Comparisons of learning state-based control using PPO and BPTT.
• Figure 5: Comparisons of learning feature-based control using PPO and BPTT.