Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Power of Input: Benchmarking Zero-Shot Sim-To-Real Transfer of Reinforcement Learning Control Policies for Quadrotor Control

Alberto Dionigi, Gabriele Costante, Giuseppe Loianno

TL;DR

The paper tackles the question of how observation-space design affects zero-shot sim-to-real transfer of DRL-based quadrotor controllers. It trains end-to-end controllers with an asymmetric actor-critic SAC framework across multiple input configurations and validates them on a real quadrotor without fine-tuning, aided by domain randomization. The main contributions are a first systematic benchmark of observation-space configurations, results showing that limited yet informative inputs—particularly world-frame position error with rotation and low-dimensional control signals—can match or exceed richer inputs, and practical guidelines for observing and controlling MAVs in real deployments. The findings have practical impact by informing sensor choices and policy design to achieve robust real-world performance without extensive tuning.

Abstract

In the last decade, data-driven approaches have become popular choices for quadrotor control, thanks to their ability to facilitate the adaptation to unknown or uncertain flight conditions. Among the different data-driven paradigms, Deep Reinforcement Learning (DRL) is currently one of the most explored. However, the design of DRL agents for Micro Aerial Vehicles (MAVs) remains an open challenge. While some works have studied the output configuration of these agents (i.e., what kind of control to compute), there is no general consensus on the type of input data these approaches should employ. Multiple works simply provide the DRL agent with full state information, without questioning if this might be redundant and unnecessarily complicate the learning process, or pose superfluous constraints on the availability of such information in real platforms. In this work, we provide an in-depth benchmark analysis of different configurations of the observation space. We optimize multiple DRL agents in simulated environments with different input choices and study their robustness and their sim-to-real transfer capabilities with zero-shot adaptation. We believe that the outcomes and discussions presented in this work supported by extensive experimental results could be an important milestone in guiding future research on the development of DRL agents for aerial robot tasks.

The Power of Input: Benchmarking Zero-Shot Sim-To-Real Transfer of Reinforcement Learning Control Policies for Quadrotor Control

TL;DR

The paper tackles the question of how observation-space design affects zero-shot sim-to-real transfer of DRL-based quadrotor controllers. It trains end-to-end controllers with an asymmetric actor-critic SAC framework across multiple input configurations and validates them on a real quadrotor without fine-tuning, aided by domain randomization. The main contributions are a first systematic benchmark of observation-space configurations, results showing that limited yet informative inputs—particularly world-frame position error with rotation and low-dimensional control signals—can match or exceed richer inputs, and practical guidelines for observing and controlling MAVs in real deployments. The findings have practical impact by informing sensor choices and policy design to achieve robust real-world performance without extensive tuning.

Abstract

In the last decade, data-driven approaches have become popular choices for quadrotor control, thanks to their ability to facilitate the adaptation to unknown or uncertain flight conditions. Among the different data-driven paradigms, Deep Reinforcement Learning (DRL) is currently one of the most explored. However, the design of DRL agents for Micro Aerial Vehicles (MAVs) remains an open challenge. While some works have studied the output configuration of these agents (i.e., what kind of control to compute), there is no general consensus on the type of input data these approaches should employ. Multiple works simply provide the DRL agent with full state information, without questioning if this might be redundant and unnecessarily complicate the learning process, or pose superfluous constraints on the availability of such information in real platforms. In this work, we provide an in-depth benchmark analysis of different configurations of the observation space. We optimize multiple DRL agents in simulated environments with different input choices and study their robustness and their sim-to-real transfer capabilities with zero-shot adaptation. We believe that the outcomes and discussions presented in this work supported by extensive experimental results could be an important milestone in guiding future research on the development of DRL agents for aerial robot tasks.

Paper Structure

This paper contains 13 sections, 8 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Quadrotor Dynamic Model
  5. Problem Formulation
  6. Deep Reinforcement Learning Approach
  7. Training and Implementation Details
  8. Experimental Results
  9. Experimental Setup and Metrics
  10. Discussion
  11. Velocity Robustness
  12. Ablation Study
  13. Conclusion

Figures (1)

  • Figure 1: Learning curves of the $\{ \mathbf{e}_\mathbf{\mathcal{W}}, \mathbf{R}, \mathbf{u}\}$ variants trained with a different observation length: we report the cumulative reward reached by each agent during the training phase.