Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes

TL;DR

This survey analyzes real-world successes of DRL in robotics, arguing that maturity varies by task and domain. It introduces a four-axis taxonomy (competencies, problem formulation, solution approach, real-world deployment level) to systematically compare papers, with emphasis on real-world validation over simulations. The review highlights mature quadruped locomotion and advancing navigation, while identifying open challenges in biped locomotion, aerial control, manipulation, HRI, and multi-robot systems. It concludes with actionable directions for stable, sample-efficient real-world RL and the potential of foundation models to accelerate deployment in real-world robotics.

Abstract

Reinforcement learning (RL), particularly its combination with deep neural networks referred to as deep RL (DRL), has shown tremendous promise across a wide range of applications, suggesting its potential for enabling the development of sophisticated robotic behaviors. Robotics problems, however, pose fundamental difficulties for the application of RL, stemming from the complexity and cost of interacting with the physical world. This article provides a modern survey of DRL for robotics, with a particular focus on evaluating the real-world successes achieved with DRL in realizing several key robotic competencies. Our analysis aims to identify the key factors underlying those exciting successes, reveal underexplored areas, and provide an overall characterization of the status of DRL in robotics. We highlight several important avenues for future work, emphasizing the need for stable and sample-efficient real-world RL paradigms, holistic approaches for discovering and integrating various competencies to tackle complex long-horizon, open-world tasks, and principled development and evaluation procedures. This survey is designed to offer insights for both RL practitioners and roboticists toward harnessing RL's power to create generally capable real-world robotic systems.

Figures (7)

• Figure 1: The four aspects of our taxonomy: (a) Robot competencies learned with DRL; (b) Problem formulation; (c) Solution approach; and (d) Levels of real-world success.
• Figure 2: Left: An overview of the three locomotion problems reviewed in Sec. \ref{['sec:locomotion']}, including quadruped cheng2023parkour and biped li_reinforcement_2024 locomotion, and quadrotor flight control hwangbo2017controlzhang2023hover; Right: Locomotion papers reviewed in Sec. \ref{['sec:locomotion']}. The color map indicates the levels of real-world success: Limited Lab, Diverse Lab, Limited Real, and Diverse Real.
• Figure 3: Left: An overview of the three navigation problems reviewed in Sec. \ref{['sec:navigation']}, including wheeled navigation xu_benchmarking_2023kahn_badgr_2021kendall2019learning, legged navigation lee2024learning, and aerial navigation song2023reaching; Right: Navigation papers reviewed in Sec. \ref{['sec:navigation']}. The color map indicates the levels of real-world success: Limited Lab, Diverse Lab, Limited Real, and Diverse Real.
• Figure 4: Top: An overview of the four manipulation problems reviewed in Sec. \ref{['sec:manipulation']}, including pick-and-place mahler2019learning, contact-rich manipulation tang2023industreal, in-hand manipulation qi2023general, and non-prehensile manipulation zhou2023learning; Bottom: Manipulation papers reviewed in Sec. \ref{['sec:manipulation']}. The color map indicates the levels of real-world success: Limited Lab, Diverse Lab, Limited Real, and Diverse Real.
• Figure 5: Top: An overview of the three MoMa challenges discussed in Sec. \ref{['sec:moma']}, including whole-body control wang2020learningfu2023deep (WBC) and short- jauhri2022robotcheng2023legs and long-horizon yokoyama2023adaptivewu2023m interactive tasks; Bottom: MoMa papers reviewed in Sec. \ref{['sec:moma']}.Color map indicates levels of real-world success: Limited Lab, Diverse Lab, Limited Real, and Diverse Real.