Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey on Robotics with Foundation Models: toward Embodied AI

Zhiyuan Xu, Kun Wu, Junjie Wen, Jinming Li, Ning Liu, Zhengping Che, Jian Tang

TL;DR

This survey addresses the integration of foundation models into embodied AI for autonomous robotic manipulation, focusing on high-level planning and low-level control. It presents a taxonomy of planning forms (structured code and PDDL-based approaches) and planning assistants (vision-grounded and feedback-enabled systems), along with concrete methods and examples. The paper also reviews datasets, simulators, and benchmarks essential for scaling embodied AI, and discusses critical challenges such as planning-control synergy, model hallucination, data efficiency, and safety, offering directions for future research. Together, these insights highlight pathways to scale foundation-model-enabled robotics toward generalized, real-world deployment and underscore the need for cross-domain collaboration.

Abstract

While the exploration for embodied AI has spanned multiple decades, it remains a persistent challenge to endow agents with human-level intelligence, including perception, learning, reasoning, decision-making, control, and generalization capabilities, so that they can perform general-purpose tasks in open, unstructured, and dynamic environments. Recent advances in computer vision, natural language processing, and multi-modality learning have shown that the foundation models have superhuman capabilities for specific tasks. They not only provide a solid cornerstone for integrating basic modules into embodied AI systems but also shed light on how to scale up robot learning from a methodological perspective. This survey aims to provide a comprehensive and up-to-date overview of foundation models in robotics, focusing on autonomous manipulation and encompassing high-level planning and low-level control. Moreover, we showcase their commonly used datasets, simulators, and benchmarks. Importantly, we emphasize the critical challenges intrinsic to this field and delineate potential avenues for future research, contributing to advancing the frontier of academic and industrial discourse.

A Survey on Robotics with Foundation Models: toward Embodied AI

TL;DR

This survey addresses the integration of foundation models into embodied AI for autonomous robotic manipulation, focusing on high-level planning and low-level control. It presents a taxonomy of planning forms (structured code and PDDL-based approaches) and planning assistants (vision-grounded and feedback-enabled systems), along with concrete methods and examples. The paper also reviews datasets, simulators, and benchmarks essential for scaling embodied AI, and discusses critical challenges such as planning-control synergy, model hallucination, data efficiency, and safety, offering directions for future research. Together, these insights highlight pathways to scale foundation-model-enabled robotics toward generalized, real-world deployment and underscore the need for cross-domain collaboration.

Abstract

While the exploration for embodied AI has spanned multiple decades, it remains a persistent challenge to endow agents with human-level intelligence, including perception, learning, reasoning, decision-making, control, and generalization capabilities, so that they can perform general-purpose tasks in open, unstructured, and dynamic environments. Recent advances in computer vision, natural language processing, and multi-modality learning have shown that the foundation models have superhuman capabilities for specific tasks. They not only provide a solid cornerstone for integrating basic modules into embodied AI systems but also shed light on how to scale up robot learning from a methodological perspective. This survey aims to provide a comprehensive and up-to-date overview of foundation models in robotics, focusing on autonomous manipulation and encompassing high-level planning and low-level control. Moreover, we showcase their commonly used datasets, simulators, and benchmarks. Importantly, we emphasize the critical challenges intrinsic to this field and delineate potential avenues for future research, contributing to advancing the frontier of academic and industrial discourse.
Paper Structure (33 sections, 1 figure, 1 table)

This paper contains 33 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Foundation Models for High-Level Planning
  3. Forms of Planning
  4. Policy Code Planning
  5. Natural Language Planning
  6. Visual-Assisted Plan
  7. Plan with Extra Knowledge
  8. Plan with Feedback
  9. Foundation Models for Low-Level Control
  10. Policy Learning
  11. Reinforcement Learning
  12. Imitation Learning
  13. Environment Modeling
  14. Forward Dynamic Learning
  15. Inverse Dynamic Learning
  16. ...and 18 more sections

Figures (1)

  • Figure 1: Taxonomy of foundation models for high-level planning and their properties.