Embodied Robot Manipulation in the Era of Foundation Models: Planning and Learning Perspectives

TL;DR

This paper addresses the challenge of robotic manipulation in the era of foundation models by proposing a two-tier abstraction: a high-level planner that organizes long-horizon tasks through language, code, motion, affordances, and 3D representations, and a low-level learning-based controller that maps perceptual inputs to actions. It introduces a unified taxonomy separating high-level planning methods (LLM-based, MLLM-based, code generation, motion planning, affordances, and 3D representations) from low-level control paradigms (learning strategies, input modeling, latent learning, and policy learning). The authors summarize state-of-the-art techniques across these axes, highlight synergies between planning and execution, and identify four core research directions: building a true robot brain, addressing data and sim-to-real challenges, advancing multimodal interaction, and ensuring safety in human–robot collaboration. The work clarifies design choices and lays out a roadmap for developing robust, scalable robotic foundation models capable of flexible, safe interaction with complex environments.

Abstract

Recent advances in vision, language, and multimodal learning have substantially accelerated progress in robotic foundation models, with robot manipulation remaining a central and challenging problem. This survey examines robot manipulation from an algorithmic perspective and organizes recent learning-based approaches within a unified abstraction of high-level planning and low-level control. At the high level, we extend the classical notion of task planning to include reasoning over language, code, motion, affordances, and 3D representations, emphasizing their role in structured and long-horizon decision making. At the low level, we propose a training-paradigm-oriented taxonomy for learning-based control, organizing existing methods along input modeling, latent representation learning, and policy learning. Finally, we identify open challenges and prospective research directions related to scalability, data efficiency, multimodal physical interaction, and safety. Together, these analyses aim to clarify the design space of modern foundation models for robotic manipulation.

Figures (7)

• Figure 1: Overview of the survey. We provide an extensive introduction to embodied manipulation, including high-level planner and low-level controller. Our introduction to the low-level controller mainly focuses on the learning-based strategy.
• Figure 2: Taxonomy of high-level planner approaches, organized by main directions (LLM-based and MLLM-based task planning, code generation, and motion planning) and supporting capabilities (affordance learning and 3D Representations).
• Figure 3: Overview of the taxonomy of high-level planners, highlighting six core components: LLM-based task planning, MLLM-based task planning, code generation, motion planning, affordance learning, and 3D scene representations. Figure are adapted from song2023llmmu2023embodiedgptliang2023codehuang2023voxposerjiang2022dittoshen2023distilled.
• Figure 4: Comparison of learning paradigms for low-level robotic manipulation. RL optimizes policies through trial-and-error interaction using reward signals, IL learns direct policy mappings from expert demonstrations, and auxiliary-task learning shapes representations via self-supervised objectives such as world modeling and goal abstraction.
• Figure 5: A taxonomy of VLA models organized by input modality (2D vs. 3D) and methodological orientation (model-oriented architectures vs. model-agnostic strategies), highlighting representative approaches across both architectural design and training or inference enhancements.