Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interaction

TL;DR

This survey analyzes how large-language and vision-language foundation models are increasingly embedded into robotic systems as intelligent intermediaries rather than end-to-end controllers. It introduces a two-axis taxonomy—model integration patterns (protocol, interface, orchestration, embedded) and agent roles (planner, orchestrator, task-specific, model-centric, generalist, systemic)—and surveys both academic and community-driven work to map current design patterns. The authors discuss practical toolkits and frameworks (MCP, LangChain, LangGraph, LlamaIndex, OpenMind OM1, ROS-related frameworks) that enable modular, memory-enabled, and safety-conscious agentic robotics, while highlighting open questions in embodiment, memory, edge-cloud deployment, and evaluation. The work emphasizes that agentic AI can augment, rather than replace, traditional robotic stacks, offering scalable, adaptable, and interoperable pathways toward embodied autonomy with real-world impact across industry and research. It also identifies critical gaps in memory, continuous data handling, and ethical governance, urging further development of robust evaluation methods and grounded, trustworthy deployments.

Abstract

Foundation models, including large language models (LLMs) and vision-language models (VLMs), have recently enabled novel approaches to robot autonomy and human-robot interfaces. In parallel, vision-language-action models (VLAs) or large behavior models (LBMs) are increasing the dexterity and capabilities of robotic systems. This survey paper reviews works that advance agentic applications and architectures, including initial efforts with GPT-style interfaces and more complex systems where AI agents function as coordinators, planners, perception actors, or generalist interfaces. Such agentic architectures allow robots to reason over natural language instructions, invoke APIs, plan task sequences, or assist in operations and diagnostics. In addition to peer-reviewed research, due to the fast-evolving nature of the field, we highlight and include community-driven projects, ROS packages, and industrial frameworks that show emerging trends. We propose a taxonomy for classifying model integration approaches and present a comparative analysis of the role that agents play in different solutions in today's literature.

Figures (3)

• Figure 1: Illustration of the main approaches to integrating large language models (LLMs), vision-language models (VLMs), and related foundation models into robotic systems. This survey focuses on the integration perspective and the role of such models within the wider robotic system where it is deployed. Our goal is to discuss The current state-of-the-art within the context of Embodied Agentic AI applications. The figure highlights how model integration differs in terms of agent architecture, interaction topology, and the resulting level of agency. In Protocol-Focused Integration, an LLM serves as a translator between user commands and predefined APIs or protocols, typically through single-shot tool calls. Interface or Agentic Integration introduces ReAct-style loops and conversational interaction, allowing the agent to reason, call tools, and respond iteratively within the physical or simulated environment. Orchestration-Oriented Integration elevates the model to a supervisory role, coordinating multiple agents, skills, or robotic subsystems through structured workflows. Finally, Direct or Embedded Integration corresponds to end-to-end or model-centric policies (e.g., VLA or LBM architectures) that directly map perception to action in continuous control loops.
• Figure 2: Progression of projects at the intersection of Generative AI and Robotics with a focus on LLMs, VLMs and VLAs.
• Figure 3: Classification of existing works by primary role and functionality.