Generative AI in Embodied Systems: System-Level Analysis of Performance, Efficiency and Scalability

TL;DR

This work provides the first system-level analysis of generative AI in embodied systems, uncovering that long-horizon tasks suffer from substantial latency primarily due to LLM-based planning and communication, memory retrieval, and execution overhead. It introduces a comprehensive workload suite spanning single- and multi-agent paradigms (centralized and decentralized) to benchmark performance and efficiency across configurations. The study reveals critical bottlenecks such as token-length growth, memory inconsistencies at scale, and exponential coordination costs, and offers concrete optimization directions including planning-guided multi-step execution, planning-then-communication, hierarchical coordination, and a dual-memory architecture. Together, these insights and recommendations lay the groundwork for more robust, scalable, and efficient embodied AI systems capable of real-world long-horizon operation.

Abstract

Embodied systems, where generative autonomous agents engage with the physical world through integrated perception, cognition, action, and advanced reasoning powered by large language models (LLMs), hold immense potential for addressing complex, long-horizon, multi-objective tasks in real-world environments. However, deploying these systems remains challenging due to prolonged runtime latency, limited scalability, and heightened sensitivity, leading to significant system inefficiencies. In this paper, we aim to understand the workload characteristics of embodied agent systems and explore optimization solutions. We systematically categorize these systems into four paradigms and conduct benchmarking studies to evaluate their task performance and system efficiency across various modules, agent scales, and embodied tasks. Our benchmarking studies uncover critical challenges, such as prolonged planning and communication latency, redundant agent interactions, complex low-level control mechanisms, memory inconsistencies, exploding prompt lengths, sensitivity to self-correction and execution, sharp declines in success rates, and reduced collaboration efficiency as agent numbers increase. Leveraging these profiling insights, we suggest system optimization strategies to improve the performance, efficiency, and scalability of embodied agents across different paradigms. This paper presents the first system-level analysis of embodied AI agents, and explores opportunities for advancing future embodied system design.

Figures (8)

• Figure 1: Embodied AI Agents Paradigm.(a) The embodied AI agents typically consist of six key modules, where sensing module perceives the environment, planning module makes high-level plans, communication module generates messages, memory module stores the agent’s action, dialogue and world knowledge, execution module generates primitive actions, and reflection module reflects actions. The single-agent embodied AI systems can be built using the (b) modularized-based paradigm or (c) end-to-end pre-trained vision-language-action model. The multi-agent embodied AI systems can be built in the (d) centralized paradigm or (e) decentralized paradigm.
• Figure 2: Embodied Agent Systems Workload Suite. Above are the workflow examples of four embodied AI agent systems. Below are the details of the benchmarked embodied agent systems, including the models used for each key building block (perception, planning, communication, memory, reflection, and execution), applications, datasets and tasks, and the extent of single or multi-agent collaboration.
• Figure 2: Runtime Latency Analysis.(a) Average runtime percentage contributed by each module per time step, and (b) Total runtime latency per task, benchmarked across 14 embodied AI workloads.
• Figure 3: Module Sensitivity Analysis. Average success rate and the number of steps taken to complete the long-horizon tasks across six single-agent and multi-agent embodied AI systems.
• Figure 4: Local Model Analysis. Task success rate and end-to-end task runtime under GPT-4 API call and Llama-3-8B local processing.