Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making

TL;DR

This paper tackles the fragmentation in evaluating LLMs for embodied decision making by introducing the Embodied Agent Interface (EAI), a standardized, object-centric, LTL-based framework that couples four fundamental ability modules: goal interpretation, subgoal decomposition, action sequencing, and transition modeling. The authors implement EAI on BEHAVIOR and VirtualHome, annotate extensive datasets, and benchmark 18 LLMs to dissect where grounding, planning, and dynamics prediction fail. Fine-grained metrics reveal that grounding NL goals to grounded states and ensuring trajectory feasibility are key bottlenecks, while modular evaluation and replanning can yield meaningful gains. The work provides a principled, extensible pathway for improving and selectively deploying LLMs in embodied AI tasks, with clear directions toward multimodal grounding, memory integration, and safer deployment.

Abstract

We aim to evaluate Large Language Models (LLMs) for embodied decision making. While a significant body of work has been leveraging LLMs for decision making in embodied environments, we still lack a systematic understanding of their performance because they are usually applied in different domains, for different purposes, and built based on different inputs and outputs. Furthermore, existing evaluations tend to rely solely on a final success rate, making it difficult to pinpoint what ability is missing in LLMs and where the problem lies, which in turn blocks embodied agents from leveraging LLMs effectively and selectively. To address these limitations, we propose a generalized interface (Embodied Agent Interface) that supports the formalization of various types of tasks and input-output specifications of LLM-based modules. Specifically, it allows us to unify 1) a broad set of embodied decision-making tasks involving both state and temporally extended goals, 2) four commonly-used LLM-based modules for decision making: goal interpretation, subgoal decomposition, action sequencing, and transition modeling, and 3) a collection of fine-grained metrics which break down evaluation into various types of errors, such as hallucination errors, affordance errors, various types of planning errors, etc. Overall, our benchmark offers a comprehensive assessment of LLMs' performance for different subtasks, pinpointing the strengths and weaknesses in LLM-powered embodied AI systems, and providing insights for effective and selective use of LLMs in embodied decision making.

Figures (64)

• Figure 1: Embodied Agent Interface unifies a broad set of tasks involving both state and temporally extended goals and four LLM-based modules for decision-making.
• Figure 2: The input and output formulation of four ability modules.
• Figure 3: Embodied Agent Interface supports a collection of fine-grained metrics and provides automatic toolkits for error analysis and benchmarking different LLMs on various embodied decision-making tasks.
• Figure 4: The overview of evaluation pipeline for four abilities. For each ability module, to provide a comprehensive evaluation for it, we isolate this single module to be handled by the LLMs while using existing data or tools for the other modules. Note that the pipeline consists of goal interpretation, action sequencing to achieve the goal, and transition modeling that predicts how each action operate the environment's state. Evaluating subgoal decomposition presents a challenge since it cannot be evaluated directly with no unified annotation strategy. To address this, we employ breadth-first search (BFS) to identify potential action sequences that accomplish each subgoal, allowing us to convert state trajectories into action sequences that can be executed in the simulator (Figure \ref{['fig:subgoal_eval_pipeline']} in Appendix). Transition modeling evaluation poses another challenge, we first annotate transition models in PDDL for $F_1$ evaluation followed with a PDDL planner to validate the feasibility of supporting potential plans. We also conduct a pipeline-based vs modularized analysis, detailed in the Appendix \ref{['sec_app:pipeline']}.
• Figure 5: Examples of different types of errors in trajectory feasibility, logic form parsing (e.g., in subgoals decomposition and transition modeling), and goal satisfaction rates.