HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Models

TL;DR

HEAL systematically investigates hallucinations in LLM-driven embodied agents executing long-horizon tasks under scene–task inconsistencies. By constructing a targeted hallucination probing set with four prompting-modification types, the authors quantify grounding failures across 12 models in VirtualHome and BEHAVIOR, using CHAIR and POPE metrics and three runs per setting. Key findings show Scene Task Contradiction yields the highest hallucination rates, cross-modal inputs mitigate but do not eliminate hallucinations, and absence of hallucination does not guarantee correct planning, with larger models showing more resilience. The work provides actionable guidance for designing robust grounded planning, highlighting the need for stronger grounding controls and reliable task rejection mechanisms in embodied AI systems.

Abstract

Large language models (LLMs) are increasingly being adopted as the cognitive core of embodied agents. However, inherited hallucinations, which stem from failures to ground user instructions in the observed physical environment, can lead to navigation errors, such as searching for a refrigerator that does not exist. In this paper, we present the first systematic study of hallucinations in LLM-based embodied agents performing long-horizon tasks under scene-task inconsistencies. Our goal is to understand to what extent hallucinations occur, what types of inconsistencies trigger them, and how current models respond. To achieve these goals, we construct a hallucination probing set by building on an existing benchmark, capable of inducing hallucination rates up to 40x higher than base prompts. Evaluating 12 models across two simulation environments, we find that while models exhibit reasoning, they fail to resolve scene-task inconsistencies-highlighting fundamental limitations in handling infeasible tasks. We also provide actionable insights on ideal model behavior for each scenario, offering guidance for developing more robust and reliable planning strategies.

Figures (5)

• Figure 1: Object hallucination rates ($C_O$) on our hallucination probing set in VirtualHome. Higher values indicate more hallucination, with Scene Task Contradiction triggering the highest rates in nearly all models.
• Figure 2: Overview of our settings. (A) The base pipeline in the existing benchmark li2024embodied begins with a scene parser that extracts structured textual scene information from raw visual input. Combined with the natural language task description, this is processed by an LLM to generate symbolic goals in Linear Temporal Logic (LTL) (see \ref{['subsec:prelim']}). (B) Examples of hallucinations. Output elements highlighted in red indicate hallucinated content that is not grounded in the scene information, i.e., inconsistent with the observed environment. These examples demonstrate that when inconsistencies arise between the scene information and the given task description, the LLM fails to reconcile the two and generates incorrect plans or object references. Given the base prompt, we systematically modify two core input components---the task description and scene information---to elicit hallucinations. Our four controlled modifications of the base prompts are: under task description variation, (i) Distractor Injection---adds non-existent objects to the task description; and under scene variation, (ii) Task Relevant Object Removal---omits key objects from the scene; (iii) Synonymous Object Substitution---replaces scene objects with synonyms; and (iv) Scene Task Contradiction—introduces conflicts between the task and scene.
• Figure 3: Example of representative base prompts from VirtualHome and BEHAVIOR environments.
• Figure 4: Prompt to get distractor injection based task description.
• Figure 5: Prompt to get synonyms for scene objects.