REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?

TL;DR

This work investigates how coreferential vagueness in human instructions affects LLM-based robot task planning. It introduces REI-Bench, a benchmark that simulates nine levels of referring-expression vagueness under three dialogue-context memory conditions, enabling systematic evaluation of planners in embodied settings. Experimental results show large performance drops when implicit REs appear, primarily due to object omissions, across multiple planners and LLMs. To address this, the authors propose Task-Oriented Context Cognition (TOCC), which resolves implicit REs and reformulates instructions for planning, outperforming aware prompting and chain-of-thought baselines and improving robustness for non-expert users like the elderly and children.

Abstract

Robot task planning decomposes human instructions into executable action sequences that enable robots to complete a series of complex tasks. Although recent large language model (LLM)-based task planners achieve amazing performance, they assume that human instructions are clear and straightforward. However, real-world users are not experts, and their instructions to robots often contain significant vagueness. Linguists suggest that such vagueness frequently arises from referring expressions (REs), whose meanings depend heavily on dialogue context and environment. This vagueness is even more prevalent among the elderly and children, who robots should serve more. This paper studies how such vagueness in REs within human instructions affects LLM-based robot task planning and how to overcome this issue. To this end, we propose the first robot task planning benchmark with vague REs (REI-Bench), where we discover that the vagueness of REs can severely degrade robot planning performance, leading to success rate drops of up to 77.9%. We also observe that most failure cases stem from missing objects in planners. To mitigate the REs issue, we propose a simple yet effective approach: task-oriented context cognition, which generates clear instructions for robots, achieving state-of-the-art performance compared to aware prompt and chains of thought. This work contributes to the research community of human-robot interaction (HRI) by making robot task planning more practical, particularly for non-expert users, e.g., the elderly and children.

Figures (13)

• Figure 1: Left: Robots using existing LLM-based task planners can understand clear instructions with explicit referring expressions (REs), but they struggle to resolve implicit REs in multi-turn dialogues. Right: We propose the REI-Bench framework that aims to study real-world HRI scenarios where coreferential vagueness exists in human instructions.
• Figure 2: Data curation pipeline of the REI dataset. From a seed instruction, we (1) generate context memory; (2) produce three context variants—Standard, Noised, Short; (3) replace explicit REs with implicit ones across varying degrees. This results in subsets reflecting nine levels of coreferential vagueness, determined by RE types (Explicit/Mixed/Implicit) and context variants.
• Figure 3: Addressing implicit referring expressions in task planning. Top row: LLM succeeds with explicit REs (“potato”), but misidentifies the object with implicit REs (“the heated one”). Middle row: a reflection prompt from humans can guide the LLM to resolve the implicit REs and identify the correct object. Bottom row: Comparison among different prompting methods, including aware prompt (AP), chain-of-thought (CoT), and our task-oriented context cognition (TOCC).
• Figure 4: Success rate (%) of two task planner frameworks (SayCan and LLM+P using three LLMs (GPT-4o-mini, LLaMA3.1-8B, and DeepSeekMath-7B) on REI dataset. Explicit, Mixed, and Implicit REs denote three levels of implicit REs in human instructions, and Standard, Noised, and Short Contexts represent three context memory types.
• Figure 5: Success rates (%) of various prompting methods applied to LLaMA 3.1-8B and Gemma 2-9B models with SayCan framework on REI dataset.