Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Asking Before Acting: Gather Information in Embodied Decision Making with Language Models

Xiaoyu Chen, Shenao Zhang, Pushi Zhang, Li Zhao, Jianyu Chen

TL;DR

The work tackles the problem that LLM-based embodied agents struggle to gather sufficient contextual information in unfamiliar environments. It introduces Asking Before Acting (ABA), a framework that interleaves natural-language information queries with actions within a Contextual MDP, and extends it with ABA-FT for improved data efficiency and long-horizon performance. By formalizing the augmentation of state/action spaces and introducing a human/external information source in the loop, the approach achieves notable performance and efficiency gains across text-based tasks, robot manipulation, and real-world image-guided tasks, with consistent improvements over baselines like ReAct and BUTLER. The results suggest that proactive information gathering via natural-language queries substantially reduces exploration burden and enhances generalization, with ABA-FT offering additional gains in complex scenarios; limitations include reliance on a controlled or well-instrumented environment and a simple human-model guesswork, pointing to future work on real hardware and robust human-interaction pipelines.

Abstract

With strong capabilities of reasoning and a broad understanding of the world, Large Language Models (LLMs) have demonstrated immense potential in building versatile embodied decision-making agents capable of executing a wide array of tasks. Nevertheless, when deployed in unfamiliar environments, we show that LLM agents encounter challenges in efficiently gathering essential information, leading to suboptimal performance. Conversely, human individuals often seek additional information from their peers prior to taking action, harnessing external knowledge to avoid unnecessary trial and error. Drawing inspiration from this behavior, we propose \textit{Asking Before Acting} (ABA), a method that empowers the agent to proactively inquire with external sources for pertinent information using natural language during their interactions within the environment. In this way, the agent is able to enhance its efficiency and performance by circumventing potentially laborious steps and combating the difficulties associated with exploration in unfamiliar environments and vagueness of the instructions. We conduct extensive experiments involving a spectrum of environments including text-based household everyday tasks, robot arm manipulation tasks, and real world open domain image based embodied tasks. The experiments involve various models from Vicuna to GPT-4. The results demonstrate that, even with modest prompts modifications, ABA exhibits substantial advantages on both performance and efficiency over baseline LLM agents. Further finetuning ABA with reformulated metadata (ABA-FT) faciliates learning the rationale for asking and allows for additional enhancements especially in tasks that baselines struggle to solve.

Asking Before Acting: Gather Information in Embodied Decision Making with Language Models

TL;DR

The work tackles the problem that LLM-based embodied agents struggle to gather sufficient contextual information in unfamiliar environments. It introduces Asking Before Acting (ABA), a framework that interleaves natural-language information queries with actions within a Contextual MDP, and extends it with ABA-FT for improved data efficiency and long-horizon performance. By formalizing the augmentation of state/action spaces and introducing a human/external information source in the loop, the approach achieves notable performance and efficiency gains across text-based tasks, robot manipulation, and real-world image-guided tasks, with consistent improvements over baselines like ReAct and BUTLER. The results suggest that proactive information gathering via natural-language queries substantially reduces exploration burden and enhances generalization, with ABA-FT offering additional gains in complex scenarios; limitations include reliance on a controlled or well-instrumented environment and a simple human-model guesswork, pointing to future work on real hardware and robust human-interaction pipelines.

Abstract

With strong capabilities of reasoning and a broad understanding of the world, Large Language Models (LLMs) have demonstrated immense potential in building versatile embodied decision-making agents capable of executing a wide array of tasks. Nevertheless, when deployed in unfamiliar environments, we show that LLM agents encounter challenges in efficiently gathering essential information, leading to suboptimal performance. Conversely, human individuals often seek additional information from their peers prior to taking action, harnessing external knowledge to avoid unnecessary trial and error. Drawing inspiration from this behavior, we propose \textit{Asking Before Acting} (ABA), a method that empowers the agent to proactively inquire with external sources for pertinent information using natural language during their interactions within the environment. In this way, the agent is able to enhance its efficiency and performance by circumventing potentially laborious steps and combating the difficulties associated with exploration in unfamiliar environments and vagueness of the instructions. We conduct extensive experiments involving a spectrum of environments including text-based household everyday tasks, robot arm manipulation tasks, and real world open domain image based embodied tasks. The experiments involve various models from Vicuna to GPT-4. The results demonstrate that, even with modest prompts modifications, ABA exhibits substantial advantages on both performance and efficiency over baseline LLM agents. Further finetuning ABA with reformulated metadata (ABA-FT) faciliates learning the rationale for asking and allows for additional enhancements especially in tasks that baselines struggle to solve.
Paper Structure (29 sections, 3 equations, 6 figures, 3 tables)

This paper contains 29 sections, 3 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Method
  4. Contextual MDP with Human / External Information Source in the Loop
  5. Algorithm
  6. Asking Before Acting
  7. Asking Before Acting with Finetuning
  8. Human Model
  9. Experiments
  10. Text-based Household Everyday Task
  11. ALFWorld
  12. Robot Arm Manipulation Tasks
  13. Real-World Open Domain Embodied Tasks with Image Inputs
  14. Related Works
  15. Language Agent
  16. ...and 14 more sections

Figures (6)

  • Figure 1: ABA (as shown in (b)) allows the agent to seek clarification about the task as well as efficiently gather necessary information. In contrast, acting without asking (as showin in (c)) may lead to task misunderstanding or laborious exploration perid. Note (a) and (b) are key frames of trajectories generated by ABA in experiments in Section \ref{['Sec:4-4Exp4']}.
  • Figure 2: Performance on Ambiguous ALFWorld and Multiround ALFWorld. The prefixes of the $x$-label Amb- and Mul- refer to Ambigous ALFWorld and multi-round ALFWorld, and two environments correspond to different $y$-axis. The legend suffixes -ID and -OOD refer to performance on in-distribution and out-of-distribution evaluation sets.
  • Figure 3: Left: Visualization of an example task instance in Task 3. Right: Average success rates across 20 trails on three manipulation tasks. Here, $x$ and $y$ refers to parameters in each task.
  • Figure 4: Visualization of real-world open domain embodied tasks. We show four trajectories and their corresponding instructions.
  • Figure 5: A failure case where the agent wrongly assumes the eggs in the garbagecan cannot be used to finish the task.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Definition 2.1
  • Definition 3.1