Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Following

Yuki Inoue, Hiroki Ohashi

TL;DR

This paper shows that embedding the physical constraints of the deployed robots into the module design is highly effective, and shows that the landmark-based object search can be replaced by an implementation that prompts pretrained large language models for landmark-object relationships, eliminating the need for collecting dedicated training data.

Abstract

Embodied Instruction Following (EIF) studies how autonomous mobile manipulation robots should be controlled to accomplish long-horizon tasks described by natural language instructions. While much research on EIF is conducted in simulators, the ultimate goal of the field is to deploy the agents in real life. This is one of the reasons why recent methods have moved away from training models end-to-end and take modular approaches, which do not need the costly expert operation data. However, as it is still in the early days of importing modular ideas to EIF, a search for modules effective in the EIF task is still far from a conclusion. In this paper, we propose to extend the modular design using knowledge obtained from two external sources. First, we show that embedding the physical constraints of the deployed robots into the module design is highly effective. Our design also allows the same modular system to work across robots of different configurations with minimal modifications. Second, we show that the landmark-based object search, previously implemented by a trained model requiring a dedicated set of data, can be replaced by an implementation that prompts pretrained large language models for landmark-object relationships, eliminating the need for collecting dedicated training data. Our proposed Prompter achieves 41.53\% and 45.32\% on the ALFRED benchmark with high-level instructions only and step-by-step instructions, respectively, significantly outperforming the previous state of the art by 5.46\% and 9.91\%.

Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Following

TL;DR

This paper shows that embedding the physical constraints of the deployed robots into the module design is highly effective, and shows that the landmark-based object search can be replaced by an implementation that prompts pretrained large language models for landmark-object relationships, eliminating the need for collecting dedicated training data.

Abstract

Embodied Instruction Following (EIF) studies how autonomous mobile manipulation robots should be controlled to accomplish long-horizon tasks described by natural language instructions. While much research on EIF is conducted in simulators, the ultimate goal of the field is to deploy the agents in real life. This is one of the reasons why recent methods have moved away from training models end-to-end and take modular approaches, which do not need the costly expert operation data. However, as it is still in the early days of importing modular ideas to EIF, a search for modules effective in the EIF task is still far from a conclusion. In this paper, we propose to extend the modular design using knowledge obtained from two external sources. First, we show that embedding the physical constraints of the deployed robots into the module design is highly effective. Our design also allows the same modular system to work across robots of different configurations with minimal modifications. Second, we show that the landmark-based object search, previously implemented by a trained model requiring a dedicated set of data, can be replaced by an implementation that prompts pretrained large language models for landmark-object relationships, eliminating the need for collecting dedicated training data. Our proposed Prompter achieves 41.53\% and 45.32\% on the ALFRED benchmark with high-level instructions only and step-by-step instructions, respectively, significantly outperforming the previous state of the art by 5.46\% and 9.91\%.
Paper Structure (27 sections, 3 equations, 3 figures, 6 tables)

This paper contains 27 sections, 3 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Embodied Intelligence
  4. Large Language Models (LLMs)
  5. Task Description
  6. Proposed Method
  7. Perception Modules
  8. Language Module
  9. Vision Module
  10. Motion Controller
  11. Local Adjustment
  12. Semantic Search Module
  13. Navigation
  14. Experiments
  15. Results
  16. ...and 12 more sections

Figures (3)

  • Figure Fig. 1: A sample episode in ALFRED. Texts in the blue boxes correspond to the low-level instructions.
  • Figure Fig. 2: Method overview. Modules in dotted squares are trained on the ALFRED data. In addition to the processed information from the perception modules, our motion controller utilizes external knowledge such as the physical constraints of the robots and LLM outputs during decision-making.
  • Figure Fig. 3: Top-down view of four rooms from the val-unseen split of ALFRED. The images are scaled according to their actual physical sizes. Green and yellow dotted areas in the figures correspond to 5 m $\times$ 5 m and 2.5 m $\times$ 2.5 m areas, respectively. The smallness of the rooms may present unexpected sim-to-real disparities, as most real environments are much larger.