DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following

TL;DR

DialFRED addresses the limited interactivity of language-guided embodied AI by enabling agents to proactively ask clarifying questions. It augments ALFRED with a large dataset of 53K human-annotated question–answer pairs and an oracle, plus data augmentation that expands sub-goal types to 25. The proposed questioner–performer framework pre-trains the questioner on human dialogue and fine-tunes it with reinforcement learning to optimize when and what to ask, while the performer executes actions using the QAs. Experimental results show that incorporating dialogue yields meaningful improvements in task success and efficiency, validating the value of dialogue in embodied instruction following and suggesting this approach can extend to other embodied tasks.

Abstract

Language-guided Embodied AI benchmarks requiring an agent to navigate an environment and manipulate objects typically allow one-way communication: the human user gives a natural language command to the agent, and the agent can only follow the command passively. We present DialFRED, a dialogue-enabled embodied instruction following benchmark based on the ALFRED benchmark. DialFRED allows an agent to actively ask questions to the human user; the additional information in the user's response is used by the agent to better complete its task. We release a human-annotated dataset with 53K task-relevant questions and answers and an oracle to answer questions. To solve DialFRED, we propose a questioner-performer framework wherein the questioner is pre-trained with the human-annotated data and fine-tuned with reinforcement learning. We make DialFRED publicly available and encourage researchers to propose and evaluate their solutions to building dialog-enabled embodied agents.

Figures (5)

• Figure 1: Example dialogue between a robot and a human user during task completion. The robot raises questions to obtain additional information (e.g., when the target location is not clear) and to resolve ambiguities (e.g., when facing two knives on the table).
• Figure 2: The annotation interface for hybrid data collection. The worker first clicks the "begin" button to watch a video clip showing the initial states of the environment. Given the instruction, the worker selects a question to help perform the task. Next, the worker clicks the "show demonstration" button to watch the expert demonstration on how to complete the task. The worker then answers their own question based on what they have learned from the videos. Finally, workers choose whether they think the questions and answers are necessary to help the agent carry out the command.
• Figure 3:
• Figure 4: The questioner-performer architecture. The questioner generates questions based on the first person image of the agent and the task instruction. The oracle answers the question based on the scene metadata. The performer takes the image, the instruction, and question and answers as input to predict actions.
• Figure 5: The Questioner model. Given the instruction and image feature $I$, our model generates question tokens $w_{1:i}$.