REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments

TL;DR

REVERIE introduces a novel embodied vision-language task that requires navigating to a remotely specified object in real indoor environments and grounding it with a bounding box. It combines a large-scale, object-annotated dataset built on Matterport3D with an Interactive Navigator-Pointer model that couples navigation decisions to object grounding through an interaction module. The results show that even state-of-the-art navigation and referring-expression methods struggle on unseen tests, with a substantial gap to human performance, highlighting the need for tighter integration between navigation and grounding. This work provides a foundation for practical human-robot collaboration by advancing evaluation benchmarks and baselines for remote object grounding in realistic 3D environments.

Abstract

One of the long-term challenges of robotics is to enable robots to interact with humans in the visual world via natural language, as humans are visual animals that communicate through language. Overcoming this challenge requires the ability to perform a wide variety of complex tasks in response to multifarious instructions from humans. In the hope that it might drive progress towards more flexible and powerful human interactions with robots, we propose a dataset of varied and complex robot tasks, described in natural language, in terms of objects visible in a large set of real images. Given an instruction, success requires navigating through a previously-unseen environment to identify an object. This represents a practical challenge, but one that closely reflects one of the core visual problems in robotics. Several state-of-the-art vision-and-language navigation, and referring-expression models are tested to verify the difficulty of this new task, but none of them show promising results because there are many fundamental differences between our task and previous ones. A novel Interactive Navigator-Pointer model is also proposed that provides a strong baseline on the task. The proposed model especially achieves the best performance on the unseen test split, but still leaves substantial room for improvement compared to the human performance.

Figures (10)

• Figure 1: REVERIE task: an agent is given a natural language instruction referring to a remote object (here in the red bounding box) in a photo-realistic 3D environment. The agent must navigate to an appropriate location and identify the object from multiple distracting candidates. The blue discs indicate nearby navigable viewpoints provided by the simulator.
• Figure 1: Several typical samples of the collected dataset, which involves various object category, goal region, path instruction, and object referring expression.
• Figure 2: Object bounding boxes (BBox) in our simulator. The BBox size and aspect ratio of the same object may change after the agent moves to another viewpoint or changes its camera view.
• Figure 2: Data collecting interface part I: instructions for AMT workers.
• Figure 3: The distribution of the number of words (left) and objects (right) in each instruction.