Spot the Difference: A Novel Task for Embodied Agents in Changing Environments

TL;DR

The paper introduces Spot the Difference, a task in which an embodied agent must detect changes in a dynamic environment starting from an outdated occupancy map. It builds a Semantic Occupancy Map dataset by generating manipulated layouts from Matterport3D and Gibson data and proposes an architecture with a mapper, pose estimator, and a hierarchical global/planner/local policy trained with PPO. A novel reward combining exploration with difference discovery guides learning, and experiments show that the CR+DR approach outperforms baselines on MP3D and Gibson benchmarks, including cases with oracle localization. This work demonstrates the value of reusing past knowledge to navigate changing environments and provides a reproducible benchmark for future research in dynamic embodied AI.

Abstract

Embodied AI is a recent research area that aims at creating intelligent agents that can move and operate inside an environment. Existing approaches in this field demand the agents to act in completely new and unexplored scenes. However, this setting is far from realistic use cases that instead require executing multiple tasks in the same environment. Even if the environment changes over time, the agent could still count on its global knowledge about the scene while trying to adapt its internal representation to the current state of the environment. To make a step towards this setting, we propose Spot the Difference: a novel task for Embodied AI where the agent has access to an outdated map of the environment and needs to recover the correct layout in a fixed time budget. To this end, we collect a new dataset of occupancy maps starting from existing datasets of 3D spaces and generating a number of possible layouts for a single environment. This dataset can be employed in the popular Habitat simulator and is fully compliant with existing methods that employ reconstructed occupancy maps during navigation. Furthermore, we propose an exploration policy that can take advantage of previous knowledge of the environment and identify changes in the scene faster and more effectively than existing agents. Experimental results show that the proposed architecture outperforms existing state-of-the-art models for exploration on this new setting.

Figures (9)

• Figure 1: Generation of alternative states of an environment: original and sample manipulated semantic maps.
• Figure 2: Overview of the proposed approach for navigation in changing environments.
• Figure 3: Value of accuracy and IoU for the different models at varying time-steps on the MP3D test set.
• Figure 4: Qualitative results comparing the performances of the CR and CR+DR agents for different episodes.
• Figure 5: Value of accuracy and IoU for the different models at varying time-steps on the MP3D validation set.