Deep Learning for Embodied Vision Navigation: A Survey

TL;DR

This survey addresses embodied visual navigation, where an agent must reach goals in 3D environments based on egocentric observations. It surveys datasets, simulators, benchmarks, and learning strategies across simulated and real-world settings, covering model-free, self-supervised, planning-based, and cross-modal approaches, including dialog-driven and audio-visual variants. It analyzes the sim-to-real domain gap, transfer learning, and learning efficiency, proposing neural SLAM, transformer-based cross-modal models, and environment-agnostic representations as promising directions. The work highlights the practical significance of progressing toward robust, real-world embodied navigation systems capable of leveraging language and interactive cues.

Abstract

"Embodied visual navigation" problem requires an agent to navigate in a 3D environment mainly rely on its first-person observation. This problem has attracted rising attention in recent years due to its wide application in autonomous driving, vacuum cleaner, and rescue robot. A navigation agent is supposed to have various intelligent skills, such as visual perceiving, mapping, planning, exploring and reasoning, etc. Building such an agent that observes, thinks, and acts is a key to real intelligence. The remarkable learning ability of deep learning methods empowered the agents to accomplish embodied visual navigation tasks. Despite this, embodied visual navigation is still in its infancy since a lot of advanced skills are required, including perceiving partially observed visual input, exploring unseen areas, memorizing and modeling seen scenarios, understanding cross-modal instructions, and adapting to a new environment, etc. Recently, embodied visual navigation has attracted rising attention of the community, and numerous works has been proposed to learn these skills. This paper attempts to establish an outline of the current works in the field of embodied visual navigation by providing a comprehensive literature survey. We summarize the benchmarks and metrics, review different methods, analysis the challenges, and highlight the state-of-the-art methods. Finally, we discuss unresolved challenges in the field of embodied visual navigation and give promising directions in pursuing future research.

Figures (13)

• Figure 1: A taxonomy of deep learning methods for embodied navigation.
• Figure 2: A demonstration of a navigation process, in which a robot move to several places to accomplish a task.
• Figure 3: The render scenes of each dataset.
• Figure 4: An illustration of an model-free visual navigation model. This model learned from imitation learning and reinforcement learning. $r_t$ is the reward and $f(s_t)$ stands for the labels calculated from the state $s_t$. And $a'$ is the label stands for the optimal action.
• Figure 5: An illustration of an end-to-end visual navigation model with self-supervised objectives. $r_t$ is the reward and $f(s_t)$ stands for the labels calculated from the state $s_t$.