Audio Visual Language Maps for Robot Navigation

TL;DR

This work introduces AVLMaps, a unified 3D map that fuses audio, visual, and language cues to enable open-vocabulary, zero-shot multimodal navigation. The method combines four localization modules into a shared voxel grid and uses cross-modal heatmap fusion guided by LLMs to disambiguate target goals described by language, images, or sounds. Empirical results in simulation and real-world experiments show improved recall and navigation success in ambiguous, multimodal scenarios, with notable gains over prior visual-language mappings. The approach highlights the potential of integrating audio with vision-language grounding to enhance robotic navigation and landmark indexing in realistic environments, while discussing limitations and avenues for lifelong multimodal learning.

Abstract

While interacting in the world is a multi-sensory experience, many robots continue to predominantly rely on visual perception to map and navigate in their environments. In this work, we propose Audio-Visual-Language Maps (AVLMaps), a unified 3D spatial map representation for storing cross-modal information from audio, visual, and language cues. AVLMaps integrate the open-vocabulary capabilities of multimodal foundation models pre-trained on Internet-scale data by fusing their features into a centralized 3D voxel grid. In the context of navigation, we show that AVLMaps enable robot systems to index goals in the map based on multimodal queries, e.g., textual descriptions, images, or audio snippets of landmarks. In particular, the addition of audio information enables robots to more reliably disambiguate goal locations. Extensive experiments in simulation show that AVLMaps enable zero-shot multimodal goal navigation from multimodal prompts and provide 50% better recall in ambiguous scenarios. These capabilities extend to mobile robots in the real world - navigating to landmarks referring to visual, audio, and spatial concepts. Videos and code are available at: https://avlmaps.github.io.

Figures (6)

• Figure 1: AVLMaps provide an open-vocabulary 3D map representation for storing cross-modal information from audio, visual, and language cues. When combined with large language models, AVLMaps consumes multimodal prompts from audio, vision and language to solve zero-shot spatial goal navigation by effectively leveraging complementary information sources to disambiguate goals.
• Figure 2: System overview. AVLMaps are constructed from RGB-D, audio, and odometry inputs, converting raw data into visual localization features, visual-language features, and audio-language features. During inference time, each module's output is unified with cross-modal reasoning, allowing users to query spatial location with multimodal information.
• Figure 3: The key idea of cross-modal reasoning is converting the prediction from different modalities into heatmaps, and then fusing them with element-wise multiplication, effectively using complementary multimodal information to resolve ambiguous prompts.
• Figure 4: Real-world experiments are conducted in a room with multiple ambiguous goals such as tables, chairs, backpacks, and paper boxes (left). We leverage dense SLAM techniques to build a 3D reconstruction of the scene from RGB-D camera data into which we anchor features from multiple foundation models (right).
• Figure 5: We artificially insert sounds with different semantics at locations shown in the image. Different sounds are played when the robot moves to these locations during mapping. Sounds are sampled from the ESC-50 dataset.