Reflex-Based Open-Vocabulary Navigation without Prior Knowledge Using Omnidirectional Camera and Multiple Vision-Language Models

TL;DR

This work addresses open-vocabulary robot navigation without prior maps or learning by fusing reflex-based control with an omnidirectional camera and pre-trained vision-language models. It processes a split 360-degree image with CLIP for global cues and Detic for local object cues, converting outputs into a directional vector that guides motion without exploration planning. The method, demonstrated on a Fetch robot, shows that combining global and local vision-language signals (ALL) yields more accurate navigation than either model alone, and can handle continuous linguistic instructions in larger environments. While promising for immediate deployment in dynamic settings, the approach relies on manually tuned parameters and currently struggles with complex room geometries and negations, indicating clear directions for future adaptive perception and obstacle-aware reflexes.

Abstract

Various robot navigation methods have been developed, but they are mainly based on Simultaneous Localization and Mapping (SLAM), reinforcement learning, etc., which require prior map construction or learning. In this study, we consider the simplest method that does not require any map construction or learning, and execute open-vocabulary navigation of robots without any prior knowledge to do this. We applied an omnidirectional camera and pre-trained vision-language models to the robot. The omnidirectional camera provides a uniform view of the surroundings, thus eliminating the need for complicated exploratory behaviors including trajectory generation. By applying multiple pre-trained vision-language models to this omnidirectional image and incorporating reflective behaviors, we show that navigation becomes simple and does not require any prior setup. Interesting properties and limitations of our method are discussed based on experiments with the mobile robot Fetch.

Figures (8)

• Figure 1: The concept of this study: simple reflex-based open-vocabulary navigation is enabled by splitting the expanded omnidirectional image and applying multiple pre-trained large-scale vision-language models.
• Figure 2: Dual-fisheye stitching for the expanded 360-degree omnidirectional image. The upper figure shows the fisheye images before processing, the middle figure shows the expanded image, and the lower figure shows the input image to vision-language models with unnecessary parts removed from the expanded image.
• Figure 3: The preliminary experiments of using large-scale vision-language models for open-vocabulary navigation. The left figure shows the split image and the recognition result, and the right graph shows the average of the transformed similarity $a$ for 10 repetitions of each instruction for CLIP and Detic: kitchen - "Go to the kitchen", microwave - "Please look at the microwave oven", bookshelf - "See the bookshelf".
• Figure 4: The configuration when mapping linguistic instructions to robot wheeled-base motion.
• Figure 5: The environmental setup of the basic experiment. The mobile robot Fetch is placed in a small area surrounded by the kitchen, microwave oven, and desk with chairs.