Sem-NaVAE: Semantically-Guided Outdoor Mapless Navigation via Generative Trajectory Priors

TL;DR

Sem-NaVAE tackles mapless outdoor navigation by combining a CVAE-based trajectory generator with an open-vocabulary semantic selector from a lightweight VLM. The method generates many diverse trajectory hypotheses conditioned on sensor data and selects among them using semantic cost maps derived from natural language constraints, with an asynchronous update mechanism to handle occlusions. Key contributions include a CVAE with a learned prior and a multi-modal training objective that encourages diversity, integration of CLIPSeg for open-vocabulary constraints, and a robust recovery strategy for when no feasible paths exist. Real-world experiments show the approach achieves superior SR, EPT, and navigation efficiency relative to baselines, enabling long-range navigation without pre-built maps.

Abstract

This work presents a mapless global navigation approach for outdoor applications. It combines the exploratory capacity of conditional variational autoencoders (CVAEs) to generate trajectories and the semantic segmentation capabilities of a lightweight visual language model (VLM) to select the trajectory to execute. Open-vocabulary segmentation is used to score and select the generated trajectories based on natural language, and a state-of-the-art local planner executes velocity commands. One of the key features of the proposed approach is its ability to generate a large variability of trajectories and to select them and navigate in real-time. The approach was validated through real-world outdoor navigation experiments, achieving superior performance compared to state-of-the-art methods. A video showing an experimental run of the system can be found in https://www.youtube.com/watch?v=i3R5ey5O2yk.

Figures (5)

• Figure 1: Examples of autonomous mapless navigation. Six timesteps are shown (1-6) with their corresponding selected and executed paths on a satellite image. Left rows show a First Person View (FPV) of the robot's onboard camera. Middle rows show an elevation map with the proposed trajectories (blue represents collisions and green dynamically feasible waypoints). Right rows show the selected trajectory projected onto an FPV costmap (in black).
• Figure 2: Sem-NaVAE overview: A generator module proposes a number of motion hypotheses based on sensory information. These hypotheses are then filtered by collisions and projected onto and FPV semantic map constructed with a lightweight VLM. Both semantic and goal-distance costs are used to then select the optimal trajectory. Instead of perdiodically updating to a new trajectory at every inference step, the new best one is compared with the currently executing one, only switching to the new when optimal.
• Figure 3: NaVAE architecture. The model takes as inputs consecutive pointclouds, past trajectory and navigation goal in polar coordinates. A pre-trained PointNet encoder serves as a feature extractor for the conditional value into the CVAE and as heatmap generator for teacher forcing in the GRU decoder. Taking $K$ samples from $p_\theta$ yields $K$ predicted trajectories.
• Figure 4: Comparison of generation baselines MTG mtg, its PointNet modification MTG', and our system Sem-NaVAE. Left rows show an elevation map with the proposed trajectories (blue represents collisions and green free-space) and right rows show the generated trajectories projected onto an FPV image.
• Figure 5: Comparison of followed trajectories when varying selection classes and costs.