VL-TGS: Trajectory Generation and Selection using Vision Language Models in Mapless Outdoor Environments

TL;DR

VL-TGS tackles mapless outdoor navigation in human-centered environments by coupling a CVAE-based trajectory generator with a Vision-Language Model–driven selector guided by visual prompts. The two-stage approach first produces multiple geometrically traversable 10 m trajectories from LiDAR and odometry, then uses RGB imagery and semantic context to choose the most human-like, rule-abiding path. Across four outdoors scenarios and a real-robot test, VL-TGS yields notable gains in traversability ($\approx$20.8%) and similarity to human teleoperation ($\approx$28.5% Fréchet distance) compared with state-of-the-art baselines. Limitations include VLM latency and robustness in dynamic scenes, motivating future work on faster models and more flexible trajectory generators.

Abstract

We present a multi-modal trajectory generation and selection algorithm for real-world mapless outdoor navigation in human-centered environments. Such environments contain rich features like crosswalks, grass, and curbs, which are easily interpretable by humans, but not by mobile robots. We aim to compute suitable trajectories that (1) satisfy the environment-specific traversability constraints and (2) generate human-like paths while navigating on crosswalks, sidewalks, etc. Our formulation uses a Conditional Variational Autoencoder (CVAE) generative model enhanced with traversability constraints to generate multiple candidate trajectories for global navigation. We develop a visual prompting approach and leverage the Visual Language Model's (VLM) zero-shot ability of semantic understanding and logical reasoning to choose the best trajectory given the contextual information about the task. We evaluate our method in various outdoor scenes with wheeled robots and compare the performance with other global navigation algorithms. In practice, we observe an average improvement of 20.81% in satisfying traversability constraints and 28.51% in terms of human-like navigation in four different outdoor navigation scenarios.

Figures (5)

• Figure 1: Trajectories generated and selected using VL-TGS in outdoor navigation. The example path includes three different types of scenarios: (A) flower bed and curb, (B) corner, and (C) crosswalk. On the top, the map pin icon marks the goal behind the building, with the red solid or dashed line highlighting the robot's path. On the bottom, candidate trajectories are marked in red lines with numbers. The green path corresponds to the trajectory computed using VL-TGS. Overall, VL-TGS is capable of generating diverse, geometrically traversable paths and selecting semantically feasible trajectories for navigation in human-centered environments.
• Figure 2: Architecture: Our approach consists of two stages: CVAE-based trajectory generation and VLM-based trajectory selection. In the first stage, our attention-based CVAE takes consecutive frames of LiDAR point clouds and robot velocities as input, generating multiple diverse trajectories. These trajectories are sorted and visually marked with lines and numbers in the robot-view RGB image. In the second stage, our VLM-based trajectory selection module identifies the best trajectory number based on semantic feasibility, ensuring it lies on the sidewalk, avoids structures, crosses at zebra crossings, and adheres to other contextual rules.
• Figure 3: Qualitative Results: The top row shows the generated trajectories using all the methods, MTG mtg in green, ViNT vint in blue, NoMaD nomad in orange, PIVOT nasiriany2024pivot in cyan, CoNVOI convoi in purple, and VL-TGS in red. The bottom row shows the candidate trajectories in gray marked with numbers and the selected trajectory in red using VL-TGS. VL-TGS can generate and select a trajectory that is both geometrically and semantically feasible.
• Figure 4: Ablation Study on the Trajectory Generator: The left shows the generated candidate trajectories (red) and the selected trajectory (green) in the robot-view image. The right shows the top-down view image of the traversability map. The cyan color represents the final selected trajectory, and the yellow color represents the human-driven trajectory. Compared with CoNVOI convoi and PIVOT nasiriany2024pivot, VL-TGS generates the trajectory closest to the human-driven one, which keeps the robot on a safe pavement surface.
• Figure 5: Ablation Study on the Trajectory Selector: Compared with MTG mtg, which selects trajectories based on the shortest distance heuristic, VL-TGS selects the trajectory closer to human-like decision-making, going around the large obstruction.