NavigScene: Bridging Local Perception and Global Navigation for Beyond-Visual-Range Autonomous Driving

TL;DR

This work tackles the gap between local perception and global navigation context in autonomous driving by introducing NavigScene, an auxiliary dataset that pairs multi-view sensor inputs with beyond-visual-range navigation guidance. It proposes three navigation-guided paradigms—Navigation-guided Reasoning (NSFT), Navigation-guided Preference Optimization (NPO), and Navigation-guided Vision-Language-Action (NVLA)—to enhance reasoning, generalization, and end-to-end driving performance. Across Q&A and end-to-end driving experiments, NavigScene yields significant improvements in perception, prediction, planning, and reasoning, with notable gains in cross-city generalization. The results demonstrate that incorporating BVR navigation knowledge enables more proactive, human-like driving behavior, bringing autonomous systems closer to reliable and safe operation in unfamiliar environments.

Abstract

Autonomous driving systems have made significant advances in Q&A, perception, prediction, and planning based on local visual information, yet they struggle to incorporate broader navigational context that human drivers routinely utilize. We address this critical gap between local sensor data and global navigation information by proposing NavigScene, an auxiliary navigation-guided natural language dataset that simulates a human-like driving environment within autonomous driving systems. Moreover, we develop three complementary paradigms to leverage NavigScene: (1) Navigation-guided Reasoning, which enhances vision-language models by incorporating navigation context into the prompting approach; (2) Navigation-guided Preference Optimization, a reinforcement learning method that extends Direct Preference Optimization to improve vision-language model responses by establishing preferences for navigation-relevant summarized information; and (3) Navigation-guided Vision-Language-Action model, which integrates navigation guidance and vision-language models with conventional driving models through feature fusion. Extensive experiments demonstrate that our approaches significantly improve performance across perception, prediction, planning, and question-answering tasks by enabling reasoning capabilities beyond visual range and improving generalization to diverse driving scenarios. This work represents a significant step toward more comprehensive autonomous driving systems capable of navigating complex, unfamiliar environments with greater reliability and safety.

Figures (8)

• Figure 1: Comparison between a) planning without global navigation guidance and b) planning with global navigation guidance. In this example, the vehicle needs to turn right at the next corner. Without beyond-view-range (BVR) knowledge from navigation, the planner makes a conservative decision to continue straight. With global BVR knowledge, it appropriately directs the vehicle to merge into the right-turn lane. Concrete examples from experiments are shown in Fig. \ref{['fig:open-vis']} and Fig. \ref{['fig:close-vis']}.
• Figure 2: Navigation guidance generation process of one scene. Part A (Visual Generation): Source and destination coordinates are calculated using the origin's coordinate and 3D translation vectors. A navigation video is constructed via Google Maps APIs, then evenly sampled to extract multiple frames. Part B (Text Generation): The multiple frames are processed by a vision-language model (GPT-4o achiam2023gpt4o) with a specialized prompt to generate several candidate responses. Self-consistency evaluation selects the highest-scoring candidate as the final navigation guidance.
• Figure 3: Comparison between a) non-navigation VLM reasoning and b) navigation-guided VLM reasoning. In our proposed navigation-guided paradigm, both the navigation guidance and question together form the prompt for VLM. (Best viewed when zoomed in.)
• Figure 4: Navigation-guided vision-language-action model for end-to-end driving. BEV features are concatenated with vision-language features generated by the frozen VLM and a learnable sparsity reduction MLP, then processed through a learnable feature fusion MLP to produce fused features for task-specific networks.
• Figure 5: Examples of question-answering on the DriveLM dataset.