Generative Planning with 3D-vision Language Pre-training for End-to-End Autonomous Driving

TL;DR

GPVL tackles end-to-end autonomous driving by bridging 3D visual perception with language through a 3D-vision language pre-training module and employing a cross-modal language model to generate decisions and precise trajectories in an autoregressive fashion. It integrates a BEVformer-based backbone with group-wise vision-language alignment and a 2D scene captioning stream to enable rich reasoning over perception and navigation prompts. On nuScenes, GPVL achieves state-of-the-art trajectory accuracy and safety metrics with real-time performance while demonstrating strong zero-shot generalization and robustness to environmental variations. This work demonstrates that language-guided perception and planning can yield safer, more interpretable, and efficient autonomous driving systems in practice.

Abstract

Autonomous driving is a challenging task that requires perceiving and understanding the surrounding environment for safe trajectory planning. While existing vision-based end-to-end models have achieved promising results, these methods are still facing the challenges of vision understanding, decision reasoning and scene generalization. To solve these issues, a generative planning with 3D-vision language pre-training model named GPVL is proposed for end-to-end autonomous driving. The proposed paradigm has two significant aspects. On one hand, a 3D-vision language pre-training module is designed to bridge the gap between visual perception and linguistic understanding in the bird's eye view. On the other hand, a cross-modal language model is introduced to generate holistic driving decisions and fine-grained trajectories with perception and navigation information in an auto-regressive manner. Experiments on the challenging nuScenes dataset demonstrate that the proposed scheme achieves excellent performances compared with state-of-the-art methods. Besides, the proposed GPVL presents strong generalization ability and real-time potential when handling high-level commands in various scenarios. It is believed that the effective, robust and efficient performance of GPVL is crucial for the practical application of future autonomous driving systems. Code is available at https://github.com/ltp1995/GPVL

Figures (3)

• Figure 1: (a) The existing end-to-end autonomous driving framework only utilizes visual information to complete perception, prediction and planning tasks. (b) The emerging LLM-injected autonomous driving models merely introduce 2D visual features and use time-consuming LLM for planning decision. (c) The designed scheme focuses on 3D-vision language pre-training and conducts the planning by a language generation style.
• Figure 2: Pipeline of GPVL for autonomous driving. The framework is divided into three parts: (1) the backbone includes a 3D-vision encoder to obtain the basic BEV feature, then it is decoded into constrained detection, motion and map features, (2) the 3D-vision language pre-training module establishes the associations between vision and language features with the group-wise alignment, (3) the cross-modal language model generates the future planning decision in an auto-regressive manner based on aligned visual feature and navigation prompt. Note that in (a), the env prompt represents the detailed descriptions of 3D bounding boxes, agent motions and BEV map on the road, and the navigation means the detailed description of high-level instruction for the self-driving car. Four kinds of 3D-vision language (VL) alignments are presented in (b) for explanation.
• Figure 3: Visualized comparison of the proposed GPVL, VAD and the ground-truth on the nuScenes dataset.