DriveTok: 3D Driving Scene Tokenization for Unified Multi-View Reconstruction and Understanding

Abstract

With the growing adoption of vision-language-action models and world models in autonomous driving systems, scalable image tokenization becomes crucial as the interface for the visual modality. However, most existing tokenizers are designed for monocular and 2D scenes, leading to inefficiency and inter-view inconsistency when applied to high-resolution multi-view driving scenes. To address this, we propose DriveTok, an efficient 3D driving scene tokenizer for unified multi-view reconstruction and understanding. DriveTok first obtains semantically rich visual features from vision foundation models and then transforms them into the scene tokens with 3D deformable cross-attention. For decoding, we employ a multi-view transformer to reconstruct multi-view features from the scene tokens and use multiple heads to obtain RGB, depth, and semantic reconstructions. We also add a 3D head directly on the scene tokens for 3D semantic occupancy prediction for better spatial awareness. With the multiple training objectives, DriveTok learns unified scene tokens that integrate semantic, geometric, and textural information for efficient multi-view tokenization. Extensive experiments on the widely used nuScenes dataset demonstrate that the scene tokens from DriveTok perform well on image reconstruction, semantic segmentation, depth prediction, and 3D occupancy prediction tasks.

Figures (7)

• Figure 1: We propose our DriveTok for multi-view scene reconstruction and understanding. Multi-view images are processed by a 3D scene encoder to produce unified scene tokens, independent of camera layout and resolution. A spatial-aware multi-view decoder renders predictions in both image and occ spaces. Through joint multi-task training, our scene tokens encode rich textural, semantic, and geometric information.
• Figure 2: Illustration of DriveTok. DriveTok processes multi-view images and scene queries through an encoder-decoder architecture to produce unified scene tokens and generate diverse autonomous driving scene reconstruction and understanding outputs.
• Figure 3: Overview of DriveTok. Surround-view images are encoded by a 3D scene encoder. view tokens (with learnable and Plücker-ray embeddings) and scene tokens interact through a spatial-aware multi-view transformer with visibility-guided scene-view attention. Joint pretraining uses image reconstruction, depth prediction, semantic prediction, and occupancy prediction.
• Figure 4: Visualizations of BEV scene tokens and images. We visualize the BEV feature maps with PCA, the ground truth labels for semantic regularization and the decoded images v.s. ground truth images. The PCA result clearly shows that our BEV scene tokens not only learns the complex textures but also models the semantic structure of the driving scenes, avoiding radial patterns in conventional methods.
• Figure 5: Visualizations of DriveTok in different tasks. We provide a holistic visualization of our DriveTok in diverse autonomous driving scene reconstruction and understanding tasks. They show that DriveTok effectively represents the overall 3D environment by constructing scene tokens, thereby maintaining strong multi-view consistency. Through joint task training, the learned scene tokens not only capture texture details in the image space but also achieve a deeper perception and understanding.