UniVision: A Unified Framework for Vision-Centric 3D Perception

TL;DR

UniVision addresses the challenge of unifying vision-centric 3D perception tasks by jointly handling occupancy prediction and 3D object detection within a single framework. It introduces an explicit-implicit view transform to fuse 2D image features into 3D voxel representations, paired with a local-global feature extraction and cross-representation interaction to leverage both voxel and BEV representations. A joint Occ-Det data augmentation strategy and a progressive loss weighting schedule stabilize and accelerate multi-task training. Empirically, UniVision achieves state-of-the-art results across four public benchmarks (nuScenes LiDAR segmentation, nuScenes detection, OpenOccupancy, Occ3D) and demonstrates meaningful cross-task improvements, establishing a strong high-performance baseline for unified vision-centric 3D perception.

Abstract

The past few years have witnessed the rapid development of vision-centric 3D perception in autonomous driving. Although the 3D perception models share many structural and conceptual similarities, there still exist gaps in their feature representations, data formats, and objectives, posing challenges for unified and efficient 3D perception framework design. In this paper, we present UniVision, a simple and efficient framework that unifies two major tasks in vision-centric 3D perception, \ie, occupancy prediction and object detection. Specifically, we propose an explicit-implicit view transform module for complementary 2D-3D feature transformation. We propose a local-global feature extraction and fusion module for efficient and adaptive voxel and BEV feature extraction, enhancement, and interaction. Further, we propose a joint occupancy-detection data augmentation strategy and a progressive loss weight adjustment strategy which enables the efficiency and stability of the multi-task framework training. We conduct extensive experiments for different perception tasks on four public benchmarks, including nuScenes LiDAR segmentation, nuScenes detection, OpenOccupancy, and Occ3D. UniVision achieves state-of-the-art results with +1.5 mIoU, +1.8 NDS, +1.5 mIoU, and +1.8 mIoU gains on each benchmark, respectively. We believe that the UniVision framework can serve as a high-performance baseline for the unified vision-centric 3D perception task. The code will be available at \url{https://github.com/Cc-Hy/UniVision}.

Figures (4)

• Figure 1: The overall architecture of UniVision. After extracting image features from the inputs, we use the Ex-Im view transform module for complementary 2D-3D feature transformation. We then propose the local-global feature extraction and fusion block for adaptive BEV and voxel feature extraction, enhancement, and interaction, which are attached to task-specific perception heads. During training, the joint Occ-Det augmentation and progressive loss weight adjustment strategy are equipped for efficient multi-task training.
• Figure 2: The Ex-Im view transform module. (a) Depth-guided Explicit Feature Lifting. (b) Query-guided Implicit Feature Sampling.
• Figure 3: Illustration of (a) joint occupancy-detection augmentation and (b) progressive loss weight adjustment strategy.
• Figure 4: Qualitative results of UniVision framework, including the detection results on the 2D image plane, the detection results on the BEV plane, and the corresponding occupancy prediction results.