ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Driving

TL;DR

This paper proposes a novel multi-modal Zero-shot Offboard Panoptic Perception (ZOPP) framework for autonomous driving scenes that integrates the powerful zero-shot recognition capabilities of vision foundation models and 3D representations derived from point clouds and demonstrates the great potential of the ZOPP for real-world scenarios.

Abstract

Offboard perception aims to automatically generate high-quality 3D labels for autonomous driving (AD) scenes. Existing offboard methods focus on 3D object detection with closed-set taxonomy and fail to match human-level recognition capability on the rapidly evolving perception tasks. Due to heavy reliance on human labels and the prevalence of data imbalance and sparsity, a unified framework for offboard auto-labeling various elements in AD scenes that meets the distinct needs of perception tasks is not being fully explored. In this paper, we propose a novel multi-modal Zero-shot Offboard Panoptic Perception (ZOPP) framework for autonomous driving scenes. ZOPP integrates the powerful zero-shot recognition capabilities of vision foundation models and 3D representations derived from point clouds. To the best of our knowledge, ZOPP represents a pioneering effort in the domain of multi-modal panoptic perception and auto labeling for autonomous driving scenes. We conduct comprehensive empirical studies and evaluations on Waymo open dataset to validate the proposed ZOPP on various perception tasks. To further explore the usability and extensibility of our proposed ZOPP, we also conduct experiments in downstream applications. The results further demonstrate the great potential of our ZOPP for real-world scenarios.

Figures (11)

• Figure 1: Overview of our proposed ZOPP. The core of ZOPP is a complete pipeline to achieve offboard panoptic perception of AD scenes, including multi-view mask track generation (red), 3D semantic and instance segmentation (orange), point cloud completion (green), 3D detection (blue), and 4D occupancy reconstruction (purple).
• Figure 2: Overview of our object association across multiple views. Multi-view images are concatenated in a panoramic order. The visual features and horizontal pixel coordinates of each object are drawn at the top and bottom of the images, respectively. Visual features $v_1$ and $v_5$ are very similar, so the location differences $d_1$ and $d_5$ contribute to the matching determination. The visual features of traffic lights are almost the same ($v_8, v_9, v_{10}$), so we can associate them with location similarities ($d_6,d_7$).
• Figure 3: Point clouds are projected into the image plane, and visualized in a color map based on the depth values (Near to Far). On the right, we compare the effect before (top) and after (bottom) our proposed parallax occlusion. Please zoom in the highlighted pink boxes to see the filtering points.
• Figure 4: Parallax Occlusion Filtering
• Figure 5: Qualitative results of our proposed ZOPP on various perception tasks in AD scenes, including 2D segmentation, 3D detection, 3D semantic segmenation, 3D panoptic segmentation, and occupancy predition.