Panoptic-PHNet: Towards Real-Time and High-Precision LiDAR Panoptic Segmentation via Clustering Pseudo Heatmap

TL;DR

Panoptic-PHNet tackles LiDAR panoptic segmentation by combining a clustering-based approach for instance center inference with a knn-transformer that enhances offset regression among thing voxels. It introduces a clustering pseudo heatmap, derived directly from shifted voxels, and a center grouping module to merge duplicate centers, eliminating the need for a learned heatmap head. A multi-scale backbone fuses fine-grained voxel features with BEV features to balance detail and context while preserving real-time speed. The method achieves state-of-the-art panoptic quality on SemanticKITTI and nuScenes, demonstrating strong generalization and practical applicability for autonomous systems.

Abstract

As a rising task, panoptic segmentation is faced with challenges in both semantic segmentation and instance segmentation. However, in terms of speed and accuracy, existing LiDAR methods in the field are still limited. In this paper, we propose a fast and high-performance LiDAR-based framework, referred to as Panoptic-PHNet, with three attractive aspects: 1) We introduce a clustering pseudo heatmap as a new paradigm, which, followed by a center grouping module, yields instance centers for efficient clustering without object-level learning tasks. 2) A knn-transformer module is proposed to model the interaction among foreground points for accurate offset regression. 3) For backbone design, we fuse the fine-grained voxel features and the 2D Bird's Eye View (BEV) features with different receptive fields to utilize both detailed and global information. Extensive experiments on both SemanticKITTI dataset and nuScenes dataset show that our Panoptic-PHNet surpasses state-of-the-art methods by remarkable margins with a real-time speed. We achieve the 1st place on the public leaderboard of SemanticKITTI and leading performance on the recently released leaderboard of nuScenes.

Figures (11)

• Figure 1: Panoptic quality vs. single frame inference latency on SemanticKITTI DBLP:behley2020benchmark. Green area indicates real-time zone, which meets 10 frame-per-second frequency. The 2D CNN based approaches DBLP:conf/cvpr/ZhouZF21DBLP:conf/iros/MiliotoBMS20, the 3D CNN based approach DBLP:conf/cvpr/Hong0Z0L21 and the combined methods DBLP:behley2020benchmark are shown in blue, red and gray respectively. Our proposed Panoptic-PHNet outperforms all other methods in PQ by a large margin and still maintains a real-time speed.
• Figure 2: The overall framework of our Panoptic-PHNet. The backbone consists of a voxel encoder, a BEV encoder and a 2D backbone network for feature extraction. The extracted BEV features are concatenated with the fine-grained voxel features as voxel representations for semantic and instance branches. In the instance branch, a knn-transformer module is introduced to model the interaction among thing voxels. A clustering pseudo heatmap is generated from the shifted thing voxels to yield instance centers followed by a center grouping module. Finally, the outputs of the two branches are combined via a voting-based scheme to obtain the panoptic segmentation results.
• Figure 3: (a) illustrates a bad offset regression for a bus that is close to the origin of the LiDAR coordinate. The shifted thing points in different color represent different instance IDs. (b) and (c) show that with our center grouping module, the bus can be appropriately integrated.
• Figure 4: The self-attention layer in our knn-transformer module.
• Figure 5: Ablation study on SemanticKITTI validation. (a) The network benefits from the two proposed components. (b) Our approach based on the clustering pseudo heatmap is faster and more accurate. (c) Fed with the same results of semantic segmentation respectively, our instance segmentation performs better than the two state-of-the-art LiDAR panoptic segmentation methods.