Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Point-Based Approach to Efficient LiDAR Multi-Task Perception

Christopher Lang, Alexander Braun, Lars Schillingmann, Abhinav Valada

TL;DR

This work introduces PAttFormer, a fully point-based transformer architecture for joint LiDAR semantic segmentation and 3D object detection. By leveraging neighborhood attention, grid-pooling, and a 3D deformable-attention detection head, the model shares parameters across tasks and avoids task-specific projections, achieving a smaller footprint and faster inference with competitive accuracy on nuScenes and KITTI. Extensive experiments show consistent gains from multi-task training, including +2.4% mAP in detection and +1.8% mIoU in segmentation, and strong data-efficiency, especially with limited labeled data. The approach advances practical autonomous driving perception by delivering efficient, scalable multi-task performance directly on raw point clouds, with potential for further gains through automated labeling and extended tasks.

Abstract

Multi-task networks can potentially improve performance and computational efficiency compared to single-task networks, facilitating online deployment. However, current multi-task architectures in point cloud perception combine multiple task-specific point cloud representations, each requiring a separate feature encoder and making the network structures bulky and slow. We propose PAttFormer, an efficient multi-task architecture for joint semantic segmentation and object detection in point clouds that only relies on a point-based representation. The network builds on transformer-based feature encoders using neighborhood attention and grid-pooling and a query-based detection decoder using a novel 3D deformable-attention detection head design. Unlike other LiDAR-based multi-task architectures, our proposed PAttFormer does not require separate feature encoders for multiple task-specific point cloud representations, resulting in a network that is 3x smaller and 1.4x faster while achieving competitive performance on the nuScenes and KITTI benchmarks for autonomous driving perception. Our extensive evaluations show substantial gains from multi-task learning, improving LiDAR semantic segmentation by +1.7% in mIou and 3D object detection by +1.7% in mAP on the nuScenes benchmark compared to the single-task models.

A Point-Based Approach to Efficient LiDAR Multi-Task Perception

TL;DR

This work introduces PAttFormer, a fully point-based transformer architecture for joint LiDAR semantic segmentation and 3D object detection. By leveraging neighborhood attention, grid-pooling, and a 3D deformable-attention detection head, the model shares parameters across tasks and avoids task-specific projections, achieving a smaller footprint and faster inference with competitive accuracy on nuScenes and KITTI. Extensive experiments show consistent gains from multi-task training, including +2.4% mAP in detection and +1.8% mIoU in segmentation, and strong data-efficiency, especially with limited labeled data. The approach advances practical autonomous driving perception by delivering efficient, scalable multi-task performance directly on raw point clouds, with potential for further gains through automated labeling and extended tasks.

Abstract

Multi-task networks can potentially improve performance and computational efficiency compared to single-task networks, facilitating online deployment. However, current multi-task architectures in point cloud perception combine multiple task-specific point cloud representations, each requiring a separate feature encoder and making the network structures bulky and slow. We propose PAttFormer, an efficient multi-task architecture for joint semantic segmentation and object detection in point clouds that only relies on a point-based representation. The network builds on transformer-based feature encoders using neighborhood attention and grid-pooling and a query-based detection decoder using a novel 3D deformable-attention detection head design. Unlike other LiDAR-based multi-task architectures, our proposed PAttFormer does not require separate feature encoders for multiple task-specific point cloud representations, resulting in a network that is 3x smaller and 1.4x faster while achieving competitive performance on the nuScenes and KITTI benchmarks for autonomous driving perception. Our extensive evaluations show substantial gains from multi-task learning, improving LiDAR semantic segmentation by +1.7% in mIou and 3D object detection by +1.7% in mAP on the nuScenes benchmark compared to the single-task models.
Paper Structure (22 sections, 2 equations, 3 figures, 8 tables)

This paper contains 22 sections, 2 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Technical Approach
  4. Problem Definition
  5. Multi-Task LiDAR PAttFormer Architecture
  6. Point-Based Attention (PAtt)
  7. Segmentation Head
  8. Detection Head
  9. Multi-Task Training
  10. Experimental Evaluation
  11. Datasets
  12. nuScenes
  13. SemanticKITTI
  14. Implementation Details
  15. 3D Object Detection Results
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Performance-latency trade-off compared to related object detection models on the nuScenes validation set. Model sizes are encoded in the blob diameter.
  • Figure 2: System architecture for a bottom-up set-up using the proposed PAtt modules that takes a raw point cloud as input and predicts a semantic class label for each scan point. In this setup, we start feature extraction on a subsampled point cloud with a larger attention window volume. We enrich the point cloud with previously masked points in the following stages while decreasing the attention window volume to balance computational complexity.
  • Figure 3: Comparison of predicted bounding boxes and segmentation labels on the 32-line scan from (left) and 64-line scan from SemanticKITTI (right) val sets.