Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PeP: a Point enhanced Painting method for unified point cloud tasks

Zichao Dong, Hang Ji, Xufeng Huang, Weikun Zhang, Xin Zhan, Junbo Chen

TL;DR

PeP tackles unified point cloud perception by introducing a refined point painting method and a language-model inspired point encoder. The two components are designed to be model-agnostic and plug-and-play, enabling stronger feature encoding and better modality alignment for both LiDAR semantic segmentation and multi-modal 3D object detection. Empirical results on KITTI and nuScenes demonstrate state-of-the-art or competitive performance, with notable gains when combining painting priors and LM-based embeddings. The work highlights the practical potential of combining segmentation-informed priors with transformer-based point representations to enhance cross-modal fusion in 3D perception.

Abstract

Point encoder is of vital importance for point cloud recognition. As the very beginning step of whole model pipeline, adding features from diverse sources and providing stronger feature encoding mechanism would provide better input for downstream modules. In our work, we proposed a novel PeP module to tackle above issue. PeP contains two main parts, a refined point painting method and a LM-based point encoder. Experiments results on the nuScenes and KITTI datasets validate the superior performance of our PeP. The advantages leads to strong performance on both semantic segmentation and object detection, in both lidar and multi-modal settings. Notably, our PeP module is model agnostic and plug-and-play. Our code will be publicly available soon.

PeP: a Point enhanced Painting method for unified point cloud tasks

TL;DR

PeP tackles unified point cloud perception by introducing a refined point painting method and a language-model inspired point encoder. The two components are designed to be model-agnostic and plug-and-play, enabling stronger feature encoding and better modality alignment for both LiDAR semantic segmentation and multi-modal 3D object detection. Empirical results on KITTI and nuScenes demonstrate state-of-the-art or competitive performance, with notable gains when combining painting priors and LM-based embeddings. The work highlights the practical potential of combining segmentation-informed priors with transformer-based point representations to enhance cross-modal fusion in 3D perception.

Abstract

Point encoder is of vital importance for point cloud recognition. As the very beginning step of whole model pipeline, adding features from diverse sources and providing stronger feature encoding mechanism would provide better input for downstream modules. In our work, we proposed a novel PeP module to tackle above issue. PeP contains two main parts, a refined point painting method and a LM-based point encoder. Experiments results on the nuScenes and KITTI datasets validate the superior performance of our PeP. The advantages leads to strong performance on both semantic segmentation and object detection, in both lidar and multi-modal settings. Notably, our PeP module is model agnostic and plug-and-play. Our code will be publicly available soon.
Paper Structure (28 sections, 1 figure, 2 tables)

This paper contains 28 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. RELATED WORK
  3. Point cloud segmentation
  4. Strong baseline: SphereFormer
  5. Point cloud detection
  6. Strong baseline: centerpoint
  7. Multi-modality 3d object detection
  8. Strong baseline: Virconv
  9. Previous point painting method
  10. ODISE
  11. Transformer based LM
  12. PointNet
  13. METHOD
  14. Overview
  15. LM-based point encoder
  16. ...and 13 more sections

Figures (1)

  • Figure 1: Overall pipeline of PeP. Our model takes both the image and point cloud as inputs. The image is processed by the segmentation model to extract semantic information. Subsequently, this semantic information is applied to the point cloud through painting, enhancing its features. The enriched point cloud is then forwarded to the language model (LM), followed by a task related decoder for further processing.