Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Attention-Guided Lidar Segmentation and Odometry Using Image-to-Point Cloud Saliency Transfer

Guanqun Ding, Nevrez Imamoglu, Ali Caglayan, Masahiro Murakawa, Ryosuke Nakamura

TL;DR

This work addresses the challenge of robust LiDAR-based perception and localization in autonomous driving by transferring saliency information from 2D images to 3D point clouds. It introduces FordSaliency for pseudo-ground-truth saliency and develops SalLiDAR for saliency-guided 3D semantic segmentation, along with SalLONet for self-supervised LiDAR odometry that leverages saliency and semantic cues to weight feature learning and pose estimation. The approach yields state-of-the-art performance on SemanticKITTI and KITTI Odometry benchmarks, demonstrating the effectiveness of cross-modal saliency knowledge transfer and dynamic/static point handling. By prioritizing salient static landmarks and suppressing dynamic points, the method improves both segmentation accuracy and odometry reliability in challenging driving scenarios with moving objects.

Abstract

LiDAR odometry estimation and 3D semantic segmentation are crucial for autonomous driving, which has achieved remarkable advances recently. However, these tasks are challenging due to the imbalance of points in different semantic categories for 3D semantic segmentation and the influence of dynamic objects for LiDAR odometry estimation, which increases the importance of using representative/salient landmarks as reference points for robust feature learning. To address these challenges, we propose a saliency-guided approach that leverages attention information to improve the performance of LiDAR odometry estimation and semantic segmentation models. Unlike in the image domain, only a few studies have addressed point cloud saliency information due to the lack of annotated training data. To alleviate this, we first present a universal framework to transfer saliency distribution knowledge from color images to point clouds, and use this to construct a pseudo-saliency dataset (i.e. FordSaliency) for point clouds. Then, we adopt point cloud-based backbones to learn saliency distribution from pseudo-saliency labels, which is followed by our proposed SalLiDAR module. SalLiDAR is a saliency-guided 3D semantic segmentation model that integrates saliency information to improve segmentation performance. Finally, we introduce SalLONet, a self-supervised saliency-guided LiDAR odometry network that uses the semantic and saliency predictions of SalLiDAR to achieve better odometry estimation. Our extensive experiments on benchmark datasets demonstrate that the proposed SalLiDAR and SalLONet models achieve state-of-the-art performance against existing methods, highlighting the effectiveness of image-to-LiDAR saliency knowledge transfer. Source code will be available at https://github.com/nevrez/SalLONet.

Attention-Guided Lidar Segmentation and Odometry Using Image-to-Point Cloud Saliency Transfer

TL;DR

This work addresses the challenge of robust LiDAR-based perception and localization in autonomous driving by transferring saliency information from 2D images to 3D point clouds. It introduces FordSaliency for pseudo-ground-truth saliency and develops SalLiDAR for saliency-guided 3D semantic segmentation, along with SalLONet for self-supervised LiDAR odometry that leverages saliency and semantic cues to weight feature learning and pose estimation. The approach yields state-of-the-art performance on SemanticKITTI and KITTI Odometry benchmarks, demonstrating the effectiveness of cross-modal saliency knowledge transfer and dynamic/static point handling. By prioritizing salient static landmarks and suppressing dynamic points, the method improves both segmentation accuracy and odometry reliability in challenging driving scenarios with moving objects.

Abstract

LiDAR odometry estimation and 3D semantic segmentation are crucial for autonomous driving, which has achieved remarkable advances recently. However, these tasks are challenging due to the imbalance of points in different semantic categories for 3D semantic segmentation and the influence of dynamic objects for LiDAR odometry estimation, which increases the importance of using representative/salient landmarks as reference points for robust feature learning. To address these challenges, we propose a saliency-guided approach that leverages attention information to improve the performance of LiDAR odometry estimation and semantic segmentation models. Unlike in the image domain, only a few studies have addressed point cloud saliency information due to the lack of annotated training data. To alleviate this, we first present a universal framework to transfer saliency distribution knowledge from color images to point clouds, and use this to construct a pseudo-saliency dataset (i.e. FordSaliency) for point clouds. Then, we adopt point cloud-based backbones to learn saliency distribution from pseudo-saliency labels, which is followed by our proposed SalLiDAR module. SalLiDAR is a saliency-guided 3D semantic segmentation model that integrates saliency information to improve segmentation performance. Finally, we introduce SalLONet, a self-supervised saliency-guided LiDAR odometry network that uses the semantic and saliency predictions of SalLiDAR to achieve better odometry estimation. Our extensive experiments on benchmark datasets demonstrate that the proposed SalLiDAR and SalLONet models achieve state-of-the-art performance against existing methods, highlighting the effectiveness of image-to-LiDAR saliency knowledge transfer. Source code will be available at https://github.com/nevrez/SalLONet.
Paper Structure (22 sections, 5 equations, 12 figures, 6 tables)

This paper contains 22 sections, 5 equations, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. LiDAR Odometry Estimation
  4. Saliency Detection on Point Cloud
  5. LiDAR Semantic Segmentation
  6. Proposed Framework
  7. Problem Formulation
  8. Framework Overview
  9. Proposed Method
  10. Experimental Analysis
  11. Implementation Details
  12. Datasets and Experimental Setup
  13. LiDAR FordSaliency Dataset
  14. SemanticKITTI Dataset
  15. KITTI Odometry Dataset
  16. ...and 7 more sections

Figures (12)

  • Figure 1: Visualization examples of SIFT lowe1999object key points and saliency maps for two consecutive frames. From left to right column: RGB images, results of SIFT lowe1999object key points and saliency maps from our FordSaliency dataset, and point clouds registered on images with saliency values.
  • Figure 2: Overview of proposed framework of image-to-LiDAR saliency knowledge transfer for 3D point cloud understanding. The 2D images saliency knowledge of RGB saliency models is transferred to 3D point clouds. Then the 3D point cloud saliency knowledge is used to attention-guided 3D point cloud understanding tasks, such as 3D semantic segmentation and LiDAR odometry estimation.
  • Figure 3: Illustration of proposed framework of image-to-LiDAR saliency knowledge transfer for attention-guided 3D point cloud understanding.
  • Figure 4: Visualization examples of point cloud and corresponding saliency pseudo-ground-truth (GT) map from our FordSaliency dataset.
  • Figure 5: Framework of proposed two-stream semantic segmentation model. The saliency prediction network is pre-trained on our FordSaliency dataset.
  • ...and 7 more figures