LHMap-loc: Cross-Modal Monocular Localization Using LiDAR Point Cloud Heat Map

TL;DR

A novel pipeline termed LHMap-loc is proposed, which achieves accurate and efficient monocular localization in LiDAR maps by designing an end-to-end online pose regression network based on optical flow estimation and spatial attention to achieve real-time monocular visual localization in a pre-built map.

Abstract

Localization using a monocular camera in the pre-built LiDAR point cloud map has drawn increasing attention in the field of autonomous driving and mobile robotics. However, there are still many challenges (e.g. difficulties of map storage, poor localization robustness in large scenes) in accurately and efficiently implementing cross-modal localization. To solve these problems, a novel pipeline termed LHMap-loc is proposed, which achieves accurate and efficient monocular localization in LiDAR maps. Firstly, feature encoding is carried out on the original LiDAR point cloud map by generating offline heat point clouds, by which the size of the original LiDAR map is compressed. Then, an end-to-end online pose regression network is designed based on optical flow estimation and spatial attention to achieve real-time monocular visual localization in a pre-built map. In addition, a series of experiments have been conducted to prove the effectiveness of the proposed method. Our code is available at: https://github.com/IRMVLab/LHMap-loc.

Figures (6)

• Figure 1: Monocular localization pipeline using LiDAR point cloud heat map (LHMap). The pipeline consists of an offline LHMap generation network to build LHMap, and an online pose regression network to achieve real-time localization with pre-built LHMap.
• Figure 2: Detailed pipeline of LHMap-loc. It includes the offline LHMap generation network and the online pose regression network. In stage 1 of the offline network, $D_{gt}$ is used to generate heat feature $H_c$, and the coarse local LHMap $M_c$ is selected by the heat value calculated by $H_c$. $D_{init}$ and $I_{offline}$ are used to generate the flow embedding $E_D$. In stage 2, the initial coarse local LHMap $M_c^{init}$ and $I_{offline}$ are used to generate the flow embedding $E_M$ and the heat feature $H_M$. Both $E_D$ and $E_M$ are used for pose supervision by spatial attention weighting. In the online pose regression network, the real-time local LHMap $M_r$ and $I_{online}$ are used to regress the real-time 6-DoF pose.
• Figure 3: The details of the regression part. Multiply flow embedding $E$ and up-sampled heat feature $H$ as inputs, and then calculate weighted features. The result is fed into $MLP_q$ and $MLP_t$ to regress 6-DoF poses
• Figure 4: Qualitative results of LiDAR-image registration on KITTIkitti dataset.
• Figure 5: 3D point cloud map of KITTI sequence 00. (a) Original LiDAR point cloud map. (b) LHMap.