Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

2D-3D Pose Tracking with Multi-View Constraints

Huai Yu, Kuangyi Chen, Wen Yang, Sebastian Scherer, Gui-Song Xia

TL;DR

This work proposes a new 2D-3D pose tracking framework, which consists of a front-end hybrid flow estimation network for consecutive frames and a back-end pose optimization module, and designs a cross-modal consistency-based loss to incorporate the multi-view constraints during the training and inference process.

Abstract

Camera localization in 3D LiDAR maps has gained increasing attention due to its promising ability to handle complex scenarios, surpassing the limitations of visual-only localization methods. However, existing methods mostly focus on addressing the cross-modal gaps, estimating camera poses frame by frame without considering the relationship between adjacent frames, which makes the pose tracking unstable. To alleviate this, we propose to couple the 2D-3D correspondences between adjacent frames using the 2D-2D feature matching, establishing the multi-view geometrical constraints for simultaneously estimating multiple camera poses. Specifically, we propose a new 2D-3D pose tracking framework, which consists: a front-end hybrid flow estimation network for consecutive frames and a back-end pose optimization module. We further design a cross-modal consistency-based loss to incorporate the multi-view constraints during the training and inference process. We evaluate our proposed framework on the KITTI and Argoverse datasets. Experimental results demonstrate its superior performance compared to existing frame-by-frame 2D-3D pose tracking methods and state-of-the-art vision-only pose tracking algorithms. More online pose tracking videos are available at \url{https://youtu.be/yfBRdg7gw5M}

2D-3D Pose Tracking with Multi-View Constraints

TL;DR

This work proposes a new 2D-3D pose tracking framework, which consists of a front-end hybrid flow estimation network for consecutive frames and a back-end pose optimization module, and designs a cross-modal consistency-based loss to incorporate the multi-view constraints during the training and inference process.

Abstract

Camera localization in 3D LiDAR maps has gained increasing attention due to its promising ability to handle complex scenarios, surpassing the limitations of visual-only localization methods. However, existing methods mostly focus on addressing the cross-modal gaps, estimating camera poses frame by frame without considering the relationship between adjacent frames, which makes the pose tracking unstable. To alleviate this, we propose to couple the 2D-3D correspondences between adjacent frames using the 2D-2D feature matching, establishing the multi-view geometrical constraints for simultaneously estimating multiple camera poses. Specifically, we propose a new 2D-3D pose tracking framework, which consists: a front-end hybrid flow estimation network for consecutive frames and a back-end pose optimization module. We further design a cross-modal consistency-based loss to incorporate the multi-view constraints during the training and inference process. We evaluate our proposed framework on the KITTI and Argoverse datasets. Experimental results demonstrate its superior performance compared to existing frame-by-frame 2D-3D pose tracking methods and state-of-the-art vision-only pose tracking algorithms. More online pose tracking videos are available at \url{https://youtu.be/yfBRdg7gw5M}
Paper Structure (19 sections, 12 equations, 12 figures, 4 tables)

This paper contains 19 sections, 12 equations, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual-only Pose Tracking
  4. 2D-3D Pose Tracking
  5. Proposed Method
  6. Hybrid Flow Estimation Network
  7. Depth Map Generation
  8. Flow Estimation
  9. Loss Function
  10. Multi-View Constraints-based Back-End Optimization
  11. Experiments
  12. Dataset and Evaluation Metrics
  13. Dataset
  14. Evaluation Metrics
  15. Experimental Setup
  16. ...and 4 more sections

Figures (12)

  • Figure 1: Illustrative examples of the proposed 2D-3D pose tracking and visual odometry on the KITTI 00 sequence. Top: The visualization of the LiDAR projection with initial pose and predicted pose. Bottom: The top view of the final trajectories of the proposed method and the visual odometry algorithm.
  • Figure 2: The proposed 2D-3D pose tracking framework. It consists of two main components: the front-end hybrid flow estimation network, and the back-end pose optimization module.
  • Figure 3: Diagram of the cross-modal consistency.
  • Figure 4: Visualization of the flow estimation. a: LiDAR projection overlay on the current camera image. b: LiDAR projection overlay on the next camera image. c: Predicted image-to-LiDAR depth flow between the current image and the LiDAR projection. d: Projected image-to-LiDAR depth flow between the next image and the LiDAR projection. e: Warped difference matrix between the predicted image-to-depth flows. f: Predicted optical flow between the two camera images.
  • Figure 5: 2D-3D pose tracking framework that loosely couples 2D-3D correspondences and 2D-2D correspondences.
  • ...and 7 more figures