Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PoseINN: Realtime Visual-based Pose Regression and Localization with Invertible Neural Networks

Zirui Zang, Ahmad Amine, Rahul Mangharam

TL;DR

PoseINN reframes visual ego-pose estimation as learning a bijective mapping between image latents and 6DoF poses via an invertible neural network, guided by NeRF-generated offline synthetic views. A VAE encodes images and a Real-NVP INN learns the forward and reverse mappings, enabling posterior sampling of poses for a given image and yielding uncertainty estimates. The approach extends LiDAR-based Local_INN to cameras, uses a fast NeRF-driven data-generation pipeline, and demonstrates competitive accuracy on public datasets while enabling real-time deployment on an embedded mobile robot with uncertainty fusion via EKF. The results show that offline, low-resolution synthetic data can match SOTA accuracy while reducing data-generation costs and providing practical uncertainty information for robust robotic localization.

Abstract

Estimating ego-pose from cameras is an important problem in robotics with applications ranging from mobile robotics to augmented reality. While SOTA models are becoming increasingly accurate, they can still be unwieldy due to high computational costs. In this paper, we propose to solve the problem by using invertible neural networks (INN) to find the mapping between the latent space of images and poses for a given scene. Our model achieves similar performance to the SOTA while being faster to train and only requiring offline rendering of low-resolution synthetic data. By using normalizing flows, the proposed method also provides uncertainty estimation for the output. We also demonstrated the efficiency of this method by deploying the model on a mobile robot.

PoseINN: Realtime Visual-based Pose Regression and Localization with Invertible Neural Networks

TL;DR

PoseINN reframes visual ego-pose estimation as learning a bijective mapping between image latents and 6DoF poses via an invertible neural network, guided by NeRF-generated offline synthetic views. A VAE encodes images and a Real-NVP INN learns the forward and reverse mappings, enabling posterior sampling of poses for a given image and yielding uncertainty estimates. The approach extends LiDAR-based Local_INN to cameras, uses a fast NeRF-driven data-generation pipeline, and demonstrates competitive accuracy on public datasets while enabling real-time deployment on an embedded mobile robot with uncertainty fusion via EKF. The results show that offline, low-resolution synthetic data can match SOTA accuracy while reducing data-generation costs and providing practical uncertainty information for robust robotic localization.

Abstract

Estimating ego-pose from cameras is an important problem in robotics with applications ranging from mobile robotics to augmented reality. While SOTA models are becoming increasingly accurate, they can still be unwieldy due to high computational costs. In this paper, we propose to solve the problem by using invertible neural networks (INN) to find the mapping between the latent space of images and poses for a given scene. Our model achieves similar performance to the SOTA while being faster to train and only requiring offline rendering of low-resolution synthetic data. By using normalizing flows, the proposed method also provides uncertainty estimation for the output. We also demonstrated the efficiency of this method by deploying the model on a mobile robot.
Paper Structure (13 sections, 2 equations, 4 figures, 3 tables)

This paper contains 13 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Pose Regression
  4. Normalizing Flows
  5. Method
  6. Generate Synthetic Views with NeRF
  7. Learning the Pose-Image Mapping
  8. Experiments
  9. Camera Pose Regression on Public Dataset
  10. Visual Localization on Real-world Mobile Robot
  11. Uncertainty Estimation
  12. Discussion & Limitations
  13. Conclusion

Figures (4)

  • Figure 1: We propose to learn a mapping between the latent space of the images and camera poses in an environment with an invertible neural network. We use NeRF to guide camera pose sampling and render synthetic images. Evaluating the reverse path of the INN outputs the full posterior distribution of camera poses given a test image.
  • Figure 2: Sampling of Novel Camera Poses. Point clouds represent high-density points in the environment. Small pyramids represent training poses, testing poses, and sampled poses.
  • Figure 3: Network Structure of the PoseINN. The forward path (solid) is from pose to image. The reverse path (dashed) is from image to pose.
  • Figure 4: Examples of training and rendered images in real-world testing (Up: Indoor, Down: Outdoor)