On Designing Consistent Covariance Recovery from a Deep Learning Visual Odometry Engine

TL;DR

This paper proposes formulating a factor graph on an implicit layer of the deep learning network to recover relative covariance estimates, which allows us to determine the covariance of the Visual Odometry (VO) solution.

Abstract

Deep learning techniques have significantly advanced in providing accurate visual odometry solutions by leveraging large datasets. However, generating uncertainty estimates for these methods remains a challenge. Traditional sensor fusion approaches in a Bayesian framework are well-established, but deep learning techniques with millions of parameters lack efficient methods for uncertainty estimation. This paper addresses the issue of uncertainty estimation for pre-trained deep-learning models in monocular visual odometry. We propose formulating a factor graph on an implicit layer of the deep learning network to recover relative covariance estimates, which allows us to determine the covariance of the Visual Odometry (VO) solution. We showcase the consistency of the deep learning engine's covariance approximation with an empirical analysis of the covariance model on the EUROC datasets to demonstrate the correctness of our formulation.

Figures (10)

• Figure 1: Covariance recovery from SLAM has utilities in many upstream tasks such as loop closure, data association, map merging, and active sensing
• Figure 2: Factor graph representation of a bundle adjustment problem. Factors are the error residuals between connected variables represented by \ref{['eq:error_residuals']}. In this diagram, $l_i$ represents landmark variables, $x_i$ represents pose variables, $t_i$ is a ternary factor with $\mathbf{X} = \{x_1, x_2, l_1\}$ , $p_i$ are binary factors with $\mathbf{X} = \{x_i, l_k\}$. like a camera projection function, and $o_i$ are binary factors with $\mathbf{X} = \{x_i, x_k\}$, $x_p$ like an odometry function, and $l_p$ are unary factors representing prior states of variables imposing initial conditions.
• Figure 3: Recovering marginal covariances of variables is selecting respective blocks in the full covariance matrix of the bundle adjustment problem.
• Figure 4: Covisibility graph of overlapping camera keyframes in a VO framework. The co-visibility graph, left, describes which frames have been successfully registered. The right figure depicts the co-visibility graph's adjacent matrix, showing how the current keyframe row $i$ is connected to other keyframe images $j$ at different columns. Generally, when the robot is exploring, the adjacency matrix is filled along the diagonal, making only a few connections with previous keyframes. The co-visibility graph directly affects the set of measurements used to form the information matrix $\mathbf{A}$. So, the co-visibility graph intuitively describes how covariance should trend for any VO odometry engine.
• Figure 5: The left plot show the trends in marginal covariances obtained from a monocular camera VO solution obtained from a deep learning visual odometry (DROID SLAM) on some sections of EUROC euroc datasets. The right plot shows the adjacency graph highlighting the nearest neighbors with which a keyframe overlaps and registers. Note that the marginal covariances of poses increase as the camera moves in one direction with some overlap to its previous frames. However, when the incoming frame overlaps past keyframes, indicated by off-diagonal elements on the adjacency graph, the covariance decreases as described in section \ref{['sec:validation']}. Also note that the rate and direction of growth of covariance directly depend on the amount of overlap. This shows that the covariance model developed using the formulation is consistent with the observed data. We have evaluated the complete datasets with similar observations throughout.