FAR: Flexible, Accurate and Robust 6DoF Relative Camera Pose Estimation

TL;DR

FAR addresses the challenge of robust and precise $6DoF$ relative pose estimation by fusing a Transformer-based learned pose with a classical solver, while recovering translation scale. It introduces a prior-guided RANSAC framework and a 6D rotation representation to enable stable fusion, along with flexible backbones (dense features or correspondences). The method delivers state-of-the-art or competitive results across Matterport3D, InteriorNet, StreetLearn, and Map-free Relocalization, and demonstrates robustness to perturbations and adaptability to different inputs and data regimes. The practical impact is a scalable, versatile pipeline that can leverage existing correspondences and features while providing reliable scale and robustness in diverse environments.

Abstract

Estimating relative camera poses between images has been a central problem in computer vision. Methods that find correspondences and solve for the fundamental matrix offer high precision in most cases. Conversely, methods predicting pose directly using neural networks are more robust to limited overlap and can infer absolute translation scale, but at the expense of reduced precision. We show how to combine the best of both methods; our approach yields results that are both precise and robust, while also accurately inferring translation scales. At the heart of our model lies a Transformer that (1) learns to balance between solved and learned pose estimations, and (2) provides a prior to guide a solver. A comprehensive analysis supports our design choices and demonstrates that our method adapts flexibly to various feature extractors and correspondence estimators, showing state-of-the-art performance in 6DoF pose estimation on Matterport3D, InteriorNet, StreetLearn, and Map-free Relocalization.

Figures (15)

• Figure 1: Precise and Robust 6DoF Pose Estimation. Correspondence Estimation + Solver methods (here LoFTR sun2021loftr, RANSAC Fischler81) produce precise outputs for moderate rotations, but are not robust to large rotations (left), and cannot produce translation scale. Learning-based methods (here LoFTR with 8-Point ViT rockwell2022 head) produce scale (right) and are more robust, but lack precision (left). FAR leverages both for precise and robust prediction, including scale.
• Figure 2: Combining Classical and Learned. Left: Solver output is precise given many inliers, but is poor when few are available; Updated solver output via FAR's prior improves robustness significantly. FAR's Transformer is less precise but more robust. The full model fuses prior-guided Solver output and Transformer output for the best of both, giving more weight to the solver when many inliers are available (right).
• Figure 3: Overview. Given dense features and correspondences, FAR's Transformer produces camera poses (in square boxes ) through a transformer (round box ) and classical solver (round box ). In the first round, the solver produces a pose $\mathbf{T}_s$. FAR's pose transformer averages this with its own prediction $\mathbf{T}_t$ via weight $\mathbf{w}$, to yield the round 1 pose $\mathbf{T}_1$. $\mathbf{T}_1$ pose serves as a prior for the classic solver, which produces an updated pose $\mathbf{T}_u$. This is combined with an additional estimate of $\mathbf{T}_t$ and weight $\mathbf{w}$ to produce the final result $\mathbf{T}$. With few correspondences, $\mathbf{T}_1$ helps solver output, while the network learns to weigh Transformer predictions more heavily; with many correspondences, solver output is often good, so the network relies mostly on solver output.
• Figure 4: Ground Truth Robustness Study on Matterport3D. Using true correspondence, the solver is nearly perfect. Adding noise or outliers, it quickly degrades, while prior-guided Updated solver is robust to outliers and the Transformer is robust to noise. FAR matches or beats all methods across settings.
• Figure 5: Evolving with Dataset Size. The Transformer learns to rely more heavily upon the solver if data is limited (40% data size), and learns to use Transformer pose estimations as data scales and performance improves (100% data size).