Large-scale visual SLAM for in-the-wild videos

TL;DR

This proposed system establishes a new baseline for visual reconstruction from casual uncontrolled videos found online, demonstrating more consistent reconstructions over longer sequences of in-the-wild videos than previously achieved.

Abstract

Accurate and robust 3D scene reconstruction from casual, in-the-wild videos can significantly simplify robot deployment to new environments. However, reliable camera pose estimation and scene reconstruction from such unconstrained videos remains an open challenge. Existing visual-only SLAM methods perform well on benchmark datasets but struggle with real-world footage which often exhibits uncontrolled motion including rapid rotations and pure forward movements, textureless regions, and dynamic objects. We analyze the limitations of current methods and introduce a robust pipeline designed to improve 3D reconstruction from casual videos. We build upon recent deep visual odometry methods but increase robustness in several ways. Camera intrinsics are automatically recovered from the first few frames using structure-from-motion. Dynamic objects and less-constrained areas are masked with a predictive model. Additionally, we leverage monocular depth estimates to regularize bundle adjustment, mitigating errors in low-parallax situations. Finally, we integrate place recognition and loop closure to reduce long-term drift and refine both intrinsics and pose estimates through global bundle adjustment. We demonstrate large-scale contiguous 3D models from several online videos in various environments. In contrast, baseline methods typically produce locally inconsistent results at several points, producing separate segments or distorted maps. In lieu of ground-truth pose data, we evaluate map consistency, execution time and visual accuracy of re-rendered NeRF models. Our proposed system establishes a new baseline for visual reconstruction from casual uncontrolled videos found online, demonstrating more consistent reconstructions over longer sequences of in-the-wild videos than previously achieved.

Figures (6)

• Figure 1: Overview of our method. Given a video stream, we extract frames from the video sequentially. We first run an efficient global SfM process to estimate the camera intrinsic parameters $K_{\mathrm{init}}$ (\ref{['sec:method:init']}). Using an off-the-shelf semantic segmentation model, we prune the potential objects in the image (\ref{['sec:dynamic_removal']}) when estimating correspondence between frames. Correspondences are estimated across frames by DPVO; we use a monocular depth estimation model to get the prior depth, which can be used to guide the bundle adjustment optimization to improve robustness (\ref{['sec:depth_ba']}). SIM(3) pose graph optimization is conducted if a loop closure is detected (\ref{['sec:loop_closure']}). Finally, we run a re-triangulation and optionally global BA to refine the camera parameters and scene geometry (\ref{['sec:post_refinement']}).
• Figure 2: The camera poses (red in the map) on the sequence "Helsingborg Seq-1" across frames 500--1000. COLMAP has a break in the trajectory (circled in blue); GLOMAP tacitly fails to register images; our method produces smooth and continuous trajectories.
• Figure 3: The rendering results at the camera pose where COLMAP breaks.
• Figure 4: Reconstruction result visualization. Our method generally achieves smooth and continuous trajectories without breaks. COLMAP often produces a model only for part of the path. GLOMAP struggles to produce consistent results in these large-scale environments. GPS tracks are provided for Lund and the two Helsingborg sequences, but note that the provided GPS data is rather inaccurate. For Uppsala, the path has been drawn manually while referencing the video. No reference data is available for Yanshan Park.
• Figure 5: Ablation study: on the "Yanshan Park" sequence, we show the effectiveness of the proposed modules. To handle in-the-wild videos, prior depth and pruning dynamics in the view greatly help to improve the robustness.