360DVO: Deep Visual Odometry for Monocular 360-Degree Camera

TL;DR

360DVO tackles robust monocular visual odometry with omnidirectional input by learning distortion-aware spherical features and enforcing pose-consistent depth through an omnidirectional differentiable bundle adjustment. The method introduces SphereResNet-based DAS-Feat to produce robust sparse patches and an ODBA module that jointly optimizes camera poses and depths under spherical reprojection constraints. A new large-scale real-world OVO dataset complements synthetic benchmarks, enabling thorough evaluation under challenging motions and lighting. Empirical results show clear gains in accuracy and robustness over state-of-the-art baselines, with a practical fast variant suitable for real-time use. The work advances OVO by integrating distortion-aware deep features with differentiable geometry tailored to 360° imagery.

Abstract

Monocular omnidirectional visual odometry (OVO) systems leverage 360-degree cameras to overcome field-of-view limitations of perspective VO systems. However, existing methods, reliant on handcrafted features or photometric objectives, often lack robustness in challenging scenarios, such as aggressive motion and varying illumination. To address this, we present 360DVO, the first deep learning-based OVO framework. Our approach introduces a distortion-aware spherical feature extractor (DAS-Feat) that adaptively learns distortion-resistant features from 360-degree images. These sparse feature patches are then used to establish constraints for effective pose estimation within a novel omnidirectional differentiable bundle adjustment (ODBA) module. To facilitate evaluation in realistic settings, we also contribute a new real-world OVO benchmark. Extensive experiments on this benchmark and public synthetic datasets (TartanAir V2 and 360VO) demonstrate that 360DVO surpasses state-of-the-art baselines (including 360VO and OpenVSLAM), improving robustness by 50% and accuracy by 37.5%. Homepage: https://chris1004336379.github.io/360DVO-homepage

Figures (10)

• Figure 1: Framework overview of 360DVO. Our method takes sequential 360-degree RGB frames as input and extracts matching features and context features using our proposed DAS-Feat module (Sec. \ref{['sec:sphresnet']}) on each of them. In DAS-Feat, the key component SphereResNet extracts distortion-resistant features, allowing patches to be cropped without deformation. After patchifying (Sec. \ref{['sec:patch']}) the matching features around their gradient maxima, we compute the correlation of patch features and context features and estimate optical flow through a recurrent network. In the ODBA module, the pose $\mathbf{T}_i$ and depth $\mathbf{d}_i$ of frame $i$ are jointly optimized by minimizing the distance between predicted patch $\mathbf{p}^{\star}_j$ (from optical flow) and reprojected patch $\mathbf{p}_i'$ on an adjacent frame $j$ (Sec. \ref{['sec:optim']}).
• Figure 2: Comparison of feature extraction between the classic CNN and the SphereCNN SphereNet. The distortion-aware convolution kernels differ in their pixel sampling manners along the height in an omnidirectional image, which is guided by the tangent projection across latitudes on the image's corresponding sphere.
• Figure 3: Sample sequences of 360DVO dataset, demonstrating representative frames, 3D trajectory, and length for each sequence.
• Figure 4: Trajectories Comparison on the 360DVO dataset in 3D space, with position variations along the X, Y, and Z axes plotted over all frames. The ground truth is shown in black dashed lines, 360DVO results in red solid lines, and OpenVSLAM results in blue solid lines.
• Figure 5: Two samples of predicted patch trajectories (ResNet Encoder vs. SphereResNet). Correct in green, incorrect in red. Patches are highlighted by yellow squares. The proposed SphereResNet yields higher tracking accuracy.