Robust Monocular Visual Odometry using Curriculum Learning

TL;DR

This work tackles the fragility of monocular visual odometry under challenging motion and visual conditions by integrating curriculum learning into the Deep Patch Visual Odometry (DPVO) framework. It introduces three CL strategies—trajectory-based progression, self-paced progression, and adaptive learning via Deep Deterministic Policy Gradient (DDPG)—to dynamically modulate learning emphasis on optical flow, translation, and rotation. Across synthetic and real-world benchmarks (including TartanAir, EuRoC, TUM-RGBD, and ICL-NUIM), the Self-Paced CL variant achieves state-of-the-art or near-state-of-the-art performance, reduces training time, and improves consistency, while Trajectory-Based CL shows particular strength on surface-reconstruction-focused data. The results demonstrate that curriculum-informed training can significantly enhance the robustness and generalization of monocular VO without sacrificing inference efficiency, and the framework is readily transferable to other VO architectures.

Abstract

Curriculum Learning (CL), drawing inspiration from natural learning patterns observed in humans and animals, employs a systematic approach of gradually introducing increasingly complex training data during model development. Our work applies innovative CL methodologies to address the challenging geometric problem of monocular Visual Odometry (VO) estimation, which is essential for robot navigation in constrained environments. The primary objective of our research is to push the boundaries of current state-of-the-art (SOTA) benchmarks in monocular VO by investigating various curriculum learning strategies. We enhance the end-to-end Deep-Patch-Visual Odometry (DPVO) framework through the integration of novel CL approaches, with the goal of developing more resilient models capable of maintaining high performance across challenging environments and complex motion scenarios. Our research encompasses several distinctive CL strategies. We develop methods to evaluate sample difficulty based on trajectory motion characteristics, implement sophisticated adaptive scheduling through self-paced weighted loss mechanisms, and utilize reinforcement learning agents for dynamic adjustment of training emphasis. Through comprehensive evaluation on the diverse synthetic TartanAir dataset and complex real-world benchmarks such as EuRoC and TUM-RGBD, our Curriculum Learning-based Deep-Patch-Visual Odometry (CL-DPVO) demonstrates superior performance compared to existing SOTA methods, including both feature-based and learning-based VO approaches. The results validate the effectiveness of integrating curriculum learning principles into visual odometry systems.

Figures (8)

• Figure 1: Distribution of difficulty scores across the TartanAir training dataset, with three difficulty levels, with dashed lines indicating difficulty thresholds at 0.44 and 0.64.
• Figure 2: Model validation set performance metrics (ATE and AUC) during training with Trajectory-Based Curriculum Learning strategy. The hard curriculum learning phase (red) improve DPVO baseline results (blue) with ATE=0.17 and AUC=0.83, while achieving comparable validation performance with only the easy and medium difficulty levels samples.
• Figure 3: Model validation set performance metrics (ATE and AUC) during training with Self-Paced Curriculum Learning strategy. During early training (step<18k), the Self-Paced approach (orange) exhibits faster and smoother improvements in both ATE and AUC metrics compared to baseline (blue). The method achieves equivalent performance with 47% fewer training steps while reaching the highest AUC (0.87) among all curriculum learning variants.
• Figure 4: Model validation set performance metrics (ATE and AUC) during training with Reinforcement Learning (DDPG) Curriculum Learning strategy.
• Figure 5: Dynamic weight progression (Flow, Pose, and Rotation) during DPVO training with RL-DDPG Curriculum Learning strategy. Flow weight maintain high values all through the training process alleviates its importance in the overall performance.