BRIGHT-VO: Brightness-Guided Hybrid Transformer for Visual Odometry with Multi-modality Refinement Module

TL;DR

This work targets robust visual odometry under challenging low-light conditions by integrating a brightness-guided Transformer front-end with a multi-modality back-end that fuses IMU data via Pose Graph Optimization. The approach introduces KiC4R, a synthetic CARLA-based low-light VO dataset, and demonstrates state-of-the-art performance on KITTI and KiC4R, including significant improvements in both normal and low-light environments. Key contributions include a brightness-aware ViT encoder, a back-end refinement that leverages IMU and GNSS corrections, and a Levenberg-Marquardt optimized PGO framework, all released as open-source. The results indicate substantial reductions in drift and improved robustness for autonomous navigation in degraded illumination, highlighting BrightVO’s practical impact for real-world VO in robotics and driving applications.

Abstract

Visual odometry (VO) plays a crucial role in autonomous driving, robotic navigation, and other related tasks by estimating the position and orientation of a camera based on visual input. Significant progress has been made in data-driven VO methods, particularly those leveraging deep learning techniques to extract image features and estimate camera poses. However, these methods often struggle in low-light conditions because of the reduced visibility of features and the increased difficulty of matching keypoints. To address this limitation, we introduce BrightVO, a novel VO model based on Transformer architecture, which not only performs front-end visual feature extraction, but also incorporates a multi-modality refinement module in the back-end that integrates Inertial Measurement Unit (IMU) data. Using pose graph optimization, this module iteratively refines pose estimates to reduce errors and improve both accuracy and robustness. Furthermore, we create a synthetic low-light dataset, KiC4R, which includes a variety of lighting conditions to facilitate the training and evaluation of VO frameworks in challenging environments. Experimental results demonstrate that BrightVO achieves state-of-the-art performance on both the KiC4R dataset and the KITTI benchmarks. Specifically, it provides an average improvement of 20% in pose estimation accuracy in normal outdoor environments and 259% in low-light conditions, outperforming existing methods. For widespread use and further development, the research work is fully open-source at https://github.com/Anastasiawd/BrightVO.

Figures (7)

• Figure 1: BrightVO estimates the motion of a camera using visual information, while also integrating IMU measurements to achieve more accurate pose estimation.
• Figure 2: The pipeline of BrightVO, which begins with the input of consecutive image frames along with other modality information. In the VONet, images are transformed into vectors and passed into the Transformer Encoder after embedding. The global features are stored in the $\texttt{cls\_token}$, which is then mapped to a 6-dimension vector via an MLP. This vector represents the output initial VO motion. In the Refinement Module, IMU measurements are integrated to motions in an integrator. The VO motion, along with IMU motion, is fed into the PGO module, where iterative optimization occurs, yielding a high-precision pose estimate.
• Figure 3: The overview of Brightness Estimator. The image at the bottom is the input. After passing through a network composed of two 1x1 convolutions and a 9x9 depth-wise convolution, we obtain the brightness feature map shown in the top image, where different RGB values represent different illumination intensities.
• Figure 4: The overview pf Transformer Layer in the VO Encoder. After patch embedding, image features are used to generate Q (queries), K (keys), and V (values). These, combined with brightness information, are employed to compute attention scores. The self-attention mechanism is then applied, and the resulting hidden states are passed through a feed-forward network to produce the final output features.
• Figure 5: The overview of 4 light conditions in KiC4R. (a) Dusk. (b) Nighttime. (c) Mid-night. (d) Extreme weather.