Towards Aerial Collaborative Stereo: Real-Time Cross-Camera Feature Association and Relative Pose Estimation for UAVs

TL;DR

The paper tackles real-time aerial collaborative stereo by addressing cross-camera feature association under changing FOVs and real-time relative pose estimation between UAVs. It introduces a dual-channel front-end that uses a guidance channel with SuperPoint+SuperGlue for high-quality matches and a prediction channel with LK-flow for rapid predictions, achieving real-time operation on NVIDIA Xavier NX. For back-end estimation, the Rel-MSCKF fuses common features with dual-VIO increments to estimate the relative pose between UAVs in real time, outperforming PGO-based back-ends in responsiveness while maintaining accuracy. Extensive experiments across simulated and real-world datasets demonstrate superior runtime, faster convergence, and robustness to asynchrony, communication dropouts, and occlusions, paving the way for remote mapping and large-scale multi-UAV perception.

Abstract

The collaborative visual perception of multiple Unmanned Aerial Vehicles (UAVs) has increasingly become a research hotspot. Compared to a single UAV equipped with a short-baseline stereo camera, multi-UAV collaborative vision offers a wide and variable baseline, providing potential benefits in flexible and large-scale depth perception. In this paper, we propose the concept of a collaborative stereo camera, where the left and right cameras are mounted on two UAVs that share an overlapping FOV. Considering the dynamic flight of two UAVs in the real world, the FOV and relative pose of the left and right cameras are continuously changing. Compared to fixed-baseline stereo cameras, this aerial collaborative stereo system introduces two challenges, which are highly real-time requirements for dynamic cross-camera stereo feature association and relative pose estimation of left and right cameras. To address these challenges, we first propose a real-time dual-channel feature association algorithm with a guidance-prediction structure. Then, we propose a Relative Multi-State Constrained Kalman Filter (Rel-MSCKF) algorithm to estimate the relative pose by fusing co-visual features and UAVs' visual-inertial odometry (VIO). Extensive experiments are performed on the popular onboard computer NVIDIA NX. Results on the resource-constrained platform show that the real-time performance of the dual-channel feature association is significantly superior to traditional methods. The convergence of Rel-MSCKF is assessed under different initial baseline errors. In the end, we present a potential application of aerial collaborative stereo for remote mapping obstacles in urban scenarios. We hope this work can serve as a foundational study for more multi-UAV collaborative vision research. Online video: https://youtu.be/avxMuOf5Qcw

Figures (28)

• Figure 1: The schema diagram of dual-channel feature association.
• Figure 2: The initialization workflow of dual-channel feature association.
• Figure 3: The update workflow of dual-channel feature association.
• Figure 4: The workflow of match fusion. ${}^{G_{4}}\boldsymbol{f}_{i}^{6}$ and ${}^{G_{4}}\boldsymbol{f}_{j}^{6}$, which are generated from the second guidance, experience twice LK-flow predictions. $\boldsymbol{f}_{i}^{6}$ and $\boldsymbol{f}_{j}^{6}$, which are generated from the first guidance, experience five times of LK-flow predictions.
• Figure 5: The Rel-MSCKF relative pose estimator in the back-end.