Self-Supervised Monocular Visual Drone Model Identification through Improved Occlusion Handling

TL;DR

This work tackles ego-motion estimation for high-speed, GPS-denied drones using purely onboard data by introducing a self-supervised learning pipeline for monocular visual odometry and a complementary neural drone dynamics model. A key advance is occlusion-aware training for PoseNet via two- and three-frame SSL losses, with a novel combination of valid-pixel masking and minimum reprojection to maintain depth and motion accuracy under heavy disocclusion. A first-of-its-kind self-supervised student model learns to predict body-frame accelerations and, when scale is recovered with a single learnable parameter $s$, outperforms its PoseNet teacher in velocity estimation, particularly at high speeds. Integrating the neural drone model into the ROVIO VIO framework yields superior odometry on aggressive 3D racing trajectories, demonstrating the practical potential of onboard, self-supervised learning to enable faster and more robust flight in unknown environments. The approach promises improved state estimation and higher-speed capabilities for real-world drones without external motion capture or calibrated environment data.

Abstract

Ego-motion estimation is vital for drones when flying in GPS-denied environments. Vision-based methods struggle when flight speed increases and close-by objects lead to difficult visual conditions with considerable motion blur and large occlusions. To tackle this, vision is typically complemented by state estimation filters that combine a drone model with inertial measurements. However, these drone models are currently learned in a supervised manner with ground-truth data from external motion capture systems, limiting scalability to different environments and drones. In this work, we propose a self-supervised learning scheme to train a neural-network-based drone model using only onboard monocular video and flight controller data (IMU and motor feedback). We achieve this by first training a self-supervised relative pose estimation model, which then serves as a teacher for the drone model. To allow this to work at high speed close to obstacles, we propose an improved occlusion handling method for training self-supervised pose estimation models. Due to this method, the root mean squared error of resulting odometry estimates is reduced by an average of 15%. Moreover, the student neural drone model can be successfully obtained from the onboard data. It even becomes more accurate at higher speeds compared to its teacher, the self-supervised vision-based model. We demonstrate the value of the neural drone model by integrating it into a traditional filter-based VIO system (ROVIO), resulting in superior odometry accuracy on aggressive 3D racing trajectories near obstacles. Self-supervised learning of ego-motion estimation represents a significant step toward bridging the gap between flying in controlled, expensive lab environments and real-world drone applications. The fusion of vision and drone models will enable higher-speed flight and improve state estimation, on any drone in any environment.

Figures (7)

• Figure 1: We introduce a self-supervised learning approach that only uses data from onboard the drone to learn ego-motion estimation. Specifically, this data consists of the inertial measurements and control commands available in the flight controller (left), and monocular camera images (right). The approach does not require any external infrastructure and allows to scale the monocular vision-based estimates.
• Figure 2: Validity image reprojection: Given three images at timestep $t-5$, $t$ and $t+5$, two reconstructions of image $I_t$ can be computed. By estimating a depth map at time t and two translation and rotation estimations, $T_{t-5\rightarrow t}$ and $T_{t+5 \rightarrow t}$. In the occluded parts, this reconstruction is undefined and creates artefacts. For instance, in the reconstructed image that uses image $t-5$ as input (top row), the pixels right and below of the gate can not be reconstructed as they were occluded in $I_{t-5}$. The reconstructed image using image $t+5$ as input (bottom row), cannot reconstruct the right edge of $I_t$ as they exited the frame. Both cases can theoretically not be correctly reconstructed, which leads to errors in the loss. This can be reduced by taking the minimum monodepth2 as the errors are on the other side of edges (MD loss) is used. Pixels that move out of the image can be ignored using a valid pixel mask (light-gray, last column) as proposed by valid_pixels. However, valid_pixels only considered a single reconstruction (top/bottom row). We propose to use the minimum per valid pixel of both error maps that solves all edge cases, and we call it the 3F scheme.
• Figure 3: Validity depth reprojection: When approaching a gate, the reprojected depth map perceives similar artefacts as reprojected images due to occlusion. This causes an error in the depth consistency loss. In depth_consistency_loss only one projection was considered and used the depth consistency loss based on the top Error Map. In our 2F method we reprojected $Dt$ as well (bottom row) using the inverse of the estimated relative pose (T), followed by taking the minimum of both reprojection errors (last column). Note that we also apply a valid pixel mask for depth consistency but do not show it here.
• Figure 4: Single learnable parameter (in T) is used to transform the scaleless PoseNet velocity to a scaled body velocity. This is then used together with the IMU and motor RPMs to estimate a residual on the accelerometer-z and drag. By integrating the specific forces and the gravitational acceleration of $9.81 m/s^2$ the drone model is forced to recover the scale correctly
• Figure 5: Our loss function improves both depth and velocity estimation when there are large (dis-)occlusions. Top left: input image when passing through a gate. Bottom left: speed, estimated with motion tracking (GT, grey), and with a PoseNet trained with our loss (green) and the original loss (blue). Top right: Depth map from PoseNet with our loss, from close by (dark blue) to far (bright yellow). Bottom right: Depth map from PoseNet trained with the original loss.