Self-Supervised Learning to Fly using Efficient Semantic Segmentation and Metric Depth Estimation for Low-Cost Autonomous UAVs

TL;DR

This work addresses vision-based autonomous flight for small indoor UAVs by integrating semantic segmentation with monocular depth estimation and an adaptive scale factor to achieve metric depth. A knowledge-distillation pipeline transfers masks from a color-based teacher to a lightweight U-Net student for real-time segmentation, while an adaptive scale factor leverages ground-plane cues and camera geometry to produce metric depth from relative predictions. The approach is validated in a controlled 5×4 m lab and a digital twin, showing improved navigation efficiency and reliability, and culminates in a 1.6M-parameter onboard student network achieving 87.5% autonomous mission success. The combination of a metric-depth aware controller and a compact, self-supervised perception stack advances practical vision-based UAV navigation on resource-constrained platforms, with implications for indoor warehouses and surveillance tasks.

Abstract

This paper presents a vision-only autonomous flight system for small UAVs operating in controlled indoor environments. The system combines semantic segmentation with monocular depth estimation to enable obstacle avoidance, scene exploration, and autonomous safe landing operations without requiring GPS or expensive sensors such as LiDAR. A key innovation is an adaptive scale factor algorithm that converts non-metric monocular depth predictions into accurate metric distance measurements by leveraging semantic ground plane detection and camera intrinsic parameters, achieving a mean distance error of 14.4 cm. The approach uses a knowledge distillation framework where a color-based Support Vector Machine (SVM) teacher generates training data for a lightweight U-Net student network (1.6M parameters) capable of real-time semantic segmentation. For more complex environments, the SVM teacher can be replaced with a state-of-the-art segmentation model. Testing was conducted in a controlled 5x4 meter laboratory environment with eight cardboard obstacles simulating urban structures. Extensive validation across 30 flight tests in a real-world environment and 100 flight tests in a digital-twin environment demonstrates that the combined segmentation and depth approach increases the distance traveled during surveillance and reduces mission time while maintaining 100% success rates. The system is further optimized through end-to-end learning, where a compact student neural network learns complete flight policies from demonstration data generated by our best-performing method, achieving an 87.5% autonomous mission success rate. This work advances practical vision-based drone navigation in structured environments, demonstrating solutions for metric depth estimation and computational efficiency challenges that enable deployment on resource-constrained platforms.

Figures (18)

• Figure 1: Comparison between real-world laboratory environment (top) and its corresponding digital-twin simulation environment (bottom). The digital-twin accurately replicates the physical environment geometry, object placement, textures and spatial relationships, enabling parallel experimentation and validation between physical and simulated domains.
• Figure 2: System architecture for our vision-based autonomous control. The framework processes RGB frames through multiple pathways: a relative depth model and segmentation model provide geometric and semantic understanding, while camera parameters enable an adaptive scale factor computation for obtaining metric depth. These components are combined to generate drone control commands (translations $\nu$ and rotations $\omega$) which are then used to train a compact student neural network through knowledge distillation which employs a mean squared error (MSE) loss for training.
• Figure 3: Representation of all present class textures in the scene. On the first row, the second image represents the background, which is represented by walls and floors. On the third row, the second and third images represent the helipad box, the zone in which the drone initiates and concludes its mission. On the fourth row, the final image represents the carpet flooring which is our fly zone.
• Figure 4: The architecture of the knowledge distillation process for the segmentation model. From the RGB frames a few coordinates for each class were manually selected and processed into $40\times40$ pixel HSV patches. These patches train an SVM model with One vs. All classification to generate class specific masks for each frame. Due to the whole SVM computational limitations for real-time processing, a U-Net is subsequently trained using the SVM-generated masks as ground truth teacher data, enabling faster segmentation for real-time applications.
• Figure 5: On the left side is the 2D image with $50$ randomly selected points from the ground segmented plane that are computed as Euclidean distances between the ground point $P"$ and the camera position $P'$. On the right side is the 3D representation of the points.