FlightBench: Benchmarking Learning-based Methods for Ego-vision-based Quadrotors Navigation

TL;DR

The paper addresses the lack of fair benchmarks for comparing learning-based and optimization-based ego-vision navigation in 3D quadrotor flight. It introduces FlightBench, an open-source benchmark with three task-difficulty metrics ($TO$, $VO$, $AOL$) and five scenarios, evaluated through comprehensive simulation and real-world experiments. Key findings show learning-based methods enable fast, aggressive flight but struggle with perception and sharp turns, while optimization-based methods yield smoother, more robust trajectories at the cost of latency; latency and sim-to-real transfer are critical factors. The benchmark and its difficulty criteria provide a reproducible framework to spur progress in vision-based quadrotor navigation and facilitate meaningful cross-method comparisons.

Abstract

Ego-vision-based navigation in cluttered environments is crucial for mobile systems, particularly agile quadrotors. While learning-based methods have shown promise recently, head-to-head comparisons with cutting-edge optimization-based approaches are scarce, leaving open the question of where and to what extent they truly excel. In this paper, we introduce FlightBench, the first comprehensive benchmark that implements various learning-based methods for ego-vision-based navigation and evaluates them against mainstream optimization-based baselines using a broad set of performance metrics. More importantly, we develop a suite of criteria to assess scenario difficulty and design test cases that span different levels of difficulty based on these criteria. Our results show that while learning-based methods excel in high-speed flight and faster inference, they struggle with challenging scenarios like sharp corners or view occlusion. Analytical experiments validate the correlation between our difficulty criteria and flight performance. Moreover, we verify the trend in flight performance within real-world environments through full-pipeline and hardware-in-the-loop experiments. We hope this benchmark and these criteria will drive future advancements in learning-based navigation for ego-vision quadrotors. Code and documentation are available at https://github.com/thu-uav/FlightBench.

Figures (6)

• Figure 1: Real-world validation of FlightBench.
• Figure 2: An overview of the FlightBench. FlightBench consists of three main components: (1) Tasks, featuring three scenarios categorized into eight difficulty levels. (2) Baselines, the core benchmarking platform supporting five ego-vision-based methods and two privileged methods. (3) Evaluation Metrics, offering a thorough suite of performance assessment metrics.
• Figure 3: Illustration of the task difficulty metrics
• Figure 4: A generic processing pipeline for hierarchical navigation systems on quadrotors.
• Figure 5: Success rate with different flight speeds.

Theorems & Definitions (5)

• Remark
• Remark
• Remark
• Remark
• Remark