FLYINGTRUST: A Benchmark for Quadrotor Navigation Across Scenarios and Vehicles

TL;DR

FLYINGTRUST proposes a simulation-first benchmark to quantify how quadrotor navigation robustness depends on both platform kinodynamics and scenario geometry. It defines two interpretable kinodynamic indicators, $TWR_{max}$ and $\alpha_{xy,max}, \alpha_{z,max}$, and derives a compact performance metric $P = TWR_{max} \cdot \alpha_{xy,max} \cdot \alpha_{z,max}$ to characterize platform capability. The framework combines 18 real and 18 virtual platform profiles with seven navigation scenarios to produce 252 platform-scenario combinations evaluated over multiple trials, using a composite, uncertainty-aware score that weighs scenario and platform importance and penalizes instability. A diverse set of optimization-based and learning-based navigation methods are benchmarked, revealing systematic interactions between kinodynamics and scene geometry and highlighting distinct failure modes across algorithms. The results underscore the need to design and evaluate navigation methods that remain robust across heterogeneous platforms and scenarios, guiding safer and more cost-effective real-world deployment.

Abstract

Visual navigation algorithms for quadrotors often exhibit a large variation in performance when transferred across different vehicle platforms and scene geometries, which increases the cost and risk of field deployment. To support systematic early-stage evaluation, we introduce FLYINGTRUST, a high-fidelity, configurable benchmarking framework that measures how platform kinodynamics and scenario structure jointly affect navigation robustness. FLYINGTRUST models vehicle capability with two compact, physically interpretable indicators: maximum thrust-to-weight ratio and axis-wise maximum angular acceleration. The benchmark pairs a diverse scenario library with a heterogeneous set of real and virtual platforms and prescribes a standardized evaluation protocol together with a composite scoring method that balances scenario importance, platform importance and performance stability. We use FLYINGTRUST to compare representative optimization-based and learning-based navigation approaches under identical conditions, performing repeated trials per platform-scenario combination and reporting uncertainty-aware metrics. The results reveal systematic patterns: navigation success depends predictably on platform capability and scene geometry, and different algorithms exhibit distinct preferences and failure modes across the evaluated conditions. These observations highlight the practical necessity of incorporating both platform capability and scenario structure into algorithm design, evaluation, and selection, and they motivate future work on methods that remain robust across diverse platforms and scenarios.

Figures (15)

• Figure 1: Representative benchmark scenario. The environment and obstacle layout, showing the quadrotor's start (green arrow) and goal (red arrow) positions. The blue line represents the straight-line reference path, and the red curve is an example of a collision-free trajectory executed by a planner.
• Figure 2: Taxonomy of quadrotor visual navigation algorithms. A flowchart of common navigation pipelines, from sensors to quadrotor. The figure contrasts the modular, optimization-based pipeline (left) with various learning-based approaches, which are categorized by the components replaced with neural networks (e.g., perception, planning, control).
• Figure 3: Overview of the FLYINGTRUST benchmarking pipeline. The benchmark pairs a fixed set of platform profiles with a fixed scenario library to form platform-scenario combinations; each combination is evaluated with multiple trials and summarized via a composite scoring scheme.
• Figure 4: Illustration of how rotor speed modulation produces the six degrees of freedom motions of a quadrotor. The central diagram illustrates forces (e.g., $T_i$, mg) and torques ($M_i$) on the vehicle. The surrounding schematics show the required changes in rotor speed (green arrows) to achieve the six canonical motions: longitudinal, lateral, vertical, pitch, roll, and yaw (red arrows).
• Figure 5: Comparison of "plus" and "cross" quadrotor layouts. This figure illustrates the two common motor configurations: (a) the "plus" layout, where motor arms align with the body's $x$ and $y$ axes, and (b) the "cross" layout, where the arms are rotated by $45^\circ$.