Fly360: Omnidirectional Obstacle Avoidance within Drone View

TL;DR

Fly360 is proposed, a two-stage perception-decision pipeline with a fixed random-yaw training strategy that achieves stable omnidirectional obstacle avoidance and outperforms forward-view baselines across all tasks.

Abstract

Obstacle avoidance in unmanned aerial vehicles (UAVs), as a fundamental capability, has gained increasing attention with the growing focus on spatial intelligence. However, current obstacle-avoidance methods mainly depend on limited field-of-view sensors and are ill-suited for UAV scenarios which require full-spatial awareness when the movement direction differs from the UAV's heading. This limitation motivates us to explore omnidirectional obstacle avoidance for panoramic drones with full-view perception. We first study an under explored problem setting in which a UAV must generate collision-free motion in environments with obstacles from arbitrary directions, and then construct a benchmark that consists of three representative flight tasks. Based on such settings, we propose Fly360, a two-stage perception-decision pipeline with a fixed random-yaw training strategy. At the perception stage, panoramic RGB observations are input and converted into depth maps as a robust intermediate representation. For the policy network, it is lightweight and used to output body-frame velocity commands from depth inputs. Extensive simulation and real-world experiments demonstrate that Fly360 achieves stable omnidirectional obstacle avoidance and outperforms forward-view baselines across all tasks. Our model is available at https://zxkai.github.io/fly360/

Figures (10)

• Figure 1: We present Fly360, a panoramic-vision-based framework for omnidirectional UAV obstacle avoidance. By mapping $360^{\circ}$ RGB inputs to control commands, Fly360 enables safe and agile navigation in complex environments without explicit mapping or specialized setups. From dynamic crowds to cluttered natural scenes and multi-UAV coordination, our method can achieve omnidirectional awareness and robust flight beyond the limitations of forward-view sensing.
• Figure 2: Overview of the experimental setting. Top: The three representative tasks used to evaluate omnidirectional obstacle avoidance:(a) Hovering maintenance, where the UAV maintains a defined position and orientation while avoiding nearby obstacles; (b) Dynamic target following, where the UAV tracks a moving object while reacting to dynamic obstacles; and (c) Fixed-trajectory filming, where the UAV follows a predefined path around a target while maintaining camera orientation. Bottom: The four high-fidelity simulation environments used in our evaluation, including (d) Park, (e) Forest, (f) Urban Street, and (g) Factory.
• Figure 3: The proposed framework unifies panoramic perception and policy learning for omnidirectional UAV obstacle avoidance. Panoramic RGB observations are first processed by a panoramic depth estimation network to produce a depth map, which is then downsampled for policy inference. The policy network fuses low-resolution depth with UAV states to predict body-frame velocity commands. Training is conducted in a differentiable simulator while inference executes the predicted commands through the onboard velocity controller and rotor-level control.
• Figure 4: Illustration of the observation vector components used in the policy network. The four components include the relative goal direction $\mathbf{d}_{\text{goal}}$, current velocity $\mathbf{v}_t$, upward orientation $\mathbf{q}^\text{up}_t$, and predefined safety radius $r$.
• Figure 5: Illustration of the proposed fixed random-yaw training strategy. In conventional free-yaw training (top), the UAV’s yaw continuously aligns with the direction of motion, and the onboard camera (yellow cone) only observes a limited forward field of view. In contrast, our training(bottom) randomly samples a yaw angle at the beginning of each rollout and keeps it constant throughout the episode. The panoramic camera (blue region) provides a full $360^{\circ}$ field of view.