Autonomous UAV Flight Navigation in Confined Spaces: A Reinforcement Learning Approach

TL;DR

This paper analyzes autonomous UAV navigation in confined industrial ducts, a safety-critical task demanding precise control. It compares on-policy PPO and off-policy SAC in a high-fidelity duct simulation to assess stability versus sample efficiency. PPO achieves a robust, collision-free end-to-end policy that completes the course, while SAC fails to reach completion, likely due to replay-buffer bias toward easier initial segments. The results suggest prioritizing training stability for reliable deployment in hazardous environments and motivate future work on sim-to-real transfer and hybrid algorithms.

Abstract

Autonomous UAV inspection of confined industrial infrastructure, such as ventilation ducts, demands robust navigation policies where collisions are unacceptable. While Deep Reinforcement Learning (DRL) offers a powerful paradigm for developing such policies, it presents a critical trade-off between on-policy and off-policy algorithms. Off-policy methods promise high sample efficiency, a vital trait for minimizing costly and unsafe real-world fine-tuning. In contrast, on-policy methods often exhibit greater training stability, which is essential for reliable convergence in hazard-dense environments. This paper directly investigates this trade-off by comparing a leading on-policy algorithm, Proximal Policy Optimization (PPO), against an off-policy counterpart, Soft Actor-Critic (SAC), for precision flight in procedurally generated ducts within a high-fidelity simulator. Our results show that PPO consistently learned a stable, collision-free policy that completed the entire course. In contrast, SAC failed to find a complete solution, converging to a suboptimal policy that navigated only the initial segments before failure. This work provides evidence that for high-precision, safety-critical navigation tasks, the reliable convergence of a well-established on-policy method can be more decisive than the nominal sample efficiency of an off-policy algorithm.

Figures (5)

• Figure 1: (a) Experimental setup for drone navigation in confined spaces, including a duct, and (b) the drone navigation evolution during training when navigating inside the digital twin of the duct shown in (a). Red indicates unsuccessful navigation, which occurs when the drone fails at the beginning of the trajectory, and green indicates successful navigation.
• Figure 2: Procedurally generated duct environment in Genesis, with a connected tubular segments with varying orientations.
• Figure 3: Simulated model used for all experiments of a Crazyflie 2, a $92 \times 92 \times 29$ mm nano-quadcopter.
• Figure 4: The drone trajectory that achieves the best performance for a model trained with PPO algorithm after 400 checkpoints.
• Figure 5: The drone trajectory that achieves the best performance for a model trained with SAC algorithm after 200 checkpoints.