N-MPC for Deep Neural Network-Based Collision Avoidance exploiting Depth Images

TL;DR

The paper tackles UAV collision avoidance using onboard depth images by introducing a neural collision predictor that is embedded into a nonlinear model predictive controller (N-MPC) as a differentiable algebraic constraint. The network is trained on simulated depth data to output a collision score for 3D points, and its output is converted into a closed-form constraint within the NLP, enabling real-time optimization at 100 Hz. Validation combines a quantitative collision-classifier analysis on simulated and real images with Gazebo simulations and real flights on the Learning-based Micro Flyer, showing robust sim-to-real transfer and practical onboard performance. This approach advances sensor-based local planning by uniting perceptual processing with optimization-based control and demonstrates open-source code for reproducibility and further development.

Abstract

This paper introduces a Nonlinear Model Predictive Control (N-MPC) framework exploiting a Deep Neural Network for processing onboard-captured depth images for collision avoidance in trajectory-tracking tasks with UAVs. The network is trained on simulated depth images to output a collision score for queried 3D points within the sensor field of view. Then, this network is translated into an algebraic symbolic equation and included in the N-MPC, explicitly constraining predicted positions to be collision-free throughout the receding horizon. The N-MPC achieves real time control of a UAV with a control frequency of 100Hz. The proposed framework is validated through statistical analysis of the collision classifier network, as well as Gazebo simulations and real experiments to assess the resulting capabilities of the N-MPC to effectively avoid collisions in cluttered environments. The associated code is released open-source along with the training images.

Figures (6)

• Figure 1: Architecture of the classifier network. The image is encoded into a latent representation $\mathbf{z}$, which is sampled from a learned distribution $\mathbf{z}\sim\mathcal{N}(\mu,\sigma)$, using a randomness variable $\epsilon$. The FC network is a coordinates-based NN, which outputs a predicted collision score $\hat{c}$ for input points $\mathbf{a}\in\mathbb{R}^3$ in the volumetric map described by $I$. The red lines highlight the part of the network to be embedded into the NMPC.
• Figure 2: The AR used in the reported experiment, LMF, and a view of the (filled) depth image from its onboard front-facing camera.
• Figure 3: $(x,y)$ motion of the AR, in blue, among the black obstacles. The orange circle and stars are the initial and goal positions. The blue dots are the position of the AR every 2s, while the red and pink arrows are resp. the reference and actual velocity vector at the corresponding time.
• Figure 4: Predicted collision score for the first (orange) and last (blue) shooting points of the receding horizon, throughout the trajectory, with its corresponding upper bound (red).
• Figure 5: Block diagram of the LMF aerial robot.