Visual Servoing NMPC Applied to UAVs for Photovoltaic Array Inspection

TL;DR

The paper tackles the challenge of conducting reliable PV-array inspections over large solar plants where photogrammetric UAV methods suffer from data bloat and resolution issues at altitude. It introduces a visual servoing framework tightly coupled with nonlinear model predictive control ($NMPC$) to keep the UAV over the PV array center at low altitude while respecting velocity and height constraints, using RGB-D line-edge features and Kalman-filtered edge estimates. The main contributions are: (i) a VS controller capable of operating within a constrained workspace via NMPC, (ii) a lightweight RGB-D line-edge feature extraction with Kalman filtering, and (iii) end-to-end validation on a DJI Matrice 100 in both simulation and real PV-site flights with real-time performance around $18.1$ ms per iteration. The results demonstrate accurate feature tracking and high-quality image capture suitable for expert or AI-based PV defect inspection, highlighting practical relevance for scalable maintenance of solar farms. This work paves the way for integrated perception and control (Perception NMPC) in autonomous UAV PV-inspection pipelines.

Abstract

The photovoltaic (PV) industry is seeing a significant shift toward large-scale solar plants, where traditional inspection methods have proven to be time-consuming and costly. Currently, the predominant approach to PV inspection using unmanned aerial vehicles (UAVs) is based on photogrammetry. However, the photogrammetry approach presents limitations, such as an increased amount of useless data during flights, potential issues related to image resolution, and the detection process during high-altitude flights. In this work, we develop a visual servoing control system applied to a UAV with dynamic compensation using a nonlinear model predictive control (NMPC) capable of accurately tracking the middle of the underlying PV array at different frontal velocities and height constraints, ensuring the acquisition of detailed images during low-altitude flights. The visual servoing controller is based on the extraction of features using RGB-D images and the Kalman filter to estimate the edges of the PV arrays. Furthermore, this work demonstrates the proposal in both simulated and real-world environments using the commercial aerial vehicle (DJI Matrice 100), with the purpose of showcasing the results of the architecture. Our approach is available for the scientific community in: https://github.com/EPVelasco/VisualServoing_NMPC

Figures (8)

• Figure 1: Pipeline of the system implemented for overflight and inspection of the PV arrays with the visual servoing controller combined with dynamic compensation and constraints based on NMPC using UAVs. The purple area shows the UAV system in conjunction with the test environment. Visual servo controllers and NMPC are explained in Algorithms \ref{['alg:knematics']} and \ref{['alg:nmpc']}, respectively. The orange area shows the Feature Extraction from the RGB-D images and is explained in Algorithm \ref{['alg:kalman_filter']}).
• Figure 2: Reference coordinate frames of the implemented system.
• Figure 3: UAV flying over the PV arrays. Figures a and d represent the simulated and real experimental environment, respectively. The bottom images b, c and d, f are the color and depth images of the PV array obtained by simulation as well as by the UAV on-board camera.
• Figure 4: Simulation results of the visual servoing controller with different values of matrix $\mathbf{W}$.
• Figure 5: Comparative results of the visual servoing controller versus the visual servoing controller combined with NMPC in a simulated environment.