CNN-Based Camera Pose Estimation and Localisation of Scan Images for Aircraft Visual Inspection

TL;DR

This work tackles the challenge of infrastructure-free camera pose estimation for aircraft exterior inspection on the airport tarmac. It introduces an end-to-end workflow that uses a PTZ camera, synthetic data generated with domain randomisation, and a geometry-aware loss (ICSC) to estimate the camera pose relative to an aircraft from a single image, enabling on-site localisation of inspection scans. A scan-path generation method translates the estimated pose into pan-tilt commands, producing 3D-labelled scan images by interpolating the aircraft surface model and mapping surface points to camera viewpoints. The method demonstrates sim-to-real transfer with real A320 data, achieving median pose errors below 0.24 m and orientation errors below 2°, and provides a practical, low-infrastructure solution for efficient visual inspection and defect localisation.

Abstract

General Visual Inspection is a manual inspection process regularly used to detect and localise obvious damage on the exterior of commercial aircraft. There has been increasing demand to perform this process at the boarding gate to minimise the downtime of the aircraft and automating this process is desired to reduce the reliance on human labour. Automating this typically requires estimating a camera's pose with respect to the aircraft for initialisation but most existing localisation methods require infrastructure, which is very challenging in uncontrolled outdoor environments and within the limited turnover time (approximately 2 hours) on an airport tarmac. Additionally, many airlines and airports do not allow contact with the aircraft's surface or using UAVs for inspection between flights, and restrict access to commercial aircraft. Hence, this paper proposes an on-site method that is infrastructure-free and easy to deploy for estimating a pan-tilt-zoom camera's pose and localising scan images. This method initialises using the same pan-tilt-zoom camera used for the inspection task by utilising a Deep Convolutional Neural Network fine-tuned on only synthetic images to predict its own pose. We apply domain randomisation to generate the dataset for fine-tuning the network and modify its loss function by leveraging aircraft geometry to improve accuracy. We also propose a workflow for initialisation, scan path planning, and precise localisation of images captured from a pan-tilt-zoom camera. We evaluate and demonstrate our approach through experiments with real aircraft, achieving root-mean-square camera pose estimation errors of less than 0.24 m and 2 degrees for all real scenes.

Figures (12)

• Figure 1: We propose an infrastructure-free and contact-less method to estimate a PTZ camera's pose with respect to an aircraft and generate a scan path to obtain scan images labelled with their location on the aircraft's surface.
• Figure 2: Overview of our proposed workflow for scanning and localising images for aircraft visual inspection on an airport tarmac between flights. Our method is easy to deploy and does not require additional infrastructure or attaching sensors or markers onto the aircraft for camera pose estimation (CPE).
• Figure 3: Summary of deep learning approach.
• Figure 4: Aircraft quadrants and proposed boundary for PTZ camera position and axes direction. Windows and pylons of an A320 are used as a visual guide to position the PTZ camera within the proposed boundary.
• Figure 5: Proposed 3D boundary for PTZ camera position in quadrant 3 with direction of axes.