FlightScope: An Experimental Comparative Review of Aircraft Detection Algorithms in Satellite Imagery

TL;DR

The paper addresses aircraft detection in satellite imagery by benchmarking a broad set of state-of-the-art detectors (YOLOv5, YOLOv8, YOLOv10, Faster RCNN, CenterNet, RetinaNet, RTMDet, DETR, Grounding DINO) trained from scratch on the HRPlanesV2 dataset and evaluated on the GDIT dataset. It demonstrates that YOLOv5 consistently achieves the highest accuracy (mAP near 0.99 and strong mAP50) across subsets, with YOLOv8 and YOLOv10 closely following, while SSD underperforms in localization. The work provides a rigorous, reproducible benchmark framework (code at the linked GitHub), enabling researchers to compare detectors under satellite-imagery-specific conditions and facilitating open-world evaluation through cross-dataset validation. The findings inform detector selection for aerial surveillance and infrastructure monitoring and contribute to advancing remote sensing object detection methodologies.

Abstract

Object detection in remotely sensed satellite pictures is fundamental in many fields such as biophysical, and environmental monitoring. While deep learning algorithms are constantly evolving, they have been mostly implemented and tested on popular ground-based taken photos. This paper critically evaluates and compares a suite of advanced object detection algorithms customized for the task of identifying aircraft within satellite imagery. Using the large HRPlanesV2 dataset, together with a rigorous validation with the GDIT dataset, this research encompasses an array of methodologies including YOLO versions 5 and 8, Faster RCNN, CenterNet, RetinaNet, RTMDet, and DETR, all trained from scratch. This exhaustive training and validation study reveal YOLOv5 as the preeminent model for the specific case of identifying airplanes from remote sensing data, showcasing high precision and adaptability across diverse imaging conditions. This research highlight the nuanced performance landscapes of these algorithms, with YOLOv5 emerging as a robust solution for aerial object detection, underlining its importance through superior mean average precision, Recall, and Intersection over Union scores. The findings described here underscore the fundamental role of algorithm selection aligned with the specific demands of satellite imagery analysis and extend a comprehensive framework to evaluate model efficacy. The benchmark toolkit and codes, available via https://github.com/toelt-llc/FlightScope_Bench, aims to further exploration and innovation in the realm of remote sensing object detection, paving the way for improved analytical methodologies in satellite imagery applications.

Figures (16)

• Figure 1: Classification of object detection methods based on (1) their architecture (one-stage, two-stage, and Transformer network), and (2) detection accuracy (in yellow) and real-time detection (blue). The red dot highlights the models that are implemented, trained, and validated in this work. The green outline (indicated as 'Overlapping' in the image) groups the models that usually perform well in both accuracy and inference time response.
• Figure 2: Basic YOLO architecture. Reproduced from diwan2023object.
• Figure 3: SSD architecture diagram. Reproduced from 10.1007/978-3-319-46448-0_2.
• Figure 4: One-stage RetinaNet architecture. Reproduced from lin_focal_2020. (a) ResNet he2016deep backbone. (b) Generation of multi-scale convolutional pyramid. This is attached to two subnetworks: (c) anchor box classification and (d) anchor box regression to ground-truth bounding box.
• Figure 5: One-stage CenterNet architecture. Reproduced from duan_centernet_2019.