How Important are Data Augmentations to Close the Domain Gap for Object Detection in Orbit?

TL;DR

An large scale experiment using a hyperparameter optimization pipeline that samples hundreds of different configurations and searches for the best set to bridge the domain gap is conducted, and two novel data augmentations specifically developed to emulate the visual effects observed in orbital imagery are proposed.

Abstract

We investigate the efficacy of data augmentations to close the domain gap in spaceborne computer vision, crucial for autonomous operations like on-orbit servicing. As the use of computer vision in space increases, challenges such as hostile illumination and low signal-to-noise ratios significantly hinder performance. While learning-based algorithms show promising results, their adoption is limited by the need for extensive annotated training data and the domain gap that arises from differences between synthesized and real-world imagery. This study explores domain generalization in terms of data augmentations -- classical color and geometric transformations, corruptions, and noise -- to enhance model performance across the domain gap. To this end, we conduct an large scale experiment using a hyperparameter optimization pipeline that samples hundreds of different configurations and searches for the best set to bridge the domain gap. As a reference task, we use 2D object detection and evaluate on the SPEED+ dataset that contains real hardware-in-the-loop satellite images in its test set. Moreover, we evaluate four popular object detectors, including Mask R-CNN, Faster R-CNN, YOLO-v7, and the open set detector GroundingDINO, and highlight their trade-offs between performance, inference speed, and training time. Our results underscore the vital role of data augmentations in bridging the domain gap, improving model performance, robustness, and reliability for critical space applications. As a result, we propose two novel data augmentations specifically developed to emulate the visual effects observed in orbital imagery. We conclude by recommending the most effective augmentations for advancing computer vision in challenging orbital environments. Code for training detectors and hyperparameter search will be made publicly available.

Figures (5)

• Figure 1: Impact of data augmentations on satellite object detection to bridge the domain gap. We observe that detectors predict more precise bounding box on real-world images when trained with more augmentation techniques applied during the training on synthetic images.
• Figure 2: Overview on our hyperparameter optimization strategy for finding the set of data augmentations yielding the best precision when trained on synthetic and tested on real-world orbital satellite images. To this end, we train a detection model with varying augmentations techniques, report an evaluation score that informs the next set of augmentations to experiment with in the consecutive trial.
• Figure 3: Overview on all our possible data augmentations, applied to the same source image (leftmost image, in the first row).
• Figure 4: Visualization of our custom space data augmentations, simulating typical visual conditions encountered on objects in orbit: stark shadow effects and strong specular reflections. Source images displayed in the left, augmented ones in the right columns.
• Figure 5: Importance and variance of individual data augmentations on the different models' precision performance. We observe varying key augmentations for each detection model. Furthermore, network size seems to impact whether severe image changing techniques benefit a model, with the larger yolo-v7 being the only detector for which those augmentations are more influential.