A New Dataset and Performance Benchmark for Real-time Spacecraft Segmentation in Onboard Computers

TL;DR

The paper addresses the challenge of real-time, onboard spacecraft segmentation under strict hardware limits by introducing SWiM, a large, diverse dataset that combines PoseBowl and Spacecrafts with two synthetic-image pipelines (background-augmented composition and TTALOS+Stable Diffusion rendering). It adopts a dual-metric evaluation using Dice and Hausdorff distance and benchmarks CPU-only YOLOv8n and YOLOv11n models under NASA-like constraints, achieving Dice around $0.92$ and Hausdorff around $0.69$ with inference times near $0.5$ s per image. The contributions include the first hardware-constrained segmentation benchmark for orbital vision, two synthetic-generation methods, a detailed dataset split into Baseline and Augmented versions, and open-source release of the SWiM dataset and performance tools. This work enables standardized, real-world evaluation of real-time onboard inspection systems, paving the way for robust autonomous servicing and proximity-operations in space, while highlighting limitations in SAM2-based mask generation for high-FoV cases.

Abstract

Spacecraft deployed in outer space are routinely subjected to various forms of damage due to exposure to hazardous environments. In addition, there are significant risks to the subsequent process of in-space repairs through human extravehicular activity or robotic manipulation, incurring substantial operational costs. Recent developments in image segmentation could enable the development of reliable and cost-effective autonomous inspection systems. While these models often require large amounts of training data to achieve satisfactory results, publicly available annotated spacecraft segmentation data are very scarce. Here, we present a new dataset of nearly 64k annotated spacecraft images that was created using real spacecraft models, superimposed on a mixture of real and synthetic backgrounds generated using NASA's TTALOS pipeline. To mimic camera distortions and noise in real-world image acquisition, we also added different types of noise and distortion to the images. Our dataset includes images with several real-world challenges, including noise, camera distortions, glare, varying lighting conditions, varying field of view, partial spacecraft visibility, brightly-lit city backgrounds, densely patterned and confounding backgrounds, aurora borealis, and a wide variety of spacecraft geometries. Finally, we finetuned YOLOv8 and YOLOv11 models for spacecraft segmentation to generate performance benchmarks for the dataset under well-defined hardware and inference time constraints to mimic real-world image segmentation challenges for real-time onboard applications in space on NASA's inspector spacecraft. The resulting models, when tested under these constraints, achieved a Dice score of 0.92, Hausdorff distance of 0.69, and an inference time of about 0.5 second. The dataset and models for performance benchmark are available at https://github.com/RiceD2KLab/SWiM.

Figures (5)

• Figure 1: Example showing segmentation mask generation using SAM 2 for an image in the PoseBowl dataset.
• Figure 2: Example showing resizing an image and merging three segmentation masks into one mask for the Spacecrafts dataset.
• Figure 3: Number of images in the baseline and augmented versions of SWiM.
• Figure 4: Sample synthetic images in the augmented dataset. Note the diversity in background and the real-world challenges, such as glare and confounding background.
• Figure 5: Overview of the YOLO Nano model architecture. The diagram illustrates the model's key components, including the backbone, the neck, and the four decoupled heads (classification, localization, objectness, and segmentation). The input image (resized to 640×640×3) passes through the backbone and neck, where features are processed and routed to the respective heads.