AI-Powered Augmented Reality for Satellite Assembly, Integration and Test

TL;DR

A technical description of the ESA project “AI for AR in Satellite AIT” which combines real-time computer vision and AR systems to assist technicians during satellite assembly, demonstrating the efficacy of AI-driven AR systems in automating critical satellite assembly tasks.

Abstract

The integration of Artificial Intelligence (AI) and Augmented Reality (AR) is set to transform satellite Assembly, Integration, and Testing (AIT) processes by enhancing precision, minimizing human error, and improving operational efficiency in cleanroom environments. This paper presents a technical description of the European Space Agency's (ESA) project "AI for AR in Satellite AIT," which combines real-time computer vision and AR systems to assist technicians during satellite assembly. Leveraging Microsoft HoloLens 2 as the AR interface, the system delivers context-aware instructions and real-time feedback, tackling the complexities of object recognition and 6D pose estimation in AIT workflows. All AI models demonstrated over 70% accuracy, with the detection model exceeding 95% accuracy, indicating a high level of performance and reliability. A key contribution of this work lies in the effective use of synthetic data for training AI models in AR applications, addressing the significant challenges of obtaining real-world datasets in highly dynamic satellite environments, as well as the creation of the Segmented Anything Model for Automatic Labelling (SAMAL), which facilitates the automatic annotation of real data, achieving speeds up to 20 times faster than manual human annotation. The findings demonstrate the efficacy of AI-driven AR systems in automating critical satellite assembly tasks, setting a foundation for future innovations in the space industry.

Figures (7)

• Figure 1: Demonstrative outputs of the AI-driven augmented reality system for satellite assembly procedures include color-coded bounding boxes: yellow, green, and blue indicate object detection, the red 3D bounding box denotes 6D pose identification, and the pink bounding box represents OCR for measurement instruments.
• Figure 2: The system architecture consists of green modules for processing on the computer, red blocks for communication, and a pink block in the headset for acquiring input data. Gray arrows represent data streams, while black arrows indicate internal data flows.
• Figure 3: Illustrative image of a technician utilizing the headset during operational procedures.
• Figure 4: SAMAL is an advanced annotation tool that accurately generates bounding boxes around objects. It effectively handles occlusions, such as user hands, and adapts to changes in object positions, making it especially valuable for augmented reality applications where hands frequently interact with objects.
• Figure 5: The chart compares the time required for manual annotation of 459 images versus SAMAL. While manual annotation took 72 minutes, SAMAL completed the same task in just 3 minutes and 20 seconds, demonstrating a substantial increase in efficiency.