Computer vision tasks for intelligent aerospace missions: An overview

TL;DR

The paper surveys DL-based perception for intelligent aerospace missions across three core tasks—pose estimation, 3D reconstruction, and recognition—highlighting how deep learning offers robustness under harsh space conditions where traditional methods struggle. It reviews relevant datasets (e.g., SPEED, SPEED+, URSO) and evaluation metrics, and discusses foundational works and practical challenges such as generalization to real space imagery and real-time onboard deployment. It then outlines advances in DL-based 3D reconstruction (including NeRF and MVS variants) and recognition (with SPARK and related CNN/Transformer approaches), along with strategies to address data scarcity via transfer learning and few-shot learning. Finally, the paper discusses deployment considerations, downstream tasks like autonomous GNC and motion estimation, and future directions including multi-sensor fusion and AI-accelerator hardware to enable onboard intelligent perception.

Abstract

Computer vision tasks are crucial for aerospace missions as they help spacecraft to understand and interpret the space environment, such as estimating position and orientation, reconstructing 3D models, and recognizing objects, which have been extensively studied to successfully carry out the missions. However, traditional methods like Kalman Filtering, Structure from Motion, and Multi-View Stereo are not robust enough to handle harsh conditions, leading to unreliable results. In recent years, deep learning (DL)-based perception technologies have shown great potential and outperformed traditional methods, especially in terms of their robustness to changing environments. To further advance DL-based aerospace perception, various frameworks, datasets, and strategies have been proposed, indicating significant potential for future applications. In this survey, we aim to explore the promising techniques used in perception tasks and emphasize the importance of DL-based aerospace perception. We begin by providing an overview of aerospace perception, including classical space programs developed in recent years, commonly used sensors, and traditional perception methods. Subsequently, we delve into three fundamental perception tasks in aerospace missions: pose estimation, 3D reconstruction, and recognition, as they are basic and crucial for subsequent decision-making and control. Finally, we discuss the limitations and possibilities in current research and provide an outlook on future developments, including the challenges of working with limited datasets, the need for improved algorithms, and the potential benefits of multi-source information fusion.

Figures (7)

• Figure 1: A comprehensive workflow to guide the readers through a multi-tiered approach to space-based perception and mission execution. (a) A single agent is equipped with different sensors and collaborates with each other to observe the designated target. (b) Through inter-communication, agents can carry out various perception tasks in a distributed manner. (c) Perception is the prerequisite for many aerospace missions. Here we list a series of classical missions.
• Figure 2: Example pictures of (a) DEOS demonstration 14 and (b) E. Deorbit demonstration (reprinted from 36, copyright © ESA/Airbus Space/OHB Systems/Thales Alenia Space).
• Figure 3: Three datasets for DL-based pose estimation. SPEED: copyright © European Space Agency 2021, SPEED+: copyright © Stanford University, URSO: copyright © Pedro F. Proença.
• Figure 4: A description for the reference frames to compute the errors. Spacecraft body reference frame ($\mathcal{B}$), camera reference frame ($\mathcal{C}$), relative position ($t_\mathcal{BC}$), relative orientation ($R_\mathcal{BC}$).
• Figure 5: Experiment images from 37.