Event-RGB Fusion for Spacecraft Pose Estimation Under Harsh Lighting

TL;DR

This work targets robust spacecraft pose estimation under harsh lighting by fusing RGB and event sensor data. It introduces a dual-channel capture rig with a beam-splitter for precise optical-temporal alignment and a learning-free fusion pipeline built on cross-modal RANSAC, augmented with cross-modal keypoint distance and dropout-based uncertainty to handle edge cases. To support research, the authors create and publicly release the FRESH dataset, combining synthetic training data with real, aligned RGB and event sequences across multiple satellite models and lighting conditions. The results show that event-RGB fusion substantially improves pose-estimation reliability in challenging scenarios, reduces PnP failures, and remains practical for onboard or near-term autonomous missions, while highlighting limitations and avenues for future work such as adding more sensors and addressing symmetry-related ambiguities.

Abstract

Spacecraft pose estimation is crucial for autonomous in-space operations, such as rendezvous, docking and on-orbit servicing. Vision-based pose estimation methods, which typically employ RGB imaging sensors, is a compelling solution for spacecraft pose estimation, but are challenged by harsh lighting conditions, which produce imaging artifacts such as glare, over-exposure, blooming and lens flare. Due to their much higher dynamic range, neuromorphic or event sensors are more resilient to extreme lighting conditions. However, event sensors generally have lower spatial resolution and suffer from reduced signal-to-noise ratio during periods of low relative motion. This work addresses these individual sensor limitations by introducing a sensor fusion approach combining RGB and event sensors. A beam-splitter prism was employed to achieve precise optical and temporal alignment. Then, a RANSAC-based technique was developed to fuse the information from the RGB and event channels to achieve pose estimation that leveraged the strengths of the two modalities. The pipeline was complemented by dropout uncertainty estimation to detect extreme conditions that affect either channel. To benchmark the performance of the proposed event-RGB fusion method, we collected a comprehensive real dataset of RGB and event data for satellite pose estimation in a laboratory setting under a variety of challenging illumination conditions. Encouraging results on the dataset demonstrate the efficacy of our event-RGB fusion approach and further supports the usage of event sensors for spacecraft pose estimation. To support community research on this topic, our dataset has been released publicly.

Figures (42)

• Figure 1: A comparison of harsh lighting conditions in real docking and rendezvous scenarios issdockingteaser1soyuzdockingteaserissdockingteaser2 (top row) along with samples from our real dataset (RGB frames in the second row and optically aligned event frames in the third row). Even in the harshest example where most of the satellite is occluded by glare/over-exposure in the third row third column, an event camera is able to see the structure of the object.
• Figure 2: Fabricated real models of the target objects: Satty, Cassini and SOHO from left to right.
• Figure 3: Real frame from our dataset showing 4 possible ambiguous locations of the same keypoint in red due to the view of the object. The keypoint can be reflected along two axes of symmetry shown by dotted lines.
• Figure 4: (a) Side view of the event-RGB capture setup. (b) Top-down view of the capture setup. (c) View of the satellite object, backdrop and light source.
• Figure 5: Zoomed in view of a pair of misaligned RGB and event frame obtained from the consumer thread outlined in Sec. \ref{['sec:temporalsynchronisation']}. (c) shows the two misaligned frame overlaid on top of each other and (d) finally show the overlay following the alignment in Sec. \ref{['sec:opticalalignment']}.