Leveraging Neural Radiance Fields for Pose Estimation of an Unknown Space Object during Proximity Operations

TL;DR

This work tackles autonomous rendezvous with unknown spacecraft targets by enabling off-the-shelf 6D pose estimation trained without a CAD model. It leverages an in-the-wild Neural Radiance Field with learnable appearance embeddings to synthesize a large, illumination-diverse training set from a sparse on-ground image collection, which is then used to train a model-based SPE network. On SPEED+ Hardware-In-The-Loop data, the NeRF-generated training approach outperforms baselines trained on few real images and approaches, or matches, CAD-based synthetic data in pose accuracy. The method demonstrates a CAD-free, model-agnostic path to robust on-board pose estimation for proximity operations, with the illumination-diversity strategy via appearance embeddings being a critical factor.

Abstract

We address the estimation of the 6D pose of an unknown target spacecraft relative to a monocular camera, a key step towards the autonomous rendezvous and proximity operations required by future Active Debris Removal missions. We present a novel method that enables an "off-the-shelf" spacecraft pose estimator, which is supposed to known the target CAD model, to be applied on an unknown target. Our method relies on an in-the wild NeRF, i.e., a Neural Radiance Field that employs learnable appearance embeddings to represent varying illumination conditions found in natural scenes. We train the NeRF model using a sparse collection of images that depict the target, and in turn generate a large dataset that is diverse both in terms of viewpoint and illumination. This dataset is then used to train the pose estimation network. We validate our method on the Hardware-In-the-Loop images of SPEED+ that emulate lighting conditions close to those encountered on orbit. We demonstrate that our method successfully enables the training of an off-the-shelf spacecraft pose estimation network from a sparse set of images. Furthermore, we show that a network trained using our method performs similarly to a model trained on synthetic images generated using the CAD model of the target.

Figures (12)

• Figure 1: Overview of the considered three-steps operation. (1) The chaser spacecraft (C) approaches the target (T) while taking pictures. (2) The images are downloaded on ground and processed to train a Spacecraft Pose Estimation network, whose weights are uploaded on the chaser. (3) The chaser finishes the operation autonomously, by relying on the trained network.
• Figure 2: Overview of the generation of an image $I'= m_{\Phi}(q,t)$ through an in-the-wild Neural Radiance Field martin2021nerffridovich2023k$m_{\Phi}$ for a given pose $(q,t)$. As explained in \ref{['sec_nerf_background']}, given a pose, i.e. rotation $q$ and translation $t$, the NeRF renders an image $I'$ by querying multiple times a MLP, which approximates the radiance field, and by aggregating the predicted color and density of the points through ray-tracing techniques. Unlike most NeRFs where the MLP takes as input a 3D position and 2 viewing angles to output the color and density of each point, In-the-wild NeRFs martin2021nerf also feeds the MLP with a learnable appearance embedding which is specific to each image. This offers the possibility to change the illumination conditions when synthesizing new images (see \ref{['sec_appearance_embeddings']}).
• Figure 3: On-ground processing pipeline. A subset of the images downloaded from the spacecraft is annotated and used to train a NeRF mildenhall2021nerf, $m_{\Phi}$. This radiance field is then used to render a large training set, which is exploited to train an off-the-shelf SPE network $f_{\Theta}$. Finally, the network weights, $\Theta$, are uploaded on the spacecraft. Variables are defined in the text.
• Figure 4: Images rendered by the same NeRF using 4 different appearance embeddings (columns 1-4). Randomizing the appearance embeddings enables the generation of images with diverse illumination conditions, regarding both the intensity and the orientation of the illumination source.
• Figure 5: (Top) Original images. (Bottom) Images after pre-processing. For each image, the average intensity of the background is subtracted and the range is extended so that the maximal intensity of both images is unchanged.