Reg-NF: Efficient Registration of Implicit Surfaces within Neural Fields

TL;DR

Reg-NF addresses the challenge of registering two neural-field representations by estimating their relative $6$-DoF pose directly on implicit surfaces, even when the scene and object models differ in scale. It introduces a bidirectional loss over multi-view surface samples, automated initialisation, and regularisation to robustly align two SDF-based neural fields (NeuS), demonstrated on a new ONR dataset with extensive ablations. The approach enables applications such as library object substitution and instance replacement in robotics, and it shows improved robustness to scale and partial observations compared with prior methods like Nerf2nerf. Limitations center on initialization sensitivity and the assumption of known object-library mappings, pointing to future work on more general warping and initialisation strategies.

Abstract

Neural fields, coordinate-based neural networks, have recently gained popularity for implicitly representing a scene. In contrast to classical methods that are based on explicit representations such as point clouds, neural fields provide a continuous scene representation able to represent 3D geometry and appearance in a way which is compact and ideal for robotics applications. However, limited prior methods have investigated registering multiple neural fields by directly utilising these continuous implicit representations. In this paper, we present Reg-NF, a neural fields-based registration that optimises for the relative 6-DoF transformation between two arbitrary neural fields, even if those two fields have different scale factors. Key components of Reg-NF include a bidirectional registration loss, multi-view surface sampling, and utilisation of volumetric signed distance functions (SDFs). We showcase our approach on a new neural field dataset for evaluating registration problems. We provide an exhaustive set of experiments and ablation studies to identify the performance of our approach, while also discussing limitations to provide future direction to the research community on open challenges in utilizing neural fields in unconstrained environments.

Figures (7)

• Figure 1: The proposed pipeline for using Reg-NF registration. Object is detected in a scene neural field (NF) and matched to an object in a library of object-centric NF. Reg-NF performs registration between the two NFs enabling neural substitution of the library object NF into the scene or the replacement of the object with another from the library. Substitution and replacement models coloured for clarity.
• Figure 2: Overview of our Reg-NF registration process. Blue and orange denotes surface sample points from the scene and library NFs for a matched object respectively. Green points represent the target alignment during optimisation. After surface extraction and an initial registration estimate, bi-directional optimisation iterates till convergence. Final output is a 6-DoF transformation matrix between models.
• Figure 3: Overview of scenes and object models in ONR dataset. Chairs objects shown in ac-room, starting clockwise from bottom-left are: chair (c), fancy chair with no pillow (fc-nop), matrix chair (mc), dining chair (dc), and fancy chair (fc). Tables shown in atbl-room from left to right are: willow table (wt), end table (et), and table (t). Objects of interest in mix-room are fc, t, and dc.
• Figure 4: Example of library replacement using Reg-NF for all objects in all scenes evaluated in Table \ref{['tbl:n2n_compare']}. Top: Original scene NF render. Bottom: Scene NF with library object NF substitutions render. Substitutions are based on Reg-NF outputs. Note, colours are added to object NFs during render to provide visual distinction between scene NF and object NFs.
• Figure 5: Example of object completion via library replacement on a scene with low coverage. Original NF (a) is unable to correctly render the back of the object as it was not seen during training. (b) shows impact of library substitution from Reg-NF registration. Geometry of the object within the scene can be fully rendered from only partial initial view.