Parameterization-driven Neural Surface Reconstruction for Object-oriented Editing in Neural Rendering

TL;DR

This work tackles intuitive editing of neural implicit surfaces within neural rendering by introducing a learnable parameterization to simple parametric domains such as spheres and polycubes. It couples a bi-directional deformation between the target surface and the parametric domain with cycle and Laplacian regularization to achieve nearly bijective mappings while controlling angle distortion, and it decomposes the radiance into view-independent material and view-dependent shading for easy editing. The approach is end-to-end and integrates with existing neural rendering pipelines, enabling 3D geometry reconstruction from multi-view images and immediate texture/shading edits without re-training, plus co-parameterization and texture transfer across objects of similar geometry. Experimental results on human heads and man-made objects demonstrate high-quality parameterizations with reduced distortion and effective pixel-level editing, though reconstruction quality lags slightly behind state-of-the-art SDF-based methods in some metrics, and the shading model makes simplifying assumptions about reflectance.

Abstract

The advancements in neural rendering have increased the need for techniques that enable intuitive editing of 3D objects represented as neural implicit surfaces. This paper introduces a novel neural algorithm for parameterizing neural implicit surfaces to simple parametric domains like spheres and polycubes. Our method allows users to specify the number of cubes in the parametric domain, learning a configuration that closely resembles the target 3D object's geometry. It computes bi-directional deformation between the object and the domain using a forward mapping from the object's zero level set and an inverse deformation for backward mapping. We ensure nearly bijective mapping with a cycle loss and optimize deformation smoothness. The parameterization quality, assessed by angle and area distortions, is guaranteed using a Laplacian regularizer and an optimized learned parametric domain. Our framework integrates with existing neural rendering pipelines, using multi-view images of a single object or multiple objects of similar geometries to reconstruct 3D geometry and compute texture maps automatically, eliminating the need for any prior information. We demonstrate the method's effectiveness on images of human heads and man-made objects.

Figures (15)

• Figure 1: Algorithmic pipeline. Initially, we learn the parametric domain $F_\text{sdf}$ using a coarse SDF $G_\text{sdf}$ of the target surface $\mathcal{S}$ during the early phase of neural rendering (in the left grey box). Once the parametric domain $\mathcal{D}$ is determined, our pipeline subsequently utilizes the bi-directional deformation $\left(F_{\text{def}} \text{~and~} F_{\text{inv-def}}\right)$, the parametric domain SDF $\left(F_{\text{sdf}}\right)$ and the radiance decomposition $\left(F_{\text{mat}} \text{~and~} F_\text{shd}\right)$. The output includes the reconstructed 3D surface $\mathcal{S}$, the decomposed radiance fields (featuring both view-dependent and view-independent components) and the map between $\mathcal{S}$ and $\mathcal{D}$. It is worth mentioning that while our framework involves two SDFs - one for the 3D surface $\mathcal{S}$ and the other for the parametric domain $\mathcal{D}$ - our network design only requires one of them. This is because one SDF can be derived from the other via either the forward or the backward deformation. To reduce the network complexity, we only adopt one geometry sub-network $F_{\text{sdf}}$, which is used to represent the parametric domain $\mathcal{D}$.
• Figure 1: Qualitative comparison with other neural parameterization methods. V.R. and D.R. stand for volume rendering and differentiable rendering, respectively.
• Figure 2: Parameterization results. Human heads are co-parameterized to a sphere owing to the clear geometric resemblance. Given the diverse geometry of cars, each is parameterized to a polycube domain (bottom three rows). From top to bottom: the input image, the normal map of the reconstructed geometry $\mathcal{S}$, the parametric domain $\mathcal{D}$, and the reconstructed surface $\mathcal{S}$. See supplementary material for additional results.
• Figure 3: Results of cross-polycube parameterization on the H3DS dataset ramon2021h3d. We learned a common polycube domain for all human heads with the parameter $k=3$ (i.e., the polycube is formed by 3 boxes) and then computed the polycube parameterization for all the human heads simultaneously. Since human heads share similar geometries, the computed polycube parameterizations are also highly consistent. The values below each figure represent the angle distortion and the area distortion, respectively.
• Figure 4: Parametrization results of NeP ma2022neural in comparison with our method. NeP heavily relies on the initial UV prior, leading to poor texture maps when such prior information is lacking. Our method learns texture maps from a set of multi-view images without any 3D tracked mesh or UV prior as input.