Sharpening Neural Implicit Functions with Frequency Consolidation Priors

TL;DR

Neural implicit SDFs struggle to capture high-frequency geometry from low-frequency observations due to network bias and data sparsity. The authors propose frequency consolidation priors that learn a mapping from a low-frequency SDF $f_L$ to a full-frequency SDF $f_F$ using a two-branch, embedding-conditioned architecture with $\boldsymbol{e}_L=[\boldsymbol{e}_F,\boldsymbol{e}_C]$ and test-time self-reconstruction of $\boldsymbol{e}_F$ to generalize to unseen inputs. They train on paired observations $(M_L,M_F)$ derived from Poisson surface reconstruction and demonstrate state-of-the-art results on ShapeNet, ABC, and ScanNet, improving sharpness and completeness of surfaces from sparse data. This approach enables sharper 3D reconstructions in real-world scenarios and provides a practical, generalizable framework for recovering high-frequency details from low-frequency geometric cues.

Abstract

Signed Distance Functions (SDFs) are vital implicit representations to represent high fidelity 3D surfaces. Current methods mainly leverage a neural network to learn an SDF from various supervisions including signed distances, 3D point clouds, or multi-view images. However, due to various reasons including the bias of neural network on low frequency content, 3D unaware sampling, sparsity in point clouds, or low resolutions of images, neural implicit representations still struggle to represent geometries with high frequency components like sharp structures, especially for the ones learned from images or point clouds. To overcome this challenge, we introduce a method to sharpen a low frequency SDF observation by recovering its high frequency components, pursuing a sharper and more complete surface. Our key idea is to learn a mapping from a low frequency observation to a full frequency coverage in a data-driven manner, leading to a prior knowledge of shape consolidation in the frequency domain, dubbed frequency consolidation priors. To better generalize a learned prior to unseen shapes, we introduce to represent frequency components as embeddings and disentangle the embedding of the low frequency component from the embedding of the full frequency component. This disentanglement allows the prior to generalize on an unseen low frequency observation by simply recovering its full frequency embedding through a test-time self-reconstruction. Our evaluations under widely used benchmarks or real scenes show that our method can recover high frequency component and produce more accurate surfaces than the latest methods. The code, data, and pre-trained models are available at \url{https://github.com/chenchao15/FCP}.

Figures (12)

• Figure 1: The concept of frequency consolidation priors. We also show averaged frequency weights across a band.
• Figure 2: The overview of our method.
• Figure 3: The illustration of low frequency observations and the full frequency coverage.
• Figure 4: Over-smoothed surfaces.
• Figure 5: Visual comparison with the state-of-the-art on ShapeNet.