FreBIS: Frequency-Based Stratification for Neural Implicit Surface Representations

TL;DR

FreBIS addresses the difficulty of capturing both coarse geometry and fine details in neural implicit surfaces by introducing frequency-based stratification with separate low-, middle-, and high-frequency encoders and a redundancy-aware weighting module to fuse their features. A decoder then predicts the SDF and appearance features, enabling accurate 3D surface reconstructions and view-dependent renderings when paired with existing backbones like VolSDF. Evaluations on BlendedMVS show consistent improvements in PSNR, SSIM, and LPIPS, along with higher-fidelity meshes, demonstrating the practical impact of frequency-aware multi-encoder representations for AR/VR and related 3D tasks.

Abstract

Neural implicit surface representation techniques are in high demand for advancing technologies in augmented reality/virtual reality, digital twins, autonomous navigation, and many other fields. With their ability to model object surfaces in a scene as a continuous function, such techniques have made remarkable strides recently, especially over classical 3D surface reconstruction methods, such as those that use voxels or point clouds. However, these methods struggle with scenes that have varied and complex surfaces principally because they model any given scene with a single encoder network that is tasked to capture all of low through high-surface frequency information in the scene simultaneously. In this work, we propose a novel, neural implicit surface representation approach called FreBIS to overcome this challenge. FreBIS works by stratifying the scene based on the frequency of surfaces into multiple frequency levels, with each level (or a group of levels) encoded by a dedicated encoder. Moreover, FreBIS encourages these encoders to capture complementary information by promoting mutual dissimilarity of the encoded features via a novel, redundancy-aware weighting module. Empirical evaluations on the challenging BlendedMVS dataset indicate that replacing the standard encoder in an off-the-shelf neural surface reconstruction method with our frequency-stratified encoders yields significant improvements. These enhancements are evident both in the quality of the reconstructed 3D surfaces and in the fidelity of their renderings from any viewpoint.

Figures (12)

• Figure 1: Overview of FreBIS: (a) Frequency-domain Representation: FreBIS works by mapping the input point coordinate to the frequency domain and encoding it via three frequency-band encoders -- one each for low, middle, and high. (b) Redundancy-aware Weighting: This module computes weights that indicate the importance of the three encoded features according to the dissimilarity of each to the other two. These weights are then used to combine the encoded features. The 3D surface is reconstructed by decoding the combined feature into a SDF value.
• Figure 2: FreBIS framework: Given an input 3D point $\boldsymbol{x}$, positional encoding maps it to the frequency domain. The output of the positional encoding is then encoded into latent feature vectors corresponding to low--, middle--, and high--frequencies, respectively ($\boldsymbol{f}_{\rm{L}}, \boldsymbol{f}_{\rm{M}}, \boldsymbol{f}_{\rm{H}}$) by leveraging our frequency-stratified encoders $\rm{Enc_L, Enc_M}$, and $\rm{Enc_H}$. The redundancy-aware weighting module takes the concatenated feature encodings ($\boldsymbol{F} = [\boldsymbol{f}_{\rm{L}}, \boldsymbol{f}_{\rm{M}}, \boldsymbol{f}_{\rm{H}}]$) and decides on the relative importance of these features according to the dissimilarity of each to the other two, estimating a normalized weight vector ($\boldsymbol{w}$). Finally, the weighted features ($\boldsymbol{F}\cdot\rm{diag}(\boldsymbol{w})$) are passed to a decoder $\rm{Dec}$ to extract a SDF value $d_{\Omega}$ and an appearance feature $\boldsymbol{f}_{\rm{RGB}}$ for the point $\boldsymbol{x}$. $\rm{MLP}_{\rm{RGB}}$ predicts $\boldsymbol{x}$'s color given the appearance feature, point position $\boldsymbol{x}$, view direction $\boldsymbol{v}$, and point normal $\nabla d_{\Omega}$.
• Figure 3: Redundancy-aware weighting module: The redundancy-aware weighting module takes the encoded frequency features and predicts a normalized importance score, following the pipeline shown in the figure, assigning a higher weight to the frequency encoding that is least similar to the other two and vice-versa.
• Figure 4: Qualitative comparison of viewpoint-based scene rendering on the BlendedMVS dataset.
• Figure 5: Qualitative comparison of surface reconstruction quality for the BlendedMVS dataset.