Neural Harmonic Textures for High-Quality Primitive Based Neural Reconstruction

Abstract

Primitive-based methods such as 3D Gaussian Splatting have recently become the state-of-the-art for novel-view synthesis and related reconstruction tasks. Compared to neural fields, these representations are more flexible, adaptive, and scale better to large scenes. However, the limited expressivity of individual primitives makes modeling high-frequency detail challenging. We introduce Neural Harmonic Textures, a neural representation approach that anchors latent feature vectors on a virtual scaffold surrounding each primitive. These features are interpolated within the primitive at ray intersection points. Inspired by Fourier analysis, we apply periodic activations to the interpolated features, turning alpha blending into a weighted sum of harmonic components. The resulting signal is then decoded in a single deferred pass using a small neural network, significantly reducing computational cost. Neural Harmonic Textures yield state-of-the-art results in real-time novel view synthesis while bridging the gap between primitive- and neural-field-based reconstruction. Our method integrates seamlessly into existing primitive-based pipelines such as 3DGUT, Triangle Splatting, and 2DGS. We further demonstrate its generality with applications to 2D image fitting and semantic reconstruction.

Figures (10)

• Figure 1: Neural Harmonic Textures for novel view synthesis. We attach learnable feature vectors (right) to the virtual vertices of bounding tetrahedra encapsulating each primitive (center). After harmonic encoding and accumulation along the ray, a small neural network decodes the resulting signal into RGB color in a deferred manner (left).
• Figure 2: Neural Harmonic Textures applied to novel-view synthesis. We virtually attach feature vectors $\mathbf{f}_i$ to the vertices of tetrahedra inscribing the Gaussian primitives. Following 3DGUT 3dgut, we evaluate the point along the ray where the projected Gaussian has maximum response. We barycentrically interpolate vertex features at that point, and encode them with sine and cosine functions into different channels. These are then alpha blended along the rest of the ray, until the resulting sum of harmonics is decoded by a shallow MLP in a single image-space pass.
• Figure 3: Illustrating our method in 2D. Each primitive is bounded by an ellipsoid in world space, which becomes a sphere in whitened canonical space (a). Considering a virtual bounding tetrahedron in this canonical space, we attach one N-dimensional feature vector $\mathbf{f}^j$ to each vertex. The primitive's contribution is evaluated at the point of maximum response $\mathbf{p}^*$ of the projected Gaussian along the intersecting ray (b). The feature vectors are barycentrically interpolated at $\mathbf{p}^*$ and encoded with sine and cosine periodic functions (c).
• Figure 4: Harmonic textures.
• Figure 5: Generality of our approach: Neural Harmonic Textures applied to different primitive-based representations, evaluated on MipNeRF360 barron2022mipnerf360.