GaussianPainter: Painting Point Cloud into 3D Gaussians with Normal Guidance

TL;DR

GaussianPainter addressess the non-uniqueness problem in 3D Gaussian Splatting by introducing normal-guided rotations and a multi-scale triplane appearance injection, enabling efficient feed-forward painting of point clouds into high-quality 3D Gaussians from a reference image. The method achieves state-of-the-art novel-view synthesis and cross-object appearance transfer on OmniObject3D and Objaverse while maintaining efficiency with a single forward pass. By constraining the Gaussian parameter space via predicted normals and bridging 2D-3D appearance through triplanes, GaussianPainter enables robust, diverse 3D content creation from point clouds. The work paves the way for scalable, controllable 3D content generation, with potential extensions to scene-scale data and text-driven appearance control.

Abstract

In this paper, we present GaussianPainter, the first method to paint a point cloud into 3D Gaussians given a reference image. GaussianPainter introduces an innovative feed-forward approach to overcome the limitations of time-consuming test-time optimization in 3D Gaussian splatting. Our method addresses a critical challenge in the field: the non-uniqueness problem inherent in the large parameter space of 3D Gaussian splatting. This space, encompassing rotation, anisotropic scales, and spherical harmonic coefficients, introduces the challenge of rendering similar images from substantially different Gaussian fields. As a result, feed-forward networks face instability when attempting to directly predict high-quality Gaussian fields, struggling to converge on consistent parameters for a given output. To address this issue, we propose to estimate a surface normal for each point to determine its Gaussian rotation. This strategy enables the network to effectively predict the remaining Gaussian parameters in the constrained space. We further enhance our approach with an appearance injection module, incorporating reference image appearance into Gaussian fields via a multiscale triplane representation. Our method successfully balances efficiency and fidelity in 3D Gaussian generation, achieving high-quality, diverse, and robust 3D content creation from point clouds in a single forward pass.

Figures (10)

• Figure 1: Given any reference image, the GaussianPainter paints point clouds into 3D Gaussians in a feed-forward network.
• Figure 2: The overview of GaussianPainter. Its first major component is Normal-guided Gaussian Painting, consisting of Point Cloud Encoder, Appearance/Normal Decoder, and the following MLPs. This component directly operates on point clouds and predicts Gaussian parameters and normal for each point, presented in Sec. \ref{['sec:gs_prediction']}. The second major component is Triplane-based Multiscale Appearance Injection, which injects appearance information from reference image into the Appearance Decoder and guides Gaussian painting, presented in Sec. \ref{['sec:injection']}.
• Figure 3: Demonstration of different Gaussian fields and Instability Score for different Gaussian parameters. SH 1 and SH 2 stand for the RGB component and the remaining component of harmonic coefficients, respectively.
• Figure 4: The illustration of converting the predicted normal to a rotation. The $xyz$-axes are rotated to $x^\prime y^\prime z^\prime$-axes, which is considered as the rotation parameter of a Gaussian.
• Figure 5: Qualitatively comparison for the task of cross-object appearance transfer on Objaverse. Our method demonstrates the superiority in transferring reasonable appearance to another object.