G-Style: Stylized Gaussian Splatting

TL;DR

𝒢‐Style, a novel algorithm designed to transfer the style of an image onto a 3D scene represented using Gaussian Splatting, is introduced, outperforming existing methods both qualitatively and quantitatively.

Abstract

We introduce G-Style, a novel algorithm designed to transfer the style of an image onto a 3D scene represented using Gaussian Splatting. Gaussian Splatting is a powerful 3D representation for novel view synthesis, as -- compared to other approaches based on Neural Radiance Fields -- it provides fast scene renderings and user control over the scene. Recent pre-prints have demonstrated that the style of Gaussian Splatting scenes can be modified using an image exemplar. However, since the scene geometry remains fixed during the stylization process, current solutions fall short of producing satisfactory results. Our algorithm aims to address these limitations by following a three-step process: In a pre-processing step, we remove undesirable Gaussians with large projection areas or highly elongated shapes. Subsequently, we combine several losses carefully designed to preserve different scales of the style in the image, while maintaining as much as possible the integrity of the original scene content. During the stylization process and following the original design of Gaussian Splatting, we split Gaussians where additional detail is necessary within our scene by tracking the gradient of the stylized color. Our experiments demonstrate that G-Style generates high-quality stylizations within just a few minutes, outperforming existing methods both qualitatively and quantitatively.

Figures (9)

• Figure 1: Overview of our method: We take a 3D scene represented using Gaussian Splatting and pre-process it to subdivide large Gaussians and normalize elongated ones. Initially, we perform a color matching between the ground truth images and the style image. Subsequently, we start the iterative stylization process. First, we optimize the colors of the Gaussians using multiple losses to capture style patterns at different scales while preserving the content of the scene. Then, we fine-tune the geometry of the scene and add details for Gaussians with a large gradient of the stylized color. We repeat the stylization and geometry fine-tuning steps until convergence. At the end, we perform an additional color matching step between the renderings of the resulting scene and the style image.
• Figure 2: Gaussian Normalization: We normalize (right) the size of narrow Gaussians to avoid multiple overlying Gaussians (left).
• Figure 3: Pre-processing: The effect of our pretraining step. Left: before pretraining, right: after pretraining. The scale of Gaussians is set to 0.25, otherwise both images would look identical.
• Figure 4: Results generated with our approach, $\mathscr{G}$-Style, for five forward-facing scenes (columns) given six style exemplars (rows).
• Figure 5: Results generated with our approach, $\mathscr{G}$-Style, for five 360° scenes (columns) given six style exemplars (rows).