ABC-GS: Alignment-Based Controllable Style Transfer for 3D Gaussian Splatting

TL;DR

ABC-GS tackles 3D style transfer for explicit 3D Gaussian Splatting representations, addressing NNFM's neglect of global style and the editability issues of implicit NeRFs. The method combines a controllable matching stage using semantic masks with a Feature Alignment Style Transfer (FAST) loss in the stylization stage, along with a color transformation and depth/regularization terms to preserve geometry. Experiments on LLFF and T&T with WikiArt and ARF styles show that ABC-GS achieves controllable, globally faithful stylization with strong multi-view consistency and real-time rendering. An ablation study confirms FAST's superiority over NNFM and related losses, and code is released for public use.

Abstract

3D scene stylization approaches based on Neural Radiance Fields (NeRF) achieve promising results by optimizing with Nearest Neighbor Feature Matching (NNFM) loss. However, NNFM loss does not consider global style information. In addition, the implicit representation of NeRF limits their fine-grained control over the resulting scenes. In this paper, we introduce ABC-GS, a novel framework based on 3D Gaussian Splatting to achieve high-quality 3D style transfer. To this end, a controllable matching stage is designed to achieve precise alignment between scene content and style features through segmentation masks. Moreover, a style transfer loss function based on feature alignment is proposed to ensure that the outcomes of style transfer accurately reflect the global style of the reference image. Furthermore, the original geometric information of the scene is preserved with the depth loss and Gaussian regularization terms. Extensive experiments show that our ABC-GS provides controllability of style transfer and achieves stylization results that are more faithfully aligned with the global style of the chosen artistic reference. Our homepage is available at https://vpx-ecnu.github.io/ABC-GS-website.

Figures (14)

• Figure 1: ABC-GS Pipeline. Given a set of content images and content masks, along with style images and style transfer type, our method first achieves a match between style and scene content through mask matching in the controllable matching stage. Subsequently, based on the matching results, color matching that is consistent with the perspective is performed between the content images and the scene Gaussians. In the stylization stage, we use the feature alignment style transfer loss to optimize the scene and introduce multiple loss terms to maintain the content and geometric information of the scene.
• Figure 2: Style Isolation. Only using either the style mask or the eroded style mask fails to prevent the leakage of the zebra texture style. Employing style isolation can effectively address this issue.
• Figure 3: FAST Loss and NNFM loss. For the calculation of FAST loss, it first tallys all pairs of k-nearest neighbors between the rendered features and the style features, which are jointly used to compute the alignment matrix $P$. Style transfer is achieved by minimizing the cosine distance between the rendered features and the aligned features. The NNFM loss directly implements style transfer by minimizing the cosine distance between each rendered feature and its nearest neighbor in the style feature.
• Figure 4: Qualitative comparisons with the baseline methods on LLFF datasets. Compared to other methods, our method better preserves the geometric information of the scene while achieving style transfer.
• Figure 5: Stylization result of compositional and semantic-aware style transfer. Our approach enables controllable style transfer.