PNeSM: Arbitrary 3D Scene Stylization via Prompt-Based Neural Style Mapping

TL;DR

A novel 3D scene stylization framework to transfer an arbitrary style to an arbitrary scene, without any style-related or scene-related re- training is proposed, which is superior to SOTA methods in both visual qual- ity and generalization.

Abstract

3D scene stylization refers to transform the appearance of a 3D scene to match a given style image, ensuring that images rendered from different viewpoints exhibit the same style as the given style image, while maintaining the 3D consistency of the stylized scene. Several existing methods have obtained impressive results in stylizing 3D scenes. However, the models proposed by these methods need to be re-trained when applied to a new scene. In other words, their models are coupled with a specific scene and cannot adapt to arbitrary other scenes. To address this issue, we propose a novel 3D scene stylization framework to transfer an arbitrary style to an arbitrary scene, without any style-related or scene-related re-training. Concretely, we first map the appearance of the 3D scene into a 2D style pattern space, which realizes complete disentanglement of the geometry and appearance of the 3D scene and makes our model be generalized to arbitrary 3D scenes. Then we stylize the appearance of the 3D scene in the 2D style pattern space via a prompt-based 2D stylization algorithm. Experimental results demonstrate that our proposed framework is superior to SOTA methods in both visual quality and generalization.

Figures (6)

• Figure 1: An overview of our method. (a) UV mapping is designed to support the complete disentanglement of geometry and appearance, which maps a 3D coordinate to a style pattern coordinate. (b) To reconstruct the original appearance, we use appearance mapping to map the style pattern coordinate along with the view direction to the radiance color $\mathbf{c_r}$. (c) A pre-trained image stylization network integrated with a visual prompt is used for stylization mapping, stylizing the appearance of the scene in the 2D style pattern space.
• Figure 2: With the cycle loss $L_{cycle}$, we encourage the bijective mapping between real-world 3D coordinate and 2D style pattern coordinate.
• Figure 3: Qualitative comparisons on LLFF dataset. We compare our method to StylizedNeRF huang2022stylizednerf, ARF zhang2022arf, INS fan2022unified and StyleRF liu2023stylerf. Our method stylizes scenes with clear geometry and competitive stylization quality.
• Figure 4: Qualitative comparisons on Tanks and Temples dataset. We compare our method to LSNV huang2021learning, Stylizing-3D-Scene chiang2022stylizing and ARF zhang2022arf. Stylized scenes generated by our method contain both precise geometry and pleasant stylization.
• Figure 5: Ablation study on direct image stylization on reconstruction appearance. (a) The results of using reconstruction appearance cubemap as content input for image stylization. (b) The results of our method (using a noise image as content input and add a visual prompt in the bottleneck of image stylization network.)