Towards Multi-View Consistent Style Transfer with One-Step Diffusion via Vision Conditioning

TL;DR

Experiments show that the proposed novel style transfer method, OSDiffST, surpasses other promising style transfer methods in synthesizing various styles for multi-view images of 3D scenes, with superior visual quality, with better structural integrity and less distortion.

Abstract

The stylization of 3D scenes is an increasingly attractive topic in 3D vision. Although image style transfer has been extensively researched with promising results, directly applying 2D style transfer methods to 3D scenes often fails to preserve the structural and multi-view properties of 3D environments, resulting in unpleasant distortions in images from different viewpoints. To address these issues, we leverage the remarkable generative prior of diffusion-based models and propose a novel style transfer method, OSDiffST, based on a pre-trained one-step diffusion model (i.e., SD-Turbo) for rendering diverse styles in multi-view images of 3D scenes. To efficiently adapt the pre-trained model for multi-view style transfer on small datasets, we introduce a vision condition module to extract style information from the reference style image to serve as conditional input for the diffusion model and employ LoRA in diffusion model for adaptation. Additionally, we consider color distribution alignment and structural similarity between the stylized and content images using two specific loss functions. As a result, our method effectively preserves the structural information and multi-view consistency in stylized images without any 3D information. Experiments show that our method surpasses other promising style transfer methods in synthesizing various styles for multi-view images of 3D scenes. Stylized images from different viewpoints generated by our method achieve superior visual quality, with better structural integrity and less distortion. The source code is available at https://github.com/YushenZuo/OSDiffST.

Figures (7)

• Figure 1: Overview of OSDiffST. The OSDiffST framework consists of two main parts: (1) Vision Condition Module, which includes a pre-trained CLIP Image Encoder $\mathcal{E}_{I}$ and a Vision Language Projector $\mathcal{P}_{\text{VL}}$. This module is used to extract the conditional embedding $c_{\text{style}}$ from the style image $I_{\text{s}}$. (2) Generative Backbone $\mathcal{S}$, which includes a pre-trained one-step stable diffusion model and LoRA adapters. This module is responsible for generating the stylized image $I_{\text{cs}}$ from the input content image $I_{\text{c}}$ and the conditional embedding $c_{\text{style}}$.
• Figure 2: Experimental results of different style transfer methods. We use two objective metrics and two subjective metrics to evaluate stylized images. 'CHD' and 'DSD' are used as objective metrics. For subjective metrics, we calculate the average score of 'Content Preservation' and 'Stylization' from the user study, which denoted as 'Avg. Content Preservation' and 'Avg. Stylization', respectively. Metrics are shown in 'CHD' / 'DSD' / 'Avg. Content Preservation' / 'Avg. Stylization'. The best and second-best results are highlighted in red and blue, respectively.
• Figure 3: Stylized results of different methods in a multi-view scenario. 'L1 dist.' denotes the L1 distance between the forward flow from stylized images and content images.
• Figure 4: Stylized results from different configurations of the color alignment loss in model training.
• Figure 5: Stylized results from different configurations of generating condition embedding.