SPAST: Arbitrary Style Transfer with Style Priors via Pre-trained Large-scale Model

TL;DR

This paper tackles arbitrary style transfer by addressing the tradeoffs between small-model efficiency and large-model quality. It introduces SPAST, a two-stage framework where Stage 1 fine-tunes a pre-trained Stable Diffusion into an Artistic Stable Diffusion (ASD) to embed rich style priors, and Stage 2 trains a dedicated arbitrary style transfer model that leverages those priors via a novel LGWSSM module. The Local-global Window Size Stylization Module (LGWSSM), comprising LWSSM and GWSSM, fuses style features into content features with region-wise and global attention, while a Style Prior Loss ($\mathcal{L}_{sp}$) extracts and enforces priors from ASD to guide stylization without increasing inference time. The approach combines content-, style-, and adversarial losses in a balanced objective, achieving higher-quality stylizations with faster inference than state-of-the-art methods, and is supported by comprehensive ablations and user studies. Overall, SPAST demonstrates that reusing priors from pre-trained large-scale models in a targeted small-model framework can deliver superior arbitrary style transfer performance with practical efficiency.

Abstract

Given an arbitrary content and style image, arbitrary style transfer aims to render a new stylized image which preserves the content image's structure and possesses the style image's style. Existing arbitrary style transfer methods are based on either small models or pre-trained large-scale models. The small model-based methods fail to generate high-quality stylized images, bringing artifacts and disharmonious patterns. The pre-trained large-scale model-based methods can generate high-quality stylized images but struggle to preserve the content structure and cost long inference time. To this end, we propose a new framework, called SPAST, to generate high-quality stylized images with less inference time. Specifically, we design a novel Local-global Window Size Stylization Module (LGWSSM)tofuse style features into content features. Besides, we introduce a novel style prior loss, which can dig out the style priors from a pre-trained large-scale model into the SPAST and motivate the SPAST to generate high-quality stylized images with short inference time.We conduct abundant experiments to verify that our proposed method can generate high-quality stylized images and less inference time compared with the SOTA arbitrary style transfer methods.

Figures (8)

• Figure 1: The first column shows the input images. The other three columns show the stylized images produced by SaNet park2019arbitrary, SaNet+SD rombach2022high, and SaNet+SP (Ours).
• Figure 2: Overview of our SPAST, which consists of two stages. Stage one: finetuning a pre-trained Stable Diffusion, obtaining an Artistic Stable Diffusion. Stage two: Training an arbitrary style transfer model with style priors.
• Figure 3: Illustration of Local-global Window Size Stylization Module (LGWSSM).
• Figure 4: Qualitative comparisons. The $1^{st}$ col shows the input images. The other columns show the stylized images generated by other SOTA methods. Please zoom in for better observation.
• Figure 5: Illustration about why the Local-global Window Size Stylization Module works.