MRStyle: A Unified Framework for Color Style Transfer with Multi-Modality Reference

TL;DR

MRStyle tackles the problem of color style transfer with guidance from either image or text prompts by unifying their style representations. It introduces IRStyle for image references and TRStyle for text references, connected through a shared style space and an efficient priors mapper that leverages Stable Diffusion priors. The approach combines an interaction dual-mapping network and a combined supervised learning pipeline to achieve artifact-free, content-preserving transfers at high resolutions, and uses synthetic data to train the text pathway in an end-to-end manner. Extensive experiments show state-of-the-art performance in both image- and text-guided color transfer, with impressive efficiency, memory economy, and open-set capabilities for text prompts.

Abstract

In this paper, we introduce MRStyle, a comprehensive framework that enables color style transfer using multi-modality reference, including image and text. To achieve a unified style feature space for both modalities, we first develop a neural network called IRStyle, which generates stylized 3D lookup tables for image reference. This is accomplished by integrating an interaction dual-mapping network with a combined supervised learning pipeline, resulting in three key benefits: elimination of visual artifacts, efficient handling of high-resolution images with low memory usage, and maintenance of style consistency even in situations with significant color style variations. For text reference, we align the text feature of stable diffusion priors with the style feature of our IRStyle to perform text-guided color style transfer (TRStyle). Our TRStyle method is highly efficient in both training and inference, producing notable open-set text-guided transfer results. Extensive experiments in both image and text settings demonstrate that our proposed method outperforms the state-of-the-art in both qualitative and quantitative evaluations.

Figures (17)

• Figure 1: Our multi-modality reference color style transfer results. (a) State-of-the-art methods DNCM ke2023neural and CAP-VST wen2023cap often produce artifacts (e.g., texture in red box) or unsatisfactory colors (e.g., content in blue box). In contrast, our method avoids artifacts and achieves better color transfer effects. (b) Our method can produce amazing stylized results on 8K images given text reference. Zoom in for better visualization.
• Figure 2: The overview of MRStyle. (a) The inference pipeline of MRStyle. (b) The combined supervised training pipeline of IRStyle. Paired supervised losses are $\mathcal{L}_{self}$ and $\mathcal{L}_{cm}$. Unpaired supervised losses are $\mathcal{L}_{style}$, $\mathcal{L}_{hist}$ and $\mathcal{L}_{content}$. The Interaction Dual-Mapping Network is detailed in Fig. \ref{['fig:dual-lut']} (c). After the training of IRStyle, we integrate the pre-trained IRStyle with stable diffusion priors to finalize our TRStyle. (c) The training pipeline of TRStyle.
• Figure 3: Architectural designs for image reference color style transfer. The feature extraction of input images has been omitted for simplicity. (a) Interaction direct-mapping network, e.g., AdaCM lin2023adacm. (b) Non-interaction dual-mapping network, e.g., DNCM ke2023neural. (c) Interaction dual-mapping network (ours).
• Figure 4: The color and illumination of the generated image in the diffusion model. (a) The denoising process during image generation. (b) The color similarity analysis under different time steps of diffusion model sampling.
• Figure 5: Qualitative comparison of the image reference setting. Our method shows the superiority in both photorealism and stylization over the other methods.