Name Your Style: An Arbitrary Artist-aware Image Style Transfer

TL;DR

TxST addresses the limitation of conventional style transfer requiring explicit style images by enabling text-driven, arbitrary artist-aware stylization. It couples CLIP-based encodings with a Positional Mapper and a Polynomial Attention module to fuse content with language-described styles, trained through contrastive objectives that align text and image representations in CLIP space. The approach achieves state-of-the-art performance across image-driven and text-driven settings, including effective mimicry of multiple artists and interactive, multi-style fusion, while preserving content structure. This work broadens practical style transfer by enabling flexible, real-time, text-guided artistic stylization with strong semantic alignment and visual quality.

Abstract

Image style transfer has attracted widespread attention in the past few years. Despite its remarkable results, it requires additional style images available as references, making it less flexible and inconvenient. Using text is the most natural way to describe the style. More importantly, text can describe implicit abstract styles, like styles of specific artists or art movements. In this paper, we propose a text-driven image style transfer (TxST) that leverages advanced image-text encoders to control arbitrary style transfer. We introduce a contrastive training strategy to effectively extract style descriptions from the image-text model (i.e., CLIP), which aligns stylization with the text description. To this end, we also propose a novel and efficient attention module that explores cross-attentions to fuse style and content features. Finally, we achieve an arbitrary artist-aware image style transfer to learn and transfer specific artistic characters such as Picasso, oil painting, or a rough sketch. Extensive experiments demonstrate that our approach outperforms the state-of-the-art methods on both image and textual styles. Moreover, it can mimic the styles of one or many artists to achieve attractive results, thus highlighting a promising direction in image style transfer.

Figures (15)

• Figure 1: Artist-aware style transfer. We show two examples of using different methods for style transfer: (a) AST (style-specific), (b) Clipstyler (fast and opti) and (c) our proposed TxST . Example (a) uses a sketch as a content to transfer Van Gogh style. Example (b) uses a photo as a content to transfer El-Greco style. AST fails to preserve the content information and CLIPstyler cannot transfer the correct styles from the text, such as the faces on the wall produced by CLIPstyler (opti) in (b). Instead, TxST successfully mimics the artists' style (curvature of Van Gogh, dark twisted lines of El-Greco) while preserving the content cues.
• Figure 2: Comparison among different CLIP based image manipulation. We show the overall training processes of a) StyleCLIP, b) Clipstyler and c) our proposed TxST for comparison. They all use CLIP image encoder $E_I$ and text encoder $E_T$ to project images and texts onto the CLIP space for distance $\Delta T$ measurement. For a), it sample latent code $\omega$ from the pre-define latent space $\mathcal{W}+$ as input to StyleGAN for generation. For b), it uses source image $I_c$ as input to the StyleNet to obtain $f(I_c)$. For c), it samples two random style images $I_s^1$ and $I_s^2$ for style transfer. By using the contrastive training loss, it minimizes the intra-class distances (text-image) $\Delta T_1$ and $\Delta T_2$, as well as maximizes the inter-class distance (image-image) $\Delta D$.
• Figure 3: Correlation between artists and paintings.(a) A set of Artist-Painting paired samples from WikiArt dataset WikiArt. The artists (Abbr.: A) are represented by the means of text, and their paintings (Abbr.: P) are in the format of color images. Different artists and their paints are given different index numbers for clear visualisation. (b) Feature relationship between artists and paintings. Features of the artists and paintings are extracted by the CLIPclip with language and visual portions, respectively. The horizontal and vertical axes are the artists and paintings respectively. The significant larger values on the diagonal elements suggests that the features from CLIPclip model are aware of the high-level painting style of different artist.
• Figure 4: Training process of the proposed TxST . We show the overall training processes of our proposed TxST . Given content images, artists' names and paintings, we process them to maximize the style's variance, as well as maximize the similarity of the same style.
• Figure 5: Structure of the Position mapper.$\textbf{R}_h$ and $\textbf{R}_w$ are two learnable position encodings for horizontal and vertical directions. Relative position encoding $\textbf{R}$ is sum of the $\textbf{R}_h$ (repeat 16 times along width dimension) and $\textbf{R}_w$ (repeat 16 times along width dimension). The input style vector can come from the image $E_{I}(I_s)$ or text $E_{t}(T_s)$. The repeated vector is then processed by three 1-by-1 linear projection layers to form a common Self-Attention (SA) non-local with query $\textbf{Q}$, key $\textbf{K}$ and value $\textbf{V}$. The relative position encoding $\textbf{R}$ is multiplied with the query $\textbf{Q}$ and attends the SA process. Finally the position mapper produces a position-aware style feature $\textbf{F}_{s}$. The dimensions in the figure are in the order of $height\times width \times channels$.