RLMiniStyler: Light-weight RL Style Agent for Arbitrary Sequential Neural Style Generation

TL;DR

RLMiniStyler addresses the cost and tuning bottlenecks of arbitrary style transfer by reframing stylization as sequential reinforcement learning with a unified encoder. It introduces an uncertainty-aware multi-task learning framework and a Hierarchical Style Representation Contrastive Loss to balance content preservation with rich style expression, enabling automatic control of stylization across sequences and resolutions. Across MS-COCO and WikiArt, it achieves competitive quality with lower computational cost and produces diverse stylized sequences from 256 to 4K, validated by qualitative comparisons and a user study. The approach offers a path toward efficient, flexible AST suitable for resource-constrained deployment and potential extension to video-style transfer.

Abstract

Arbitrary style transfer aims to apply the style of any given artistic image to another content image. Still, existing deep learning-based methods often require significant computational costs to generate diverse stylized results. Motivated by this, we propose a novel reinforcement learning-based framework for arbitrary style transfer RLMiniStyler. This framework leverages a unified reinforcement learning policy to iteratively guide the style transfer process by exploring and exploiting stylization feedback, generating smooth sequences of stylized results while achieving model lightweight. Furthermore, we introduce an uncertainty-aware multi-task learning strategy that automatically adjusts loss weights to adapt to the content and style balance requirements at different training stages, thereby accelerating model convergence. Through a series of experiments across image various resolutions, we have validated the advantages of RLMiniStyler over other state-of-the-art methods in generating high-quality, diverse artistic image sequences at a lower cost. Codes are available at https://github.com/fengxiaoming520/RLMiniStyler.

Figures (5)

• Figure 1: Illustration of our arbitrary style sequence generation process. Top Left: Content and Style Images (5 style examples). Right: The sequence number of the results. Content images are progressively stylized with increasing strength along prediction sequences (see the index). Our method allows for easy control over stylization degree, preserving content details in early sequences and synthesizing more style patterns in later sequences, resulting in a user-friendly approach.
• Figure 2: Overview of the RLMiniStyler model. Top: The state $y_t$ is initialized with the content image $I_c$ and the style image $I_s$. Latent-action ${\bf x}_t$ is sampled from a high-dimensional Gaussian distribution and is concatenated with the critic's output. It is estimated by the policy $P_\kappa$: ${\bf x_t} \sim P_\kappa({\bf x}_t|{\bf y}_t)$. The predicted moving image $I_m^{t+1}$ is generated by builder $B_\tau$. 'Pull' and 'Push' refer to minimize and maximize the distance between two feature maps, respectively. Note that the pre-trained VGG network is used only to extract features for calculating rewards and losses during training.Bottom: The structure of the actor and the builder. And $Sign_{1,2,3,4}$ refer to the style signals derived from the calculation of style features. Different colors in the network represent different network architectures, and details of the network structure can be found in supplementary materials.
• Figure 3: The overlap between style and content images, and the illustration from Pre-trained Encoder to Non-pretrained Encoder. In the figure, 'S', 'C', and 'O' represent the style image, content image, and stylized output, respectively. The images shown in (a) are drawn from both content and style datasets, but the boundaries between them are so blurred that it's challenging to clearly distinguish their original sources. Most existinig style transfer methods typically employ two encoding approaches: one directly utilizes single complex pre-trained encoder (b), while the other trains separate encoders for content and style (c). In contrast, our method adopts a novel approach, using single mini-unified policy for both content and style (d). We detail the specifics of this unified policy as shown in Fig. \ref{['fig:pipline']}.
• Figure 4: Qualitative Comparison with several AST algorithms in 256 pixel resolution. The 1st and 2nd columns present the content and style images, respectively. The subsequent four columns display the results from the current SOTA AST methods. The three columns immediately following showcase the results of the lightweight methods. Lastly, we present the sequential stylization results generated by our method, including sequences 1st, 5th, and 10th.
• Figure 5: Ablation Study Results Comparing the Impact of RL, Automatic Multi-task Learning (AML), and Hierarchical Style Representation Contrastive Loss (HSRCL) vs. Style Signal Contrastive Loss on Style Transfer Performance. The visual comparison underscores the contributions of RL, AML, and HSRCL to the fidelity and stability of stylized results across sequences. More results are presented in the supplementary materials.