Color and Texture Dual Pipeline Lightweight Style Transfer

TL;DR

This paper addresses the challenge of simultaneously transferring color and texture in style transfer. It introduces CTDP, a lightweight dual-pipeline network with shallow and deep branches plus a fusion decoder, trained with branch-specific Gram losses, a masked total variation loss, and a texture-intensity control parameter, enabling concurrent color and texture transfer. Key contributions include the first controllable texture intensity for color transfer, a novel MTv-based texture suppression mechanism, and insights from input smoothing and feature visualization that explain texture formation. The approach achieves state-of-the-art results in both color and texture transfer while maintaining a highly compact model (color transfer branch ~20k parameters), offering practical benefits for fast, high-quality stylization.

Abstract

Style transfer methods typically generate a single stylized output of color and texture coupling for reference styles, and color transfer schemes may introduce distortion or artifacts when processing reference images with duplicate textures. To solve the problem, we propose a Color and Texture Dual Pipeline Lightweight Style Transfer CTDP method, which employs a dual pipeline method to simultaneously output the results of color and texture transfer. Furthermore, we designed a masked total variation loss to suppress artifacts and small texture representations in color transfer results without affecting the semantic part of the content. More importantly, we are able to add texture structures with controllable intensity to color transfer results for the first time. Finally, we conducted feature visualization analysis on the texture generation mechanism of the framework and found that smoothing the input image can almost completely eliminate this texture structure. In comparative experiments, the color and texture transfer results generated by CTDP both achieve state-of-the-art performance. Additionally, the weight of the color transfer branch model size is as low as 20k, which is 100-1500 times smaller than that of other state-of-the-art models.

Figures (9)

• Figure 1: Add a texture structure with controllable intensity to the color transfer result, and the following values represent the intensity modulation parameter $\lambda_{d}$ of the texture feature.
• Figure 2: Architecture illustration of the proposed CTDP. See Section 3 for details.
• Figure 3: Ablation study of feature decoding consistency loss.$\mathcal{L}_{mtv}$ ensures that shallow features in the shallow and fusion decoders' outputs yield similar results.
• Figure 4: Gradually suppress the texture by overlaying three methods.
• Figure 5: Visualization of feature maps for the first and third convolutions of three methods.