FISTNet: FusIon of STyle-path generative Networks for Facial Style Transfer

TL;DR

FISTNet tackles data-efficient, multi-style facial style transfer by fusing pre-trained style-path networks through a dual-path architecture (intrinsic and extrinsic). It leverages Modulative Residual Blocks and a Gated Mapping Unit to preserve facial identity while fusing diverse styles, using curriculum learning to stabilize training. Experiments on CelebA-HQ, Celeb, and RDGAN demonstrate strong performance with limited data, achieving competitive FID scores and superior user-study results, while supporting high-resolution outputs at $1024\\times1024$. The approach enables flexible, exemplar-free multi-style transfer suitable for XR/Metaverse avatar creation and NFT art, with future work aimed at broader style sets and localized stylization.

Abstract

With the surge in emerging technologies such as Metaverse, spatial computing, and generative AI, the application of facial style transfer has gained a lot of interest from researchers as well as startups enthusiasts alike. StyleGAN methods have paved the way for transfer-learning strategies that could reduce the dependency on the huge volume of data that is available for the training process. However, StyleGAN methods have the tendency of overfitting that results in the introduction of artifacts in the facial images. Studies, such as DualStyleGAN, proposed the use of multipath networks but they require the networks to be trained for a specific style rather than generating a fusion of facial styles at once. In this paper, we propose a FusIon of STyles (FIST) network for facial images that leverages pre-trained multipath style transfer networks to eliminate the problem associated with lack of huge data volume in the training phase along with the fusion of multiple styles at the output. We leverage pre-trained styleGAN networks with an external style pass that use residual modulation block instead of a transform coding block. The method also preserves facial structure, identity, and details via the gated mapping unit introduced in this study. The aforementioned components enable us to train the network with very limited amount of data while generating high-quality stylized images. Our training process adapts curriculum learning strategy to perform efficient, flexible style and model fusion in the generative space. We perform extensive experiments to show the superiority of FISTNet in comparison to existing state-of-the-art methods.

Figures (8)

• Figure 1: The work proposes FISTNet that performs high resolution facial style transfer, i.e. 1024$\times$1024. The existing works either overfit styles onto the faces that do not preserve facial structure and characteristics, such as Resolution Dependent GAN, DualStyleGAN, and Toonify VToonify, or do not transfer diverse styles such as AnimeGAN and white-box cartoon representations Whitebox. In addition, studies such as AnimeGAN introduce artifacts into facial images. Aforementioned works can also generate different styles of images along with the proposed work, respectively.
• Figure 2: The proposed FISTNet network architecture.
• Figure 3: Results of FISTNet after each fine-tuning stage
• Figure 4: Qualitative Comparison for style transfer with state-of-the-art works. The right side of the line shows the results from existing methods that require an input style while the left side of the line compares with the works that require exemplar images.
• Figure 5: Sketch style transfer using FISTNet