AEANet: Affinity Enhanced Attentional Networks for Arbitrary Style Transfer

TL;DR

A affinity-enhanced attentional network, which include the content affinity-enhanced attention (CAEA) module, the style affinity-enhanced attention (SAEA) module, and the hybrid attention (HA) module, is proposed, which better preserves the affinity with content and style images.

Abstract

Arbitrary artistic style transfer is a research area that combines rational academic study with emotive artistic creation. It aims to create a new image from a content image according to a target artistic style, maintaining the content's textural structural information while incorporating the artistic characteristics of the style image. However, existing style transfer methods often significantly damage the texture lines of the content image during the style transformation. To address these issues, we propose affinity-enhanced attentional network, which include the content affinity-enhanced attention (CAEA) module, the style affinity-enhanced attention (SAEA) module, and the hybrid attention (HA) module. The CAEA and SAEA modules first use attention to enhance content and style representations, followed by a detail enhanced (DE) module to reinforce detail features. The hybrid attention module adjusts the style feature distribution based on the content feature distribution. We also introduce the local dissimilarity loss based on affinity attention, which better preserves the affinity with content and style images. Experiments demonstrate that our work achieves better results in arbitrary style transfer than other state-of-the-art methods.

Figures (5)

• Figure 1: Utilizing our arbitrary style transfer model, the generated images exhibit substantial enhancements in visual quality and a marked improvement in content affinity.
• Figure 2: Overview of our framework. We input the content image $I_c$ and the style image $I_s$ into a pre-trained VGG encoder, generating the corresponding features $F_c$ and $F_s$. These features $F_c$ and $F_s$ are fed into the CAEA and SAEA modules, respectively. The HA module adjusts the style features $F_{ss}$ based on the distribution of the content features $F_{cc}$ to obtain the stylized features $F_{cs}$, which are decoded by a decoder symmetrically designed to the encoder to generate the resulting image $I_{cs}$.
• Figure 3: Qualitative comparisons of style transfer results with SOTA methods
• Figure 4: content style trade-off
• Figure 5: Ablation study results. The first column shows the input content and style images, the second column shows the stylization results from the complete model, and the third column shows the results without the $L_{LDL}$ loss functions ($L_{LDL}$ is $\mathcal{L}_{LD\_Content}$ add $\mathcal{L}_{LD\_Style}$ ).