An Analysis for Image-to-Image Translation and Style Transfer

TL;DR

The paper addresses the common confusion between image-to-image translation and style transfer by dissecting their goals, data regimes, training paradigms, and evaluation criteria. It details image-to-image translation as domain-driven mappings (often using GANs with cycle-consistency or contrastive constraints) and style transfer as single-image fusion of content and style through pre-trained content encoders and style decoders, with losses that separately constrain content and style. Key contributions include a structured comparison across technology type, training mode, and evaluation, plus discussion of diffusion-augmented methods (e.g., InST, GMU) that blur traditional boundaries and enable shape changes in addition to texture transfer. The analysis provides a framework for researchers to select methods based on domain scope, desired semantic changes, and evaluation targets, highlighting practical implications for broadening the frontier of arbitrary image processing in the era of diffusion models.

Abstract

With the development of generative technologies in deep learning, a large number of image-to-image translation and style transfer models have emerged at an explosive rate in recent years. These two technologies have made significant progress and can generate realistic images. However, many communities tend to confuse the two, because both generate the desired image based on the input image and both cover the two definitions of content and style. In fact, there are indeed significant differences between the two, and there is currently a lack of clear explanations to distinguish the two technologies, which is not conducive to the advancement of technology. We hope to serve the entire community by introducing the differences and connections between image-to-image translation and style transfer. The entire discussion process involves the concepts, forms, training modes, evaluation processes, and visualization results of the two technologies. Finally, we conclude that image-to-image translation divides images by domain, and the types of images in the domain are limited, and the scope involved is small, but the conversion ability is strong and can achieve strong semantic changes. Style transfer divides image types by single image, and the scope involved is large, but the transfer ability is limited, and it transfers more texture and color of the image.

Figures (6)

• Figure 1: (a) Image-to-image translation (w/o reference). In each set of images, the left side is the input image and the right side is the generated image. The six tasks shown require a total of six models. The models and generated results are from CycleGAN zhu2017unpaired. (b) Image-to-image translation (w/ reference). In each group of images, the first row is the reference image, and the second row is the generated image. The three tasks shown require a total of three models. The models and generated results are from DSMAP chang2020domain. (c) Style transfer. The first column is the content image, the first row is the style reference image, and the rest are generated images. The generation task shown requires a total of one model. The model and generation results are from AdaAttN liu2021adaattn.
• Figure 2: (a) The generation process of the generative adversarial network. The noise $z$ is converted to the image domain $Y$ through the model. (b) The generation process of image-to-image translation. The image domain $X$ is converted to the image domain $Y$ through the model. (c) Data distribution. The distribution of the generated images and the real images.
• Figure 3: The comparison between unsupervised image translation and arbitrary style transfer includes three parts: technology type, training mode and effect evaluation. $X$ and $Y$ represent images from two different domains.
• Figure 4: Comparison of the generative effects of unsupervised image translation and arbitrary style transfer. The left side shows the conversion between cats and dogs, and the right side shows the conversion from photos to Monet paintings. AdaIN huang2017arbitrary is a model for arbitrary style transfer, and DSMAP chang2020domain is a model for unsupervised image translation. The generative results are derived from DSMAP chang2020domain.
• Figure 5: Comparison of the effects of style loss and adversarial loss. DualAST chen2021dualast is a complete model that includes adversarial loss and style loss. w/o GAN retains the style loss and removes the adversarial loss, while w/o Style loss retains the adversarial loss and removes the style loss. The generated results are from DualAST chen2021dualast.