Multimodal Unsupervised Image-to-Image Translation

TL;DR

The paper introduces MUNIT, a multimodal unsupervised image-to-image translation framework that decomposes images into a shared content code and domain-specific style codes to generate diverse outputs without paired data. Translation is performed by recombining a source content code with a randomly sampled target-style code, enabling continuous multimodality and example-guided style control. The authors provide theoretical results showing latent and joint distribution matching and a style-augmented cycle consistency as a weaker alternative to strict cycle-consistency for multimodal translation. Empirical results on multiple datasets show that MUNIT achieves higher diversity and image quality than unsupervised baselines and competitive performance with supervised methods, with notable ability to control translation style via example images.

Abstract

Unsupervised image-to-image translation is an important and challenging problem in computer vision. Given an image in the source domain, the goal is to learn the conditional distribution of corresponding images in the target domain, without seeing any pairs of corresponding images. While this conditional distribution is inherently multimodal, existing approaches make an overly simplified assumption, modeling it as a deterministic one-to-one mapping. As a result, they fail to generate diverse outputs from a given source domain image. To address this limitation, we propose a Multimodal Unsupervised Image-to-image Translation (MUNIT) framework. We assume that the image representation can be decomposed into a content code that is domain-invariant, and a style code that captures domain-specific properties. To translate an image to another domain, we recombine its content code with a random style code sampled from the style space of the target domain. We analyze the proposed framework and establish several theoretical results. Extensive experiments with comparisons to the state-of-the-art approaches further demonstrates the advantage of the proposed framework. Moreover, our framework allows users to control the style of translation outputs by providing an example style image. Code and pretrained models are available at https://github.com/nvlabs/MUNIT

Figures (12)

• Figure 1: An illustration of our method. (a) Images in each domain $\mathcal{X}_{i}$ are encoded to a shared content space $\mathcal{C}$ and a domain-specific style space $\mathcal{S}_{i}$. Each encoder has an inverse decoder omitted from this figure. (b) To translate an image in $\mathcal{X}_{1}$ (e.g., a leopard) to $\mathcal{X}_{2}$ (e.g., domestic cats), we recombine the content code of the input with a random style code in the target style space. Different style codes lead to different outputs.
• Figure 2: Model overview. Our image-to-image translation model consists of two auto-encoders (denoted by red and blue arrows respectively), one for each domain. The latent code of each auto-encoder is composed of a content code $c$ and a style code $s$. We train the model with adversarial objectives (dotted lines) that ensure the translated images to be indistinguishable from real images in the target domain, as well as bidirectional reconstruction objectives (dashed lines) that reconstruct both images and latent codes.
• Figure 3: Our auto-encoder architecture. The content encoder consists of several strided convolutional layers followed by residual blocks. The style encoder contains several strided convolutional layers followed by a global average pooling layer and a fully connected layer. The decoder uses a MLP to produce a set of AdaIN huang2017adain parameters from the style code. The content code is then processed by residual blocks with AdaIN layers, and finally decoded to the image space by upsampling and convolutional layers.
• Figure 4: Qualitative comparison on edges $\rightarrow$ shoes. The first column shows the input and ground truth output. Each following column shows $3$ random outputs from a method.
• Figure 5: Example results of (a) edges $\leftrightarrow$ shoes and (b) edges $\leftrightarrow$ handbags.

Theorems & Definitions (9)

• proposition thmcounterproposition
• proposition thmcounterproposition
• proposition thmcounterproposition
• proposition thmcounterproposition
• proof
• proof
• proof
• proof
• proof