Progressive Energy-Based Cooperative Learning for Multi-Domain Image-to-Image Translation

TL;DR

This work addresses unpaired multi-domain image-to-image translation by introducing PMD-CoopNets, a unified energy-based framework that couples a multi-head descriptor with a diversified image generator (style generator, style encoder, and style-controlled translator). The descriptor defines a multi-domain distribution and guides the translator via MCMC teaching, while the generator provides informed initializations and diverse style-driven outputs; learning is stabilized by regularizers and a progressive growth strategy across resolutions. Core contributions include a single multi-head EBM for multiple domains, a transferable style-conditioned translator capable of one-to-many mappings, and a progressive cooperative learning algorithm that significantly improves efficiency and scalability for high-resolution translation. Empirical results on CelebA-HQ and AFHQ demonstrate competitive performance with GAN-based methods and state-of-the-art results among energy-based translation models, with notable gains in diversity and stability.

Abstract

This paper studies a novel energy-based cooperative learning framework for multi-domain image-to-image translation. The framework consists of four components: descriptor, translator, style encoder, and style generator. The descriptor is a multi-head energy-based model that represents a multi-domain image distribution. The components of translator, style encoder, and style generator constitute a diversified image generator. Specifically, given an input image from a source domain, the translator turns it into a stylised output image of the target domain according to a style code, which can be inferred by the style encoder from a reference image or produced by the style generator from a random noise. Since the style generator is represented as an domain-specific distribution of style codes, the translator can provide a one-to-many transformation (i.e., diversified generation) between source domain and target domain. To train our framework, we propose a likelihood-based multi-domain cooperative learning algorithm to jointly train the multi-domain descriptor and the diversified image generator (including translator, style encoder, and style generator modules) via multi-domain MCMC teaching, in which the descriptor guides the diversified image generator to shift its probability density toward the data distribution, while the diversified image generator uses its randomly translated images to initialize the descriptor's Langevin dynamics process for efficient sampling.

Figures (8)

• Figure 1: Diagram of energy-based cooperative learning for multi-domain image-to-image translation. The framework consists of a style generator, a style encoder, a translator and a descriptor. The first three components (i.e., style generator, style encoder, and translator) form a diversified image generator. Given a input source image, the translator can transform it into a target domain, which is specified by a style code. The style code can be obtained by sampling from the domain-specific style generator or extracted from a reference image by the style encoder. The descriptor is a multi-domain image distribution, which plays the role of guiding the translation such that the translated images can match the observed images in the target domain in terms of statistical property. All components are trained simultaneously in a cooperative learning scheme. The descriptor learns from the multi-domain training images by maximizing the data likelihood, while utilizing MCMC teaching to guide the training of the diversified image generator, which consists of a translator, a style encoder, and a style generator.
• Figure 2: An illustration of the progressive strategy for the style encoder $E$, translator $T$, and descriptor $D$. Boxes in dark grey represent well-trained modules at resolution level $s-1$, while blocks in light gray represent the newly added parameters at the current resolution level $s$. The expansion of the model involves removing some incompatible parameters (depicted as dark grey boxes with dashed boundaries) and adding new parameters (depicted as light grey boxes). The output of the module that needs to be removed and the output of the module that needs to be added are fused using a transition factor $\omega$, This factor starts from 0 and gradually increases to 1, controlling the percentage of contribution from the old and new modules. Left: style encoder. Middle: style-controlled image-to-image translator. Right: descriptor.
• Figure 3: Qualitative results of diverse image generation for human face on CelebA-HQ dataset (left) and animal face on AFHQ dataset (right) are shown in this figure. Each column displays one example of one-to-many image generation. The first row displays source images. The rest four rows show different translated images, which are obtained by using four style codes randomly generated by the style generator. The style generator produces style codes by randomly sampling from Gaussian distribution.
• Figure 4: Architecture of proposed networks
• Figure 5: Generation results on human face in different resolution.