Variational Transfer Learning using Cross-Domain Latent Modulation

TL;DR

This work tackles the challenge of transferring learned representations across domains by introducing Cross-Domain Latent Modulation (CDLM) within a variational autoencoder. CDLM constructs a Transfer Latent Space (TLS) and applies cross-domain modulation to reparameterize latent variables using deep representations from the other domain, aided by adversarial alignment and a consistency loss. The approach yields reduced KL-divergence between domain latent distributions and demonstrates competitive performance on unsupervised domain adaptation and image-to-image translation, supported by qualitative visualizations and quantitative metrics like A-distance. The framework is relatively simple yet effective, with potential extensions to multi-domain and heterogeneous transfer scenarios. The results suggest CDLM provides a principled, scalable path to cross-domain generative modeling and adaptation.

Abstract

To successfully apply trained neural network models to new domains, powerful transfer learning solutions are essential. We propose to introduce a novel cross-domain latent modulation mechanism to a variational autoencoder framework so as to achieve effective transfer learning. Our key idea is to procure deep representations from one data domain and use it to influence the reparameterization of the latent variable of another domain. Specifically, deep representations of the source and target domains are first extracted by a unified inference model and aligned by employing gradient reversal. The learned deep representations are then cross-modulated to the latent encoding of the alternative domain, where consistency constraints are also applied. In the empirical validation that includes a number of transfer learning benchmark tasks for unsupervised domain adaptation and image-to-image translation, our model demonstrates competitive performance, which is also supported by evidence obtained from visualization.

Figures (10)

• Figure 1: Illustration of the latent space manipulation. We can get the latent space by a deep autoencoder structure ($\mathbf{E}_u$ stands for an encoder and $\mathbf{D}$ for a decoder). Our aim is then to design a transformation to transfer the latent space ($\bm z_{tr}$) for the source and the target. The source data flow ($\bm x_s, \bm \hat{x}_s$) is in purple and the target ($\bm x_t, \bm \hat{x}_t$) is in blue.
• Figure 2: Architectural view of the proposed model Hou2021. It encourages an image from the target domain (blue hexagon) to be transformed to a corresponding image in the source domain (black hexagon). The transfer latent distributions $p(\bm {\ddot{z}}_{ts}|\bm x_t, \bm h_s)$ and $p(\bm{\ddot{z}}_{st}|\bm x_s, \bm h_t)$ are learned and are used to generate corresponding images by the desired decoder. The deep representations are integrated into the reparameterization transformation with standard Gaussian auxiliary noise. Blue lines are for the target domain and black ones are for the source domain.
• Figure 3: Graphical model of CDLM. The dashed lines are inference process and the solid lines are generative process.
• Figure 4: Modulation effect on the latent mean vectors: transfer means $\bm\mu_{st}$ and $\bm\mu_{ts}$ are pushed closer compared with the original domain latent means $\bm\mu_{s}$ and $\bm\mu_{t}$.
• Figure 5: Visualization for the adaptations. 6 different tasks are illustrated. For each task, the first row shows target images and the second row shows the adapted images with source-like style.

Theorems & Definitions (1)

• Definition 1