Transfer Learning with Pre-trained Conditional Generative Models

TL;DR

Addresses transfer learning when source and target label spaces do not overlap, source data are unavailable, and architectures can differ. Proposes a two-stage approach using pre-trained conditional generative models: pseudo pre-training (PP) to initialize target architectures from synthetic source-conditioned data, and pseudo semi-supervised learning (P-SSL) that treats target-related pseudo samples as unlabeled data in SSL. Empirically, PP+P-SSL outperforms scratch and naïve knowledge-distillation baselines across multiple target architectures and datasets, and extends to object detection, illustrating practical applicability without standard transfer-learning prerequisites. Limitations arise when the source-target distribution gap is large, and future work may optimize generative models toward the target distribution to further boost performance while keeping the method simple.

Abstract

Transfer learning is crucial in training deep neural networks on new target tasks. Current transfer learning methods always assume at least one of (i) source and target task label spaces overlap, (ii) source datasets are available, and (iii) target network architectures are consistent with source ones. However, holding these assumptions is difficult in practical settings because the target task rarely has the same labels as the source task, the source dataset access is restricted due to storage costs and privacy, and the target architecture is often specialized to each task. To transfer source knowledge without these assumptions, we propose a transfer learning method that uses deep generative models and is composed of the following two stages: pseudo pre-training (PP) and pseudo semi-supervised learning (P-SSL). PP trains a target architecture with an artificial dataset synthesized by using conditional source generative models. P-SSL applies SSL algorithms to labeled target data and unlabeled pseudo samples, which are generated by cascading the source classifier and generative models to condition them with target samples. Our experimental results indicate that our method can outperform the baselines of scratch training and knowledge distillation.

Figures (8)

• Figure 1: Proposed transfer learning methods leveraging conditional source generative model $G_s$. (a) We produce initial weights of a target architecture $\mathcal{A}_t$ by training a source classifier $C_s^{\mathcal{A}_t}$ with pairs of conditional sample $\hat{x}_s \sim G_s(y_s)$ and uniformly sampled target label $y_s$. (b) We penalize a target classifier $C_t^{\mathcal{A}_t}$ with unsupervised loss derived from SSL method by applying a pseudo sample $x_{s\leftarrow t}$ while supervised training on target dataset $\mathcal{D}_t$. $x_{s\leftarrow t}$ is sampled from $G_s$ conditioned by pseudo source label $y_{s\leftarrow t} = C^{\mathcal{A}_s}_s(x_t)$.
• Figure 2: Pseudo conditional sampling. We obtain a pseudo soft label $y_{s\leftarrow t}$ by applying a target data $x_t$ to a source classifier $C^{\mathcal{A}_s}_s$, and then generate a target-related sample $x_{s\leftarrow t}$ from a source generative model $G_s$ conditioned by $y_{s\leftarrow t}$. In this example, $C^{\mathcal{A}_s}_s$ output $y_{s\leftarrow t}$ from the input car image $x_t$ of Hummer class by interpreting $x_t$ as a mixture of source car classes (Jeep, Limousine, MovingVan, etc.), and then $G_s$ generate a target-related car image from $y_{s\leftarrow t}$.
• Figure 2: Pseudo conditional sampling
• Figure 3: Correlation between FID and accuracy gain
• Figure 4: Top-1 accuracy of P-SSL when scaling pseudo dataset size