Improved Input Reprogramming for GAN Conditioning

TL;DR

This work defines GAN conditioning as transforming a pretrained unconditional GAN into a CGAN using labeled data, and shows that input reprogramming is most effective when labels are scarce. It introduces InRep+, an improved input reprogramming framework that uses a shared, invertible modifier and a PU-based discriminator loss with weight-sharing discriminators to scale to many classes. Theoretical analysis confirms optimal conditional distribution learning under ideal conditions, while comprehensive experiments demonstrate strong gains over baselines, particularly at very low supervision and under label noise or class imbalance. The approach offers practical gains in memory efficiency and robustness, enabling effective conditional generation without retraining or substantial labeling effort. The findings have potential implications for conditioning other unconditional generative models and connect concepts with prompt-tuning in downstream tasks.

Abstract

We study the GAN conditioning problem, whose goal is to convert a pretrained unconditional GAN into a conditional GAN using labeled data. We first identify and analyze three approaches to this problem -- conditional GAN training from scratch, fine-tuning, and input reprogramming. Our analysis reveals that when the amount of labeled data is small, input reprogramming performs the best. Motivated by real-world scenarios with scarce labeled data, we focus on the input reprogramming approach and carefully analyze the existing algorithm. After identifying a few critical issues of the previous input reprogramming approach, we propose a new algorithm called InRep+. Our algorithm InRep+ addresses the existing issues with the novel uses of invertible neural networks and Positive-Unlabeled (PU) learning. Via extensive experiments, we show that InRep+ outperforms all existing methods, particularly when label information is scarce, noisy, and/or imbalanced. For instance, for the task of conditioning a CIFAR10 GAN with 1% labeled data, InRep+ achieves an average Intra-FID of 76.24, whereas the second-best method achieves 114.51.

Figures (7)

• Figure 1: GAN conditioning setting (illustrated with two-class MNIST). GAN conditioning algorithms convert an unconditional generator $G$ (top left) into a conditional generator $G'$ (right) using a labeled dataset (bottom left). On the two-class MNIST data, unconditional generator $G$ uniformly generates images of $0$s or $1$s from random noise vector ${\mathbf{z}}$. The labeled data contains class-conditional images with labels $0$ and $1$. The output of the GAN conditioning algorithm is the conditional generator $G'$ that generates samples of class $y$ from random noise ${\mathbf{z}}$ and the provided class label $y$ in $\{y_0=0, y_1=1\}$.
• Figure 2: Modular design of InRep+ (Improved Input Reprogramming) framework. Given a fixed unconditional generator $G$, InRep+ learns a modifier network $M$ and conditional discriminator networks $D_{y}$, $y \in {\mathbb{Y}}$. Each $D_{y}$ is built on a weight-sharing network $D$ followed by a class-specific linear head $H_{y}$, i.e. $D_{y} = H_{y}\circ D$. We embed each label $y$ into a vector ${\mathbf{y}}$ with the embedding module $E$, then concatenate ${\mathbf{y}}$ to a random noise vector ${\mathbf{u}}$ to get vector ${\mathbf{z}}$. The modifier $M$ converts ${\mathbf{z}}$ into a $y$-conditional noise ${\mathbf{z}}_y$ so that $G({\mathbf{z}}_y)$ is a sample of class $y$ (a $0$-image of $\text{fake}_{y}$ in the illustration). We train our networks using GAN training.
• Figure 3: Highlights of design differences between GAN-Reprogramlee2020 and InRep+ (ours).GAN-Reprogram (left) learns a pair of modifier $M_i$ and discriminator $D_i$ per each of $n$ conditions. InRep+ (right) makes three improvements over GAN-Reprogram: i) using a single conditional modifier $M$ and sharing weights between $\{D_i\}_{1}^n$ to improve memory scalability when $n$ is large, ii) adopting the invertible architecture for modifier network $M$ to prevent class-conditional latent regions from overlapping, iii) replacing the standard GAN loss with a new PU-based loss to overcome the false rejection issue.
• Figure 4: Visualization of InRep+'s discriminator loss on real data. We visualize the loss with different values of $\pi_y$ (in $[0, 1]$) and $D_y(x)$ (in $[0, 1]$). Compared to the standard negative log-loss ($\pi_{y}=0$), the new loss ($\pi_{y}>0$) more strongly encourages $D_{y}$ to predict values near $1$ for positive samples.
• Figure 5: Conditional samples from InRep+ on different datasets. (a) Gaussian mixture: We synthesize a mixture of four Gaussian distributions on two-dimensional space with 10000.0 samples. Four Gaussian components have unit variance and their means are $(0, 2), (-2, 0), (0, -2), (2, 0)$, respectively. The synthesized data is visualized in the left figure (Real). The central figure visualizes the distribution learned by unconditional GAN (UGAN). The right figure (InRep+) shows the conditional samples from InRep+. As we can see, the InRep+'s distribution is highly similar to the real data distribution. Also, it covers all four distribution modes. (b) MNIST: We visualize class-conditional samples of CGANs learned by InRep+, each row per class. Our samples have correct labels, with high-quality and diverse shapes. (c) Face data: Unconditional GAN model is StyleGAN pretrained on the FFHQ dataset, which contains high-resolution face images. We use CelebA data as the labeled data, with two classes: wearing glasses and male. Most conditional samples are high-quality and with correct labels. We further show conditional samples of CIFAR10 in Fig. \ref{['fig:cifar10_visual_comparison']}.

Theorems & Definitions (19)

• Lemma 1: ACGAN provably fails on a non-separable dataset
• Lemma 2: ACGAN provably fails on a separable dataset
• Lemma 3: ProjGAN provably fails on a two-class dataset
• Remark 1
• Proposition 1
• Theorem 1: Adapted from goodfellow2014generative
• Remark 2: Fairness in the data generation
• Lemma 1: W-ACGAN fails to learn a correct Gaussian mixture, full statement
• proof
• Lemma 2: W-ACGAN with gradient descent fails to learn a separable distribution, full statement