Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

TL;DR

This work shows that state-of-the-art GAN models – such as they are being publicly released by researchers and industry – can be used for a range of applications beyond unconditional image generation, by an iterative scheme that also allows gaining control over the image generation process despite the highly non-linear latent spaces of the latest GAN model.

Abstract

While Generative Adversarial Networks (GANs) show increasing performance and the level of realism is becoming indistinguishable from natural images, this also comes with high demands on data and computation. We show that state-of-the-art GAN models -- such as they are being publicly released by researchers and industry -- can be used for a range of applications beyond unconditional image generation. We achieve this by an iterative scheme that also allows gaining control over the image generation process despite the highly non-linear latent spaces of the latest GAN models. We demonstrate that this opens up the possibility to re-use state-of-the-art, difficult to train, pre-trained GANs with a high level of control even if only black-box access is granted. Our work also raises concerns and awareness that the use cases of a published GAN model may well reach beyond the creators' intention, which needs to be taken into account before a full public release. Code is available at https://github.com/hui-po-wang/hijackgan.

Figures (10)

• Figure 1: An overview of the proposed framework. The framework takes two steps to reuse GANs: (a) train a proxy model to distill information from pre-trained models, circumventing accessing the gradients of pre-trained models; (b) identify and iteratively traverse a non-linear trajectory under the guidance of gradients.
• Figure 2: Unconditional attribute manipulation on PGGAN (left) and StyleGAN (right) with respect to Eyeglasses, Gender, Smile, and Age. We compare our method to Linear and InterfaceGAN.
• Figure 3: Logit changes over steps on StyleGAN. From left to right: Eyeglasses, Gender, Smile, Age, Narrow Eyes, Blond Hair, and Pale Skin. The solid lines represent the predictions of the target attributes while the dot lines represent the mean values over all the other non-target attribute predictions. Zoom in for better visualization.
• Figure 4: Unconditional rare attribute manipulation on StyleGAN with respect to Bald, Narrow Eyes, Pale Skin, and Blond Hair. Each set consists of input (left), our method (middle), and InterfaceGAN (right).
• Figure 5: Conditional attribute manipulation on PGGAN (left) and StyleGAN (right) with respect to Eyeglasses, Gender, Smile, and Age. We compare our method to InterfaceGAN.