Spatial Steerability of GANs via Self-Supervision from Discriminator

TL;DR

This work tackles the challenge of spatial controllability in GANs by introducing SpatialGAN, a self-supervised framework that injects randomly sampled Gaussian heatmaps into a generator's intermediate layers and aligns them with the discriminator's GradCAM-based attention loss $\mathcal{L}_{align}$. The key idea is that the discriminator naturally attends to informative image regions, and steering the generator with heatmaps guided by this attention yields intuitive spatial edits (move, remove, or recolor regions) without requiring per-model attribute annotations. The authors further enhance realism by integrating a DragGAN-based coarse-to-fine manipulation pipeline, allowing fast, coarse spatial edits that are refined via point-based optimization. Across faces, outdoor scenes, and multi-object indoor scenes, SpatialGAN improves synthesis quality (FID improvements) and demonstrates robust, hierarchical spatial control, with ablations confirming the importance of hierarchical heatmaps, indoor-specific encoding, and the self-supervised alignment strategy.

Abstract

Generative models make huge progress to the photorealistic image synthesis in recent years. To enable human to steer the image generation process and customize the output, many works explore the interpretable dimensions of the latent space in GANs. Existing methods edit the attributes of the output image such as orientation or color scheme by varying the latent code along certain directions. However, these methods usually require additional human annotations for each pretrained model, and they mostly focus on editing global attributes. In this work, we propose a self-supervised approach to improve the spatial steerability of GANs without searching for steerable directions in the latent space or requiring extra annotations. Specifically, we design randomly sampled Gaussian heatmaps to be encoded into the intermediate layers of generative models as spatial inductive bias. Along with training the GAN model from scratch, these heatmaps are being aligned with the emerging attention of the GAN's discriminator in a self-supervised learning manner. During inference, users can interact with the spatial heatmaps in an intuitive manner, enabling them to edit the output image by adjusting the scene layout, moving, or removing objects. Moreover, we incorporate DragGAN into our framework, which facilitates fine-grained manipulation within a reasonable time and supports a coarse-to-fine editing process. Extensive experiments show that the proposed method not only enables spatial editing over human faces, animal faces, outdoor scenes, and complicated multi-object indoor scenes but also brings improvement in synthesis quality. Code, models, and demo video are available at https://genforce.github.io/SpatialGAN/.

Figures (14)

• Figure 1: Illustration of Spatial Manipulations. Our method enables various spatial manipulations for image generation, like moving a bed, a cat, or a building (green arrow), controlling the image layout, removing a drawing (yellow box), and changing the local appearance (blue box).
• Figure 2: Illustration of SpatialGAN. We conduct spatial encoding in $G$ and align its spatial awareness with $D$ attention maps. Specifically, we randomly sample spatial heatmaps and encode them into $G$ via the spatial encoding layer (SEL). To implement the alignment during training, we calculate $D$ attention maps over the generated samples via GradCAM.
• Figure 3: Spatial visual attention at the intermediate layers of the discriminator, visualized by GradCAM. A bright color indicates a strong contribution to the final score. '$64\times64$' indicates being upsampled from a $64\times64$ feature map. The samples are the real images and the images generated by StyleGAN2 stylegan2.
• Figure 4: User interface for interactive editing. Users can drag the Gaussian centers to alter heatmaps and synthesize an image corresponding to the new heatmap. In addition to that users can also change the sigma value of the heatmap center and the local style corresponding to that center. The Bedroom model (left) uses the same heatmap for all the resolutions while the Cat model (right) uses hierarchical heatmaps. Readers are suggested to view the demo video of interactive editing on our project page.
• Figure 5: Controlling the room layout by using the same spatial heatmap. For each row, we adopt different latent codes but use the same spatial heatmap. The images in the same row show a similar spatial layout, while their appearances are different. The appearances include colour, texture, lighting, and so on.