Unlocking Pre-trained Image Backbones for Semantic Image Synthesis

TL;DR

DP-SIMS tackles semantic image synthesis by combining pre-trained discriminative backbones as fixed encoders in a UNet-like discriminator with a feature-conditioning pipeline, and a generator that uses cross-attention to inject noise into multi-scale label-map features for enhanced diversity. The approach introduces a multi-faceted loss suite, including a pixel-wise focal GAN loss, a patch-level contrastive loss, and a diversity constraint, to boost both fidelity and variety. It achieves state-of-the-art results on ADE-20K, COCO-Stuff, and Cityscapes with two orders of magnitude faster inference than diffusion models. The work demonstrates that leveraging discriminative pre-training in GAN frameworks can yield high-quality, diverse, and layout-faithful images suitable for practical deployment and data augmentation tasks.

Abstract

Semantic image synthesis, i.e., generating images from user-provided semantic label maps, is an important conditional image generation task as it allows to control both the content as well as the spatial layout of generated images. Although diffusion models have pushed the state of the art in generative image modeling, the iterative nature of their inference process makes them computationally demanding. Other approaches such as GANs are more efficient as they only need a single feed-forward pass for generation, but the image quality tends to suffer on large and diverse datasets. In this work, we propose a new class of GAN discriminators for semantic image synthesis that generates highly realistic images by exploiting feature backbone networks pre-trained for tasks such as image classification. We also introduce a new generator architecture with better context modeling and using cross-attention to inject noise into latent variables, leading to more diverse generated images. Our model, which we dub DP-SIMS, achieves state-of-the-art results in terms of image quality and consistency with the input label maps on ADE-20K, COCO-Stuff, and Cityscapes, surpassing recent diffusion models while requiring two orders of magnitude less compute for inference.

Figures (14)

• Figure 1: Images generated with models trained on COCO-Stuff, comparing our approach to state-of-the-art methods OASIS, SDM, and PITI, along with inference times to generate a single image. Our approach combines high-quality samples with low-latency sampling.
• Figure 2: Architecture of our discriminator model. The encoder consists of a pre-trained feature backbone $\mathcal{F}_\theta$ (left), residual blocks at full image resolution (top), and trained feature decoder that aggregates the multi-scale features from the frozen backbone (right).
• Figure 3: Our generator architecture consist of two components. (i) A conditional image generation network (top) that takes a low-resolution label map as input and produces the full-resolution output image. (ii) A semantic map encoding network (bottom) that takes the full resolution label map as input and produces multi-scale features that are used to modulate the intermediate features of the image generation network.
• Figure 4: Qualitative comparison with prior work on the Cityscapes dataset. We show the results of OASIS oasis, SDM sdm, and our approach along with the corresponding label map used for generating each image. Note that our method generates more coherent objects with realistic textures in comparison.
• Figure 5: Images generated by varying the noise vector with DP-SIMS trained on COCO-Stuff and using a ConvNext-L backbone.