High-Resolution Complex Scene Synthesis with Transformers

TL;DR

The paper presents a pure likelihood framework for high-resolution complex scene synthesis from coarse layouts by combining a VQGAN-based discrete latent space with an autoregressive transformer conditioned on tokenized layouts. By compressing images into discrete tokens and modeling their distribution in latent space, the approach avoids auxiliary losses and intermediate steps typical of prior methods. It achieves state-of-the-art FID/SceneFID on COCO-Stuff and Visual Genome, scales to 512×512 outputs, and demonstrates zero-shot transfer from Open Images to COCO. The results indicate that latent-space autoregressive modeling, guided by adversarially learned compression, effectively captures large-scale scene structure while suppressing texture detail in the latent representation. This yields high-quality, layout-consistent scene synthesis with practical implications for scalable, high-resolution image generation from coarse scene specifications.

Abstract

The use of coarse-grained layouts for controllable synthesis of complex scene images via deep generative models has recently gained popularity. However, results of current approaches still fall short of their promise of high-resolution synthesis. We hypothesize that this is mostly due to the highly engineered nature of these approaches which often rely on auxiliary losses and intermediate steps such as mask generators. In this note, we present an orthogonal approach to this task, where the generative model is based on pure likelihood training without additional objectives. To do so, we first optimize a powerful compression model with adversarial training which learns to reconstruct its inputs via a discrete latent bottleneck and thereby effectively strips the latent representation of high-frequency details such as texture. Subsequently, we train an autoregressive transformer model to learn the distribution of the discrete image representations conditioned on a tokenized version of the layouts. Our experiments show that the resulting system is able to synthesize high-quality images consistent with the given layouts. In particular, we improve the state-of-the-art FID score on COCO-Stuff and on Visual Genome by up to 19% and 53% and demonstrate the synthesis of images up to 512 x 512 px on COCO and Open Images.

Figures (8)

• Figure 1: From houseplants to fire hydrants, from skaters to kite surfers, from sheep to giraffes: Our model knows a broad range of objects and, given a bounding box layout, generates high-resolution images. It adds realistic context and details (for instance, short beachwear or marked sheep) and relates them in a meaningful way (for instance, skaters grabbing their board or people reading books close to them). These 512$\times$512 pixel samples from our model trained on COCO data caesar_2018_cocostuff are best viewed zoomed in.
• Figure 2: Method. Encoder and decoder are trained in a first stage as a VQGAN taming_vqgan and encode image content in a compact, discrete latent space. a) An autoregressive transformer is then trained to model the distribution of tokens within the latent space conditioned on object layouts. These layouts are received in the tokenized fashion shown on the right and are simply prepended to the image tokens. b) After training, the codebook vectors sampled by the transformer are then passed through the decoder for image rendering.
• Figure 3: Qualitative comparison on 256 and 512 px with the SOTA model LostGAN-v2 sun_2020_learning_lost_gan_v_2. Our model produces images with higher visual quality and consistency. Best viewed zoomed in.
• Figure 4: Zero-shot scene image generation. We sample COCO test layouts with a model trained on full COCO (only things classes) and on Open Images on 256$\times$256 px (330 classes). In order to do so, we filter the layouts for the 80 things classes that are known to both our models. The COCO model is specialized on this dataset and thus presents a higher visual quality. The OpenImages model (not fully converged) generalizes well to COCO layouts and demonstrates a higher diversity.
• Figure 5: Qualitative comparison with existing architectures for 128 px. Our model produces consistent images with objects in the desired positions and high visual quality. We especially highlight the ability to produce sharp textures. Comparison model samples taken from sylvain_oc_gan.