Generating Multiple Objects at Spatially Distinct Locations

TL;DR

The paper addresses the need for GANs to generate images containing multiple objects at specified locations without requiring full semantic scene layouts. It introduces an object pathway, applied per bounding box, and a complementary global pathway, enabling per-object feature generation and global scene layout while the generator and discriminator share these pathways. Experiments on Multi-MNIST, CLEVR, and MS-COCO show the model can control object identity, location, and count, generalize to novel configurations, and improve image quality as measured by IS and FID. Ablation studies and analysis reveal the importance of both pathways and highlight limitations such as bounding-box overlaps and reliance on provided boxes, suggesting future work on automatic layout extraction and enhanced merging.

Abstract

Recent improvements to Generative Adversarial Networks (GANs) have made it possible to generate realistic images in high resolution based on natural language descriptions such as image captions. Furthermore, conditional GANs allow us to control the image generation process through labels or even natural language descriptions. However, fine-grained control of the image layout, i.e. where in the image specific objects should be located, is still difficult to achieve. This is especially true for images that should contain multiple distinct objects at different spatial locations. We introduce a new approach which allows us to control the location of arbitrarily many objects within an image by adding an object pathway to both the generator and the discriminator. Our approach does not need a detailed semantic layout but only bounding boxes and the respective labels of the desired objects are needed. The object pathway focuses solely on the individual objects and is iteratively applied at the locations specified by the bounding boxes. The global pathway focuses on the image background and the general image layout. We perform experiments on the Multi-MNIST, CLEVR, and the more complex MS-COCO data set. Our experiments show that through the use of the object pathway we can control object locations within images and can model complex scenes with multiple objects at various locations. We further show that the object pathway focuses on the individual objects and learns features relevant for these, while the global pathway focuses on global image characteristics and the image background.

Figures (11)

• Figure 1: Both the generator and the discriminator of our model consist of a global and an object pathway. The global pathway focuses on global image characteristics, such as the background, while the object pathway is responsible for modeling individual objects at their specified location.
• Figure 2: Multi-MNIST images generated by the model. Training included only images with three individual normal-sized digits. Highlighted bounding boxes and yellow ground truth for visualization.
• Figure 3: Images from the CLEVR data set. The left image of each pair shows the rendered image according to specific attributes. The right image of each pair is the image generated by our model.
• Figure 4: Examples of images generated from the given caption from the MS-COCO data set. A) shows the original images and the respective image captions, B) shows images generated by our StackGAN+OP (with the corresponding bounding boxes for visualization), and C) shows images generated by the original StackGAN zhang2017stackgan
• Figure 5: Examples of images generated from the given caption from the MS-COCO data set. A) shows the original images and the respective image captions, B) shows images generated by our StackGAN+OP (with the corresponding bounding boxes for visualization) with the object pathway enabled, C) shows images generated by the our StackGAN+OP when the object pathway is disabled, and D) shows images generated by the our StackGAN+OP when the global pathway is disabled.