LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators

TL;DR

LayoutGAN directly generates structured graphic layouts by modeling inter-element relations among labeled 2D elements. It introduces a self-attention-based generator and two discriminators, including a differentiable wireframe rendering layer that enables CNN-based evaluation of layout alignment in image space. Across MNIST point layouts, document layouts, clipart scenes, tangrams, and mobile layouts, the wireframe discriminator yields superior alignment, reduced overlap, and more coherent relational structures. This approach decouples layout generation from pixel rendering, enabling scalable, permutation-invariant layout synthesis with strong cross-domain performance.

Abstract

Layout is important for graphic design and scene generation. We propose a novel Generative Adversarial Network, called LayoutGAN, that synthesizes layouts by modeling geometric relations of different types of 2D elements. The generator of LayoutGAN takes as input a set of randomly-placed 2D graphic elements and uses self-attention modules to refine their labels and geometric parameters jointly to produce a realistic layout. Accurate alignment is critical for good layouts. We thus propose a novel differentiable wireframe rendering layer that maps the generated layout to a wireframe image, upon which a CNN-based discriminator is used to optimize the layouts in image space. We validate the effectiveness of LayoutGAN in various experiments including MNIST digit generation, document layout generation, clipart abstract scene generation and tangram graphic design.

Figures (14)

• Figure 1: Overall architecture of LayoutGAN. The generator takes as input graphic elements with randomly sampled class probabilities and geometric parameters from Uniform and Gaussian distribution respectively. An encoder embeds the input and feeds them into the stacked relation module, which refines the embedded features of each element in a coordinative manner by considering its semantic and spatial relations with all the other elements. Finally, a decoder decodes the refined features back to class probabilities and geometric parameters. The wireframe rendering discriminator feeds the generated results to a differentiable wireframe rendering layer which raterizes the input graphic elements into 2D wireframe images, upon which a CNN is applied for layout optimization.
• Figure 1: Visualization of document layout samples and their corresponding real document pages.
• Figure 2: Wireframe rendering of different polygons (point, rectangle and triangle). The black grids represent grids of target image. The orange dots/dotted lines represent the graphic element mapped onto the image grid. The blue solid lines represent the rasterized wireframes expressed as differentiable functions of graphic elements in terms of both class probilities and geometric parameters.
• Figure 2: Document layout generation.
• Figure 3: Results on MNIST digit generation.