StructLayoutFormer:Conditional Structured Layout Generation via Structure Serialization and Disentanglement

TL;DR

This work presents StructLayoutFormer, a novel Transformer-based approach for conditional structured layout generation that achieves conditional structured layout generation and produces realistic layout structures explicitly and compares its approach with existing data-driven layout generation approaches by including post-processing for structure extraction.

Abstract

Structured layouts are preferable in many 2D visual contents (\eg, GUIs, webpages) since the structural information allows convenient layout editing. Computational frameworks can help create structured layouts but require heavy labor input. Existing data-driven approaches are effective in automatically generating fixed layouts but fail to produce layout structures. We present StructLayoutFormer, a novel Transformer-based approach for conditional structured layout generation. We use a structure serialization scheme to represent structured layouts as sequences. To better control the structures of generated layouts, we disentangle the structural information from the element placements. Our approach is the first data-driven approach that achieves conditional structured layout generation and produces realistic layout structures explicitly. We compare our approach with existing data-driven layout generation approaches by including post-processing for structure extraction. Extensive experiments have shown that our approach exceeds these baselines in conditional structured layout generation. We also demonstrate that our approach is effective in extracting and transferring layout structures. The code is publicly available at %\href{https://github.com/Teagrus/StructLayoutFormer} {https://github.com/Teagrus/StructLayoutFormer}.

Figures (12)

• Figure 1: GUI layouts (top) and their major internal structures (below). In these examples, all images are under grid nodes, and their arrangements can be automatically adjusted to different screen sizes.
• Figure 2: An illustration of layout representation and serialization. a) An example GUI layout. b) The layout's visible elements. c) The layout structure and the corresponding layout sequence. d) The visualization of the layout hierarchies.
• Figure 3: An example of recovering a layout sequence to a layout tree. a) In the layout sequence, $\mathbf{N}_1$, $\mathbf{N}_2$, and $\mathbf{N}_3$ (in magenta) are internal nodes; $\mathbf{N}_4$, $\mathbf{N}_5$, and $\mathbf{N}_6$ (in green) are leaf nodes; $\mathbf{N}_1$, $\mathbf{N}_3$, $\mathbf{N}_4$, and $\mathbf{N}_6$ (with frames) are the last children of their parents. All this information is stored in the nodes. b) Detailed recovery procedure. The * symbol represents the position where the next predicted element will be placed.
• Figure 4: An overview of our model. a) An example of layout generation conditioned on element types and sizes. The purple node is the target of the current round of node prediction. b) Structure encoder is a VAE that encodes a structure sequence into a latent structure code $z$. c) Context encoder encodes the context information of $\mathbf{N}_k$(the node model predicts now) to a context code $e_k$. The context information includes its parent, sibling, and the predicted nodes. d) Conditional layout generator selects the proper condition according to structure $z$ and the context code $e_k$ then predicts attributes of the new element $\mathbf{N}_k$.
• Figure 5: Examples of four typical internal nodes. a) LinearLayout. All elements are arranged linearly in the horizontal or vertical direction. b) RelativeLayout. Elements can be represented flexibly in a non-linear manner. c) ListView. Elements are arranged in a list and are usually of equal size. d) GridView. Elements are arranged in a grid and are usually of equal size.