LayoutDiT: Exploring Content-Graphic Balance in Layout Generation with Diffusion Transformer

TL;DR

LayoutDiT is introduced, an effective framework that balances content and graphic features to generate high-quality, visually appealing layouts and achieves superior performance in both constrained and unconstrained settings, significantly outperforming existing methods.

Abstract

Layout generation is a foundation task of graphic design, which requires the integration of visual aesthetics and harmonious expression of content delivery. However, existing methods still face challenges in generating precise and visually appealing layouts, including blocking, overlapping, small-sized, or spatial misalignment. We found that these methods overlook the crucial balance between learning content-aware and graphic-aware features. This oversight results in their limited ability to model the graphic structure of layouts and generate reasonable layout arrangements. To address these challenges, we introduce LayoutDiT, an effective framework that balances content and graphic features to generate high-quality, visually appealing layouts. Specifically, we first design an adaptive factor that optimizes the model's awareness of the layout generation space, balancing the model's performance in both content and graphic aspects. Secondly, we introduce a graphic condition, the saliency bounding box, to bridge the modality difference between images in the visual domain and layouts in the geometric parameter domain. In addition, we adapt a diffusion transformer model as the backbone, whose powerful generative capability ensures the quality of layout generation. Benefiting from the properties of diffusion models, our method excels in constrained settings without introducing additional constraint modules. Extensive experimental results demonstrate that our method achieves superior performance in both constrained and unconstrained settings, significantly outperforming existing methods.

Figures (15)

• Figure 1: (a) Given a background image, our method generates reasonable and visually appealing layouts, which can be turned into a beautiful brand logo and advertising text via rendering. (b) Layouts generated by SOTA methods suffer from issues such as blocking key image areas and overlapping with each other. In contrast, our approach produces a layout that shows well-structured graphic design and seamlessly integrates with the image content.
• Figure 2: The overview of our framework. The inputs are Gaussian noise $l_T$, image $I$ with its saliency map $S$, and saliency bounding box $B$. The layout encoder and decoder serve as the main backbone, both composed of a series of transformer blocks. Image features $F_I$ and box features $F_B$ are extracted by the image encoder and bounding encoder respectively, and are incorporated into the backbone through cross-attention modules. The CGBFP module takes $F_I$ and layout representations $F_L$ as inputs to predict a balance factor $\omega$, which modulates the cross-attention interactions. Finally, the framework generates high-quality and visually appealing layouts $l_0$.
• Figure 3: Exploration of the factor $w$ on PKU dataset. The factor $w$ enhances the model’s awareness of layout generation space, thereby achieving optimal content-graphic balance. In contrast, using a constant factor often leads to poor layout performance in either aspect.
• Figure 4: Qualitative comparison of unconstrained generation on PKU and CGL. Compared to other approaches, LayoutDiT more effectively handles issues of blocking, overlapping, misalignment and small-sized elements while preserving layout diversity.
• Figure 5: Examples of ablation studies on module contributions and their effects.