Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generation

Jaejung Seol, Seojun Kim, Jaejun Yoo

TL;DR

PosterLlama addresses the challenge of content-aware poster layout generation by reformatting layout elements as HTML sequences and leveraging the design knowledge embedded in large language models. It introduces a two-stage vision-language training pipeline with a visual encoder adapter and an HTML-based generation target, augmented by depth-guided data augmentation to mitigate data scarcity and inpainting artifacts. Empirical results show PosterLlama achieving state-of-the-art or competitive performance across graphic and content measures on CGL and PKU datasets, while demonstrating strong conditional generation capabilities and robustness to data leakage. The method offers practical utility through a one-click poster generation pipeline and proves effective even with limited data, making it suitable for real-world design workflows.

Abstract

Visual layout plays a critical role in graphic design fields such as advertising, posters, and web UI design. The recent trend towards content-aware layout generation through generative models has shown promise, yet it often overlooks the semantic intricacies of layout design by treating it as a simple numerical optimization. To bridge this gap, we introduce PosterLlama, a network designed for generating visually and textually coherent layouts by reformatting layout elements into HTML code and leveraging the rich design knowledge embedded within language models. Furthermore, we enhance the robustness of our model with a unique depth-based poster augmentation strategy. This ensures our generated layouts remain semantically rich but also visually appealing, even with limited data. Our extensive evaluations across several benchmarks demonstrate that PosterLlama outperforms existing methods in producing authentic and content-aware layouts. It supports an unparalleled range of conditions, including but not limited to unconditional layout generation, element conditional layout generation, layout completion, among others, serving as a highly versatile user manipulation tool.

PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generation

TL;DR

PosterLlama addresses the challenge of content-aware poster layout generation by reformatting layout elements as HTML sequences and leveraging the design knowledge embedded in large language models. It introduces a two-stage vision-language training pipeline with a visual encoder adapter and an HTML-based generation target, augmented by depth-guided data augmentation to mitigate data scarcity and inpainting artifacts. Empirical results show PosterLlama achieving state-of-the-art or competitive performance across graphic and content measures on CGL and PKU datasets, while demonstrating strong conditional generation capabilities and robustness to data leakage. The method offers practical utility through a one-click poster generation pipeline and proves effective even with limited data, making it suitable for real-world design workflows.

Abstract

Visual layout plays a critical role in graphic design fields such as advertising, posters, and web UI design. The recent trend towards content-aware layout generation through generative models has shown promise, yet it often overlooks the semantic intricacies of layout design by treating it as a simple numerical optimization. To bridge this gap, we introduce PosterLlama, a network designed for generating visually and textually coherent layouts by reformatting layout elements into HTML code and leveraging the rich design knowledge embedded within language models. Furthermore, we enhance the robustness of our model with a unique depth-based poster augmentation strategy. This ensures our generated layouts remain semantically rich but also visually appealing, even with limited data. Our extensive evaluations across several benchmarks demonstrate that PosterLlama outperforms existing methods in producing authentic and content-aware layouts. It supports an unparalleled range of conditions, including but not limited to unconditional layout generation, element conditional layout generation, layout completion, among others, serving as a highly versatile user manipulation tool.
Paper Structure (40 sections, 12 figures, 5 tables)

This paper contains 40 sections, 12 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Content-Agnostic Layout Generation
  4. Content-Aware Layout Generation
  5. Method
  6. Input Output Sequence Formatting
  7. Layout Formation
  8. HTML Formatting
  9. Training Method
  10. Depth-Guided Poster Augmentation
  11. Experiments
  12. Experimental setting
  13. Datasets
  14. Baselines
  15. Implementation details
  16. ...and 25 more sections

Figures (12)

  • Figure 1: (a) The overall training step of PosterLlama. The first stage of training (top) involves training the adapter to learn visual alignment. In the second stage of training (bottom), we fine-tune the model for HTML formatted layout generation, utilizing the frozen adapter. (b) Illustration of input-output sequence in PosterLlama
  • Figure 2: (a) The illustration of depth-guided poster augmentation. The estimated depth map and caption undergo multiple augmentations with the ControlNet, and the top-k similarity samples with real images are selected as augmentation samples. (b) The illustration of augmented samples.
  • Figure 3: The data-leakage example of inpainted area. In addition to the original layout Red box area is deliberately erased and inpainted. Note that RADM puts the layouts precisely at the inpainted area.
  • Figure 4: Visualization of PosterLlama across five conditional generations, with the conditions for layout generation positioned at the top. The order of elements is $(c, x, y, w, h)$.
  • Figure 5: The example of proposed poster generation pipeline. The layout of scene text generation is generated by PosterLlama. Note that the right two samples exhibit unsatisfactory outcomes, attributed to inappropriate bounding box sizes.
  • ...and 7 more figures