VASCAR: Content-Aware Layout Generation via Visual-Aware Self-Correction

TL;DR

This paper proposes the training-free Visual-Aware Self-Correction LAyout GeneRation (VASCAR), which enables LVLMs to iteratively refine their outputs with reference to rendered layout images, which are visualized as colored bounding boxes on poster background (i.e., canvas).

Abstract

Large language models (LLMs) have proven effective for layout generation due to their ability to produce structure-description languages, such as HTML or JSON. In this paper, we argue that while LLMs can perform reasonably well in certain cases, their intrinsic limitation of not being able to perceive images restricts their effectiveness in tasks requiring visual content, e.g., content-aware layout generation. Therefore, we explore whether large vision-language models (LVLMs) can be applied to content-aware layout generation. To this end, inspired by the iterative revision and heuristic evaluation workflow of designers, we propose the training-free Visual-Aware Self-Correction LAyout GeneRation (VASCAR). VASCAR enables LVLMs (e.g., GPT-4o and Gemini) iteratively refine their outputs with reference to rendered layout images, which are visualized as colored bounding boxes on poster background (i.e., canvas). Extensive experiments and user study demonstrate VASCAR's effectiveness, achieving state-of-the-art (SOTA) layout generation quality. Furthermore, the generalizability of VASCAR across GPT-4o and Gemini demonstrates its versatility.

Figures (17)

• Figure 1: The designers' workflow follows an iterative revision process li2024revision and heuristics based on guidelines duan2024generating. The conversation highlights the ability of LVLMs to evaluate and identify design issues on rendered problematic layouts, and provide valuable feedback. This capability is leveraged appropriately to improve content-aware layout generation, like the workflow of designers.
• Figure 2: An overview of VASCAR. First, ICL retriever uses the saliency map of the query image to retrieve several ICL examples from the dataset. The retrieved samples are transformed to rendered images along with the text layout in HTML format. We input all the ICL examples and the query into a frozen LVLM to generate the layout in HTML format. The LVLM is queried iteratively with optimized prompts by layout scorer and suggester. After $I$ iterations, VASCAR return the candidate with the highest score.
• Figure 3: Visual comparison of baselines and VASCAR with different values of $I$. More examples can be found in Appendix.
• Figure 3: Quantitative result of five constrained generation tasks on the PKU and CGL test splits. Best and second-best results are in bold and underline, respectively.
• Figure 4: The trends of each metric based on the number of self-correction $I$.