Automatic Layout Planning for Visually-Rich Documents with Instruction-Following Models

TL;DR

This work introduces DocLap, a multimodal instruction-following framework for automatic layout planning of visually-rich documents. By modeling the task as aligning a sequence of visual components $\{ {\bm{i}}_1, ..., {\bm{i}}_n \}$ onto a canvas of size $({\bm{w}}, {\bm{h}})$ for a given application ${\bm{a}}$, and by encoding per-component layout via CSS-like tokens (top, left, width, height, layer), the method supports coordinates, recovery, and planning tasks. DocLap extends mPLUG-Owl with Llama-7b and CLIP features, introduces a visual abstractor of 64 tokens, and augments the Llama vocabulary with numeric tokens $0$ to $128$ to handle layout-specific reasoning. Evaluations on Crello and PosterLayout show DocLap competitive performance against few-shot GPT-4V and competitive baselines, demonstrating the potential of instruction-following models to democratize design automation while highlighting trade-offs in accuracy, object size, and occlusion. The results suggest a practical path toward accessible, task-conditioned layout generation for visually-rich documents, with future work addressing robustness to components, richer evaluation metrics, and ethical use considerations.

Abstract

Recent advancements in instruction-following models have made user interactions with models more user-friendly and efficient, broadening their applicability. In graphic design, non-professional users often struggle to create visually appealing layouts due to limited skills and resources. In this work, we introduce a novel multimodal instruction-following framework for layout planning, allowing users to easily arrange visual elements into tailored layouts by specifying canvas size and design purpose, such as for book covers, posters, brochures, or menus. We developed three layout reasoning tasks to train the model in understanding and executing layout instructions. Experiments on two benchmarks show that our method not only simplifies the design process for non-professionals but also surpasses the performance of few-shot GPT-4V models, with mIoU higher by 12% on Crello. This progress highlights the potential of multimodal instruction-following models to automate and simplify the design process, providing an approachable solution for a wide range of design tasks on visually-rich documents.

Figures (5)

• Figure 1: An example of a model conducting automatic layout planning following human-provided instructions and arranging visual contents for design purpose.
• Figure 2: Example inputs and outputs of the three layout reasoning tasks. (a) and (b) are examples from Crello Yamaguchi2021CanvasVAELT, while (c) is an example from PosterLayout Hsu2023PosterLayoutAN.
• Figure 3: (a) mIoU variation with the number of visual components in design templates. (b) IoU correlation with the relative size of a single visual component. Both plots pertain to Crello.
• Figure 4: Qualitative comparisons for layout planning results on Crello. GPT-4V w/ 1-shot learning.
• Figure 5: Qualitative comparisons for layout planning results on PosterLayout. GPT-4V w/ 1-shot learning.