Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aggregated Structural Representation with Large Language Models for Human-Centric Layout Generation

Jiongchao Jin, Shengchu Zhao, Dajun Chen, Wei Jiang, Yong Li

TL;DR

This paper addresses the challenge of automated UI layout generation by preserving structural information while enabling rich generative capabilities. It proposes Aggregation Structural Representation (ASR), which fuses graph-based hierarchical representations with a multimodal large language model via an editable, human-editable relation matrix to guide generation. The method demonstrates strong quantitative gains on the RICO dataset (e.g., higher $mIoU$ and lower $RE$) and favorable perceptual results from a crowd study, while also enabling diverse outputs through relation-feature sampling. ASR thus provides a scalable, human-in-the-loop framework with potential extension to scene and floorplan understanding, offering practical impact for multi-device UI design and beyond.

Abstract

Time consumption and the complexity of manual layout design make automated layout generation a critical task, especially for multiple applications across different mobile devices. Existing graph-based layout generation approaches suffer from limited generative capability, often resulting in unreasonable and incompatible outputs. Meanwhile, vision based generative models tend to overlook the original structural information, leading to component intersections and overlaps. To address these challenges, we propose an Aggregation Structural Representation (ASR) module that integrates graph networks with large language models (LLMs) to preserve structural information while enhancing generative capability. This novel pipeline utilizes graph features as hierarchical prior knowledge, replacing the traditional Vision Transformer (ViT) module in multimodal large language models (MLLM) to predict full layout information for the first time. Moreover, the intermediate graph matrix used as input for the LLM is human editable, enabling progressive, human centric design generation. A comprehensive evaluation on the RICO dataset demonstrates the strong performance of ASR, both quantitatively using mean Intersection over Union (mIoU), and qualitatively through a crowdsourced user study. Additionally, sampling on relational features ensures diverse layout generation, further enhancing the adaptability and creativity of the proposed approach.

Aggregated Structural Representation with Large Language Models for Human-Centric Layout Generation

TL;DR

This paper addresses the challenge of automated UI layout generation by preserving structural information while enabling rich generative capabilities. It proposes Aggregation Structural Representation (ASR), which fuses graph-based hierarchical representations with a multimodal large language model via an editable, human-editable relation matrix to guide generation. The method demonstrates strong quantitative gains on the RICO dataset (e.g., higher and lower ) and favorable perceptual results from a crowd study, while also enabling diverse outputs through relation-feature sampling. ASR thus provides a scalable, human-in-the-loop framework with potential extension to scene and floorplan understanding, offering practical impact for multi-device UI design and beyond.

Abstract

Time consumption and the complexity of manual layout design make automated layout generation a critical task, especially for multiple applications across different mobile devices. Existing graph-based layout generation approaches suffer from limited generative capability, often resulting in unreasonable and incompatible outputs. Meanwhile, vision based generative models tend to overlook the original structural information, leading to component intersections and overlaps. To address these challenges, we propose an Aggregation Structural Representation (ASR) module that integrates graph networks with large language models (LLMs) to preserve structural information while enhancing generative capability. This novel pipeline utilizes graph features as hierarchical prior knowledge, replacing the traditional Vision Transformer (ViT) module in multimodal large language models (MLLM) to predict full layout information for the first time. Moreover, the intermediate graph matrix used as input for the LLM is human editable, enabling progressive, human centric design generation. A comprehensive evaluation on the RICO dataset demonstrates the strong performance of ASR, both quantitatively using mean Intersection over Union (mIoU), and qualitatively through a crowdsourced user study. Additionally, sampling on relational features ensures diverse layout generation, further enhancing the adaptability and creativity of the proposed approach.

Paper Structure

This paper contains 23 sections, 5 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Graph Neural Network
  4. Large Language Model
  5. Layout Generation
  6. Methods
  7. Graph Definition $\mathcal{G}_{<N, E>}$
  8. Graph Representation Module $\mathscr{F}_\mathcal{G}$
  9. Relation Matrix $M$
  10. Large Language Model Aggregation
  11. Experiments
  12. Experiment Settings
  13. Metrics and subtask settings
  14. Quantitative Results
  15. Perceptual Results
  16. ...and 8 more sections

Figures (4)

  • Figure 1: ASR extracts hierarchical representations from graphs with semantic and positional relationships, then aggregates them with a large language model for layout generation. Human interaction during the process enables customization, resulting in diverse, user-centered layouts adaptable to practical design needs.
  • Figure 2: Overview pipeline of ASR. The top subfigure shows the training phase of ASR while the bottom subfigure indicates the inference phase of ASR.
  • Figure 3: The explanation of semantic relations and positional relations
  • Figure 4: The gallery of qualitative results in different tasks