APD-Agents: A Large Language Model-Driven Multi-Agents Collaborative Framework for Automated Page Design

TL;DR

This paper addresses the heavy workload of mobile app page layout design by proposing APD-agents, a large-language-model-driven multi-agent framework that outputs structured JSON layouts compatible with tools like Sketch and Figma. The system decomposes design into specialized agents (Orchestrator, SemanticParser, TemplateRetrieval, PrimaryLayout, RecursiveComponent) that collaboratively produce a coarse-to-fine page design and recursively refine sub-elements, guided by few-shot exemplars. It introduces a retrieval-based design-spec summarization mechanism to adapt exemplars to each page and demonstrates state-of-the-art performance on the RICO dataset across multiple metrics. The work facilitates editable, consistent, and scalable automated page design with potential for broader deployment in design pipelines.

Abstract

Layout design is a crucial step in developing mobile app pages. However, crafting satisfactory designs is time-intensive for designers: they need to consider which controls and content to present on the page, and then repeatedly adjust their size, position, and style for better aesthetics and structure. Although many design software can now help to perform these repetitive tasks, extensive training is needed to use them effectively. Moreover, collaborative design across app pages demands extra time to align standards and ensure consistent styling. In this work, we propose APD-agents, a large language model (LLM) driven multi-agent framework for automated page design in mobile applications. Our framework contains OrchestratorAgent, SemanticParserAgent, PrimaryLayoutAgent, TemplateRetrievalAgent, and RecursiveComponentAgent. Upon receiving the user's description of the page, the OrchestratorAgent can dynamically can direct other agents to accomplish users' design task. To be specific, the SemanticParserAgent is responsible for converting users' descriptions of page content into structured data. The PrimaryLayoutAgent can generate an initial coarse-grained layout of this page. The TemplateRetrievalAgent can fetch semantically relevant few-shot examples and enhance the quality of layout generation. Besides, a RecursiveComponentAgent can be used to decide how to recursively generate all the fine-grained sub-elements it contains for each element in the layout. Our work fully leverages the automatic collaboration capabilities of large-model-driven multi-agent systems. Experimental results on the RICO dataset show that our APD-agents achieve state-of-the-art performance.

Figures (8)

• Figure 1: An example of page design with Sketch. The elements of one page should be layered so that the designers can modify them via Sketch, Figma or other design software.
• Figure 2: Our APD-agents framework.
• Figure 3: Example of the input data and output schema. In the output format, the sub elements in key "layers" have the same json format as the output.
• Figure 4: Example of template retrieval process. For toolbar to be generated, we retrieve from the element template library and select the most similar one (framed in green in the figure) as the element template to add in the prompt.
• Figure 5: Examples of design specification summary. When generating the layout of the first level elements, we send the retrieval results into a LLM to get the design specifications. Prompts for generating design specifications is provided in the appendix.