MP-GUI: Modality Perception with MLLMs for GUI Understanding

TL;DR

This work addresses the challenge of GUI understanding by introducing MP-GUI, a dual-visual-clues MLLM that employs three GUI-tailored perceivers (textual, graphical, spatial) and a Fusion Gate to adaptively fuse signals for task-specific GUI knowledge. A multi-stage training strategy and a novel Spatial Relationship Prediction task, along with a data synthesis pipeline, guide explicit learning of GUI semantics despite limited data. Empirical results show MP-GUI achieves strong performance across diverse GUI benchmarks and excels at grounding small UI elements, demonstrating robust GUI perception with significantly less GUI-specific data than prior methods. The contribution lies in combining tailored perceptual modules with semantic-aware fusion and synthetic data generation to advance practical GUI understanding in real-world applications.

Abstract

Graphical user interface (GUI) has become integral to modern society, making it crucial to be understood for human-centric systems. However, unlike natural images or documents, GUIs comprise artificially designed graphical elements arranged to convey specific semantic meanings. Current multi-modal large language models (MLLMs) already proficient in processing graphical and textual components suffer from hurdles in GUI understanding due to the lack of explicit spatial structure modeling. Moreover, obtaining high-quality spatial structure data is challenging due to privacy issues and noisy environments. To address these challenges, we present MP-GUI, a specially designed MLLM for GUI understanding. MP-GUI features three precisely specialized perceivers to extract graphical, textual, and spatial modalities from the screen as GUI-tailored visual clues, with spatial structure refinement strategy and adaptively combined via a fusion gate to meet the specific preferences of different GUI understanding tasks. To cope with the scarcity of training data, we also introduce a pipeline for automatically data collecting. Extensive experiments demonstrate that MP-GUI achieves impressive results on various GUI understanding tasks with limited data.

Figures (14)

• Figure 1: (a) Graphical, textual and spatial oriented tasks on GUI understanding. (b) Some classic GUI spatial structure forms.
• Figure 2: Overview of our MP-GUI. MP-GUI consists of three parts: (1) a vision backbone providing visual clues of the screenshot; (2) a TGS-Perception Fusion Module including three GUI-tailored perceivers for extracting specific GUI modality signals and a Fusion Gate for dynamically fusing these signals based on task semantics to produce GUI-tailored visual clues; and (3) an LLM generating results relying on screen visual clues, GUI-tailored visual clues, and task semantic signal.
• Figure 3: Examples of Spatial Relationship Prediction (SRP) task.
• Figure 4: Comparison of the grounding results of various methods on UI elements of different sizes under RefExp refexp. The proportion of $k\%$ indicates that $\frac{w\times h}{W \times H} \leq k\%$, where $w$ and $h$ represent the width and height of UI elements, and $W$ and $H$ represent the screen resolution.
• Figure 5: Case studies on basic GUI understanding benchmark (\ref{['sec: experimental_setting']}). Accurately described answer is marked in green, while inaccurately and incompletely described ones in red and orange.