MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agents

TL;DR

MPR-GUI-Bench introduces the first multilingual, fine-grained GUI perception and reasoning benchmark, revealing clear English superiority and substantial non-English gaps in LVLM-based GUI agents. To close this gap, the authors propose GUI-XLI, a training-free cross-lingual representation intervention that leverages a GUI-XL-Memory to align latent spaces across languages. Results show consistent multilingual improvements across open- and closed-source LVLMs, with notable gains in complex reasoning tasks and lower inference latency. The work provides a practical, generalizable approach to broaden GUI agents' global applicability while offering a detailed benchmark for fine-grained P&R evaluation.

Abstract

With the advancement of computational resources, Large Vision-Language Models (LVLMs) exhibit impressive Perception and Reasoning (P&R) performance on Graphical User Interface (GUI) tasks. However, although they demonstrate strong P&R capabilities in English GUI scenarios, their performance in multilingual settings has received little attention, which limits their global applications. Moreover, existing studies on GUI tasks lack fine-grained analyses, including widget functions and elements' spatial relationships, which are fundamental for more targeted improvements. To tackle these issues, we propose MPR-GUI-Bench, a Multilingual fine-grained Perception and Reasoning GUI Benchmark to evaluate GUI agents' P&R capabilities. Evaluation results demonstrate that LVLMs exhibit significantly worse P&R performance in non-English languages than in English. To address these gaps, we propose GUI-XLI, a GUI Cross-Lingual Intervention method that applies interventions to the hidden states at P&R capability-related layers to mitigate the gaps between English and other languages, building on previous research showing that the hidden states of different language inputs exhibit significant differences in the latent space. Experimental results indicate that our method improves GUI agents' multilingual P&R capability by 6.5% on average.

Figures (12)

• Figure 1: Performance of GUI agents on our MPR-GUI-Bench benchmark. The left figure illustrates that all GUI agents exhibit the strongest performance in English, while the right one exhibits their fine-grained P&R capabilities across multiple dimensions.
• Figure 2: An overview of the MPR-GUI-Bench construction pipeline in §\ref{['Step 1']}. Step 1 Screenshot Collection: Annotators collect parallel screenshots across 6 languages and diverse GUI scenarios. Step 2 Candidate VQA Lists Construction: Manually designed prompts and the English screenshots are fed to GPT-4o to construct candidate VQA lists. Step 3 Manually Checking: Annotators manually check the candidate VQA lists to ensure quality. Step 4 Expansion & Final Check: English QAs and Non-English screenshots are provided to GPT-4o for multi-lingual parallel VQA expansion, followed by annotators' manual check to ensure cross-lingual consistency.
• Figure 3: The composition of MPR-GUI-Bench. As shown with gray numbers, We generated 2156 samples for each language, specifically 351 samples for the first 6 dimensions, and 25 for each of the last 2 dimensions.
• Figure 4: t-SNE Visualization of Multilingual Hidden State before and after applied GUI-XLI.
• Figure 5: Line chart of grid search on MPR-GUI-Bench for intervention strength $\alpha$ and layer $l$ on Zh and JA. The upper two figures present the grid search results for $l$, and the lower two present those for $\alpha$.