CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMs

TL;DR

CORE addresses the privacy risk of uploading full mobile UI context to cloud LLMs by introducing an asymmetric cloud-local collaboration framework. It uses layout-aware XML block partitioning to structure UI content, a co-planning phase where local LLMs propose sub-tasks and cloud LLMs select the best, and a multi-round co-decision-making phase where the cloud progressively accumulates blocks for confident decisions. Empirical results on DroidTask and AndroidLab show up to 55.6% UI exposure reduction with small task performance penalties, along with substantial reductions in sensitive UI elements uploaded (up to 70.49%), demonstrating a practical privacy-preserving trade-off. The approach is modality-agnostic and extensible to multimodal inputs, suggesting broad applicability for privacy-preserving mobile automation.

Abstract

Mobile agents rely on Large Language Models (LLMs) to plan and execute tasks on smartphone user interfaces (UIs). While cloud-based LLMs achieve high task accuracy, they require uploading the full UI state at every step, exposing unnecessary and often irrelevant information. In contrast, local LLMs avoid UI uploads but suffer from limited capacity, resulting in lower task success rates. We propose $\textbf{CORE}$, a $\textbf{CO}$llaborative framework that combines the strengths of cloud and local LLMs to $\textbf{R}$educe UI $\textbf{E}$xposure, while maintaining task accuracy for mobile agents. CORE comprises three key components: (1) $\textbf{Layout-aware block partitioning}$, which groups semantically related UI elements based on the XML screen hierarchy; (2) $\textbf{Co-planning}$, where local and cloud LLMs collaboratively identify the current sub-task; and (3) $\textbf{Co-decision-making}$, where the local LLM ranks relevant UI blocks, and the cloud LLM selects specific UI elements within the top-ranked block. CORE further introduces a multi-round accumulation mechanism to mitigate local misjudgment or limited context. Experiments across diverse mobile apps and tasks show that CORE reduces UI exposure by up to 55.6% while maintaining task success rates slightly below cloud-only agents, effectively mitigating unnecessary privacy exposure to the cloud. The code is available at https://github.com/Entropy-Fighter/CORE.

Figures (11)

• Figure 1: Given the user query "Search NIPS", only the search bar on the Chrome start page is task-relevant. Uploading just this block to the agent is sufficient for correct decision-making, indicating that other UI blocks (basic buttons, browsing history, recommendations) are unnecessary exposure.
• Figure 2: Overview of our CORE pipeline. It has three stages: on-device pre-processing, co-planning, and co-decision-making. In the latter two stages, it leverages the cloud LLM's strength of strong reasoning abilities and the local LLM's strength of full access to the UI blocks to collaborate.
• Figure 3: Our layout-ware block partitioning on a Calendar page. By comparing the ancestors of all important nodes at the same tree depth, we group them into blocks. Different depths yield different partition granularities; we select a suitable one to obtain 3 coherent blocks.
• Figure 4: Our design vs. Baselines from latency and cloud LLM token usage.
• Figure 5: Success and reduction rates of the GPT-4o baseline and our method (using best-performing setting) across apps on both datasets. Note: On the right, GPT-4o appears as a single point due to 0% reduction. Dashed arcs indicate the overall performance on the entire dataset.