See-Control: A Multimodal Agent Framework for Smartphone Interaction with a Robotic Arm

TL;DR

The paper tackles the problem of platform-agnostic, privacy-preserving smartphone operation by introducing the Embodied Smartphone Operation (ESO) task and See-Control, a framework that enables direct physical interaction with smartphones via a low-DoF robotic arm without ADB. It contributes an ESO benchmark (155 tasks across 8 domains), a baseline MLLM-based embodied agent that reasons from screen captures, and a richly annotated dataset of operation episodes to support future Vision–Language–Action development. Experimental results show feasibility and reveal key challenges in visual grounding and cross-app tasks, complemented by a user study underscoring privacy and platform concerns with traditional software-bridge methods. The work positions embodied interaction as a valuable complement to software-based smartphone agents, broadening deployment scenarios for home robotics and informing safer, cross-platform automation strategies.

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have enabled their use as intelligent agents for smartphone operation. However, existing methods depend on the Android Debug Bridge (ADB) for data transmission and action execution, limiting their applicability to Android devices. In this work, we introduce the novel Embodied Smartphone Operation (ESO) task and present See-Control, a framework that enables smartphone operation via direct physical interaction with a low-DoF robotic arm, offering a platform-agnostic solution. See-Control comprises three key components: (1) an ESO benchmark with 155 tasks and corresponding evaluation metrics; (2) an MLLM-based embodied agent that generates robotic control commands without requiring ADB or system back-end access; and (3) a richly annotated dataset of operation episodes, offering valuable resources for future research. By bridging the gap between digital agents and the physical world, See-Control provides a concrete step toward enabling home robots to perform smartphone-dependent tasks in realistic environments.

Figures (4)

• Figure 1: Overview of the See-Control Framework. The framework consists of three main components: a benchmark, an MLLM-based embodied agent, and a richly labeled dataset of smartphone operation episodes. See-Control reframes mobile HCI automation from a software-bridge problem into an embodied perception-and-action challenge, aligning with the way people naturally interact with phones. It not only overcomes the platform limitations present in existing research but also provides a more secure solution.
• Figure 2: Pipeline of the See-Control Agent and its Visual Perception Module. The top panel shows how the agent processes user instructions and screen images to generate actions. The bottom panel provides a specific example of how the agent's reasoning leverages specialized visual grounding tools and visual prompting to accurately locate and interact with an UI element, demonstrating its ability to bridge user instructions with physical control.
• Figure 3: An example of the execution pathway of the See-Control agent during task completion, highlighting its comprehensive capabilities in task planning, reasoning, and visual perception.
• Figure 4: An example of the See-Control agent’s thinking and decision-making process, in which the agent interacts with the environment to gather observations for each state, performs reasoning, and converts its decisions into precise actions for the robotic arm. Due to space constraints, only the first four steps of the task are presented.