iSHIFT: Lightweight Slow-Fast GUI Agent with Adaptive Perception

TL;DR

iSHIFT introduces a compact ~$2.5\text{B}$ multimodal GUI agent that unifies implicit latent thinking with an on-demand Visual Perception Module to realize adaptive slow-fast reasoning. By embedding internal deliberation and a lightweight perception trigger within a single model, it achieves state-of-the-art or near state-of-the-art performance on multiple GUI benchmarks while substantially reducing parameter count. The approach demonstrates strong generalization, efficiency gains, and robustness to annotation noise, offering a practical path toward resource-aware GUI automation. This work highlights implicit adaptive reasoning as a scalable strategy for high-fidelity, fast GUI interaction in real-world applications.

Abstract

Multimodal Large Language Models (MLLMs) show strong potential for interpreting and interacting with complex, pixel-rich Graphical User Interface (GUI) environments. However, building agents that are both efficient for high-level tasks and precise for fine-grained interactions remains challenging. GUI agents must perform routine actions efficiently while also handling tasks that demand exact visual grounding, yet existing approaches struggle when accuracy depends on identifying specific interface elements. These MLLMs also remain large and cannot adapt their reasoning depth to the task at hand. In this work, we introduce iSHIFT: Implicit Slow-fast Hybrid Inference with Flexible Tokens, a lightweight agent that integrates latent thinking (implicit chain-of-thought) with a perception control module. iSHIFT enables an MLLM to switch between a slow mode, which leverages detailed visual grounding for high precision and a fast mode that uses global cues for efficiency. Special perception tokens guide attention to relevant screen regions, allowing the model to decide both how to reason and where to focus. Despite its compact 2.5B size, iSHIFT matches state-of-the-art performance on multiple benchmark datasets.

Figures (31)

• Figure 1: Left: Compared to recent state-of-the-art methods on Android In The Wild Benchmark NEURIPS2023_bbbb6308, our approach achieves a competitive performance despite smaller size. Right: iSHIFT achieves this by adaptively switching between slow and fast modes based on task needs.
• Figure 2: Overview of our proposed method. The MLLM takes a screenshot and task query as input and begins with the Fast Path. It utilizes the Latent Thinking Tokens (<bot>...<eot>) for implicit thinking to assess if the context is sufficient for direct action. If not, it switches to the Slow Path by generating Latent Perception Tokens (<bop>, <ctrl>, <eop>). This invokes the Visual Perception Module which extract localized image features ($z_p$) for precise and grounded action generation.
• Figure 3: Qualitative comparison of GUI action sequences. The figure shows step-by-step actions taken SeeClick (red), ShowUI (orange), and iSHIFT (blue) in comparison to Ground Truth (green). Each colored hand or arrow represents the predicted interaction, including touch and lift points across interface states, and the ROI is highlighted with a yellow bounding box at each step. iSHIFT aligns closely with the correct action sequence, demonstrating superior spatial precision and decision accuracy compared to prior methods.
• Figure 4: Efficiency comparison (Accuracy-to-Parameter ratio) of iSHIFT against subset wise state-of-the-art agents.
• Figure 5: Divergence of distributions of predicted action from the Ground Truth across different iSHIFT variants (Smaller bar is better), This demonstrates the adaptive model's superior action fidelity and significantly lower error rates across all actions.