ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents

TL;DR

ToolTok reframes GUI agent control from coordinate regression to multi-step pathfinding using discrete, learnable tool tokens. By coupling spherical semantic anchoring with a three-stage curriculum that progresses from semantic grounding to simplified perception and finally real-world navigation, ToolTok achieves strong data efficiency and robust generalization on diverse GUI benchmarks, even with substantially fewer training samples than prior post-training approaches. The approach yields state-of-the-art performance among 4B models and competitive results with much larger models, highlighting the value of explicit tool semantics and structured reasoning. This paradigm shift has practical implications for building more generalizable, data-efficient GUI automation and could transfer to other domains requiring learned token-alignment with pre-trained backbones.

Abstract

Existing GUI agent models relying on coordinate-based one-step visual grounding struggle with generalizing to varying input resolutions and aspect ratios. Alternatives introduce coordinate-free strategies yet suffer from learning under severe data scarcity. To address the limitations, we propose ToolTok, a novel paradigm of multi-step pathfinding for GUI agents, where operations are modeled as a sequence of progressive tool usage. Specifically, we devise tools aligned with human interaction habits and represent each tool using learnable token embeddings. To enable efficient embedding learning under limited supervision, ToolTok introduces a semantic anchoring mechanism that grounds each tool with semantically related concepts as natural inductive bias. To further enable a pre-trained large language model to progressively acquire tool semantics, we construct an easy-to-hard curriculum consisting of three tasks: token definition question-answering, pure text-guided tool selection, and simplified visual pathfinding. Extensive experiments on multiple benchmarks show that ToolTok achieves superior performance among models of comparable scale (4B) and remains competitive with a substantially larger model (235B). Notably, these results are obtained using less than 1% of the training data required by other post-training approaches. In addition, ToolTok demonstrates strong generalization across unseen scenarios. Our training & inference code is open-source at https://github.com/ZephinueCode/ToolTok.

Figures (8)

• Figure 1: Comparison of GUI agent interaction paradigms. Left: One-step visual grounding with direct coordinate prediction. Right: ToolTok, which performs multi-step visual pathfinding via discrete tool tokens, leading to improved robustness, data efficiency, and semantic interpretability.
• Figure 2: Performance comparison under varying resolutions and aspect ratios. While baselines degrade significantly in extreme cases, our model demonstrates superior robustness using less than 1% of the training data.
• Figure 3: The main pipeline of ToolTok. Featuring discrete action tokens, an efficient token initialization method of spherical semantic anchoring, and a curriculum learning schedule featuring three stages.
• Figure 4: Visualization for multiple experiment cases of different resolutions, ratios and usage scenarios. Our models generate intuitive and easy to understand paths while also supporting more varied and human-like actions. Row 1 samples show the superior semantic robustness of our models (self correction, shortcutting, etc.), while row 2 samples show emergent generalization capabilities where our models output valid but untrained actions.
• Figure 5: Test-time scaling results of our models. As the maximum allowed step counts increases, the performance of our models significantly improves.