AppSelectBench: Application-Level Tool Selection Benchmark

TL;DR

AppSelectBench introduces a dedicated benchmark for application-level tool selection in computer-using agents, addressing the gap left by API-level benchmarks. It combines a multi-stage user-task generation pipeline with unified evaluation protocols across random, heuristic, zero-shot, few-shot, and retrieval-augmented prompting, evaluated on over 100 desktop applications. The study provides extensive analysis of model performance, revealing significant cross-category confusions and highlighting that retrieval augmentation and few-shot prompts offer notable gains for many models, while high-capacity models still struggle with consistent inter-application reasoning. The work establishes a foundation for advancing inter-application planning and CUA reliability, with a publicly available source and a clear path toward multi-application coordination in future work.

Abstract

Computer Using Agents (CUAs) are increasingly equipped with external tools, enabling them to perform complex and realistic tasks. For CUAs to operate effectively, application selection, which refers to deciding which application to use before invoking fine-grained tools such as APIs, is a fundamental capability. It determines whether the agent initializes the correct environment, avoids orchestration confusion, and efficiently focuses on relevant context. However, existing benchmarks primarily assess fine-grained API selection, offering limited insight into whether models can reason across and choose between different applications. To fill this gap, we introduce AppSelectBench, a comprehensive benchmark for evaluating application selection in CUAs. AppSelectBench contains a novel user task generation pipeline that produces realistic, diverse, and semantically grounded user intents at scale, together with unified evaluation protocols covering random, heuristic, zero-shot, few-shot, and retrieval-augmented-settings. AppSelectBench covers one hundred widely used desktop applications and includes more than one hundred thousand realistic, diverse, and semantically grounded user tasks. Extensive experiments across both closed-source and open-source large language models reveal systematic strengths and weaknesses in inter-application reasoning, showing that even the most capable models still struggle to make consistent application choices. Together, these results establish AppSelectBench as a foundation for studying and advancing application level reasoning, an essential yet underexplored capability of intelligent CUAs. The source is available at https://microsoft.github.io/appselectbench/.

Figures (10)

• Figure 1: The overview of the AppSelectBench framework, including three core stages: (1) a multi-step generation pipeline producing over 100k realistic tasks; (2) a parallelized tool-selection core where multiple LLMs are tested against various protocols; and (3) the resulting in-depth performance analysis.
• Figure 2: The three levels of task complexity: (1) Singleton Application for tasks solvable by a single tool; (2) Sequential Execution represented by a linear chain (e.g., Browser $\rightarrow$ PowerPoint); (3) Parallel or Branched Execution with multiple outgoing or incoming edges.
• Figure 3: Distribution of the 100 desktop applications in AppSelectBench across 12 high-level categories.
• Figure 4: User task generation pipeline that composes atomic actions into workflows and narrates them into natural-language instructions.
• Figure 5: Instruction Narrator with step drop-out and LLM paraphrasing to create realistic user task descriptions.