Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents

Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

TL;DR

AXIS introduces an API-first framework for large language model–based agents to dramatically reduce latency and cognitive burden in human-agent interactions. By automatically exploring applications to build and expand APIs, AXIS replaces multi-step UI interactions with API calls while preserving reliability and accuracy. The framework structures task execution into trajectory collection, skill generation, and skill validation, enabling scalable creation of API-driven capabilities. Empirical results in Microsoft Word show substantial gains in speed (65-70% faster task completion) and cognitive relief (38-53% reduction) with near-human accuracy, highlighting the practical impact of API-first HACI and paving the way toward an Agent OS that converts applications into autonomous agents.

Abstract

Multimodal large language models (MLLMs) have enabled LLM-based agents to directly interact with application user interfaces (UIs), enhancing agents' performance in complex tasks. However, these agents often suffer from high latency and low reliability due to the extensive sequential UI interactions. To address this issue, we propose AXIS, a novel LLM-based agents framework that prioritize actions through application programming interfaces (APIs) over UI actions. This framework also facilitates the creation and expansion of APIs through automated exploration of applications. Our experiments on Microsoft Word demonstrate that AXIS reduces task completion time by 65%-70% and cognitive workload by 38%-53%, while maintaining accuracy of 97%-98% compared to humans. Our work contributes to a new human-agent-computer interaction (HACI) framework and explores a fresh UI design principle for application providers to turn applications into agents in the era of LLMs, paving the way towards an agent-centric operating system (Agent OS).

AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents

TL;DR

AXIS introduces an API-first framework for large language model–based agents to dramatically reduce latency and cognitive burden in human-agent interactions. By automatically exploring applications to build and expand APIs, AXIS replaces multi-step UI interactions with API calls while preserving reliability and accuracy. The framework structures task execution into trajectory collection, skill generation, and skill validation, enabling scalable creation of API-driven capabilities. Empirical results in Microsoft Word show substantial gains in speed (65-70% faster task completion) and cognitive relief (38-53% reduction) with near-human accuracy, highlighting the practical impact of API-first HACI and paving the way toward an Agent OS that converts applications into autonomous agents.

Abstract

Multimodal large language models (MLLMs) have enabled LLM-based agents to directly interact with application user interfaces (UIs), enhancing agents' performance in complex tasks. However, these agents often suffer from high latency and low reliability due to the extensive sequential UI interactions. To address this issue, we propose AXIS, a novel LLM-based agents framework that prioritize actions through application programming interfaces (APIs) over UI actions. This framework also facilitates the creation and expansion of APIs through automated exploration of applications. Our experiments on Microsoft Word demonstrate that AXIS reduces task completion time by 65%-70% and cognitive workload by 38%-53%, while maintaining accuracy of 97%-98% compared to humans. Our work contributes to a new human-agent-computer interaction (HACI) framework and explores a fresh UI design principle for application providers to turn applications into agents in the era of LLMs, paving the way towards an agent-centric operating system (Agent OS).
Paper Structure (49 sections, 1 equation, 14 figures, 11 tables)

This paper contains 49 sections, 1 equation, 14 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-based UI Agent
  4. Agent Operating System
  5. UI design in LLM era
  6. Preliminary
  7. Design of AXIS
  8. Stage I: Trajectory Collection
  9. Stage II: Skill Generation
  10. Stage III: Skill Validation
  11. Feasibility Study
  12. SKill Exploration
  13. Task Completion
  14. User Study
  15. Experimental Metrics
  16. ...and 34 more sections

Figures (14)

  • Figure 1: An illustration comparing task completion methods: manual operation, UI Agent, and our approach AXIS. Manual operation risks wrong trails if users are unfamiliar with the UI. The UI Agent requires numerous sequential interactions. Our AXIS efficiently completes the task with a single API call.
  • Figure 2: Overview of AXIS framework. AXIS first collects interaction trajectories in Follower or Explorer mode. Then, the explored trajectories are used to generate skills and translate them into skill code. The skill validation stage then validates skills in the real environment. Note that the dashed boxes refer to the interaction between agents and application environment.
  • Figure 3: The results of NASA Workload and learn efforts on L1 and L2 tasks of user study. Bars indicate standard errors (**: p < 0.01, ***: p < 0.001)
  • Figure 4: The results of subjective preference on L1 and L2 tasks of user study.
  • Figure 5: The figure of the introduction page of manual mode in user study. Each participant was instructed to follow the steps to finish the task manually.
  • ...and 9 more figures