Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Every Software as an Agent: Blueprint and Case Study

Mengwei Xu

TL;DR

This work addresses the limitations of traditional API- and GUI-based software agents by proposing JiT-Codegen, a whitebox framework that grants LLMs access to a program's source code and runtime context to generate and inject executable action code directly into the software. The architecture comprises a Code Agent and an Execution Sandbox that interact iteratively to turn natural language instructions into precise in-software actions, leveraging runtime data, UI elements, and databases. Case studies on Electron-based desktop apps demonstrate feasibility, with GPT-4o achieving an 80% task completion rate across 10 tasks, indicating the potential for significantly more capable and efficient software agents. The paper discusses safety, accuracy, deployment considerations (on-cloud vs on-device), and multi-agent extensions, outlining a roadmap toward scalable, collaborative software automation that can understand, operate, and even think to meet complex user needs.

Abstract

The rise of (multimodal) large language models (LLMs) has shed light on software agent -- where software can understand and follow user instructions in natural language. However, existing approaches such as API-based and GUI-based agents are far from satisfactory at accuracy and efficiency aspects. Instead, we advocate to endow LLMs with access to the software internals (source code and runtime context) and the permission to dynamically inject generated code into software for execution. In such a whitebox setting, one may better leverage the software context and the coding ability of LLMs. We then present an overall design architecture and case studies on two popular web-based desktop applications. We also give in-depth discussion of the challenges and future directions. We deem that such a new paradigm has the potential to fundamentally overturn the existing software agent design, and finally creating a digital world in which software can comprehend, operate, collaborate, and even think to meet complex user needs.

Every Software as an Agent: Blueprint and Case Study

TL;DR

This work addresses the limitations of traditional API- and GUI-based software agents by proposing JiT-Codegen, a whitebox framework that grants LLMs access to a program's source code and runtime context to generate and inject executable action code directly into the software. The architecture comprises a Code Agent and an Execution Sandbox that interact iteratively to turn natural language instructions into precise in-software actions, leveraging runtime data, UI elements, and databases. Case studies on Electron-based desktop apps demonstrate feasibility, with GPT-4o achieving an 80% task completion rate across 10 tasks, indicating the potential for significantly more capable and efficient software agents. The paper discusses safety, accuracy, deployment considerations (on-cloud vs on-device), and multi-agent extensions, outlining a roadmap toward scalable, collaborative software automation that can understand, operate, and even think to meet complex user needs.

Abstract

The rise of (multimodal) large language models (LLMs) has shed light on software agent -- where software can understand and follow user instructions in natural language. However, existing approaches such as API-based and GUI-based agents are far from satisfactory at accuracy and efficiency aspects. Instead, we advocate to endow LLMs with access to the software internals (source code and runtime context) and the permission to dynamically inject generated code into software for execution. In such a whitebox setting, one may better leverage the software context and the coding ability of LLMs. We then present an overall design architecture and case studies on two popular web-based desktop applications. We also give in-depth discussion of the challenges and future directions. We deem that such a new paradigm has the potential to fundamentally overturn the existing software agent design, and finally creating a digital world in which software can comprehend, operate, collaborate, and even think to meet complex user needs.

Paper Structure

This paper contains 10 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The Status Quo and Their Inadequacies
  3. A Vision of Software Agent
  4. Case Studies of Desktop Applications
  5. Challenges and Future Directions
  6. Safeguarding against Vulnerable Action Code
  7. Improving Action Code Accuracy
  8. On-Cloud vs On-Devices Codegen
  9. Multi-Software Agent Systems
  10. Conclusions

Figures (5)

  • Figure 1: An illustrative example showing the difference between GUI-based agent and our proposal of JiT-Codegen-based agent. The software tested is a popular open-sourced markdown editor.
  • Figure 2: An overview of software agent using JiT-Codegen.
  • Figure 3: A comparison of software agents from the perspective of state machines.
  • Figure 4: Examples of the action code generated by different LLMs.
  • Figure 5: An example showing how multiple software agents collectively solve a user task.