Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interaction

Bin Lei, Weitai Kang, Zijian Zhang, Winson Chen, Xi Xie, Shan Zuo, Mimi Xie, Ali Payani, Mingyi Hong, Yan Yan, Caiwen Ding

TL;DR

InfantAgent-Next addresses the need for a generalist agent capable of multimodal computer interaction by unifying tool-based and pure-vision approaches within a modular, memory-aware architecture. It routes subtasks to specialized models (planning, tool selection, execution, visual grounding, audio, etc.), maintains a unified dialogue history, and uses dynamic toolkits to optimize performance. Empirical results across OSWorld, SWE-Bench, and GAIA demonstrate competitive to state-of-the-art performance, with strong open-source standings on SWE-Bench and GAIA. The work provides open-source code, modular toolkit, and evaluation scripts to accelerate research in multimodal agents.

Abstract

This paper introduces \textsc{InfantAgent-Next}, a generalist agent capable of interacting with computers in a multimodal manner, encompassing text, images, audio, and video. Unlike existing approaches that either build intricate workflows around a single large model or only provide workflow modularity, our agent integrates tool-based and pure vision agents within a highly modular architecture, enabling different models to collaboratively solve decoupled tasks in a step-by-step manner. Our generality is demonstrated by our ability to evaluate not only pure vision-based real-world benchmarks (i.e., OSWorld), but also more general or tool-intensive benchmarks (e.g., GAIA and SWE-Bench). Specifically, we achieve $\mathbf{7.27\%}$ accuracy on OSWorld, higher than Claude-Computer-Use. Codes and evaluation scripts are open-sourced at https://github.com/bin123apple/InfantAgent.

InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interaction

TL;DR

InfantAgent-Next addresses the need for a generalist agent capable of multimodal computer interaction by unifying tool-based and pure-vision approaches within a modular, memory-aware architecture. It routes subtasks to specialized models (planning, tool selection, execution, visual grounding, audio, etc.), maintains a unified dialogue history, and uses dynamic toolkits to optimize performance. Empirical results across OSWorld, SWE-Bench, and GAIA demonstrate competitive to state-of-the-art performance, with strong open-source standings on SWE-Bench and GAIA. The work provides open-source code, modular toolkit, and evaluation scripts to accelerate research in multimodal agents.

Abstract

This paper introduces \textsc{InfantAgent-Next}, a generalist agent capable of interacting with computers in a multimodal manner, encompassing text, images, audio, and video. Unlike existing approaches that either build intricate workflows around a single large model or only provide workflow modularity, our agent integrates tool-based and pure vision agents within a highly modular architecture, enabling different models to collaboratively solve decoupled tasks in a step-by-step manner. Our generality is demonstrated by our ability to evaluate not only pure vision-based real-world benchmarks (i.e., OSWorld), but also more general or tool-intensive benchmarks (e.g., GAIA and SWE-Bench). Specifically, we achieve accuracy on OSWorld, higher than Claude-Computer-Use. Codes and evaluation scripts are open-sourced at https://github.com/bin123apple/InfantAgent.
Paper Structure (29 sections, 7 figures, 4 tables, 2 algorithms)

This paper contains 29 sections, 7 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work Comparison
  3. InfantAgent-Next
  4. Architecture
  5. Component Design
  6. Mouse Click & File Edit
  7. Experiment
  8. Vision Ability
  9. Logic Reasoning Ability
  10. General Task Performance
  11. Ablation on Visual Grounding design
  12. Evaluation on File Editing
  13. Conclusion
  14. Case Analysis
  15. Workflow Prompt Templates
  16. ...and 14 more sections

Figures (7)

  • Figure 1: Three real-world task examples addressed by InfantAgent-Next, each requiring different modality capabilities. Input requests are shown in the block, actions taken by the agent are shown in the block, and execution results appear directly below the action block.
  • Figure 2: InfantAgent-Next architecture overview. : User input argument and request. Environment related icons:: Agent Interaction Environment. : Terminal interface. : GNOME desktop. : Jupyter. Models related icons:: Load Workflow models. : Planning model : Tool Selection model : Execution model : Planner. : Tool Selector. : Executer. Tools related icon:: Load Toolsets. : Tool models argument: Vision_model_name. : Tool models argument: Audio_model_name. : Tool models argument: Video_model_name. : Multimodal toolkit. : Other Toolkits. Memory related icons:: Load agent memory. : Request from the user. : Agent Memory Cache, used for storing all memories.
  • Figure 3: We conduct an ablation study on the Iterative Region Cropping setup from four perspectives. (a) Vary the region width. (b) Vary the width and height ratios of the cropped region (relative to the full image) (c) Vary the height. (d) Vary the number of iteration
  • Figure 4: Evaluation on a subset of SWE-Bench-Verified.
  • Figure 5: Cases analysis. Zoom in to view the detailed content in the screenshot.
  • ...and 2 more figures