Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations

Gaurav Verma, Rachneet Kaur, Nishan Srishankar, Zhen Zeng, Tucker Balch, Manuela Veloso

TL;DR

The AdaptAgent framework is introduced that enables both proprietary and open-weights multimodal web agents to adapt to new websites and domains using few human demonstrations, and unlocks a complementary axis for developing widely applicable multimodal web agents beyond large-scale pre-training and fine-tuning.

Abstract

State-of-the-art multimodal web agents, powered by Multimodal Large Language Models (MLLMs), can autonomously execute many web tasks by processing user instructions and interacting with graphical user interfaces (GUIs). Current strategies for building web agents rely on (i) the generalizability of underlying MLLMs and their steerability via prompting, and (ii) large-scale fine-tuning of MLLMs on web-related tasks. However, web agents still struggle to automate tasks on unseen websites and domains, limiting their applicability to enterprise-specific and proprietary platforms. Beyond generalization from large-scale pre-training and fine-tuning, we propose building agents for few-shot adaptability using human demonstrations. We introduce the AdaptAgent framework that enables both proprietary and open-weights multimodal web agents to adapt to new websites and domains using few human demonstrations (up to 2). Our experiments on two popular benchmarks -- Mind2Web & VisualWebArena -- show that using in-context demonstrations (for proprietary models) or meta-adaptation demonstrations (for meta-learned open-weights models) boosts task success rate by 3.36% to 7.21% over non-adapted state-of-the-art models, corresponding to a relative increase of 21.03% to 65.75%. Furthermore, our additional analyses (a) show the effectiveness of multimodal demonstrations over text-only ones, (b) shed light on the influence of different data selection strategies during meta-learning on the generalization of the agent, and (c) demonstrate the effect of number of few-shot examples on the web agent's success rate. Overall, our results unlock a complementary axis for developing widely applicable multimodal web agents beyond large-scale pre-training and fine-tuning, emphasizing few-shot adaptability.

AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations

TL;DR

The AdaptAgent framework is introduced that enables both proprietary and open-weights multimodal web agents to adapt to new websites and domains using few human demonstrations, and unlocks a complementary axis for developing widely applicable multimodal web agents beyond large-scale pre-training and fine-tuning.

Abstract

State-of-the-art multimodal web agents, powered by Multimodal Large Language Models (MLLMs), can autonomously execute many web tasks by processing user instructions and interacting with graphical user interfaces (GUIs). Current strategies for building web agents rely on (i) the generalizability of underlying MLLMs and their steerability via prompting, and (ii) large-scale fine-tuning of MLLMs on web-related tasks. However, web agents still struggle to automate tasks on unseen websites and domains, limiting their applicability to enterprise-specific and proprietary platforms. Beyond generalization from large-scale pre-training and fine-tuning, we propose building agents for few-shot adaptability using human demonstrations. We introduce the AdaptAgent framework that enables both proprietary and open-weights multimodal web agents to adapt to new websites and domains using few human demonstrations (up to 2). Our experiments on two popular benchmarks -- Mind2Web & VisualWebArena -- show that using in-context demonstrations (for proprietary models) or meta-adaptation demonstrations (for meta-learned open-weights models) boosts task success rate by 3.36% to 7.21% over non-adapted state-of-the-art models, corresponding to a relative increase of 21.03% to 65.75%. Furthermore, our additional analyses (a) show the effectiveness of multimodal demonstrations over text-only ones, (b) shed light on the influence of different data selection strategies during meta-learning on the generalization of the agent, and (c) demonstrate the effect of number of few-shot examples on the web agent's success rate. Overall, our results unlock a complementary axis for developing widely applicable multimodal web agents beyond large-scale pre-training and fine-tuning, emphasizing few-shot adaptability.

Paper Structure

This paper contains 18 sections, 4 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Few-Shot Adaptation with Human Demonstrations
  4. Experimental Protocol and Details
  5. Results
  6. Discussion and Limitations
  7. Appendix
  8. Detailed Related Work
  9. UI/Web Agents
  10. Few-shot learning with LLMs
  11. Learning from demonstrations
  12. First-order approximation of MAML for Multimodal Web Agents
  13. Benchmark Details
  14. Mind2Web
  15. VisualWebArena
  16. ...and 3 more sections

Figures (3)

  • Figure 1: AdaptAgent for few-shot adaptation of web agents that are based on proprietary and open-weights multimodal LLMs. Left: For proprietary MLLM-based web agents, we include the multimodal human demonstration as in-context examples. Right: For web agents based on open-weights MLLMs, we first learn a better prior using meta-learning and then use few-shot human demonstrations for faster adaptation.
  • Figure 2: Additional analyses. Left: Ablation study on demonstration modality in SeeAct*. Center: Comparison of overall SR across meta-learning adaptation strategies in CogAgent. Right: Variation in performance with different numbers of in-context demonstrations; numbers are inset in the bars.
  • Figure 3: Visual depiction of the protocol used for meta-learning using the Mind2Web train set (left), and the meta-adaptation done on cross-domain and cross-website evaluation sets (top-right). For completeness, we also show the conventional fine-tuning strategy (bottom-right).