Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning

TL;DR

Navigating WebAI investigates how to train agents to complete web tasks using large language models and reinforcement learning. It shows that pure supervised LLM approaches can be data-efficient but struggle with authentic HTML comprehension, while RL-based methods require substantial data; to address this, the authors propose a hybrid SL+RL framework with hierarchical planning and multimodal inputs evaluated on the Miniwob++ benchmark. The study demonstrates that a hierarchical WebN-T5 variant achieves about 43% average accuracy in supervised settings, narrows the performance gap to RL baselines, and highlights memorization as a key pitfall that must be mitigated via action-history processing and reference randomization. Overall, the work advances a grounded, scalable direction for web-navigation agents and delineates practical challenges and future research avenues in multimodal planning and transferability across web interfaces.

Abstract

Recent advancements in language models have demonstrated remarkable improvements in various natural language processing (NLP) tasks such as web navigation. Supervised learning (SL) approaches have achieved impressive performance while utilizing significantly less training data compared to previous methods. However, these SL-based models fall short when compared to reinforcement learning (RL) approaches, which have shown superior results. In this paper, we propose a novel approach that combines SL and RL techniques over the MiniWoB benchmark to leverage the strengths of both methods. We also address a critical limitation in previous models' understanding of HTML content, revealing a tendency to memorize target elements rather than comprehend the underlying structure. To rectify this, we propose methods to enhance true understanding and present a new baseline of results. Our experiments demonstrate that our approach outperforms previous SL methods on certain tasks using less data and narrows the performance gap with RL models, achieving 43.58\% average accuracy in SL and 36.69\% when combined with a multimodal RL approach. This study sets a new direction for future web navigation and offers insights into the limitations and potential of language modeling for computer tasks.

Figures (15)

• Figure 1: Comparison of Existing Models Regarding the Web Navigation Task over the Miniwob Benchmark. Average comparison over the tasks proposed on the Miniwob benchmark between different training techniques and architectures humphreys2022datadrivengur2023understandingliu2018reinforcementkim2023language.
• Figure 2: Example of Miniwob Episodes. Each opened episode is timed and alongside it, a discounted reward is computed. These episodes cover a wide range of tasks, and in our case, we select a subset of 40 episodes that are suited to work with language models in the fashion of Gur et al. gur2023understanding.
• Figure 3: Structure of Action History as Proposed by Gur et al. gur2023understanding.
• Figure 4: Structure of T5 Input in its traditional form as Proposed by Gur et al. gur2023understanding.
• Figure 5: T5 Hierarchical Inference Process. We first infer the model to devise a navigation plan from the initial utterance, then iterate through the subtask instructions individually to infer the current action at each time step while evaluating the state of the episode by means of the computed reward and terminal state.