Adversarial Environment Generation for Learning to Navigate the Web

TL;DR

Problem: train RL agents to navigate unknown real-world websites with high-dimensional, dynamic state-action spaces. Approach: an adversarial, regret-based curriculum design (AEG) built on PAIRED and extended as Flexible b-PAIRED, using an autoregressive adversary to compose websites from DOM primitives in the gMiniWoB framework, with budgeted control to match agent progress. Contributions: (i) gMiniWoB open-source benchmark, (ii) Flexible b-PAIRED with stable regret estimates and budget enforcement, (iii) empirical evidence of progressively harder curricula and strong generalization to unseen sites, often achieving >80% success on simpler tasks. Impact: enables scalable, automatic curriculum design for complex, compositional web navigation tasks and broader RL auto-curriculum design in high-dimensional environments.

Abstract

Learning to autonomously navigate the web is a difficult sequential decision making task. The state and action spaces are large and combinatorial in nature, and websites are dynamic environments consisting of several pages. One of the bottlenecks of training web navigation agents is providing a learnable curriculum of training environments that can cover the large variety of real-world websites. Therefore, we propose using Adversarial Environment Generation (AEG) to generate challenging web environments in which to train reinforcement learning (RL) agents. We provide a new benchmarking environment, gMiniWoB, which enables an RL adversary to use compositional primitives to learn to generate arbitrarily complex websites. To train the adversary, we propose a new technique for maximizing regret using the difference in the scores obtained by a pair of navigator agents. Our results show that our approach significantly outperforms prior methods for minimax regret AEG. The regret objective trains the adversary to design a curriculum of environments that are "just-the-right-challenge" for the navigator agents; our results show that over time, the adversary learns to generate increasingly complex web navigation tasks. The navigator agents trained with our technique learn to complete challenging, high-dimensional web navigation tasks, such as form filling, booking a flight etc. We show that the navigator agent trained with our proposed Flexible b-PAIRED technique significantly outperforms competitive automatic curriculum generation baselines -- including a state-of-the-art RL web navigation approach -- on a set of challenging unseen test environments, and achieves more than 80% success rate on some tasks.

Figures (8)

• Figure 1: Samples of generated web pages from selected websites taken from early, middle, and late snapshots of the training (a-c) and unseen test "Login" website (d). Over time, the number of pages in a website decreases but the density of elements in a page increases with more task-oriented elements.
• Figure 2: An example underspecified DOM tree template (b) and its instantiations (a,c) with different values. (*) indicates a variable; either an element or one of its attributes. (a) is used in Page 1 and (c) is used in Page 2 in Figure \ref{['fig:rollout']}.
• Figure 3: A sample rollout of the adversary for compositional environment generation for web navigation problem. An initial observation (Obs) is given at the beginning of the rollout. $f_0$, $f_K$, $f_L$, $f_P$, and $f_I$ denote networks for encoding initial observation, generating number of pages, page indices,primitives, and encoding LSTM inputs, respectively.
• Figure 4: Comparison of PAIRED dennis2020emergent and Flexible PAIRED with and without budget enforcing; averaged over 4 difficulty levels. (f): Percentage of active primitives over training steps.
• Figure 5: Aggregated task success rate comparison of Flexible b-PAIRED and baseline models on test environments with increasing difficulty levels. See Appendix \ref{['ap:detailed-results']} for detailed results.