DOM-Q-NET: Grounded RL on Structured Language

TL;DR

DOM-Q-NET tackles web navigation by modeling the web page as a DOM graph and learning a factorized, goal-conditioned Q-function with three action streams (DOM selection, word token, and mode). A graph neural network builds local, neighbor, and global embeddings that feed three Q-networks, with goal-attention to integrate instruction context; this enables solving large action spaces and transferring knowledge across tasks. The approach achieves competitive or superior performance on MiniWoB without demonstrations and exhibits about 2x sample efficiency in multitask settings, including solving a long-horizon task like choose-date. The work demonstrates practical improvements in data efficiency and generalization for structured-language grounded RL in web interfaces, with broad implications for scalable, instruction-following agents on the web.

Abstract

Building agents to interact with the web would allow for significant improvements in knowledge understanding and representation learning. However, web navigation tasks are difficult for current deep reinforcement learning (RL) models due to the large discrete action space and the varying number of actions between the states. In this work, we introduce DOM-Q-NET, a novel architecture for RL-based web navigation to address both of these problems. It parametrizes Q functions with separate networks for different action categories: clicking a DOM element and typing a string input. Our model utilizes a graph neural network to represent the tree-structured HTML of a standard web page. We demonstrate the capabilities of our model on the MiniWoB environment where we can match or outperform existing work without the use of expert demonstrations. Furthermore, we show 2x improvements in sample efficiency when training in the multi-task setting, allowing our model to transfer learned behaviours across tasks.

Figures (24)

• Figure 1: Given the web page on the right, its DOM tree representation is shown as a graph where each DOM represents a node from $V$. Different colors indicate different tag attributes of DOMs. DOMs are embedded as a local module, ${\bm{e}}_{local}$, and propagated by a GNN to produce a neighbor module, ${\bm{e}}_{neighbor}$. The global module, ${\bm{e}}_{global}$, is aggregated from the neighbor module. The $Q_{dom}$ stream uses all three modules whereas $Q_{token}$ and $Q_{mode}$ streams only use the global module. Here, Q values of the 'submit' and 'sr' token are computed by $Q_{dom}$ and $Q_{token}$ respectively.
• Figure 2: A successful trajectory executed by our model for search-engine. $S_i$ is the state, and $A_i=(a_{dom}, a_{token}, a_{mode})$ is a tuple of actions for the three distinct categories of actions at timestep i. DOM($x$) represents the index of the corresponding element $x$ in the web page.
• Figure 3: Performance comparisons of DOM-Q-NET with shi2017worldliu2018reinforcement
• Figure 4: Multitask Comparisons: 9-multitask DOM-Q-NET with goal-attention consistently has better sample efficiency. Results for other tasks are shown in Appendix \ref{['appendix:multitask']}. g_a=goal-attention.
• Figure 5: Comparisons in sample efficiency for 2 hard tasks, social-media (left) and search-engine (right), by multitask learning. 9_multitask refers to the tasks discussed in Figure \ref{['fig:results']}