Do BERTs Learn to Use Browser User Interface? Exploring Multi-Step Tasks with Unified Vision-and-Language BERTs

TL;DR

The paper asks whether BERT-like models can learn to perform multi-step tasks through a browser UI by framing tasks as GUI pages and actions. It introduces BUI-BERT, a memory-augmented extension that fuses vision and language inputs with action-prediction losses, trained via pre-training for actions and multi-task learning across GLUE, SQuAD, VQA, and SA-style tasks. Results show the model can learn GUI manipulation and retain linguistic capabilities, achieving a meaningful fraction of non-GUI baselines, while generalization to unseen tasks remains limited. This work demonstrates the feasibility of transferring linguistic knowledge to action sequences in GUI environments and highlights the need for more data-efficient, generalizable approaches and reinforcement learning to improve exploration in GUI tasks.

Abstract

Pre-trained Transformers are good foundations for unified multi-task models owing to their task-agnostic representation. Pre-trained Transformers are often combined with text-to-text framework to execute multiple tasks by a single model. Performing a task through a graphical user interface (GUI) is another candidate to accommodate various tasks, including multi-step tasks with vision and language inputs. However, few papers combine pre-trained Transformers with performing through GUI. To fill this gap, we explore a framework in which a model performs a task by manipulating the GUI implemented with web pages in multiple steps. We develop task pages with and without page transitions and propose a BERT extension for the framework. We jointly trained our BERT extension with those task pages, and made the following observations. (1) The model learned to use both task pages with and without page transition. (2) In four out of five tasks without page transitions, the model performs greater than 75% of the performance of the original BERT, which does not use browsers. (3) The model did not generalize effectively on unseen tasks. These results suggest that we can fine-tune BERTs to multi-step tasks through GUIs, and there is room for improvement in their generalizability. Code will be available online.

Figures (13)

• Figure 1: Framework comparison. (a) Typical frameworks assume single-step tasks in which a model takes a sequence of text and images to generate an output. (b) In our WoB-like framework, we make a task as web pages, which allow structured contents, hyperlinks and scripts. The page design decides how to submit an answer (e.g., choose a button or input text). A model completes a task in multiple steps using the browser user interface (BUI). The model take a screenshot to output an action for each step. (e.g., click or keystroke).
• Figure 2: Three types of task page and the examples. (a) Pre-training for actions. In the area task, a blank space exists between the instructions and the buttons so that a model needs to scroll until the buttons are visible. (b) Single-page tasks. The rectangles outlined by the blue dotted lines represent the initial visible area. (c) Multi-page tasks. Models can make page transitions within the child frames embedded in the outer page.
• Figure 3: Example pages for the SA-A task. All the SA tasks share the page design. The task pages include an initial page, search result, and detail page. The result changes depending on the query. Models are required to jump between those pages to answer the question.
• Figure 4: Overview of BUI-BERT. This figure shows how the model predicts an action for the step $t$. (a) The model consists of a pre-trained BERT (Fusion BERT), which takes vision, language, memory, and some auxiliary tokens to output the next action. Fusion BERT adds position and segment embeddings in the same way as the original BERT (omitted in the view). (b) The model obtains action (three tokens), vision ($L$ tokens) and language embeddings ($N'$ tokens from $N$ words) with deep neural networks.
• Figure 5: Screen examples form the BUI version of VQAv2 (left) and that of SQuADv2 (right). Those are screenshot that the BUI models receive. The blue dash rectangles show the initial visible area for the models. The instructions and answer forms are common for the all examples.