Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning UI Navigation through Demonstrations composed of Macro Actions

Wei Li

TL;DR

The paper addresses robust UI navigation across diverse apps by learning from human demonstrations. It introduces a UI-element-based state representation and a macro-action policy, trained with DQfD or BC, augmented by demo augmentation and screenshot-based demonstrations to achieve high success with relatively few demonstrations. Key contributions include a transformer-based policy over UI elements, a hierarchical macro-action framework, and an error-driven demo collection loop that dramatically reduces the required human demonstrations while enabling generalization to unseen apps. This approach offers a scalable path toward production-ready UI automation across Android and web interfaces, lowering the need for hand-crafted macros and enabling cross-application applicability.

Abstract

We have developed a framework to reliably build agents capable of UI navigation. The state space is simplified from raw-pixels to a set of UI elements extracted from screen understanding, such as OCR and icon detection. The action space is restricted to the UI elements plus a few global actions. Actions can be customized for tasks and each action is a sequence of basic operations conditioned on status checks. With such a design, we are able to train DQfD and BC agents with a small number of demonstration episodes. We propose demo augmentation that significantly reduces the required number of human demonstrations. We made a customization of DQfD to allow demos collected on screenshots to facilitate the demo coverage of rare cases. Demos are only collected for the failed cases during the evaluation of the previous version of the agent. With 10s of iterations looping over evaluation, demo collection, and training, the agent reaches a 98.7\% success rate on the search task in an environment of 80+ apps and websites where initial states and viewing parameters are randomized.

Learning UI Navigation through Demonstrations composed of Macro Actions

TL;DR

The paper addresses robust UI navigation across diverse apps by learning from human demonstrations. It introduces a UI-element-based state representation and a macro-action policy, trained with DQfD or BC, augmented by demo augmentation and screenshot-based demonstrations to achieve high success with relatively few demonstrations. Key contributions include a transformer-based policy over UI elements, a hierarchical macro-action framework, and an error-driven demo collection loop that dramatically reduces the required human demonstrations while enabling generalization to unseen apps. This approach offers a scalable path toward production-ready UI automation across Android and web interfaces, lowering the need for hand-crafted macros and enabling cross-application applicability.

Abstract

We have developed a framework to reliably build agents capable of UI navigation. The state space is simplified from raw-pixels to a set of UI elements extracted from screen understanding, such as OCR and icon detection. The action space is restricted to the UI elements plus a few global actions. Actions can be customized for tasks and each action is a sequence of basic operations conditioned on status checks. With such a design, we are able to train DQfD and BC agents with a small number of demonstration episodes. We propose demo augmentation that significantly reduces the required number of human demonstrations. We made a customization of DQfD to allow demos collected on screenshots to facilitate the demo coverage of rare cases. Demos are only collected for the failed cases during the evaluation of the previous version of the agent. With 10s of iterations looping over evaluation, demo collection, and training, the agent reaches a 98.7\% success rate on the search task in an environment of 80+ apps and websites where initial states and viewing parameters are randomized.

Paper Structure

This paper contains 17 sections, 1 equation, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Technical Details
  4. Screen representation
  5. Action Space
  6. Neural network architecture
  7. Macro actions
  8. Evaluation, demo collection, and training
  9. Demo augmentation
  10. Demos from screenshots
  11. Experimental Results
  12. Evaluation tasks
  13. AndroidInstall
  14. AndroidSearch
  15. Agent performance
  16. ...and 2 more sections

Figures (12)

  • Figure 1: A screen is represented as a list of UI elements, of which each is encoded by concatenating various feature vectors of the associated attributes.
  • Figure 2: Action space
  • Figure 3: Neural network architecture
  • Figure 4: An example macro action: focus_and_type. (a) click to set focus. (b) wait for blinking cursor. (c) type text. (d) press enter and wait screen changes.
  • Figure 5: The loop for building UI navigation agents.
  • ...and 7 more figures