Lexi: Self-Supervised Learning of the UI Language

TL;DR

Lexi introduces a self-supervised visio-linguistic approach to learning UI language from instructional manuals, circumventing the need for UI metadata. It combines a UI-aware detector with a text-enriched region representation and four pre-training tasks, including a novel POS tagging objective, to capture UI-specific semantics. The UICaption dataset provides diverse, functional captions paired with UI images, enabling cross-platform learning. Evaluations across UI action entailment, instruction-based image retrieval, grounding, and UI entity recognition show Lexi outperforming strong baselines and ablations highlight the importance of ITA and text detection for performance.

Abstract

Humans can learn to operate the user interface (UI) of an application by reading an instruction manual or how-to guide. Along with text, these resources include visual content such as UI screenshots and images of application icons referenced in the text. We explore how to leverage this data to learn generic visio-linguistic representations of UI screens and their components. These representations are useful in many real applications, such as accessibility, voice navigation, and task automation. Prior UI representation models rely on UI metadata (UI trees and accessibility labels), which is often missing, incompletely defined, or not accessible. We avoid such a dependency, and propose Lexi, a pre-trained vision and language model designed to handle the unique features of UI screens, including their text richness and context sensitivity. To train Lexi we curate the UICaption dataset consisting of 114k UI images paired with descriptions of their functionality. We evaluate Lexi on four tasks: UI action entailment, instruction-based UI image retrieval, grounding referring expressions, and UI entity recognition.

Figures (5)

• Figure 1: Examples of how-to and support websites. In addition to text, they contain UI screenshots and icon images, which we use to generate image-caption pairs for training Lexi.
• Figure 2: Our proposed VL transformer encoder, trained using 4 pre-training tasks: masked language modeling, masked region modeling with KL divergence, image-text alignment, and POS tagging.
• Figure 3: Examples of two Lexi's downstream tasks.
• Figure 4: Examples of correct (first two) and wrong (last two) predictions for the grounding referring expressions task. The referring expression is above the UI screen, and ground truth UI elements (with a green border) and Lexi's predictions (with an orange border) are below.
• Figure 5: Samples from UI action entailment with Lexi-6 predictions for Zoom and Teams. The correct pair is the first one in each row. Correct predictions are highlighted in green and wrong ones in red.