Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training a Vision Language Model as Smartphone Assistant

Nicolai Dorka, Janusz Marecki, Ammar Anwar

TL;DR

This work tackles instruction-based control of mobile devices via the UI by training vision-language models that generate text-encoded actions from natural language instructions and a history of screen images. It evaluates two backbones, LLama+ViT and Qwen-VL, and demonstrates that a history of screenshots plus OCR-pretraining substantially improves action accuracy on the Android in the Wild (AitW) benchmark, achieving state-of-the-art partial-match scores. The approach leverages LoRA fine-tuning to adapt pretrained vision-language models to the device-control task while maintaining a compact training footprint. The results suggest strong potential for UI-driven assistants that generalize across apps and pave the way for extending the approach to desktop environments.

Abstract

Addressing the challenge of a digital assistant capable of executing a wide array of user tasks, our research focuses on the realm of instruction-based mobile device control. We leverage recent advancements in large language models (LLMs) and present a visual language model (VLM) that can fulfill diverse tasks on mobile devices. Our model functions by interacting solely with the user interface (UI). It uses the visual input from the device screen and mimics human-like interactions, encompassing gestures such as tapping and swiping. This generality in the input and output space allows our agent to interact with any application on the device. Unlike previous methods, our model operates not only on a single screen image but on vision-language sentences created from sequences of past screenshots along with corresponding actions. Evaluating our method on the challenging Android in the Wild benchmark demonstrates its promising efficacy and potential.

Training a Vision Language Model as Smartphone Assistant

TL;DR

This work tackles instruction-based control of mobile devices via the UI by training vision-language models that generate text-encoded actions from natural language instructions and a history of screen images. It evaluates two backbones, LLama+ViT and Qwen-VL, and demonstrates that a history of screenshots plus OCR-pretraining substantially improves action accuracy on the Android in the Wild (AitW) benchmark, achieving state-of-the-art partial-match scores. The approach leverages LoRA fine-tuning to adapt pretrained vision-language models to the device-control task while maintaining a compact training footprint. The results suggest strong potential for UI-driven assistants that generalize across apps and pave the way for extending the approach to desktop environments.

Abstract

Addressing the challenge of a digital assistant capable of executing a wide array of user tasks, our research focuses on the realm of instruction-based mobile device control. We leverage recent advancements in large language models (LLMs) and present a visual language model (VLM) that can fulfill diverse tasks on mobile devices. Our model functions by interacting solely with the user interface (UI). It uses the visual input from the device screen and mimics human-like interactions, encompassing gestures such as tapping and swiping. This generality in the input and output space allows our agent to interact with any application on the device. Unlike previous methods, our model operates not only on a single screen image but on vision-language sentences created from sequences of past screenshots along with corresponding actions. Evaluating our method on the challenging Android in the Wild benchmark demonstrates its promising efficacy and potential.
Paper Structure (16 sections, 1 equation, 1 figure, 2 tables)

This paper contains 16 sections, 1 equation, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models for Decision Making
  4. Instruction-based Mobile Device Control
  5. Method
  6. Vision Language Models
  7. LLama + ViT
  8. Qwen-VL
  9. Action Representation for Mobile Devices
  10. Training
  11. Experiments
  12. Android in the Wild Dataset
  13. Comparative Evaluation
  14. Discussion
  15. Android in the Wild Benchmark
  16. ...and 1 more sections

Figures (1)

  • Figure 1: Visualization of our approach. We create a sequence of embedding vectors for the VLM from the instruction, the history of screenshots, and the history of actions which are first translated to natural language which is then encoded into token embeddings. Depending on the vision encoder the number of vision embedding vectors can vary.