Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents

TL;DR

MONDAY tackles the scarcity and rapid evolution of real-world mobile OS navigation data by automatically extracting cross-platform task sequences from instructional videos. It introduces a three-component pipeline—OCR-based scene transition detection, robust UI element detection, and a three-step action identification process—to build the MONDAY dataset of 313K annotated frames from 20K videos. Empirical results show strong cross-platform generalization when pre-trained on MONDAY, including significant gains on unseen platforms (e.g., Windows Mobile) and improvements over baselines across AitW and AMEX benchmarks. The work enables scalable, cost-effective dataset expansion and contributes toward more robust GUI visual agents capable of operating across diverse mobile interfaces in real-world settings.

Abstract

Recent advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs) have sparked significant interest in developing GUI visual agents. We introduce MONDAY (Mobile OS Navigation Task Dataset for Agents from YouTube), a large-scale dataset of 313K annotated frames from 20K instructional videos capturing diverse real-world mobile OS navigation across multiple platforms. Models that include MONDAY in their pre-training phases demonstrate robust cross-platform generalization capabilities, consistently outperforming models trained on existing single OS datasets while achieving an average performance gain of 18.11%p on an unseen mobile OS platform. To enable continuous dataset expansion as mobile platforms evolve, we present an automated framework that leverages publicly available video content to create comprehensive task datasets without manual annotation. Our framework comprises robust OCR-based scene detection (95.04% F1score), near-perfect UI element detection (99.87% hit ratio), and novel multi-step action identification to extract reliable action sequences across diverse interface configurations. We contribute both the MONDAY dataset and our automated collection framework to facilitate future research in mobile OS navigation.

Figures (17)

• Figure 1: Example screens from our MONDAY dataset: (a) mobile OS navigation sequence showing how to turn off incognito mode on Google Maps. Our dataset captures real-world mobile OS navigation procedures across different platforms and configurations, enabling effective generalization; (b) iOS interfaces across different versions and user configurations, including light/dark mode, custom control center, and accessibility settings; (c) Android interfaces with various themes, app layouts, and different resolutions.
• Figure 2: MONDAY dataset collection framework for mobile OS task dataset for agents from YouTube. Given a video, we first detect scene transitions (Section \ref{['mobvid:subsec:dataset_scene_transition_detection']}) and then identify actions in a 3-step process (Section \ref{['mobvid:subsec:dataset_action_annotation']}): (1) scene summary, (2) initial action identification with SoM representation , and (3) refined action identification for precise localization. In all three steps, we leverage narrations to disambiguate between multiple UI elements of similar effects. The final coordinate is set to the center of the bounding box of the selected UI element.
• Figure 3: Extracted position of the frame from each scene transition detection. Dotted vertical lines represent ground truth transition points, with frames from the same transition segment marked in identical colors. Vision-based methods often miss transitions when visual changes are subtle, whereas our OCR-based method reliably detects them.
• Figure 4: Comparison between our UI element detection module and OmniParser lu-arxiv24. Ours successfully detects a broader range of UI elements, including home screen icons and bottom-positioned UI elements that OmniParser frequently misses.
• Figure 5: Identified actions between different ablation methods. Our multi-image 3-step approach outperforms simplified variants.