Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Macro Mining from Interaction Traces at Scale

Forrest Huang, Gang Li, Tao Li, Yang Li

TL;DR

This work tackles the challenge of extracting high-level, executable macros that capture user tasks from large-scale mobile interaction traces. It introduces an LLM-grounded two-stage pipeline—task extraction and action/trace merging—to produce semantically described, replayable macros and a Macro Replayer for execution on Android devices. The authors demonstrate the approach on RICO and Rehearsal datasets, extracting tens of thousands of macros (e.g., 23,117 from RICO) and achieving meaningful task descriptions and a substantial live-execution success rate (76.7%). They also release open-source macro datasets and discuss applications in interactive task understanding, UI automation, and how-to knowledge sharing, while acknowledging limitations and outlining future improvements. Overall, the method enables scalable discovery and execution of cross-app tasks, with potential to empower automation and knowledge transfer in mobile UX research and practice.

Abstract

Macros are building block tasks of our everyday smartphone activity (e.g., "login", or "booking a flight"). Effectively extracting macros is important for understanding mobile interaction and enabling task automation. These macros are however difficult to extract at scale as they can be comprised of multiple steps yet hidden within programmatic components of mobile apps. In this paper, we introduce a novel approach based on Large Language Models (LLMs) to automatically extract semantically meaningful macros from both random and user-curated mobile interaction traces. The macros produced by our approach are automatically tagged with natural language descriptions and are fully executable. We conduct multiple studies to validate the quality of extracted macros, including user evaluation, comparative analysis against human-curated tasks, and automatic execution of these macros. These experiments and analyses show the effectiveness of our approach and the usefulness of extracted macros in various downstream applications.

Automatic Macro Mining from Interaction Traces at Scale

TL;DR

This work tackles the challenge of extracting high-level, executable macros that capture user tasks from large-scale mobile interaction traces. It introduces an LLM-grounded two-stage pipeline—task extraction and action/trace merging—to produce semantically described, replayable macros and a Macro Replayer for execution on Android devices. The authors demonstrate the approach on RICO and Rehearsal datasets, extracting tens of thousands of macros (e.g., 23,117 from RICO) and achieving meaningful task descriptions and a substantial live-execution success rate (76.7%). They also release open-source macro datasets and discuss applications in interactive task understanding, UI automation, and how-to knowledge sharing, while acknowledging limitations and outlining future improvements. Overall, the method enables scalable discovery and execution of cross-app tasks, with potential to empower automation and knowledge transfer in mobile UX research and practice.

Abstract

Macros are building block tasks of our everyday smartphone activity (e.g., "login", or "booking a flight"). Effectively extracting macros is important for understanding mobile interaction and enabling task automation. These macros are however difficult to extract at scale as they can be comprised of multiple steps yet hidden within programmatic components of mobile apps. In this paper, we introduce a novel approach based on Large Language Models (LLMs) to automatically extract semantically meaningful macros from both random and user-curated mobile interaction traces. The macros produced by our approach are automatically tagged with natural language descriptions and are fully executable. We conduct multiple studies to validate the quality of extracted macros, including user evaluation, comparative analysis against human-curated tasks, and automatic execution of these macros. These experiments and analyses show the effectiveness of our approach and the usefulness of extracted macros in various downstream applications.
Paper Structure (34 sections, 2 equations, 5 figures, 5 tables)

This paper contains 34 sections, 2 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Definition of Macros and Prior Work
  3. Semantic Understanding of Individual UI Elements and Screens
  4. Developer-defined Abstractions of UIs
  5. Defining Macros through demonstration, End-User programming, and Task-based Applications
  6. LLM-based UI Applications
  7. Macro Extraction and Execution
  8. Notation, Extraction Inputs, and Extraction Outputs
  9. Task Extraction
  10. Task Discovery
  11. Action Grounding
  12. Parameter-finding
  13. Action/Trace Merging and Optimal Path-finding
  14. Overall Graph Structure
  15. Node Identification and Merging
  16. ...and 19 more sections

Figures (5)

  • Figure 1: Full macro extraction system for a single example macro: 'Create a reminder'. Our system extracts K macros from all N traces for each app in the actual datasets.
  • Figure 2: Macro extraction results for apps in the Rehearsal dataset.
  • Figure 3: Macro extraction results for apps in the RICO dataset.
  • Figure 4: Limitations of our current macro mining system illustrated by a single Macro example.
  • Figure 5: Examples with low Rouge-L scores.