DroidRetriever: An Autonomous Navigation and Information Integration System Facilitating Mobile Sensemaking

TL;DR

DroidRetriever tackles the fragmented information landscape on mobile devices by automating cross-app navigation and information integration to support mobile sensemaking. It uses a three-module, multi-LLM framework (task decomposition, UI navigation, and report synthesis) to autonomously locate, extract, and present information with citations, while allowing user interventions to ensure transparency and trust. Across controlled and real-world studies, the system demonstrates faster information gathering, higher coverage, and lower redundancy compared to manual approaches and several baselines, albeit with some navigation latency and reliance on model planning. The work highlights the practical benefits of human-in-the-loop design for high-stakes tasks, provides a pathway toward open-source deployment, and discusses future directions for handling dynamic content and privacy in mobile contexts.

Abstract

Users regularly rely on mobile applications for their daily information needs, and mobile sensemaking is prevalent in various domains such as education, healthcare, business intelligence, and emergency response, where timely and context-aware information-processing and decision-making is critical. However, valuable information is often scattered across the closed ecosystems within various applications, posing challenges for traditional search engines to retrieve data openly and in real-time. Additionally, due to limitations such as mobile device screen sizes, language differences, and unfamiliarity with specific applications and domain knowledge, users have to frequently switch between multiple applications and spend substantial time locating and integrating the information. To address these challenges, we present DroidRetriever, a system for cross-application information retrieval to facilitate mobile sensemaking. DroidRetriever can automatically navigate to relevant interfaces based on users' natural language commands, capture screenshots, extract and integrate information, and finally present the results. Our experimental results demonstrate that DroidRetriever can extract and integrate information with near-human accuracy while significantly reducing processing time. Furthermore, with minimal user intervention, DroidRetriever effectively corrects and completes various information retrieval tasks, substantially reducing the user's workload. Our summary of the motivations for intervention and the discussion of their necessity provide valuable implications for future research. We will open-source our code upon acceptance of the paper.

Figures (9)

• Figure 1: Foraging-Sensemaking Loop Pirolli2005 (top) can be divided into two main loops: the foraging loop and the sensemaking loop. The automated workflow of DroidRetriever (bottom) aligns with the Foraging-Sensemaking Loop.
• Figure 2: DroidRetriever is an in-app information retrieval and sensemaking system that employs a multi-LLM framework. It comprises three key modules: task decomposition, UI navigation, and report synthesis.
• Figure 3: Illustration of three search modes: focused, list-view, and multi-page.
• Figure 4: The design of interface for intervention.
• Figure 5: The workflow of intervention and feedback. Here, $Intervention^a$ means user needs to perform gesture operations during intervention, such as tapping and inputting; $Intervention^b$ indicates user wants to take a screenshot and save the current interface to the search result database; $Intervention^c$ signifies user intends to terminate the UI navigation.