Vital Insight: Assisting Experts' Context-Driven Sensemaking of Multi-modal Personal Tracking Data Using Visualization and Human-In-The-Loop LLM

TL;DR

This work tackles the challenge of translating multi-modal passive sensing data into high-level, context-aware insights for health and behavior. It presents Vital Insight (VI), an LLM-assisted visual analytics prototype that enables human-in-the-loop sensemaking by integrating direct data representations with AI-generated inferences across hourly and daily scales. Through three iterative user studies with 21 experts, the authors derive an expert sensemaking model, identify design implications, and demonstrate that AI augmentation can boost user satisfaction and perceived usefulness while highlighting the need for evidence provenance and careful anomaly handling. The study contributes a practical framework for AI-augmented visualization in health sensing, including an open-source prototype and a nuanced discussion of trust, interoperability, and socio-technical considerations for real-world deployment.

Abstract

Passive tracking methods, such as phone and wearable sensing, have become dominant in monitoring human behaviors in modern ubiquitous computing studies. While there have been significant advances in machine-learning approaches to translate periods of raw sensor data to model momentary behaviors, (e.g., physical activity recognition), there still remains a significant gap in the translation of these sensing streams into meaningful, high-level, context-aware insights that are required for various applications (e.g., summarizing an individual's daily routine). To bridge this gap, experts often need to employ a context-driven sensemaking process in real-world studies to derive insights. This process often requires manual effort and can be challenging even for experienced researchers due to the complexity of human behaviors. We conducted three rounds of user studies with 21 experts to explore solutions to address challenges with sensemaking. We follow a human-centered design process to identify needs and design, iterate, build, and evaluate Vital Insight (VI), a novel, LLM-assisted, prototype system to enable human-in-the-loop inference (sensemaking) and visualizations of multi-modal passive sensing data from smartphones and wearables. Using the prototype as a technology probe, we observe experts' interactions with it and develop an expert sensemaking model that explains how experts move between direct data representations and AI-supported inferences to explore, question, and validate insights. Through this iterative process, we also synthesize and discuss a list of design implications for the design of future AI-augmented visualization systems to better assist experts' sensemaking processes in multi-modal health sensing data.

Figures (9)

• Figure 1: Eight components in VI-1 powered by visualization and LLM: time selection, main visualization, user profile, user check-in (user conversation with chatbot), LLM daily summary, LLM question, LLM anomaly detection, LLM hourly summary
• Figure 2: Overview of the structure of LLM augmentation for VI-1 and VI-2.
• Figure 3: Survey results from the initial user testing (Study 2).
• Figure 4: Time ranges of LLM detected heart rate and respiration anomalies from 11/18/2024 to 11/24/2024. The horizontal lines represent the start and end time of the anomalies, and the different colors represent 10 LLM test runs.
• Figure 5: Design of the new iteration of Vital Insight (VI-2). Top left: user testing input; bottom left: main visualization and LLM detected occurrences; top right: user profile; middle right: Day in a glance generated by LLM; bottom right: user check-in conversation.