PrISM-Observer: Intervention Agent to Help Users Perform Everyday Procedures Sensed using a Smartwatch

TL;DR

PrISM-Observer tackles the problem of missed or misordered steps in everyday procedures by delivering proactive, context-aware interventions via a smartwatch. The approach combines multimodal sensing (audio+motion) with stochastic modeling of user behavior to forecast when to intervene, offering two intervention types: remind in advance and notify if forgotten. Across three task datasets and a real-time cooking study with 10 participants, the framework achieves meaningful timing improvements and favorable user perceptions, despite HAR accuracy being imperfect. The work demonstrates the practical viability of passive, proactive task support for cognitively challenged users and outlines a path toward broader health applications and platform extensibility.

Abstract

We routinely perform procedures (such as cooking) that include a set of atomic steps. Often, inadvertent omission or misordering of a single step can lead to serious consequences, especially for those experiencing cognitive challenges such as dementia. This paper introduces PrISM-Observer, a smartwatch-based, context-aware, real-time intervention system designed to support daily tasks by preventing errors. Unlike traditional systems that require users to seek out information, the agent observes user actions and intervenes proactively. This capability is enabled by the agent's ability to continuously update its belief in the user's behavior in real-time through multimodal sensing and forecast optimal intervention moments and methods. We first validated the steps-tracking performance of our framework through evaluations across three datasets with different complexities. Then, we implemented a real-time agent system using a smartwatch and conducted a user study in a cooking task scenario. The system generated helpful interventions, and we gained positive feedback from the participants. The general applicability of PrISM-Observer to daily tasks promises broad applications, for instance, including support for users requiring more involved interventions, such as people with dementia or post-surgical patients.

Figures (13)

• Figure 1: Process of selecting steps for intervention and their types. The system suggests the possibility of the notify if forgotten intervention based on the HAR accuracy of the step. Refer to Section \ref{['sec:proposed-formulation']} for details.
• Figure 2: Example $D_t^{\hat{s}}$ transition during a session of the cooking task, where $\hat{s}$ is $s_4$ (performed early), $s_{10}$ (performed in the middle), and $s_{13}$ (performed later). The orange line is the ground truth (actual remaining time till the target step). Two uncertainties affect the distribution: current belief about the user state and future user trajectory.
• Figure 3: Example transitions of estimated remaining time $E[D_t^{\hat{s}}]$ and entropy $H[D_t^{\hat{s}}]$ from different sessions with different target steps and how our intervention policy works. The y-axis for the entropy graph starts from 2.0 for visualization. $K^-$ and $K^+$ are system parameters for the timing offsets of the remind in advance and notify if forgotten interventions, respectively.
• Figure 4: Transition graph of the three tasks: wound care, cooking, and latte-making. Each task has a different level of graph complexity, i.e., the number of branches. Each arrow points to a possible transition step, and the opacity of the arrow represents the probability of the transition.
• Figure 5: Example error calculation in the forecasted and actual timing of the target step in Study 1.