Multimodal Physical Activity Forecasting in Free-Living Clinical Settings: Hunting Opportunities for Just-in-Time Interventions

TL;DR

This research develops multimodal long short-term memory network models, which are capable of forecasting the number of step counts of a patient up to 24 hours in advance by examining data from activity and engagement modalities and designs goal-based forecasting models to predict whether a person's next-day steps will be over a certain threshold.

Abstract

Objective: This research aims to develop a lifestyle intervention system, called MoveSense, that forecasts a patient's activity behavior to allow for early and personalized interventions in real-world clinical environments. Methods: We conducted two clinical studies involving 58 prediabetic veterans and 60 patients with obstructive sleep apnea to gather multimodal behavioral data using wearable devices. We develop multimodal long short-term memory (LSTM) network models, which are capable of forecasting the number of step counts of a patient up to 24 hours in advance by examining data from activity and engagement modalities. Furthermore, we design goal-based forecasting models to predict whether a person's next-day steps will be over a certain threshold. Results: Multimodal LSTM with early fusion achieves 33% and 37% lower mean absolute errors than linear regression and ARIMA respectively on the prediabetes dataset. LSTM also outperforms linear regression and ARIMA with a margin of 13% and 32% on the sleep dataset. Multimodal forecasting models also perform with 72% and 79% accuracy on the prediabetes dataset and sleep dataset respectively on goal-based forecasting. Conclusion: Our experiments conclude that multimodal LSTM models with early fusion are better than multimodal LSTM with late fusion and unimodal LSTM models and also than ARIMA and linear regression models. Significance: We address an important and challenging task of time-series forecasting in uncontrolled environments. Effective forecasting of a person's physical activity can aid in designing adaptive behavioral interventions to keep the user engaged and adherent to a prescribed routine.

Figures (5)

• Figure 1: The overview of our MoveSense system and its components: a wearable wristband, a lifestyle intervention app, a forecasting model, and an adaptive intervention agent.
• Figure 2: Screenshots from the BeWell24 and SleepWell24 intervention applications of our MoveSense system. The BeWell24 app has the features of monitoring physical activity, diet, etc. The SleepWell24 app has an additional component for monitoring the usage of continuous positive airway pressure (CPAP) devices.
• Figure 3: A generic formulation of the multimodal time-series forecasting problem of the MoveSense system. (a) A multimodal feature $x_t$ at time step $t$ is the collection of two features from two modalities, $u_t$ and $v_t$. (b) The forecasting function $f$ takes the values of the last $w$ time steps for the multimodal feature $x$ to forecast the value of $y_{t+1}$, which is the value of the forecast variable $y$ at time step $t+1$. (c) In an early fusion method, modalities $u$ and $v$ are combined at an early stage of the neural network. (d) In a late fusion method, representations of modalities $u$ and $v$ are combined at one of the final layers of the neural network.
• Figure 4: Block diagrams of MoveSense's backbones, LSTM-based multimodal forecasting models. (a) Multimodal forecasting with early fusion. Here, the input is an early-concatenated feature set that contains the input from both the engagement and physical activity modalities. (b) Multimodal forecasting model with late fusion. The neural network processes the inputs from the engagement modality and the activity modality independently until they are concatenated in a later layer.
• Figure 5: Performances of the MoveSense system's LSTM early fusion and LSTM late fusion multimodal models for steps forecasting on the prediabetes and sleep study participants.