Leveraging LLMs to Predict Affective States via Smartphone Sensor Features

TL;DR

This study investigates predicting affective states from smartphone sensor data through grounding large language models (LLMs) in digital phenotyping tasks. By testing zero-shot and few-shot prompting (with chain-of-thought variants) on weekly I-PANAS-SF items collected from 10 Australian university students over 17 weeks, the authors demonstrate that LLMs can infer affective states from multisensor activity descriptions. Zero-shot predictions yield moderate accuracy ($MAE_{overall} \approx 1.65$ out of 5, $\epsilon_{overall} \approx 40.9\%$), while few-shot prompts substantially improve performance, achieving near parity between positive and negative affects ($MAE_{pos} \approx 0.75$, $MAE_{neg} \approx 0.74$) and reduced variance. These results highlight the potential of LLMs for digital phenotyping and affective state prediction, while also signaling challenges for negative affect and the value of expanding datasets and richer context in prompts.

Abstract

As mental health issues for young adults present a pressing public health concern, daily digital mood monitoring for early detection has become an important prospect. An active research area, digital phenotyping, involves collecting and analysing data from personal digital devices such as smartphones (usage and sensors) and wearables to infer behaviours and mental health. Whilst this data is standardly analysed using statistical and machine learning approaches, the emergence of large language models (LLMs) offers a new approach to make sense of smartphone sensing data. Despite their effectiveness across various domains, LLMs remain relatively unexplored in digital mental health, particularly in integrating mobile sensor data. Our study aims to bridge this gap by employing LLMs to predict affect outcomes based on smartphone sensing data from university students. We demonstrate the efficacy of zero-shot and few-shot embedding LLMs in inferring general wellbeing. Our findings reveal that LLMs can make promising predictions of affect measures using solely smartphone sensing data. This research sheds light on the potential of LLMs for affective state prediction, emphasizing the intricate link between smartphone behavioral patterns and affective states. To our knowledge, this is the first work to leverage LLMs for affective state prediction and digital phenotyping tasks.

Figures (4)

• Figure 1: Prompt for Zero-Shot Tasks
• Figure 2: Prompt for Few-Shot Tasks
• Figure 3: Learning curve for data from two participants
• Figure 4: Four participant examples of the linear relationship between MAEs for predicting positive and negative affects.