MoodCapture: Depression Detection Using In-the-Wild Smartphone Images

TL;DR

It is shown that a random forest trained with face landmarks can classify samples as depressed or non-depressed and predict raw PHQ-8 scores effectively, providing critical insights into privacy concerns that inform the future design of in-the-wild image-based mental health assessment tools.

Abstract

MoodCapture presents a novel approach that assesses depression based on images automatically captured from the front-facing camera of smartphones as people go about their daily lives. We collect over 125,000 photos in the wild from N=177 participants diagnosed with major depressive disorder for 90 days. Images are captured naturalistically while participants respond to the PHQ-8 depression survey question: \textit{``I have felt down, depressed, or hopeless''}. Our analysis explores important image attributes, such as angle, dominant colors, location, objects, and lighting. We show that a random forest trained with face landmarks can classify samples as depressed or non-depressed and predict raw PHQ-8 scores effectively. Our post-hoc analysis provides several insights through an ablation study, feature importance analysis, and bias assessment. Importantly, we evaluate user concerns about using MoodCapture to detect depression based on sharing photos, providing critical insights into privacy concerns that inform the future design of in-the-wild image-based mental health assessment tools.

Figures (7)

• Figure 1: MoodCapture Framework: Users answer the PHQ-8 depression survey questions using the MoodCapture Android App while the app takes bursts of photos using the front-facing camera on the smartphone (top-left). Image characteristics are analysed using factors, such as, illumination, indoor vs. outdoors, phone angle, dominant image color, and background objects (top-right). Given that raw images compromise privacy, these characteristics provide insights into the types of features our machine learning and deep learning model infer. Finally, OpenFace features are extracted to train machine learning models, while raw images are used to train deep learning models (bottom). Depression classification is a binary predictor that classifies an image as depressed or not depressed, whereas PHQ-8 score prediction is a regression model that predicts raw PHQ-8 scores.
• Figure 2: PHQ-8 application screens for each item: Images are always captured while users respond to the PHQ-8 depression survey question (highlighted in cyan): "I have felt down, depressed, or hopeless". While users consent to have photos taken using the front-facing camera during the operation of the MoodCapture app they are not informed exactly when these photos are captured to promote in the moment naturalistic and authentic images.
• Figure 3: PHQ-8 score statistics: Figure (a) depicts the distribution of the PHQ-8 score reported by the participant and the corresponding label (i.e., Depression or No Depression). Figure (b) shows the variability of PHQ-8 scores among participants over the duration of the study (Cronbach's $\pmb{\alpha=0.85}$).
• Figure 4: Background objects: Word Cloud showing the range of objects detected in the background of the images captured. (Acc=91.72 ; $\pmb{\kappa}$=0.70)
• Figure 5: SHAP plots describing the top 10 features for the classification and regression tasks. The best performing random forest trained using 3D landmark features is evaluated using SHAP. The features are x and y axis with the numbers (0-indexed) corresponding to facial landmarks sagonas2013300sagonas2016300.