Personalized Emotion Detection from Floor Vibrations Induced by Footsteps

TL;DR

This work addresses privacy and intrusiveness barriers in emotion recognition by introducing EmotionVibe, which infers valence and arousal from footstep-induced floor vibrations. It combines gait- and vibration-related features within a two-stage modeling framework and personalizes predictions for each target by weighting training data according to gait similarity, achieving MAEs around 1.11 for valence and 1.07 for arousal in real-world tests. The approach demonstrates nonintrusive emotion monitoring suitable for smart homes and mental health applications, with robust ablation and robustness analyses showing consistent gains from feature fusion and personalization. Practical impact includes a scalable sensing paradigm using low-cost geophones and a data-efficient learning regimen that can adapt to individual gait differences while maintaining privacy.

Abstract

Emotion recognition is critical for various applications such as early detection of mental health disorders and emotion based smart home systems. Previous studies used various sensing methods for emotion recognition, such as wearable sensors, cameras, and microphones. However, these methods have limitations in long term domestic, including intrusiveness and privacy concerns. To overcome these limitations, this paper introduces a nonintrusive and privacy friendly personalized emotion recognition system, EmotionVibe, which leverages footstep induced floor vibrations for emotion recognition. The main idea of EmotionVibe is that individuals' emotional states influence their gait patterns, subsequently affecting the floor vibrations induced by their footsteps. However, there are two main research challenges: 1) the complex and indirect relationship between human emotions and footstep induced floor vibrations and 2) the large between person variations within the relationship between emotions and gait patterns. To address these challenges, we first empirically characterize this complex relationship and develop an emotion sensitive feature set including gait related and vibration related features from footstep induced floor vibrations. Furthermore, we personalize the emotion recognition system for each user by calculating gait similarities between the target person (i.e., the person whose emotions we aim to recognize) and those in the training dataset and assigning greater weights to training people with similar gait patterns in the loss function. We evaluated our system in a real-world walking experiment with 20 participants, summing up to 37,001 footstep samples. EmotionVibe achieved the mean absolute error (MAE) of 1.11 and 1.07 for valence and arousal score estimations, respectively, reflecting 19.0% and 25.7% error reduction compared to the baseline method.

Figures (16)

• Figure 1: EmotionVibe recognizes human emotions by analyzing the footstep-induced floor vibration patterns, which are affected by emotions.
• Figure 2: Main Intuition of EmotionVibe. Emotion impacts gait pattern, which in turn affects footstep-induced floor vibrations.
• Figure 3: Description of Emotion Using Valence-Arousal Model russell1980circumplex.
• Figure 4: Representative example for footstep-induced floor vibration patterns characterization under different emotional states: (a, c) Average vibration signals for Person A and Person B across various emotional states. Vibration amplitudes are higher during emotions with high valence for person A. (b, d) Variations in step frequency for Person A and Person B under different emotional states. Step frequency is larger during high arousal emotional states for person A and lower during high valence low arousal states for person B. Both individuals exhibit distinct footstep-induced vibration patterns across different emotional states. In addition, individual variability is not negligible.
• Figure 5: Feature variance across the four emotion classes. Each cell indicates the deviation from the overall average for a given feature under specific emotional conditions. Positive values (towards 1.00) suggest that the feature increases in the corresponding emotion classes, while negative values (towards -1.00) denote that the feature decreases in the corresponding emotion classes. The effectiveness of these features is evident in their varying performance across different emotion classes.