Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoding Emotions: Unveiling Facial Expressions through Acoustic Sensing with Contrastive Attention

Guangjing Wang, Juexing Wang, Ce Zhou, Weikang Ding, Huacheng Zeng, Tianxing Li, Qiben Yan

TL;DR

This paper introduces FacER+, an active acoustic facial expression recognition system, which eliminates the requirement for external microphone arrays and develops a contrastive external attention-based model to consistently learn expression features across different users, reducing the distribution differences.

Abstract

Expression recognition holds great promise for applications such as content recommendation and mental healthcare by accurately detecting users' emotional states. Traditional methods often rely on cameras or wearable sensors, which raise privacy concerns and add extra device burdens. In addition, existing acoustic-based methods struggle to maintain satisfactory performance when there is a distribution shift between the training dataset and the inference dataset. In this paper, we introduce FacER+, an active acoustic facial expression recognition system, which eliminates the requirement for external microphone arrays. FacER+ extracts facial expression features by analyzing the echoes of near-ultrasound signals emitted between the 3D facial contour and the earpiece speaker on a smartphone. This approach not only reduces background noise but also enables the identification of different expressions from various users with minimal training data. We develop a contrastive external attention-based model to consistently learn expression features across different users, reducing the distribution differences. Extensive experiments involving 20 volunteers, both with and without masks, demonstrate that FacER+ can accurately recognize six common facial expressions with over 90% accuracy in diverse, user-independent real-life scenarios, surpassing the performance of the leading acoustic sensing methods by 10%. FacER+ offers a robust and practical solution for facial expression recognition.

Decoding Emotions: Unveiling Facial Expressions through Acoustic Sensing with Contrastive Attention

TL;DR

This paper introduces FacER+, an active acoustic facial expression recognition system, which eliminates the requirement for external microphone arrays and develops a contrastive external attention-based model to consistently learn expression features across different users, reducing the distribution differences.

Abstract

Expression recognition holds great promise for applications such as content recommendation and mental healthcare by accurately detecting users' emotional states. Traditional methods often rely on cameras or wearable sensors, which raise privacy concerns and add extra device burdens. In addition, existing acoustic-based methods struggle to maintain satisfactory performance when there is a distribution shift between the training dataset and the inference dataset. In this paper, we introduce FacER+, an active acoustic facial expression recognition system, which eliminates the requirement for external microphone arrays. FacER+ extracts facial expression features by analyzing the echoes of near-ultrasound signals emitted between the 3D facial contour and the earpiece speaker on a smartphone. This approach not only reduces background noise but also enables the identification of different expressions from various users with minimal training data. We develop a contrastive external attention-based model to consistently learn expression features across different users, reducing the distribution differences. Extensive experiments involving 20 volunteers, both with and without masks, demonstrate that FacER+ can accurately recognize six common facial expressions with over 90% accuracy in diverse, user-independent real-life scenarios, surpassing the performance of the leading acoustic sensing methods by 10%. FacER+ offers a robust and practical solution for facial expression recognition.

Paper Structure

This paper contains 28 sections, 11 equations, 15 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Acoustic Signal
  5. Attention Mechanism
  6. Contrastive Learning
  7. System Design
  8. Acoustic Sensing Design
  9. Signal Generator
  10. Noise Removal
  11. Domain Gap for Expression Recognition
  12. Acoustic Sensing Data Augmentation
  13. Contrastive Attention-based Domain Adaptation
  14. Pseudo Label Generation
  15. Attention-based Expression Learning
  16. ...and 13 more sections

Figures (15)

  • Figure 1: Facial expression recognition using a smartphone.
  • Figure 2: Illustration of Preliminaries.
  • Figure 3: The raw signal and the signal after noise removal.
  • Figure 4: The raw signal and the signal after noise removal.
  • Figure 5: The spectrogram of six expressions in 50 milliseconds. The first row is from a woman without a mask, the second row is from a man without a mask, and the third row is from the same man with a mask.
  • ...and 10 more figures