Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing the Efficacy of Deep Learning Approaches for Facial Expression Recognition in Individuals with Intellectual Disabilities

F. Xavier Gaya-Morey, Silvia Ramis, Jose M. Buades-Rubio, Cristina Manresa-Yee

TL;DR

A set of 12 convolutional neural networks are trained in different approaches for recognizing facial expressions in individuals with intellectual disabilities through tailored user-specific training methodologies, which enable the models to effectively address the unique expressions of each user.

Abstract

Facial expression recognition has gained significance as a means of imparting social robots with the capacity to discern the emotional states of users. The use of social robotics includes a variety of settings, including homes, nursing homes or daycare centers, serving to a wide range of users. Remarkable performance has been achieved by deep learning approaches, however, its direct use for recognizing facial expressions in individuals with intellectual disabilities has not been yet studied in the literature, to the best of our knowledge. To address this objective, we train a set of 12 convolutional neural networks in different approaches, including an ensemble of datasets without individuals with intellectual disabilities and a dataset featuring such individuals. Our examination of the outcomes, both the performance and the important image regions for the models, reveals significant distinctions in facial expressions between individuals with and without intellectual disabilities, as well as among individuals with intellectual disabilities. Remarkably, our findings show the need of facial expression recognition within this population through tailored user-specific training methodologies, which enable the models to effectively address the unique expressions of each user.

Assessing the Efficacy of Deep Learning Approaches for Facial Expression Recognition in Individuals with Intellectual Disabilities

TL;DR

A set of 12 convolutional neural networks are trained in different approaches for recognizing facial expressions in individuals with intellectual disabilities through tailored user-specific training methodologies, which enable the models to effectively address the unique expressions of each user.

Abstract

Facial expression recognition has gained significance as a means of imparting social robots with the capacity to discern the emotional states of users. The use of social robotics includes a variety of settings, including homes, nursing homes or daycare centers, serving to a wide range of users. Remarkable performance has been achieved by deep learning approaches, however, its direct use for recognizing facial expressions in individuals with intellectual disabilities has not been yet studied in the literature, to the best of our knowledge. To address this objective, we train a set of 12 convolutional neural networks in different approaches, including an ensemble of datasets without individuals with intellectual disabilities and a dataset featuring such individuals. Our examination of the outcomes, both the performance and the important image regions for the models, reveals significant distinctions in facial expressions between individuals with and without intellectual disabilities, as well as among individuals with intellectual disabilities. Remarkably, our findings show the need of facial expression recognition within this population through tailored user-specific training methodologies, which enable the models to effectively address the unique expressions of each user.
Paper Structure (22 sections, 7 figures, 1 table)

This paper contains 22 sections, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Methods
  4. Datasets
  5. Image preprocessing
  6. Models
  7. XAI approach
  8. Experiments
  9. First experiment: Training on FER datasets
  10. Second experiment: training on MuDERI
  11. Third experiment: explaining results
  12. Results
  13. First experiment: training on FER datasets
  14. Second experiment: training on MuDERI
  15. Third experiment: explaining results
  16. ...and 7 more sections

Figures (7)

  • Figure 1: Considerations for the scenarios: each user has video clips for each expression (happiness, sadness and anger). The clips are then divided into frames.
  • Figure 2: Box-plot showing the accuracy by network and evaluation dataset for the $k=15$ trainings on FER-DB5. The big gap in accuracy between evaluating the models on Google FE-Test and on MuDERI can be appreciated.
  • Figure 3: Accuracy on the Google FE-Test dataset of the $k$ (with $k=15$) trainings performed on FER-DB5 and MuDERI, by network. In this case, the low performance on Google FE-Test of the models trained on MuDERI should be noted.
  • Figure 4: Average per-class F1 score obtained on Google FE-Test by the trainings on MuDERI. Note the great difference among classes.
  • Figure 5: Accuracy of the different training scenarios on MuDERI, by network. We have added the average results obtained by the 15 trainings on FER-DB5 for comparison.
  • ...and 2 more figures