Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Weaknesses of Facial Emotion Recognition Systems

Aleksandra Jamróz, Patrycja Wysocka, Piotr Garbat

TL;DR

The paper interrogates the generalization of leading facial emotion recognition systems by training models on one dataset and testing on others, revealing significant cross-dataset degradation. It analyzes three state-of-the-art FER models—POSTER++ poster_v2, DMUE, and DAN—across AffectNet, RAF-DB, and ExpW, and shows that accuracy can overstate real-world performance due to dataset biases and class imbalance. Key findings include persistent difficulties with minority emotions, label biases across in-the-wild datasets, and limited transferability between datasets. The work advocates data-centric improvements, balanced labeling, and multimodal approaches to improve FER robustness in real-world deployments.

Abstract

Emotion detection from faces is one of the machine learning problems needed for human-computer interaction. The variety of methods used is enormous, which motivated an in-depth review of articles and scientific studies. Three of the most interesting and best solutions are selected, followed by the selection of three datasets that stood out for the diversity and number of images in them. The selected neural networks are trained, and then a series of experiments are performed to compare their performance, including testing on different datasets than a model was trained on. This reveals weaknesses in existing solutions, including differences between datasets, unequal levels of difficulty in recognizing certain emotions and the challenges in differentiating between closely related emotions.

Weaknesses of Facial Emotion Recognition Systems

TL;DR

The paper interrogates the generalization of leading facial emotion recognition systems by training models on one dataset and testing on others, revealing significant cross-dataset degradation. It analyzes three state-of-the-art FER models—POSTER++ poster_v2, DMUE, and DAN—across AffectNet, RAF-DB, and ExpW, and shows that accuracy can overstate real-world performance due to dataset biases and class imbalance. Key findings include persistent difficulties with minority emotions, label biases across in-the-wild datasets, and limited transferability between datasets. The work advocates data-centric improvements, balanced labeling, and multimodal approaches to improve FER robustness in real-world deployments.

Abstract

Emotion detection from faces is one of the machine learning problems needed for human-computer interaction. The variety of methods used is enormous, which motivated an in-depth review of articles and scientific studies. Three of the most interesting and best solutions are selected, followed by the selection of three datasets that stood out for the diversity and number of images in them. The selected neural networks are trained, and then a series of experiments are performed to compare their performance, including testing on different datasets than a model was trained on. This reveals weaknesses in existing solutions, including differences between datasets, unequal levels of difficulty in recognizing certain emotions and the challenges in differentiating between closely related emotions.
Paper Structure (12 sections, 2 figures, 4 tables)

This paper contains 12 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. General concept
  4. Experiments
  5. FER Models
  6. Datasets
  7. Training and testing
  8. Conclusions
  9. Minority classes
  10. Datasets differences
  11. Truth about accuracy
  12. Possible improvements

Figures (2)

  • Figure 1: Class distributions in chosen datasets. Each set contains more images marked as happy, with neutral coming in second place. All other classes have significantly fewer samples. The collections were created by bulk downloading images from the internet and it is suspected that happy and neutral images are the most abundant on the web, as they are the images people upload most frequently.
  • Figure 2: Results of POSTER++ models evaluation on AffectNet dataset. These highlight differences between the datasets, exposing classes that are confused with each other most often. A) The most confused pairing between RAF-DB and AffectNet is surprise with fear, and sad with neutral and disgust. B) There are a great deal of images labelled neutral in ExpW, which has resulted in over-classification as neutral, rather than more varied results. It can also be seen that the least frequent classes in ExpW (angry, disgust and fear) have drastically the worst results.