Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi Loss-based Feature Fusion and Top Two Voting Ensemble Decision Strategy for Facial Expression Recognition in the Wild

Guangyao Zhou, Yuanlun Xie, Yiqin Fu, Zhaokun Wang

TL;DR

This paper proposes one novel single model named R18+FAML, as well as one ensemble model named T2V to improve the performance of the FER in the wild, and proposes one ensemble strategy, i.e., the Top Two Voting (T2V) to support the classification of FER, which can consider more classification information comprehensively.

Abstract

Facial expression recognition (FER) in the wild is a challenging task affected by the image quality and has attracted broad interest in computer vision. There is no research using feature fusion and ensemble strategy for FER simultaneously. Different from previous studies, this paper applies both internal feature fusion for a single model and feature fusion among multiple networks, as well as the ensemble strategy. This paper proposes one novel single model named R18+FAML, as well as one ensemble model named R18+FAML-FGA-T2V to improve the performance of the FER in the wild. Based on the structure of ResNet18 (R18), R18+FAML combines internal Feature fusion and three Attention blocks using Multiple Loss functions (FAML) to improve the diversity of the feature extraction. To improve the performance of R18+FAML, we propose a Feature fusion among networks based on the Genetic Algorithm (FGA), which can fuse the convolution kernels for feature extraction of multiple networks. On the basis of R18+FAML and FGA, we propose one ensemble strategy, i.e., the Top Two Voting (T2V) to support the classification of FER, which can consider more classification information comprehensively. Combining the above strategies, R18+FAML-FGA-T2V can focus on the main expression-aware areas. Extensive experiments demonstrate that our single model R18+FAML and the ensemble model R18+FAML-FGA-T2V achieve the accuracies of $\left( 90.32, 62.17, 65.83 \right)\%$ and $\left( 91.59, 63.27, 66.63 \right)\%$ on three challenging unbalanced FER datasets RAF-DB, AffectNet-8 and AffectNet-7 respectively, both outperforming the state-of-the-art results.

Multi Loss-based Feature Fusion and Top Two Voting Ensemble Decision Strategy for Facial Expression Recognition in the Wild

TL;DR

This paper proposes one novel single model named R18+FAML, as well as one ensemble model named T2V to improve the performance of the FER in the wild, and proposes one ensemble strategy, i.e., the Top Two Voting (T2V) to support the classification of FER, which can consider more classification information comprehensively.

Abstract

Facial expression recognition (FER) in the wild is a challenging task affected by the image quality and has attracted broad interest in computer vision. There is no research using feature fusion and ensemble strategy for FER simultaneously. Different from previous studies, this paper applies both internal feature fusion for a single model and feature fusion among multiple networks, as well as the ensemble strategy. This paper proposes one novel single model named R18+FAML, as well as one ensemble model named R18+FAML-FGA-T2V to improve the performance of the FER in the wild. Based on the structure of ResNet18 (R18), R18+FAML combines internal Feature fusion and three Attention blocks using Multiple Loss functions (FAML) to improve the diversity of the feature extraction. To improve the performance of R18+FAML, we propose a Feature fusion among networks based on the Genetic Algorithm (FGA), which can fuse the convolution kernels for feature extraction of multiple networks. On the basis of R18+FAML and FGA, we propose one ensemble strategy, i.e., the Top Two Voting (T2V) to support the classification of FER, which can consider more classification information comprehensively. Combining the above strategies, R18+FAML-FGA-T2V can focus on the main expression-aware areas. Extensive experiments demonstrate that our single model R18+FAML and the ensemble model R18+FAML-FGA-T2V achieve the accuracies of and on three challenging unbalanced FER datasets RAF-DB, AffectNet-8 and AffectNet-7 respectively, both outperforming the state-of-the-art results.
Paper Structure (15 sections, 13 equations, 8 figures, 6 tables, 1 algorithm)

This paper contains 15 sections, 13 equations, 8 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Facial Expression Recognition
  4. Feature Fusion and Ensemble Strategies
  5. Network and Methodologies
  6. Network Architecture and Training Strategies
  7. Feature Fusion between Models based on GA
  8. Ensemble Strategy: T2V
  9. Experiments
  10. Experiments Design and Datasets
  11. Ablation Studies
  12. The Comparison with the State-of-the-Art
  13. Universality Studies of T2V Compared with Other Ensembles
  14. Attention Visualization
  15. Conclusion and Future work

Figures (8)

  • Figure 1: The architecture of the proposed ResNet18+FAML.
  • Figure 2: The diagrams of ensemble and fusion strategies for facial expression recognition: integrating the interest areas of multiple networks to focus on main expression-aware areas for higher accuracy.
  • Figure 3: The frameworks of the proposed R18+FAML-FGA-T2V.
  • Figure 4: The architecture of ResNet18 with three Attention Blocks.
  • Figure 5: The confusion matrices (%) of R18, R18+FAML and R18+FAML-FGA-T2V on RAF-DB and AffectNet-8.
  • ...and 3 more figures