Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A PCA based Keypoint Tracking Approach to Automated Facial Expressions Encoding

Shivansh Chandra Tripathi, Rahul Garg

TL;DR

This work tackles the challenge of manual Facial Action Coding System (FACS) labeling by introducing an unsupervised PCA-based framework that derives data-driven AUs (PCA AUs) from facial keypoint movements. By preprocessing 68-point keypoints across datasets and applying PCA, the authors obtain interpretable movement components that align with facial muscles and generalize across DISFA, BP4D-Spontaneous, and CK+. They show that PCA AUs explain a large portion of variance in test data (around 92.83% on average) and compare favorably with FACS AUs, especially when the number of components is small, while acknowledging that interpretability is incomplete for some components. The approach promises scalable, real-time facial expression analysis with cross-dataset applicability, and the authors release code to encourage further research and validation.

Abstract

The Facial Action Coding System (FACS) for studying facial expressions is manual and requires significant effort and expertise. This paper explores the use of automated techniques to generate Action Units (AUs) for studying facial expressions. We propose an unsupervised approach based on Principal Component Analysis (PCA) and facial keypoint tracking to generate data-driven AUs called PCA AUs using the publicly available DISFA dataset. The PCA AUs comply with the direction of facial muscle movements and are capable of explaining over 92.83 percent of the variance in other public test datasets (BP4D-Spontaneous and CK+), indicating their capability to generalize facial expressions. The PCA AUs are also comparable to a keypoint-based equivalence of FACS AUs in terms of variance explained on the test datasets. In conclusion, our research demonstrates the potential of automated techniques to be an alternative to manual FACS labeling which could lead to efficient real-time analysis of facial expressions in psychology and related fields. To promote further research, we have made code repository publicly available.

A PCA based Keypoint Tracking Approach to Automated Facial Expressions Encoding

TL;DR

This work tackles the challenge of manual Facial Action Coding System (FACS) labeling by introducing an unsupervised PCA-based framework that derives data-driven AUs (PCA AUs) from facial keypoint movements. By preprocessing 68-point keypoints across datasets and applying PCA, the authors obtain interpretable movement components that align with facial muscles and generalize across DISFA, BP4D-Spontaneous, and CK+. They show that PCA AUs explain a large portion of variance in test data (around 92.83% on average) and compare favorably with FACS AUs, especially when the number of components is small, while acknowledging that interpretability is incomplete for some components. The approach promises scalable, real-time facial expression analysis with cross-dataset applicability, and the authors release code to encourage further research and validation.

Abstract

The Facial Action Coding System (FACS) for studying facial expressions is manual and requires significant effort and expertise. This paper explores the use of automated techniques to generate Action Units (AUs) for studying facial expressions. We propose an unsupervised approach based on Principal Component Analysis (PCA) and facial keypoint tracking to generate data-driven AUs called PCA AUs using the publicly available DISFA dataset. The PCA AUs comply with the direction of facial muscle movements and are capable of explaining over 92.83 percent of the variance in other public test datasets (BP4D-Spontaneous and CK+), indicating their capability to generalize facial expressions. The PCA AUs are also comparable to a keypoint-based equivalence of FACS AUs in terms of variance explained on the test datasets. In conclusion, our research demonstrates the potential of automated techniques to be an alternative to manual FACS labeling which could lead to efficient real-time analysis of facial expressions in psychology and related fields. To promote further research, we have made code repository publicly available.
Paper Structure (12 sections, 5 figures, 1 table)

This paper contains 12 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Facial Keypoint-Based Automated Coding System
  3. Geometric Corrections
  4. Feature Generation
  5. Principal Component Analysis (PCA)
  6. Feature Generation for FACS AUs
  7. Experiments
  8. Datasets
  9. Evaluation Metrics
  10. PCA AUs
  11. Comparison of PCA AUs with FACS AUs
  12. Conclusion

Figures (5)

  • Figure 1: Keypoint locations for the 68-keypoint template in the three datasets. Yellow keypoints indicate the standard affine co-ordinates
  • Figure 2: Train VE and the average of the Test VE on remaining two datasets when PCA is trained on either DISFA, CK+ or BP4D-Spontaneous with $k=1$ to $k=136$ (x-axis on log scale)
  • Figure 3: PCA AUs- the red color represents the keypoint position on a neutral face, green on an expression face and blue is a non-interpretable movement. The arrow shows the direction of movement of keypoints. (Face image source: BP4D-Spontaneous zhang2014bp4d)
  • Figure 4: Muscle movements generating AUs that exclude head movements as given by FACS
  • Figure 5: Comparison of PCA AUs, pure AUs and comb AUs (x-axis on log scale)