Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Vision Language Models for Facial Attribute Recognition: Emotion, Race, Gender, and Age

Nouar AlDahoul, Myles Joshua Toledo Tan, Harishwar Reddy Kasireddy, Yasir Zaki

TL;DR

Vision language models such as generative pre-trained transformer (GPT), GEMINI, GEMINI, large language and vision assistant (LLAVA), PaliGemma, and Microsoft Florence2 are proposed to recognize facial attributes such as race, gender, age, and emotion from images with human faces to show that VLMs are competitive if not superior to traditional techniques.

Abstract

Technologies for recognizing facial attributes like race, gender, age, and emotion have several applications, such as surveillance, advertising content, sentiment analysis, and the study of demographic trends and social behaviors. Analyzing demographic characteristics based on images and analyzing facial expressions have several challenges due to the complexity of humans' facial attributes. Traditional approaches have employed CNNs and various other deep learning techniques, trained on extensive collections of labeled images. While these methods demonstrated effective performance, there remains potential for further enhancements. In this paper, we propose to utilize vision language models (VLMs) such as generative pre-trained transformer (GPT), GEMINI, large language and vision assistant (LLAVA), PaliGemma, and Microsoft Florence2 to recognize facial attributes such as race, gender, age, and emotion from images with human faces. Various datasets like FairFace, AffectNet, and UTKFace have been utilized to evaluate the solutions. The results show that VLMs are competitive if not superior to traditional techniques. Additionally, we propose "FaceScanPaliGemma"--a fine-tuned PaliGemma model--for race, gender, age, and emotion recognition. The results show an accuracy of 81.1%, 95.8%, 80%, and 59.4% for race, gender, age group, and emotion classification, respectively, outperforming pre-trained version of PaliGemma, other VLMs, and SotA methods. Finally, we propose "FaceScanGPT", which is a GPT-4o model to recognize the above attributes when several individuals are present in the image using a prompt engineered for a person with specific facial and/or physical attributes. The results underscore the superior multitasking capability of FaceScanGPT to detect the individual's attributes like hair cut, clothing color, postures, etc., using only a prompt to drive the detection and recognition tasks.

Exploring Vision Language Models for Facial Attribute Recognition: Emotion, Race, Gender, and Age

TL;DR

Vision language models such as generative pre-trained transformer (GPT), GEMINI, GEMINI, large language and vision assistant (LLAVA), PaliGemma, and Microsoft Florence2 are proposed to recognize facial attributes such as race, gender, age, and emotion from images with human faces to show that VLMs are competitive if not superior to traditional techniques.

Abstract

Technologies for recognizing facial attributes like race, gender, age, and emotion have several applications, such as surveillance, advertising content, sentiment analysis, and the study of demographic trends and social behaviors. Analyzing demographic characteristics based on images and analyzing facial expressions have several challenges due to the complexity of humans' facial attributes. Traditional approaches have employed CNNs and various other deep learning techniques, trained on extensive collections of labeled images. While these methods demonstrated effective performance, there remains potential for further enhancements. In this paper, we propose to utilize vision language models (VLMs) such as generative pre-trained transformer (GPT), GEMINI, large language and vision assistant (LLAVA), PaliGemma, and Microsoft Florence2 to recognize facial attributes such as race, gender, age, and emotion from images with human faces. Various datasets like FairFace, AffectNet, and UTKFace have been utilized to evaluate the solutions. The results show that VLMs are competitive if not superior to traditional techniques. Additionally, we propose "FaceScanPaliGemma"--a fine-tuned PaliGemma model--for race, gender, age, and emotion recognition. The results show an accuracy of 81.1%, 95.8%, 80%, and 59.4% for race, gender, age group, and emotion classification, respectively, outperforming pre-trained version of PaliGemma, other VLMs, and SotA methods. Finally, we propose "FaceScanGPT", which is a GPT-4o model to recognize the above attributes when several individuals are present in the image using a prompt engineered for a person with specific facial and/or physical attributes. The results underscore the superior multitasking capability of FaceScanGPT to detect the individual's attributes like hair cut, clothing color, postures, etc., using only a prompt to drive the detection and recognition tasks.

Paper Structure

This paper contains 36 sections, 13 figures, 24 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Challenges in facial attribute datasets
  4. Deep Learning for facial attributes recognition
  5. Emerging role of LLMs in image processing
  6. Research Motivation
  7. Materials and Methods
  8. Dataset Overview
  9. FairFace dataset
  10. Emotion dataset
  11. UTK-Face dataset
  12. DiverseFaces
  13. Methods
  14. OpenAI GPT-4o
  15. Google GEMINI-1.5
  16. ...and 21 more sections

Figures (13)

  • Figure 1: Several samples from each race category to show the challenging contents of this dataset. The races are as follows: first row: Black, second row: Indian, third row: Middle Eastern, fourth row: Latinx_Hispanic, fifth row: White, sixth row: East Asian, and seventh row: Southeast Asian.
  • Figure 2: Several samples from the AffectNet dataset.
  • Figure 3: Several samples of UTKFace dataset.
  • Figure 4: Few samples of the DiverseFaces dataset.
  • Figure 5: The Proposed Solution block diagram.
  • ...and 8 more figures