Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Evaluation of a Visual Question Answering Strategy for Zero-shot Facial Expression Recognition in Still Images

Modesto Castrillón-Santana, Oliverio J Santana, David Freire-Obregón, Daniel Hernández-Sosa, Javier Lorenzo-Navarro

TL;DR

This work tackles zero-shot facial expression recognition by leveraging locally executable visual language models through a visual question answering framework. By crafting task-specific prompts and mapping diverse VLM outputs to basic emotions (with auxiliary mapping via ChatGPT), the approach enables zero-shot FER without fine-tuning. The authors comprehensively evaluate multiple VLM architectures across AffectNet7, FERPlus, and RAF-DB, showing substantial prompt sensitivity yet identifying configurations (notably PaliGemma 3b-mix-448 and LLAMA 3.2:11B) that yield strong cross-dataset performance, surpassing some traditional FER baselines. The study highlights the potential and limitations of prompt-driven VLMs for FER, underscoring the need for further work to stabilize and generalize zero-shot FER in real-world settings.

Abstract

Facial expression recognition (FER) is a key research area in computer vision and human-computer interaction. Despite recent advances in deep learning, challenges persist, especially in generalizing to new scenarios. In fact, zero-shot FER significantly reduces the performance of state-of-the-art FER models. To address this problem, the community has recently started to explore the integration of knowledge from Large Language Models for visual tasks. In this work, we evaluate a broad collection of locally executed Visual Language Models (VLMs), avoiding the lack of task-specific knowledge by adopting a Visual Question Answering strategy. We compare the proposed pipeline with state-of-the-art FER models, both integrating and excluding VLMs, evaluating well-known FER benchmarks: AffectNet, FERPlus, and RAF-DB. The results show excellent performance for some VLMs in zero-shot FER scenarios, indicating the need for further exploration to improve FER generalization.

An Evaluation of a Visual Question Answering Strategy for Zero-shot Facial Expression Recognition in Still Images

TL;DR

This work tackles zero-shot facial expression recognition by leveraging locally executable visual language models through a visual question answering framework. By crafting task-specific prompts and mapping diverse VLM outputs to basic emotions (with auxiliary mapping via ChatGPT), the approach enables zero-shot FER without fine-tuning. The authors comprehensively evaluate multiple VLM architectures across AffectNet7, FERPlus, and RAF-DB, showing substantial prompt sensitivity yet identifying configurations (notably PaliGemma 3b-mix-448 and LLAMA 3.2:11B) that yield strong cross-dataset performance, surpassing some traditional FER baselines. The study highlights the potential and limitations of prompt-driven VLMs for FER, underscoring the need for further work to stabilize and generalize zero-shot FER in real-world settings.

Abstract

Facial expression recognition (FER) is a key research area in computer vision and human-computer interaction. Despite recent advances in deep learning, challenges persist, especially in generalizing to new scenarios. In fact, zero-shot FER significantly reduces the performance of state-of-the-art FER models. To address this problem, the community has recently started to explore the integration of knowledge from Large Language Models for visual tasks. In this work, we evaluate a broad collection of locally executed Visual Language Models (VLMs), avoiding the lack of task-specific knowledge by adopting a Visual Question Answering strategy. We compare the proposed pipeline with state-of-the-art FER models, both integrating and excluding VLMs, evaluating well-known FER benchmarks: AffectNet, FERPlus, and RAF-DB. The results show excellent performance for some VLMs in zero-shot FER scenarios, indicating the need for further exploration to improve FER generalization.
Paper Structure (9 sections, 2 equations, 5 figures, 2 tables)

This paper contains 9 sections, 2 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Previous work
  3. Proposal
  4. Pipeline
  5. Experimental evaluation
  6. VLMs
  7. Datasets and metrics
  8. Results
  9. Conclusions

Figures (5)

  • Figure 1: Graphical outline of the sample processing pipeline (image source thispersondoesnotexist.com)
  • Figure : (a) AffectNet7
  • Figure : (a) AffectNet7
  • Figure : (b) FERPlus
  • Figure : (c) RAF-DB