Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Contextual Emotion Recognition using Large Vision Language Models

Yasaman Etesam, Özge Nilay Yalçın, Chuxuan Zhang, Angelica Lim

TL;DR

This work tackles contextual emotion recognition by leveraging both captioning plus large language models and end-to-end vision-language models. It compares a narrative captioning pipeline (NarraCap) and strong caption baselines against zero-shot and fine-tuned VLMs including CLIP, GPT-4 Vision, and LLaVA on the EMOTIC dataset. Key findings show that zero-shot GPT-4 Vision with prompt engineering can surpass traditional EMOTIC baselines, while fine-tuning LLaVA on a modest, augmented dataset yields the best overall F1 score, underscoring the strong generalization and data efficiency of VLMs for this task. The results highlight significant potential for emotionally aware robotics and agents, while also outlining limitations and directions for richer captioning and improved handling of multi-person scenes.

Abstract

"How does the person in the bounding box feel?" Achieving human-level recognition of the apparent emotion of a person in real world situations remains an unsolved task in computer vision. Facial expressions are not enough: body pose, contextual knowledge, and commonsense reasoning all contribute to how humans perform this emotional theory of mind task. In this paper, we examine two major approaches enabled by recent large vision language models: 1) image captioning followed by a language-only LLM, and 2) vision language models, under zero-shot and fine-tuned setups. We evaluate the methods on the Emotions in Context (EMOTIC) dataset and demonstrate that a vision language model, fine-tuned even on a small dataset, can significantly outperform traditional baselines. The results of this work aim to help robots and agents perform emotionally sensitive decision-making and interaction in the future.

Contextual Emotion Recognition using Large Vision Language Models

TL;DR

This work tackles contextual emotion recognition by leveraging both captioning plus large language models and end-to-end vision-language models. It compares a narrative captioning pipeline (NarraCap) and strong caption baselines against zero-shot and fine-tuned VLMs including CLIP, GPT-4 Vision, and LLaVA on the EMOTIC dataset. Key findings show that zero-shot GPT-4 Vision with prompt engineering can surpass traditional EMOTIC baselines, while fine-tuning LLaVA on a modest, augmented dataset yields the best overall F1 score, underscoring the strong generalization and data efficiency of VLMs for this task. The results highlight significant potential for emotionally aware robotics and agents, while also outlining limitations and directions for richer captioning and improved handling of multi-person scenes.

Abstract

"How does the person in the bounding box feel?" Achieving human-level recognition of the apparent emotion of a person in real world situations remains an unsolved task in computer vision. Facial expressions are not enough: body pose, contextual knowledge, and commonsense reasoning all contribute to how humans perform this emotional theory of mind task. In this paper, we examine two major approaches enabled by recent large vision language models: 1) image captioning followed by a language-only LLM, and 2) vision language models, under zero-shot and fine-tuned setups. We evaluate the methods on the Emotions in Context (EMOTIC) dataset and demonstrate that a vision language model, fine-tuned even on a small dataset, can significantly outperform traditional baselines. The results of this work aim to help robots and agents perform emotionally sensitive decision-making and interaction in the future.
Paper Structure (17 sections, 1 equation, 2 figures, 4 tables)

This paper contains 17 sections, 1 equation, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Image Captioning and Large Language Model
  5. Narrative Captioning
  6. ExpansionNet Captioning
  7. Caption to Emotion using LLM (Zero Shot)
  8. Mistral (Fine-Tuned)
  9. End-to-End Vision Language Models
  10. CLIP (Zero Shot)
  11. GPT-4 Vision and LLaVA (Zero Shot)
  12. LLaVA (Fine-Tuned)
  13. Prompt Engineering
  14. Experiments
  15. Results and Discussion
  16. ...and 2 more sections

Figures (2)

  • Figure 1: In this paper, we evaluated two distinct zero-shot approaches: a) using CLIP to generate captions from images, followed by providing these captions alongside a prompt to large language models (LLMs) to obtain emotion labels. b) directly providing the image and a prompt to vision language models (VLMs) and requesting the emotion labels.
  • Figure 2: CLIP saliency maps using Grad-Cam selvaraju2017grad. It identifies the regions in an input image that most significantly influence the classification score by leveraging the gradients of the score relative to the last convolutional layer's feature map.