Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoning

Yasaman Etesam, Özge Nilay Yalçın, Chuxuan Zhang, Angelica Lim

TL;DR

This work proposes multiple methods to incorporate the emotional reasoning capabilities by constructing “narrative captions” relevant to emotion perception, that includes contextual and physical signal descriptors that focuses on “Who”, “What”, “Where” and “How” questions related to the image and emotions of the individual.

Abstract

The emotional theory of mind problem requires facial expressions, body pose, contextual information and implicit commonsense knowledge to reason about the person's emotion and its causes, making it currently one of the most difficult problems in affective computing. In this work, we propose multiple methods to incorporate the emotional reasoning capabilities by constructing "narrative captions" relevant to emotion perception, that includes contextual and physical signal descriptors that focuses on "Who", "What", "Where" and "How" questions related to the image and emotions of the individual. We propose two distinct ways to construct these captions using zero-shot classifiers (CLIP) and fine-tuning visual-language models (LLaVA) over human generated descriptors. We further utilize these captions to guide the reasoning of language (GPT-4) and vision-language models (LLaVa, GPT-Vision). We evaluate the use of the resulting models in an image-to-language-to-emotion task. Our experiments showed that combining the "Fast" narrative descriptors and "Slow" reasoning of language models is a promising way to achieve emotional theory of mind.

Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoning

TL;DR

This work proposes multiple methods to incorporate the emotional reasoning capabilities by constructing “narrative captions” relevant to emotion perception, that includes contextual and physical signal descriptors that focuses on “Who”, “What”, “Where” and “How” questions related to the image and emotions of the individual.

Abstract

The emotional theory of mind problem requires facial expressions, body pose, contextual information and implicit commonsense knowledge to reason about the person's emotion and its causes, making it currently one of the most difficult problems in affective computing. In this work, we propose multiple methods to incorporate the emotional reasoning capabilities by constructing "narrative captions" relevant to emotion perception, that includes contextual and physical signal descriptors that focuses on "Who", "What", "Where" and "How" questions related to the image and emotions of the individual. We propose two distinct ways to construct these captions using zero-shot classifiers (CLIP) and fine-tuning visual-language models (LLaVA) over human generated descriptors. We further utilize these captions to guide the reasoning of language (GPT-4) and vision-language models (LLaVa, GPT-Vision). We evaluate the use of the resulting models in an image-to-language-to-emotion task. Our experiments showed that combining the "Fast" narrative descriptors and "Slow" reasoning of language models is a promising way to achieve emotional theory of mind.
Paper Structure (15 sections, 5 figures, 1 table)

This paper contains 15 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Narrative Captioning
  5. Human-generated Captions
  6. Emotion Inference: Fast and Slow
  7. Experiments
  8. Trained Baseline Model
  9. Fast Framework
  10. Fast and Slow Framework
  11. Vision-Language Model (VLM) Baselines
  12. Results and Discussion
  13. Limitations
  14. Conclusion
  15. Ethical Considerations

Figures (5)

  • Figure 1: Fast frameworks might perceive the body pose with raised arms and predict the emotions of 'surprise' and 'fear'. Slow networks could further reason about context and future implications.
  • Figure 2: Qualitative results of EMOTIC images, ground truth (GT) labels, captions and inferred labels from example models from "Fast" and "Fast & Slow" framework.
  • Figure 3: Example failure cases of NarraCap (X.a) that was correctly captioned by trained LLaVA captions (X.b).
  • Figure 4: Examples for correct (Left) and incorrect (Right) grounding of NarraCapsXL, where captions change depending on the red bounding box.
  • Figure 5: CLIP saliency maps of example images and their resulting captions generated by trained LLaVA model. Captions fail to capture person in the bounding box, even though it explains the scene correctly.