Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recognizing Emotion Regulation Strategies from Human Behavior with Large Language Models

Philipp Müller, Alexander Heimerl, Sayed Muddashir Hossain, Lea Siegel, Jan Alexandersson, Patrick Gebhard, Elisabeth André, Tanja Schneeberger

TL;DR

This work addresses the challenge of automatically classifying emotion regulation strategies, focusing on shame, by leveraging instruction-tuned large language models (LLMs) with prompts derived from multimodal behavioral data. Using the Deep corpus, the authors fine-tune Llama2-7B and Gemma with LoRA and evaluate in a cross-subject LOSO setting, achieving strong performance (notably remaining robust when post-interaction introspection data is unavailable). A Bayesian Network baseline is included for comparison, and results indicate that LLMs excel in scenarios with incomplete introspective data, primarily by leveraging verbal and contextual cues. The findings highlight the potential for practical affective computing systems to recognize internal regulation strategies in realistic interactions, while outlining limitations and avenues for extending to additional emotions and automated feature extraction.

Abstract

Human emotions are often not expressed directly, but regulated according to internal processes and social display rules. For affective computing systems, an understanding of how users regulate their emotions can be highly useful, for example to provide feedback in job interview training, or in psychotherapeutic scenarios. However, at present no method to automatically classify different emotion regulation strategies in a cross-user scenario exists. At the same time, recent studies showed that instruction-tuned Large Language Models (LLMs) can reach impressive performance across a variety of affect recognition tasks such as categorical emotion recognition or sentiment analysis. While these results are promising, it remains unclear to what extent the representational power of LLMs can be utilized in the more subtle task of classifying users' internal emotion regulation strategy. To close this gap, we make use of the recently introduced \textsc{Deep} corpus for modeling the social display of the emotion shame, where each point in time is annotated with one of seven different emotion regulation classes. We fine-tune Llama2-7B as well as the recently introduced Gemma model using Low-rank Optimization on prompts generated from different sources of information on the \textsc{Deep} corpus. These include verbal and nonverbal behavior, person factors, as well as the results of an in-depth interview after the interaction. Our results show, that a fine-tuned Llama2-7B LLM is able to classify the utilized emotion regulation strategy with high accuracy (0.84) without needing access to data from post-interaction interviews. This represents a significant improvement over previous approaches based on Bayesian Networks and highlights the importance of modeling verbal behavior in emotion regulation.

Recognizing Emotion Regulation Strategies from Human Behavior with Large Language Models

TL;DR

This work addresses the challenge of automatically classifying emotion regulation strategies, focusing on shame, by leveraging instruction-tuned large language models (LLMs) with prompts derived from multimodal behavioral data. Using the Deep corpus, the authors fine-tune Llama2-7B and Gemma with LoRA and evaluate in a cross-subject LOSO setting, achieving strong performance (notably remaining robust when post-interaction introspection data is unavailable). A Bayesian Network baseline is included for comparison, and results indicate that LLMs excel in scenarios with incomplete introspective data, primarily by leveraging verbal and contextual cues. The findings highlight the potential for practical affective computing systems to recognize internal regulation strategies in realistic interactions, while outlining limitations and avenues for extending to additional emotions and automated feature extraction.

Abstract

Human emotions are often not expressed directly, but regulated according to internal processes and social display rules. For affective computing systems, an understanding of how users regulate their emotions can be highly useful, for example to provide feedback in job interview training, or in psychotherapeutic scenarios. However, at present no method to automatically classify different emotion regulation strategies in a cross-user scenario exists. At the same time, recent studies showed that instruction-tuned Large Language Models (LLMs) can reach impressive performance across a variety of affect recognition tasks such as categorical emotion recognition or sentiment analysis. While these results are promising, it remains unclear to what extent the representational power of LLMs can be utilized in the more subtle task of classifying users' internal emotion regulation strategy. To close this gap, we make use of the recently introduced \textsc{Deep} corpus for modeling the social display of the emotion shame, where each point in time is annotated with one of seven different emotion regulation classes. We fine-tune Llama2-7B as well as the recently introduced Gemma model using Low-rank Optimization on prompts generated from different sources of information on the \textsc{Deep} corpus. These include verbal and nonverbal behavior, person factors, as well as the results of an in-depth interview after the interaction. Our results show, that a fine-tuned Llama2-7B LLM is able to classify the utilized emotion regulation strategy with high accuracy (0.84) without needing access to data from post-interaction interviews. This represents a significant improvement over previous approaches based on Bayesian Networks and highlights the importance of modeling verbal behavior in emotion regulation.
Paper Structure (20 sections, 2 figures, 4 tables)

This paper contains 20 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model of Emotions and Emotion Regulation
  4. LLMs and Emotion Understanding
  5. Corpus
  6. Approach
  7. Multi-modal LLM Approach
  8. Prompt generation
  9. Context information
  10. Utilized LLMs
  11. Training Details
  12. Testing
  13. Bayesian Network Model
  14. Evaluation
  15. Overall Results
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Example prompt consisting of situational context, nonverbal behavior, verbalized introspection and personal context. The situational context incorporates a transcript (below the dotted line).
  • Figure 2: The Deep-BN schema constructed based on the Deep method information. The ground truth of emotion regulation is also part of the verbalized introspection. But since it represents the ground truth (and not a potential input to the model), it is colored pink. The emotion regulation strategies are based on nathanson1994shame, while internal emotion component and the experienced emotion are based on the Differential Emotions Scale izard1974differential and PANAS-X watson1994panas.