Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Large Language Models Aid in Annotating Speech Emotional Data? Uncovering New Frontiers

Siddique Latif, Muhammad Usama, Mohammad Ibrahim Malik, Björn W. Schuller

TL;DR

This study investigates whether large language models can effectively annotate speech emotion data to bolster SER performance. By enriching text prompts with audio context (average energy, pitch, gender) and discrete VQ-VAE speech codes, the authors demonstrate that few-shot ChatGPT annotations significantly improve within-corpus SER and, when used for data augmentation, yield notable gains on MELD-derived tasks and cross-corpus scenarios. The work provides a practical pathway to leverage LLMs for annotation at lower cost, while acknowledging limitations in generalization and the need for combining LLM outputs with human annotations. Overall, the approach reveals new frontiers for LLM-assisted annotation in affective computing and highlights the potential of audio-contextual LLM prompts to enhance speech emotion classification.

Abstract

Despite recent advancements in speech emotion recognition (SER) models, state-of-the-art deep learning (DL) approaches face the challenge of the limited availability of annotated data. Large language models (LLMs) have revolutionised our understanding of natural language, introducing emergent properties that broaden comprehension in language, speech, and vision. This paper examines the potential of LLMs to annotate abundant speech data, aiming to enhance the state-of-the-art in SER. We evaluate this capability across various settings using publicly available speech emotion classification datasets. Leveraging ChatGPT, we experimentally demonstrate the promising role of LLMs in speech emotion data annotation. Our evaluation encompasses single-shot and few-shots scenarios, revealing performance variability in SER. Notably, we achieve improved results through data augmentation, incorporating ChatGPT-annotated samples into existing datasets. Our work uncovers new frontiers in speech emotion classification, highlighting the increasing significance of LLMs in this field moving forward.

Can Large Language Models Aid in Annotating Speech Emotional Data? Uncovering New Frontiers

TL;DR

This study investigates whether large language models can effectively annotate speech emotion data to bolster SER performance. By enriching text prompts with audio context (average energy, pitch, gender) and discrete VQ-VAE speech codes, the authors demonstrate that few-shot ChatGPT annotations significantly improve within-corpus SER and, when used for data augmentation, yield notable gains on MELD-derived tasks and cross-corpus scenarios. The work provides a practical pathway to leverage LLMs for annotation at lower cost, while acknowledging limitations in generalization and the need for combining LLM outputs with human annotations. Overall, the approach reveals new frontiers for LLM-assisted annotation in affective computing and highlights the potential of audio-contextual LLM prompts to enhance speech emotion classification.

Abstract

Despite recent advancements in speech emotion recognition (SER) models, state-of-the-art deep learning (DL) approaches face the challenge of the limited availability of annotated data. Large language models (LLMs) have revolutionised our understanding of natural language, introducing emergent properties that broaden comprehension in language, speech, and vision. This paper examines the potential of LLMs to annotate abundant speech data, aiming to enhance the state-of-the-art in SER. We evaluate this capability across various settings using publicly available speech emotion classification datasets. Leveraging ChatGPT, we experimentally demonstrate the promising role of LLMs in speech emotion data annotation. Our evaluation encompasses single-shot and few-shots scenarios, revealing performance variability in SER. Notably, we achieve improved results through data augmentation, incorporating ChatGPT-annotated samples into existing datasets. Our work uncovers new frontiers in speech emotion classification, highlighting the increasing significance of LLMs in this field moving forward.
Paper Structure (19 sections, 5 equations, 3 figures, 2 tables)

This paper contains 19 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. VQ-VAE for Speech Code Generation
  5. Emotion Label Annotation using LLMs
  6. Speech Emotion Classifier
  7. Experimental Setup
  8. Datasets
  9. IEMOCAP
  10. MSP-IMPROV
  11. MELD
  12. Speech Features
  13. Hyperparameters
  14. Experiments and Results
  15. Within Corpus Experiments
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Model Diagram of the VQ-VAE
  • Figure 2: Comparing the classification performance (UAR %) using training data annotated by ChatGPT and original IEMOCAP labels.
  • Figure 3: Comparing the classier performance (UAR %) with data augmentation.