Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Emotion Annotations in the Era of Large Language Models

Minxue Niu, Yara El-Tawil, Amrit Romana, Emily Mower Provost

TL;DR

The paper addresses the cost and subjectivity of human emotion annotations and evaluates GPT-4 as a scalable, complementary annotator. It conducts a multi-dataset, human-evaluation study to compare GPT-4's zero-shot emotion labels with human labels and investigates two integration strategies: pre-filtering label spaces and post-filtering samples. Key findings show GPT-4 generally aligns with human judgments and is often preferred by evaluators, especially with larger label spaces, while pre-filtering reduces cognitive load and preserves coverage and post-filtering can improve downstream model performance with less data. The work demonstrates practical pathways for incorporating LLMs into emotion annotation pipelines, highlights the need to rethink ground-truth standards, and points to promising directions for future evaluation metrics and annotation workflows.

Abstract

Modern affective computing systems rely heavily on datasets with human-annotated emotion labels, for training and evaluation. However, human annotations are expensive to obtain, sensitive to study design, and difficult to quality control, because of the subjective nature of emotions. Meanwhile, Large Language Models (LLMs) have shown remarkable performance on many Natural Language Understanding tasks, emerging as a promising tool for text annotation. In this work, we analyze the complexities of emotion annotation in the context of LLMs, focusing on GPT-4 as a leading model. In our experiments, GPT-4 achieves high ratings in a human evaluation study, painting a more positive picture than previous work, in which human labels served as the only ground truth. On the other hand, we observe differences between human and GPT-4 emotion perception, underscoring the importance of human input in annotation studies. To harness GPT-4's strength while preserving human perspective, we explore two ways of integrating GPT-4 into emotion annotation pipelines, showing its potential to flag low-quality labels, reduce the workload of human annotators, and improve downstream model learning performance and efficiency. Together, our findings highlight opportunities for new emotion labeling practices and suggest the use of LLMs as a promising tool to aid human annotation.

Rethinking Emotion Annotations in the Era of Large Language Models

TL;DR

The paper addresses the cost and subjectivity of human emotion annotations and evaluates GPT-4 as a scalable, complementary annotator. It conducts a multi-dataset, human-evaluation study to compare GPT-4's zero-shot emotion labels with human labels and investigates two integration strategies: pre-filtering label spaces and post-filtering samples. Key findings show GPT-4 generally aligns with human judgments and is often preferred by evaluators, especially with larger label spaces, while pre-filtering reduces cognitive load and preserves coverage and post-filtering can improve downstream model performance with less data. The work demonstrates practical pathways for incorporating LLMs into emotion annotation pipelines, highlights the need to rethink ground-truth standards, and points to promising directions for future evaluation metrics and annotation workflows.

Abstract

Modern affective computing systems rely heavily on datasets with human-annotated emotion labels, for training and evaluation. However, human annotations are expensive to obtain, sensitive to study design, and difficult to quality control, because of the subjective nature of emotions. Meanwhile, Large Language Models (LLMs) have shown remarkable performance on many Natural Language Understanding tasks, emerging as a promising tool for text annotation. In this work, we analyze the complexities of emotion annotation in the context of LLMs, focusing on GPT-4 as a leading model. In our experiments, GPT-4 achieves high ratings in a human evaluation study, painting a more positive picture than previous work, in which human labels served as the only ground truth. On the other hand, we observe differences between human and GPT-4 emotion perception, underscoring the importance of human input in annotation studies. To harness GPT-4's strength while preserving human perspective, we explore two ways of integrating GPT-4 into emotion annotation pipelines, showing its potential to flag low-quality labels, reduce the workload of human annotators, and improve downstream model learning performance and efficiency. Together, our findings highlight opportunities for new emotion labeling practices and suggest the use of LLMs as a promising tool to aid human annotation.

Paper Structure

This paper contains 28 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Emotion Label Spaces
  4. Challenges in Obtaining Human Annotations
  5. Emotional Capability of LLMs
  6. Datasets
  7. GPT-4's Emotion Annotation Capability
  8. GPT-4 Prompting
  9. Human Evaluation Study
  10. Sample Selection
  11. Crowdsourcing Experiments
  12. Label Distributions and Agreement Analysis
  13. Human Evaluation Results
  14. Preference
  15. Perceived Accuracy Ratings
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Disagreements between Human and GPT-4 Annotations, visualized as confusion matrices.
  • Figure 2: Proportion of human evaluators' votes favoring human annotations versus GPT-4 annotations, across datasets.
  • Figure 3: Perceived accuracy ratings on a scale of 1 (totally inaccurate) to 7 (totally accurate).
  • Figure 4: Comparison of the cognitive load on different label sets. The bars show the mean of the ratings, and error bars show the standard deviation. Significance tests are run between Pre-Filter and Large/Small sets. Dependent t-test is used in (a) because each annotator provided an overall rating for the whole session, and annotators were assigned different samples. Independent t-test is used in (b) because we measured the avg. time spent on each sample, and the sets share the same samples. *:p$<$0.1, **:p$<$0.05.