Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing the Reliability and Validity of GPT-4 in Annotating Emotion Appraisal Ratings

Deniss Ruder, Andero Uusberg, Kairit Sirts

TL;DR

This study evaluates GPT-4 as a reader-annotator for 21 emotion appraisal dimensions using the crowd-enVent dataset, directly comparing its outputs to human annotations. Reliability is demonstrated via multiple runs with $RMSE$ around $0.61$ and mean $\rho$ around $0.87$, while accuracy against experiencer-annotators is near or slightly better than human reader-annotators, comparable to RoBERTa baselines. Majority voting over five completions substantially improves GPT-4 performance (RMSE down to about $1.12$), though confidence-based tie-breaking provides little benefit; adding an emotion-prediction step generally harms performance, and longer event descriptions correlate with higher accuracy (optimal around 400–500 characters). The findings support using GPT-4 as a cost-effective annotator for large psychology datasets, while highlighting practical guidelines for prompt design and data quality alongside important ethical considerations.

Abstract

Appraisal theories suggest that emotions arise from subjective evaluations of events, referred to as appraisals. The taxonomy of appraisals is quite diverse, and they are usually given ratings on a Likert scale to be annotated in an experiencer-annotator or reader-annotator paradigm. This paper studies GPT-4 as a reader-annotator of 21 specific appraisal ratings in different prompt settings, aiming to evaluate and improve its performance compared to human annotators. We found that GPT-4 is an effective reader-annotator that performs close to or even slightly better than human annotators, and its results can be significantly improved by using a majority voting of five completions. GPT-4 also effectively predicts appraisal ratings and emotion labels using a single prompt, but adding instruction complexity results in poorer performance. We also found that longer event descriptions lead to more accurate annotations for both model and human annotator ratings. This work contributes to the growing usage of LLMs in psychology and the strategies for improving GPT-4 performance in annotating appraisals.

Assessing the Reliability and Validity of GPT-4 in Annotating Emotion Appraisal Ratings

TL;DR

This study evaluates GPT-4 as a reader-annotator for 21 emotion appraisal dimensions using the crowd-enVent dataset, directly comparing its outputs to human annotations. Reliability is demonstrated via multiple runs with around and mean around , while accuracy against experiencer-annotators is near or slightly better than human reader-annotators, comparable to RoBERTa baselines. Majority voting over five completions substantially improves GPT-4 performance (RMSE down to about ), though confidence-based tie-breaking provides little benefit; adding an emotion-prediction step generally harms performance, and longer event descriptions correlate with higher accuracy (optimal around 400–500 characters). The findings support using GPT-4 as a cost-effective annotator for large psychology datasets, while highlighting practical guidelines for prompt design and data quality alongside important ethical considerations.

Abstract

Appraisal theories suggest that emotions arise from subjective evaluations of events, referred to as appraisals. The taxonomy of appraisals is quite diverse, and they are usually given ratings on a Likert scale to be annotated in an experiencer-annotator or reader-annotator paradigm. This paper studies GPT-4 as a reader-annotator of 21 specific appraisal ratings in different prompt settings, aiming to evaluate and improve its performance compared to human annotators. We found that GPT-4 is an effective reader-annotator that performs close to or even slightly better than human annotators, and its results can be significantly improved by using a majority voting of five completions. GPT-4 also effectively predicts appraisal ratings and emotion labels using a single prompt, but adding instruction complexity results in poorer performance. We also found that longer event descriptions lead to more accurate annotations for both model and human annotator ratings. This work contributes to the growing usage of LLMs in psychology and the strategies for improving GPT-4 performance in annotating appraisals.

Paper Structure

This paper contains 22 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Research Questions
  3. Q1: Is GPT-4 reliable in generating emotion appraisal ratings?
  4. Q2: How accurate are the GPT-4 appraisal ratings compared to the ratings given by the human reader-annotators?
  5. Q3: Can majority voting with or without a confidence tiebreaker improve GPT-4 accuracy?
  6. Q4: Does adding an emotion prediction task to the prompt impact appraisal ratings accuracy?
  7. Q5: How does the length of the event description affect the accuracy of the appraisal ratings for both the GPT-4 and the human reader-annotators?
  8. Method
  9. Data
  10. Model Setup
  11. Prompt
  12. Context:
  13. Instruction:
  14. Description:
  15. Results
  16. ...and 7 more sections

Figures (1)

  • Figure 1: Average RMSE of texts with different lengths. The x-axis labels show the end of the bin in characters: the first bin contains texts with length up to 100 characters, the second between 100 and 200 characters, etc. The secondary y-axis plots the number of texts in each bin, except for the first bin, where the number of texts was ca 800 and thus did not fit to the plot.