Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning

Petter Törnberg

TL;DR

The paper finds that ChatGPT-4 has achieves higher accuracy, higher reliability, and equal or lower bias than the human classifiers, and suggests that LLM will have substantial impact on the use of textual data in the social sciences, by enabling interpretive research at a scale.

Abstract

This paper assesses the accuracy, reliability and bias of the Large Language Model (LLM) ChatGPT-4 on the text analysis task of classifying the political affiliation of a Twitter poster based on the content of a tweet. The LLM is compared to manual annotation by both expert classifiers and crowd workers, generally considered the gold standard for such tasks. We use Twitter messages from United States politicians during the 2020 election, providing a ground truth against which to measure accuracy. The paper finds that ChatGPT-4 has achieves higher accuracy, higher reliability, and equal or lower bias than the human classifiers. The LLM is able to correctly annotate messages that require reasoning on the basis of contextual knowledge, and inferences around the author's intentions - traditionally seen as uniquely human abilities. These findings suggest that LLM will have substantial impact on the use of textual data in the social sciences, by enabling interpretive research at a scale.

ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning

TL;DR

The paper finds that ChatGPT-4 has achieves higher accuracy, higher reliability, and equal or lower bias than the human classifiers, and suggests that LLM will have substantial impact on the use of textual data in the social sciences, by enabling interpretive research at a scale.

Abstract

This paper assesses the accuracy, reliability and bias of the Large Language Model (LLM) ChatGPT-4 on the text analysis task of classifying the political affiliation of a Twitter poster based on the content of a tweet. The LLM is compared to manual annotation by both expert classifiers and crowd workers, generally considered the gold standard for such tasks. We use Twitter messages from United States politicians during the 2020 election, providing a ground truth against which to measure accuracy. The paper finds that ChatGPT-4 has achieves higher accuracy, higher reliability, and equal or lower bias than the human classifiers. The LLM is able to correctly annotate messages that require reasoning on the basis of contextual knowledge, and inferences around the author's intentions - traditionally seen as uniquely human abilities. These findings suggest that LLM will have substantial impact on the use of textual data in the social sciences, by enabling interpretive research at a scale.
Paper Structure (5 sections, 3 figures)

This paper contains 5 sections, 3 figures.

Table of Contents

  1. ChatGPT and the rise of Generative AI
  2. Method: Using ChatGPT to classify Twitter messages
  3. Results
  4. Discussion

Figures (3)

  • Figure 1: A normalized density histogram of the response accuracy by crowd worker, experts, and the LLM models. The dashes lines show the mean response accuracy for LLMs and experts. The combined MTurk line shows the accuracy of the majority response for each question, as this is how crowd-worker answers tend to be employed. As can be seen, the LLMs outperform all individual human classifiers. The mean accuracy lines for the two temperatures of the LLM nearly precisely overlap.
  • Figure 2: The Krippendorf's Alpha measure of intercoder reliability, with 95% bootstrap confidence interval. The LLM offers substantially higher levels of reliability than the human coders.
  • Figure 3: All coders are biased to guessing Democrat over Republican. The LLMs and experts are similar in the level of bias, while the MTurk classifiers have a significantly stronger bias.