Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Advancing Annotation of Stance in Social Media Posts: A Comparative Analysis of Large Language Models and Crowd Sourcing

Mao Li, Frederick Conrad

TL;DR

This paper evaluates stance annotation in social media by benchmarking eight LLMs and crowd-sourced judgments on citizenship-related tweets from the 2020 Census. It introduces a System 1 System 2 explicitness framework and uses human judgment variability as a proxy for text explicitness, analyzing agreement via logistic regression and bootstrapped F1 across corpora. The findings show LLMs perform best on explicit, System 1 style posts and struggle with implicit, System 2 inferences, with Few-Shot prompting helping but not eliminating gaps. The work advocates a hybrid approach combining human expertise with scalable LLM predictions and offers practical guidance on prompts and corpus-aware annotation to improve robustness and scalability of stance detection.

Abstract

In the rapidly evolving landscape of Natural Language Processing (NLP), the use of Large Language Models (LLMs) for automated text annotation in social media posts has garnered significant interest. Despite the impressive innovations in developing LLMs like ChatGPT, their efficacy, and accuracy as annotation tools are not well understood. In this paper, we analyze the performance of eight open-source and proprietary LLMs for annotating the stance expressed in social media posts, benchmarking their performance against human annotators' (i.e., crowd-sourced) judgments. Additionally, we investigate the conditions under which LLMs are likely to disagree with human judgment. A significant finding of our study is that the explicitness of text expressing a stance plays a critical role in how faithfully LLMs' stance judgments match humans'. We argue that LLMs perform well when human annotators do, and when LLMs fail, it often corresponds to situations in which human annotators struggle to reach an agreement. We conclude with recommendations for a comprehensive approach that combines the precision of human expertise with the scalability of LLM predictions. This study highlights the importance of improving the accuracy and comprehensiveness of automated stance detection, aiming to advance these technologies for more efficient and unbiased analysis of social media.

Advancing Annotation of Stance in Social Media Posts: A Comparative Analysis of Large Language Models and Crowd Sourcing

TL;DR

This paper evaluates stance annotation in social media by benchmarking eight LLMs and crowd-sourced judgments on citizenship-related tweets from the 2020 Census. It introduces a System 1 System 2 explicitness framework and uses human judgment variability as a proxy for text explicitness, analyzing agreement via logistic regression and bootstrapped F1 across corpora. The findings show LLMs perform best on explicit, System 1 style posts and struggle with implicit, System 2 inferences, with Few-Shot prompting helping but not eliminating gaps. The work advocates a hybrid approach combining human expertise with scalable LLM predictions and offers practical guidance on prompts and corpus-aware annotation to improve robustness and scalability of stance detection.

Abstract

In the rapidly evolving landscape of Natural Language Processing (NLP), the use of Large Language Models (LLMs) for automated text annotation in social media posts has garnered significant interest. Despite the impressive innovations in developing LLMs like ChatGPT, their efficacy, and accuracy as annotation tools are not well understood. In this paper, we analyze the performance of eight open-source and proprietary LLMs for annotating the stance expressed in social media posts, benchmarking their performance against human annotators' (i.e., crowd-sourced) judgments. Additionally, we investigate the conditions under which LLMs are likely to disagree with human judgment. A significant finding of our study is that the explicitness of text expressing a stance plays a critical role in how faithfully LLMs' stance judgments match humans'. We argue that LLMs perform well when human annotators do, and when LLMs fail, it often corresponds to situations in which human annotators struggle to reach an agreement. We conclude with recommendations for a comprehensive approach that combines the precision of human expertise with the scalability of LLM predictions. This study highlights the importance of improving the accuracy and comprehensiveness of automated stance detection, aiming to advance these technologies for more efficient and unbiased analysis of social media.
Paper Structure (21 sections, 1 equation, 3 figures, 3 tables)

This paper contains 21 sections, 1 equation, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Data Sets
  4. Selected Models
  5. Hardware
  6. Prompt
  7. Statistical Analysis
  8. Results
  9. Performance of LLMs
  10. H1. Agreement Between LLM and Human Annotations in System 1 vs. System 2 Texts
  11. H2. Influence of Topic and Opinion on Text Categorization as System 1 or 2
  12. H3. LLM Performance Variations in Stance Detection Across Different Corpora
  13. Conclusion and Discussion
  14. Limitations
  15. Appendix
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Distributions of mean and standard deviation of human judgments
  • Figure 2: Box Plot and Pairwise T-Test for different scenarios
  • Figure 3: Bootstrapped F1 scores