Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Data

Poli Apollinaire Nemkova, Solomon Ubani, Mark V. Albert

TL;DR

The paper benchmarks five LLMs (GPT-4, GPT-3.5, Claude-2, LLaMA-3, Mistral-7B) on detecting references to human rights violations in multilingual social media posts (Russian/Ukrainian). It compares zero-shot and few-shot prompting in English and Russian against a gold-standard, double-annotated 1000-post dataset, revealing GPT-4's strongest overall performance and notable gains from language-aligned prompts; open-source models show potential under few-shot and with language-specific prompts. The study identifies error patterns related to ambiguity and indirect references and highlights practical guidance for deploying LLMs in high-stakes multilingual domains, including cost/robustness trade-offs. It also points to future directions in improving multilingual capabilities of smaller models and refining prompting strategies.

Abstract

In the era of increasingly sophisticated natural language processing (NLP) systems, large language models (LLMs) have demonstrated remarkable potential for diverse applications, including tasks requiring nuanced textual understanding and contextual reasoning. This study investigates the capabilities of multiple state-of-the-art LLMs - GPT-3.5, GPT-4, LLAMA3, Mistral 7B, and Claude-2 - for zero-shot and few-shot annotation of a complex textual dataset comprising social media posts in Russian and Ukrainian. Specifically, the focus is on the binary classification task of identifying references to human rights violations within the dataset. To evaluate the effectiveness of these models, their annotations are compared against a gold standard set of human double-annotated labels across 1000 samples. The analysis includes assessing annotation performance under different prompting conditions, with prompts provided in both English and Russian. Additionally, the study explores the unique patterns of errors and disagreements exhibited by each model, offering insights into their strengths, limitations, and cross-linguistic adaptability. By juxtaposing LLM outputs with human annotations, this research contributes to understanding the reliability and applicability of LLMs for sensitive, domain-specific tasks in multilingual contexts. It also sheds light on how language models handle inherently subjective and context-dependent judgments, a critical consideration for their deployment in real-world scenarios.

Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Data

TL;DR

The paper benchmarks five LLMs (GPT-4, GPT-3.5, Claude-2, LLaMA-3, Mistral-7B) on detecting references to human rights violations in multilingual social media posts (Russian/Ukrainian). It compares zero-shot and few-shot prompting in English and Russian against a gold-standard, double-annotated 1000-post dataset, revealing GPT-4's strongest overall performance and notable gains from language-aligned prompts; open-source models show potential under few-shot and with language-specific prompts. The study identifies error patterns related to ambiguity and indirect references and highlights practical guidance for deploying LLMs in high-stakes multilingual domains, including cost/robustness trade-offs. It also points to future directions in improving multilingual capabilities of smaller models and refining prompting strategies.

Abstract

In the era of increasingly sophisticated natural language processing (NLP) systems, large language models (LLMs) have demonstrated remarkable potential for diverse applications, including tasks requiring nuanced textual understanding and contextual reasoning. This study investigates the capabilities of multiple state-of-the-art LLMs - GPT-3.5, GPT-4, LLAMA3, Mistral 7B, and Claude-2 - for zero-shot and few-shot annotation of a complex textual dataset comprising social media posts in Russian and Ukrainian. Specifically, the focus is on the binary classification task of identifying references to human rights violations within the dataset. To evaluate the effectiveness of these models, their annotations are compared against a gold standard set of human double-annotated labels across 1000 samples. The analysis includes assessing annotation performance under different prompting conditions, with prompts provided in both English and Russian. Additionally, the study explores the unique patterns of errors and disagreements exhibited by each model, offering insights into their strengths, limitations, and cross-linguistic adaptability. By juxtaposing LLM outputs with human annotations, this research contributes to understanding the reliability and applicability of LLMs for sensitive, domain-specific tasks in multilingual contexts. It also sheds light on how language models handle inherently subjective and context-dependent judgments, a critical consideration for their deployment in real-world scenarios.
Paper Structure (10 sections, 2 figures)

This paper contains 10 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Experiment Design
  4. Dataset
  5. Method
  6. Results
  7. Error Analysis and Discussion
  8. Model Performance Overview
  9. Limitations
  10. Conclusion

Figures (2)

  • Figure 1: Performance metrics for GPT 3.5, GPT 4.0, LLaMA-3, Mistral-7B, and Claude-2 in Few-Shot and Zero-Shot Prompting scenarios using English and Russian prompts. Bold values indicate the highest performance for each column across all rows.
  • Figure 2: Performance metrics for LLaMA-3 and GPT-4.0 on the agreement and disagreement sets in Few-Shot and Zero-Shot Prompting scenarios using English and Russian prompts. Bold values indicate the highest performance for each column across all rows.