Large language models struggle with ethnographic text annotation

TL;DR

This study critically assesses whether seven state-of-the-art LLMs can reliably annotate ethnographic texts for a large set of ritual features. Using zero-shot, multi-task, and ensemble prompting on two ethnographic datasets (morphospace and synchrony), the authors find that LLM performance remains far below the reliability threshold needed for automated annotation, with long, interpretive, and regionally variable texts posing major challenges. Human inter-coder reliability closely tracks LLM accuracy, suggesting that fundamental ambiguity in ethnographic coding constrains automation and that larger gains require improved alignment with interpretive reasoning or domain-specific coding schemes. The findings temper optimism about LLM-driven thick-to-thin data conversion in cross-cultural research and emphasize cautious, expert-guided use of automation for concrete, binary features, reserving interpretive tasks for human annotation.

Abstract

Large language models (LLMs) have shown promise for automated text annotation, raising hopes that they might accelerate cross-cultural research by extracting structured data from ethnographic texts. We evaluated 7 state-of-the-art LLMs on their ability to annotate 121 ritual features across 567 ethnographic excerpts. Performance was limited, falling well below levels required for reliable automated annotation. Longer texts, features requiring ordinal distinctions, and ambiguous constructs proved particularly difficult. Human inter-coder reliability set an approximate ceiling on LLM accuracy: features that human coders found difficult to agree upon were also difficult for LLMs. Yet even on features where humans reliably agreed, models fell short of human performance. Our findings suggest that LLMs cannot yet substitute for human expertise in ethnographic annotation.

Figures (7)

• Figure 1: Prompting strategy and ensembling improve LLM annotation performance across 115 morphospace features. Mean F1 score for each model under three conditions: baseline (zero-shot prompting), MTP (multi-task prompting), and MTP+Ensemble (majority voting across MTP runs). Error bars indicate 95% confidence intervals. MTP consistently outperforms baseline prompting, with ensembling providing additional gains for most models. Due to cost restrictions, only open-source models included ensembling.
• Figure 2: LLM performance across morphospace categories and synchrony features. (a) Heatmap showing mean F1 score for each model–condition combination (columns) across feature categories (rows). Categories are ordered by average performance. Duration, comprising a single feature, was evaluated under baseline conditions only as multi-turn prompting requires multi-feature iteration. Performance is highest for well-defined categorical features and lowest for subjective, context-dependent attributes and multi-option features. (b) Feature-level F1 scores for synchrony feature annotations across models and conditions. Features requiring identification of unambiguous coordinated actions (e.g., dancing and singing) show higher performance than those involving nuanced judgements about group coordination (e.g., generic synchronous movement). This pattern suggests that feature complexity partly drives classification difficulty.
• Figure 3: Human inter-coder reliability predicts LLM performance. (a) Cohen's $\kappa$ measuring agreement between two independent human coders for each synchrony feature. (b) Mean F1 score across all LLM models for the same features; error bars show the range across models. Colour indicates difficulty category based on human agreement: features that humans find difficult to code reliably (low $\kappa$) also prove challenging for LLMs, suggesting inherent ambiguity in the ethnographic texts and/or task rather than model-specific limitations.
• Figure 4: Feature-level F1 performance on all 115 ritual features. RitualDuration was a single-feature category and thus only the baseline condition could be evaluated. Both Claude Sonnet 4.5 and Perplexity Sonar refused to answer questions pertaining to sexual intercourse due to model constraints.
• Figure 5: LLM–LLM agreement heatmap. Mean Cohen's $\kappa$ is reported across all model pairs.