Scaling Public Health Text Annotation: Zero-Shot Learning vs. Crowdsourcing for Improved Efficiency and Labeling Accuracy

TL;DR

The paper addresses scalable labeling of public health text by comparing zero-shot LLM labeling using GPT-4 Turbo to crowd-sourced annotation on 12,000 tweets about physical activity, sedentary behavior, and sleep problems, formalizing the task with $\mathcal{D}=\{(x_i,y_i)\}$ and evaluating $\alpha$, $\tau$, and $\kappa$ against expert gold labels $y_i^{(\mathrm{EXP})}$. It introduces a multi-stage workflow with prompt engineering to generate $y_i^{(\mathrm{LLM})}$ and compares it to $y_i^{(\mathrm{AMT})}$, reporting accuracy, time, and cost across topics. Key findings show LLMs can rival or exceed crowdworkers on straightforward signals but struggle with nuanced, domain-specific cues (e.g., sleep medications); cost and speed advantages favor LLM-based labeling for large-scale tasks. The work supports a hybrid annotation strategy that leverages automated speed for clear cases while preserving human oversight for ambiguous signals, enabling scalable public health surveillance with maintained label quality.

Abstract

Public health researchers are increasingly interested in using social media data to study health-related behaviors, but manually labeling this data can be labor-intensive and costly. This study explores whether zero-shot labeling using large language models (LLMs) can match or surpass conventional crowd-sourced annotation for Twitter posts related to sleep disorders, physical activity, and sedentary behavior. Multiple annotation pipelines were designed to compare labels produced by domain experts, crowd workers, and LLM-driven approaches under varied prompt-engineering strategies. Our findings indicate that LLMs can rival human performance in straightforward classification tasks and significantly reduce labeling time, yet their accuracy diminishes for tasks requiring more nuanced domain knowledge. These results clarify the trade-offs between automated scalability and human expertise, demonstrating conditions under which LLM-based labeling can be efficiently integrated into public health research without undermining label quality.

Figures (1)

• Figure 1: Overview of the LLM-based tweet annotation pipeline implemented in this paper. The dataset $X$ is partitioned into smaller chunks $x_{i,k}$ to fit GPT-4's context window. Annotation guidelines are extracted from the instruction PDF and structured into a prompt $P = g(D)$. GPT-4 generates tweet labels $y_i^{(\mathrm{LLM})}$, which undergo validation and correction in the output check stage. The final structured labels are evaluated by comparing $y^{(\mathrm{LLM})}$ with expert ($y^{(\mathrm{EXP})}$) and AMT-generated ($y^{(\mathrm{AMT})}$) labels using accuracy ($\alpha$), time ($\tau$), and cost ($\kappa$).