If in a Crowdsourced Data Annotation Pipeline, a GPT-4

TL;DR

The paper investigates GPT-4’s labeling performance against a rigorously conducted MTurk crowdsourcing pipeline using the CODA-19 scheme on 3,177 sentence segments from 200 papers. It demonstrates that GPT-4 typically outperforms the best MTurk pipeline (83.6% vs 81.5%), and shows that integrating GPT-4 with crowd labels via advanced aggregation can reach up to 87.5% accuracy. The study also reveals that crowd and GPT-4 strengths can be complementary, particularly when crowds excel in the Finding/Contribution class, enabling improvements through aggregation. Methodologically, it combines two interfaces, expert gold standards, multiple label-cleaning strategies, and eight aggregation algorithms to provide a robust, real-world evaluation of crowdsourcing in the era of large language models. The findings inform best practices for hybrid human-AI labeling pipelines and highlight the potential for more efficient, high-quality annotations with targeted expert input and sophisticated aggregation.

Abstract

Recent studies indicated GPT-4 outperforms online crowd workers in data labeling accuracy, notably workers from Amazon Mechanical Turk (MTurk). However, these studies were criticized for deviating from standard crowdsourcing practices and emphasizing individual workers' performances over the whole data-annotation process. This paper compared GPT-4 and an ethical and well-executed MTurk pipeline, with 415 workers labeling 3,177 sentence segments from 200 scholarly articles using the CODA-19 scheme. Two worker interfaces yielded 127,080 labels, which were then used to infer the final labels through eight label-aggregation algorithms. Our evaluation showed that despite best practices, MTurk pipeline's highest accuracy was 81.5%, whereas GPT-4 achieved 83.6%. Interestingly, when combining GPT-4's labels with crowd labels collected via an advanced worker interface for aggregation, 2 out of the 8 algorithms achieved an even higher accuracy (87.5%, 87.0%). Further analysis suggested that, when the crowd's and GPT-4's labeling strengths are complementary, aggregating them could increase labeling accuracy.

Figures (10)

• Figure 1: The basic worker interface, individually designed by one of the authors, has a focus on prioritizing task simplicity and user-friendliness.
• Figure 2: The advanced worker interface, adopted from CODA-19 huang-etal-2020-coda, incorporates advanced features such as a visual feedback button, color-coded annotation view, and a time lock mechanism to deter hasty spam submissions.
• Figure 3: A density histogram of the response of all individual crowd workers' accuracy (filtered by the Exclude-By-Worker strategy.) The dash lines show majority vote accuracy and the best aggregation algorithm accuracy on both basic and advanced interface response, the accuracy of GPT-4 (t=0.2 and t=1.0), and the accuracy of CS Expert. The accuracy uses the Bio Expert as the gold standard.
• Figure 4: Aggregation Methods for All Workers, Exclude-By-Worker, and Exclude-By-Batch. Among the various models and strategies employed, only the combination of the One-Coin Dawid-Skene aggregation method and workers in the Advanced Interface group using the Exclude-By-Worker approach demonstrated performance that closely approached that of GPT-4. In contrast, the other models utilizing different strategies were unable to achieve GPT-4's performance level.
• Figure 5: Exclude-By-Worker simulation results applied to different aggregation models. One-Coin Dawid-Skene and MACE algorithms for a combination of advanced interface results had the best accuracy and outperformed GPT-4 at temperature 0.2 (see (d) (f)).