Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy F. Chen, Zhengyuan Liu, Diyi Yang

TL;DR

<3-5 sentence high-level summary>CoAnnotating addresses the cost-quality tension in data annotation by enabling uncertainty-guided collaboration between humans and LLMs. It introduces instance-level uncertainty measures (self-evaluation and entropy) and a diversity-promoting prompt set to estimate LLM annotating expertise, followed by three allocation strategies and Pareto-front-based strategy selection. Across six NLP tasks, the approach shows improved cost efficiency and notes where outsourcing is advantageous or detrimental, with insights into prompt design and LLM reliability. The framework provides a practical guide for scalable, cost-conscious dataset construction in real-world NLP pipelines.

Abstract

Annotated data plays a critical role in Natural Language Processing (NLP) in training models and evaluating their performance. Given recent developments in Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot capability on many text-annotation tasks, comparable with or even exceeding human annotators. Such LLMs can serve as alternatives for manual annotation, due to lower costs and higher scalability. However, limited work has leveraged LLMs as complementary annotators, nor explored how annotation work is best allocated among humans and LLMs to achieve both quality and cost objectives. We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Under this framework, we utilize uncertainty to estimate LLMs' annotation capability. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline. For code implementation, see https://github.com/SALT-NLP/CoAnnotating.

CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

TL;DR

<3-5 sentence high-level summary>CoAnnotating addresses the cost-quality tension in data annotation by enabling uncertainty-guided collaboration between humans and LLMs. It introduces instance-level uncertainty measures (self-evaluation and entropy) and a diversity-promoting prompt set to estimate LLM annotating expertise, followed by three allocation strategies and Pareto-front-based strategy selection. Across six NLP tasks, the approach shows improved cost efficiency and notes where outsourcing is advantageous or detrimental, with insights into prompt design and LLM reliability. The framework provides a practical guide for scalable, cost-conscious dataset construction in real-world NLP pipelines.

Abstract

Annotated data plays a critical role in Natural Language Processing (NLP) in training models and evaluating their performance. Given recent developments in Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot capability on many text-annotation tasks, comparable with or even exceeding human annotators. Such LLMs can serve as alternatives for manual annotation, due to lower costs and higher scalability. However, limited work has leveraged LLMs as complementary annotators, nor explored how annotation work is best allocated among humans and LLMs to achieve both quality and cost objectives. We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Under this framework, we utilize uncertainty to estimate LLMs' annotation capability. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline. For code implementation, see https://github.com/SALT-NLP/CoAnnotating.
Paper Structure (29 sections, 2 equations, 5 figures, 3 tables)

This paper contains 29 sections, 2 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Weak Supervision
  4. LLMs for Annotation
  5. Human-Machine Collaboration for Dataset Creation
  6. CoAnnotating Framework
  7. Prompt Construction
  8. Uncertainty Computation
  9. Self-Evaluation.
  10. Entropy.
  11. Work Allocation Strategies
  12. Random Allocation.
  13. Self-Evaluation Guided Allocation.
  14. Entropy Guided Allocation.
  15. Strategy Selection
  16. ...and 14 more sections

Figures (5)

  • Figure 1: CoAnnotating framework. It differs from previous work by considering how to allocate data within the same dataset to humans and ChatGPT by obtaining responses from ChatGPT using different variations of prompts and estimating ChatGPT's annotation expertise with the use of uncertainty metrics such as entropy.
  • Figure 2: Workflow of CoAnnotating. The framework consists of uncertainty-guided expertise estimation, work allocation, and cost performance Pareto analysis. With insights gained from Pareto analysis on the pilot dataset, uncertainty-guided work allocation can be applied on the original unlabeled dataset to achieve greater cost efficiency.
  • Figure 3: Distribution of entropy and confidence values.
  • Figure 4: Scatter plots of the average alignment of ChatGPT's annotation with human annotation for train data against the threshold. We vary the threshold for different metrics during work allocation to investigate the effectiveness of different metrics in quantifying ChatGPT's annotation capability.
  • Figure 5: Pareto curves under different allocation strategies (random, entropy guided, self-evaluation guided). The Pareto frontier is highlighted, illustrating the optimal choices that are Pareto efficient.