Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models for Data Annotation and Synthesis: A Survey

Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, Huan Liu

TL;DR

The survey tackles the use of large language models for data annotation and synthesis by presenting a structured taxonomy that spans annotation generation, assessment, and utilization. It details methods for constructing instruction-response data, labels, rationales, and feedback (pairwise and textual), as well as domain-specific data, while examining evaluation, filtering, and learning-with-synthetic-annotations strategies. It further discusses alignment, fine-tuning, and inference-time use, and addresses societal impacts, ethics, challenges, and efficiency considerations. The work aims to guide researchers and practitioners through practical techniques, benchmarking resources, and open questions to advance LLM-driven data annotation and synthesis.

Abstract

Data annotation and synthesis generally refers to the labeling or generating of raw data with relevant information, which could be used for improving the efficacy of machine learning models. The process, however, is labor-intensive and costly. The emergence of advanced Large Language Models (LLMs), exemplified by GPT-4, presents an unprecedented opportunity to automate the complicated process of data annotation and synthesis. While existing surveys have extensively covered LLM architecture, training, and general applications, we uniquely focus on their specific utility for data annotation. This survey contributes to three core aspects: LLM-Based Annotation Generation, LLM-Generated Annotations Assessment, and LLM-Generated Annotations Utilization. Furthermore, this survey includes an in-depth taxonomy of data types that LLMs can annotate, a comprehensive review of learning strategies for models utilizing LLM-generated annotations, and a detailed discussion of the primary challenges and limitations associated with using LLMs for data annotation and synthesis. Serving as a key guide, this survey aims to assist researchers and practitioners in exploring the potential of the latest LLMs for data annotation, thereby fostering future advancements in this critical field.

Large Language Models for Data Annotation and Synthesis: A Survey

TL;DR

The survey tackles the use of large language models for data annotation and synthesis by presenting a structured taxonomy that spans annotation generation, assessment, and utilization. It details methods for constructing instruction-response data, labels, rationales, and feedback (pairwise and textual), as well as domain-specific data, while examining evaluation, filtering, and learning-with-synthetic-annotations strategies. It further discusses alignment, fine-tuning, and inference-time use, and addresses societal impacts, ethics, challenges, and efficiency considerations. The work aims to guide researchers and practitioners through practical techniques, benchmarking resources, and open questions to advance LLM-driven data annotation and synthesis.

Abstract

Data annotation and synthesis generally refers to the labeling or generating of raw data with relevant information, which could be used for improving the efficacy of machine learning models. The process, however, is labor-intensive and costly. The emergence of advanced Large Language Models (LLMs), exemplified by GPT-4, presents an unprecedented opportunity to automate the complicated process of data annotation and synthesis. While existing surveys have extensively covered LLM architecture, training, and general applications, we uniquely focus on their specific utility for data annotation. This survey contributes to three core aspects: LLM-Based Annotation Generation, LLM-Generated Annotations Assessment, and LLM-Generated Annotations Utilization. Furthermore, this survey includes an in-depth taxonomy of data types that LLMs can annotate, a comprehensive review of learning strategies for models utilizing LLM-generated annotations, and a detailed discussion of the primary challenges and limitations associated with using LLMs for data annotation and synthesis. Serving as a key guide, this survey aims to assist researchers and practitioners in exploring the potential of the latest LLMs for data annotation, thereby fostering future advancements in this critical field.
Paper Structure (23 sections, 3 figures, 4 tables)

This paper contains 23 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. LLM-Based Annotation Generation
  4. Instruction & Response
  5. Label
  6. Rationale
  7. Pairwise Feedback
  8. Textual Feedback
  9. Other Domain-Specific Data
  10. LLM-Generated Annotations Assessment
  11. Evaluating LLM-Generated Annotations
  12. Filtering & Selection
  13. LLM-Generated Annotations Utilization
  14. Supervised Fine-Tuning
  15. Alignment Tuning
  16. ...and 8 more sections

Figures (3)

  • Figure 1: The proposed taxonomy of existing research on LLM for data annotation.
  • Figure 2: The examples for LLM-based annotation generation.
  • Figure 3: Stack AI dashboard. They provide a visual interface for users to design and track the AI workflow.