Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Labelling with Open-source LLMs using Dynamic Label Schema Integration

Thomas Walshe, Sae Young Moon, Chunyang Xiao, Yawwani Gunawardana, Fran Silavong

TL;DR

The paper addresses the high cost and privacy risks of creating labeled data by leveraging open-source LLMs for zero-shot labeling. It introduces Retrieval Augmented Classification (RAC), a method that ranks label descriptions by semantic relatedness and performs iterative binary inferences over the top labels, with early stopping and abstention to balance quality and coverage. Empirical results on public datasets show that incorporating label descriptions improves labeling quality, and truncated RAC offers a practical trade-off between accuracy and data coverage, while latency remains manageable. On an internal 61-category dataset, RAC demonstrates substantial labeling capability, though achieving enterprise-grade quality may require stronger models or few-shot enhancement. Overall, RAC provides a privacy-conscious, scalable framework for automatic labeling using in-house open-source LLMs, with clear pathways to further improvements.

Abstract

Acquiring labelled training data remains a costly task in real world machine learning projects to meet quantity and quality requirements. Recently Large Language Models (LLMs), notably GPT-4, have shown great promises in labelling data with high accuracy. However, privacy and cost concerns prevent the ubiquitous use of GPT-4. In this work, we explore effectively leveraging open-source models for automatic labelling. We identify integrating label schema as a promising technology but found that naively using the label description for classification leads to poor performance on high cardinality tasks. To address this, we propose Retrieval Augmented Classification (RAC) for which LLM performs inferences for one label at a time using corresponding label schema; we start with the most related label and iterates until a label is chosen by the LLM. We show that our method, which dynamically integrates label description, leads to performance improvements in labelling tasks. We further show that by focusing only on the most promising labels, RAC can trade off between label quality and coverage - a property we leverage to automatically label our internal datasets.

Automatic Labelling with Open-source LLMs using Dynamic Label Schema Integration

TL;DR

The paper addresses the high cost and privacy risks of creating labeled data by leveraging open-source LLMs for zero-shot labeling. It introduces Retrieval Augmented Classification (RAC), a method that ranks label descriptions by semantic relatedness and performs iterative binary inferences over the top labels, with early stopping and abstention to balance quality and coverage. Empirical results on public datasets show that incorporating label descriptions improves labeling quality, and truncated RAC offers a practical trade-off between accuracy and data coverage, while latency remains manageable. On an internal 61-category dataset, RAC demonstrates substantial labeling capability, though achieving enterprise-grade quality may require stronger models or few-shot enhancement. Overall, RAC provides a privacy-conscious, scalable framework for automatic labeling using in-house open-source LLMs, with clear pathways to further improvements.

Abstract

Acquiring labelled training data remains a costly task in real world machine learning projects to meet quantity and quality requirements. Recently Large Language Models (LLMs), notably GPT-4, have shown great promises in labelling data with high accuracy. However, privacy and cost concerns prevent the ubiquitous use of GPT-4. In this work, we explore effectively leveraging open-source models for automatic labelling. We identify integrating label schema as a promising technology but found that naively using the label description for classification leads to poor performance on high cardinality tasks. To address this, we propose Retrieval Augmented Classification (RAC) for which LLM performs inferences for one label at a time using corresponding label schema; we start with the most related label and iterates until a label is chosen by the LLM. We show that our method, which dynamically integrates label description, leads to performance improvements in labelling tasks. We further show that by focusing only on the most promising labels, RAC can trade off between label quality and coverage - a property we leverage to automatically label our internal datasets.
Paper Structure (18 sections, 5 figures, 8 tables)

This paper contains 18 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Preliminary Studies
  3. Leveraging label description for classification
  4. Experiment settings
  5. LLM-based binary classification results
  6. Retrieval Augmented Classification
  7. Results on public datasets
  8. Datasets
  9. Full and Truncated RAC
  10. Analysis
  11. Latency investigation
  12. Label Distillation
  13. Results on internal datasets
  14. Related Work
  15. Conclusion
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Retrieval Augmented Classification (RAC) during inference: the (bottom) retrieval part retrieved most relevant category and its corresponding description; (top) LLM classification leverages these information and performs binary classification iteratively.
  • Figure 2: Prompt used to generate a category label using the text from samples belonging to the same class. Prompt design follows shah2023using. Jinja2 templating is used to insert data into the prompt.
  • Figure 3: Prompt used to generate a category description using the text from samples belonging to the same class. Prompt design follows shah2023using. Jinja2 templating is used to insert data into the prompt.
  • Figure 4: Prompt used to classify a sample against a category described by the label and description using No-COT configuration. Jinja2 templating is used to insert data into the prompt.
  • Figure 5: Prompt used to classify a sample against a category described by the label and description using COT configuration. We encourage a rationale to be produced by including "explain your answer step-by-step". Jinja2 templating is used to insert data into the prompt.