Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost

Parikshit Bansal, Amit Sharma

TL;DR

The paper tackles NLP generalization under domain shifts with limited labeled data by introducing EAGLE, a method that uses LLMs to annotate strategically chosen inputs. It introduces Conditional Informativeness, a deviation-based criterion comparing base and finetuned model scores to guide which unlabeled inputs should be annotated within a fixed budget. Empirical results on semantic similarity and semantic search show that CI-driven sampling consistently outperforms traditional uncertainty-based sampling and even surpasses annotating the full target domain in some cases, improving both in-domain and target-domain accuracy. The work demonstrates a practical workflow for cost-effective, LLM-assisted domain adaptation, with broad implications for deploying robust NLP systems under data constraints.

Abstract

State-of-the-art supervised NLP models achieve high accuracy but are also susceptible to failures on inputs from low-data regimes, such as domains that are not represented in training data. As an approximation to collecting ground-truth labels for the specific domain, we study the use of large language models (LLMs) for annotating inputs and improving the generalization of NLP models. Specifically, given a budget for LLM annotations, we present an algorithm for sampling the most informative inputs to annotate and retrain the NLP model. We find that popular active learning strategies such as uncertainty-based sampling do not work well. Instead, we propose a sampling strategy based on the difference in prediction scores between the base model and the finetuned NLP model, utilizing the fact that most NLP models are finetuned from a base model. Experiments with classification (semantic similarity) and ranking (semantic search) tasks show that our sampling strategy leads to significant gains in accuracy for both the training and target domains.

Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost

TL;DR

The paper tackles NLP generalization under domain shifts with limited labeled data by introducing EAGLE, a method that uses LLMs to annotate strategically chosen inputs. It introduces Conditional Informativeness, a deviation-based criterion comparing base and finetuned model scores to guide which unlabeled inputs should be annotated within a fixed budget. Empirical results on semantic similarity and semantic search show that CI-driven sampling consistently outperforms traditional uncertainty-based sampling and even surpasses annotating the full target domain in some cases, improving both in-domain and target-domain accuracy. The work demonstrates a practical workflow for cost-effective, LLM-assisted domain adaptation, with broad implications for deploying robust NLP systems under data constraints.

Abstract

State-of-the-art supervised NLP models achieve high accuracy but are also susceptible to failures on inputs from low-data regimes, such as domains that are not represented in training data. As an approximation to collecting ground-truth labels for the specific domain, we study the use of large language models (LLMs) for annotating inputs and improving the generalization of NLP models. Specifically, given a budget for LLM annotations, we present an algorithm for sampling the most informative inputs to annotate and retrain the NLP model. We find that popular active learning strategies such as uncertainty-based sampling do not work well. Instead, we propose a sampling strategy based on the difference in prediction scores between the base model and the finetuned NLP model, utilizing the fact that most NLP models are finetuned from a base model. Experiments with classification (semantic similarity) and ranking (semantic search) tasks show that our sampling strategy leads to significant gains in accuracy for both the training and target domains.
Paper Structure (49 sections, 15 equations, 3 figures, 11 tables)

This paper contains 49 sections, 15 equations, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs for data augmentation.
  4. Semantic similarity with limited labeled data.
  5. Active Learning.
  6. Conditional informativeness criterion for sampling LLM annotations
  7. Background: Building NLP classifiers using base models
  8. Semantic Similarity.
  9. Finetuning on Base model.
  10. A domain adaptation case study: Which inputs to annotate?
  11. AUC under different sampling strategies.
  12. Implications.
  13. Conditional Informativeness metric
  14. EAGLE: Enhanced Generalization using LLM Annotations
  15. The EAGLE Algorithm
  16. ...and 34 more sections

Figures (3)

  • Figure 1: Enhanced Generalization using LLM Annotations. Illustration of our algorithm using the duplicate question detection task. We propose a sampling strategy based on deviation of an NLP model's similarity score from the base model, called (base model)-conditional informativeness. Inputs are sampled using this strategy (Step 2), annotated using an LLM (Step 3) and then added to the training set of the NLP model. Our sampling strategy performs significantly better than random or active learning-based strategies.
  • Figure 2: Gain in AUC on including LLM-annotated and GT-based augmentations on Quora dataset. The orange line is gain in AUC with uncertainty based sampling (shaded region shows std err.). We divide the unlabeled data into 20 quantiles, based on the Conditional Informativeness metric. Conditional Informativeness increases from left to right. For LLMs, uncertainty is not a good method for sampling and Conditional Informativeness based sampling is better, while for GT-based augmentations uncertainty based sampling is better.
  • Figure 3: LLM vs Finetuned accuracy across quantiles