Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Similarity-Based Domain Adaptation with LLMs

Jie He, Wendi Zhou, Xiang Lorraine Li, Jeff Z. Pan

TL;DR

This work tackles unsupervised domain adaptation by avoiding source-domain retraining and instead annotating target-domain data with an LLM using kNN-based augmentation. It then distills knowledge to smaller language models via two losses: a label-distribution alignment loss and a similarity-alignment loss that transfers representation structure, achieving state-of-the-art-like gains on cross-domain sentiment tasks. The approach demonstrates strong empirical gains, notably a 2.44% accuracy improvement over the previous SOTA across eight task setups, and highlights the importance of both target-label quality and representation-level supervision. This offers a practical pathway to deploy effective cross-domain NLP systems in resource-constrained settings, leveraging LLMs for data annotation while maintaining efficient small-model inference.

Abstract

Unsupervised domain adaptation leverages abundant labeled data from various source domains to generalize onto unlabeled target data. Prior research has primarily focused on learning domain-invariant features across the source and target domains. However, these methods often require training a model using source domain data, which is time-consuming and can limit model usage for applications with different source data. This paper introduces a simple framework that utilizes the impressive generalization capabilities of Large Language Models (LLMs) for target data annotation without the need of source model training, followed by a novel similarity-based knowledge distillation loss. Our extensive experiments on cross-domain text classification reveal that our framework achieves impressive performance, specifically, 2.44\% accuracy improvement when compared to the SOTA method.

Similarity-Based Domain Adaptation with LLMs

TL;DR

This work tackles unsupervised domain adaptation by avoiding source-domain retraining and instead annotating target-domain data with an LLM using kNN-based augmentation. It then distills knowledge to smaller language models via two losses: a label-distribution alignment loss and a similarity-alignment loss that transfers representation structure, achieving state-of-the-art-like gains on cross-domain sentiment tasks. The approach demonstrates strong empirical gains, notably a 2.44% accuracy improvement over the previous SOTA across eight task setups, and highlights the importance of both target-label quality and representation-level supervision. This offers a practical pathway to deploy effective cross-domain NLP systems in resource-constrained settings, leveraging LLMs for data annotation while maintaining efficient small-model inference.

Abstract

Unsupervised domain adaptation leverages abundant labeled data from various source domains to generalize onto unlabeled target data. Prior research has primarily focused on learning domain-invariant features across the source and target domains. However, these methods often require training a model using source domain data, which is time-consuming and can limit model usage for applications with different source data. This paper introduces a simple framework that utilizes the impressive generalization capabilities of Large Language Models (LLMs) for target data annotation without the need of source model training, followed by a novel similarity-based knowledge distillation loss. Our extensive experiments on cross-domain text classification reveal that our framework achieves impressive performance, specifically, 2.44\% accuracy improvement when compared to the SOTA method.

Paper Structure

This paper contains 15 sections, 5 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. $k$NN-based Annotation
  5. Distillation Objectives
  6. Experiment
  7. Baselines
  8. Results
  9. Ablation Studies
  10. Case Study
  11. Conclusion
  12. Experimental Configuration
  13. Models and parameter setting
  14. Dataset
  15. Analysis: Impact of loss type

Figures (1)

  • Figure 1: Proposed framework Overview. LLM is used to construct the source data store, as shown in the upper left corner. Given target data, we first obtain LLM annotations via $k$NN augmented comparison to the data store, then fine-tune an SLM with two training losses: classification consistency loss $\mathcal{L}_1$ and similarity consistency loss $\mathcal{L}_2$.