Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DEFT: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection

Devleena Das, Vivek Khetan

TL;DR

The paper tackles data efficiency in fine-tuning pre-trained language models for text-editing by proposing DEFT-UCS, which uses unsupervised core-set selection to form a compact yet representative training subset $D_c$ from a larger dataset $D$. It builds a pipeline where a base set $D_{base}$ is augmented with $D_c$ derived from clustering embeddings (notably via Sentence-T5) and distance-based sampling, to yield $M_{DEFT-UCS}$ that matches or exceeds the performance of the state-of-the-art CoEDIT while using only a fraction of the data. Across eight evaluation datasets covering six edit tasks, DEFT-UCS with as little as $32.5\%$ of $D_{CoEDIT}$ achieves competitive SARI and ROUGE-L scores and gains in human-perceived edit quality. The work demonstrates the practicality of unsupervised core-set-based data pruning for NLP tasks, discusses limitations of manually chosen hyperparameters, and points to future work on automatic parameter selection and broader task applicability.

Abstract

Recent advances have led to the availability of many pre-trained language models (PLMs); however, a question that remains is how much data is truly needed to fine-tune PLMs for downstream tasks? In this work, we introduce DEFT-UCS, a data-efficient fine-tuning framework that leverages unsupervised core-set selection to identify a smaller, representative dataset that reduces the amount of data needed to fine-tune PLMs for downstream tasks. We examine the efficacy of DEFT-UCS in the context of text-editing LMs, and compare to the state-of-the art text-editing model, CoEDIT. Our results demonstrate that DEFT-UCS models are just as accurate as CoEDIT, across eight different datasets consisting of six different editing tasks, while finetuned on 70% less data.

DEFT: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection

TL;DR

The paper tackles data efficiency in fine-tuning pre-trained language models for text-editing by proposing DEFT-UCS, which uses unsupervised core-set selection to form a compact yet representative training subset from a larger dataset . It builds a pipeline where a base set is augmented with derived from clustering embeddings (notably via Sentence-T5) and distance-based sampling, to yield that matches or exceeds the performance of the state-of-the-art CoEDIT while using only a fraction of the data. Across eight evaluation datasets covering six edit tasks, DEFT-UCS with as little as of achieves competitive SARI and ROUGE-L scores and gains in human-perceived edit quality. The work demonstrates the practicality of unsupervised core-set-based data pruning for NLP tasks, discusses limitations of manually chosen hyperparameters, and points to future work on automatic parameter selection and broader task applicability.

Abstract

Recent advances have led to the availability of many pre-trained language models (PLMs); however, a question that remains is how much data is truly needed to fine-tune PLMs for downstream tasks? In this work, we introduce DEFT-UCS, a data-efficient fine-tuning framework that leverages unsupervised core-set selection to identify a smaller, representative dataset that reduces the amount of data needed to fine-tune PLMs for downstream tasks. We examine the efficacy of DEFT-UCS in the context of text-editing LMs, and compare to the state-of-the art text-editing model, CoEDIT. Our results demonstrate that DEFT-UCS models are just as accurate as CoEDIT, across eight different datasets consisting of six different editing tasks, while finetuned on 70% less data.
Paper Structure (41 sections, 7 figures, 6 tables, 1 algorithm)

This paper contains 41 sections, 7 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Efficient Fine-Tuning of LLMs
  4. Core-Set Selection & Dataset Distillation
  5. Instruction Tuning for Text-Editing
  6. Problem Formulation
  7. DEFT-UCS Framework
  8. Clustering in UCS
  9. Sampling $\boldsymbol{D_{c}}$ in UCS
  10. DEFT-UCS Applied to Text-Editing
  11. $\boldsymbol{D_{Base}}$ in CoEDIT
  12. DEFT-UCS Hyperparameters
  13. Dataset Embedding
  14. Model Fine-Tuning
  15. Experiments
  16. ...and 26 more sections

Figures (7)

  • Figure 1: Our DEFT-UCS framework utilizes unsupervised core-set selection (UCS) to find a core-set of data $D_{c}$, as well as initial seed data, $D_{base}$ to produce a fine-tuned PLM, $M_{DEFT-UCS}$.
  • Figure 2: Comparisons between the CoEDIT model raheja2023coedit, LIMA-inspired model $M_{LIMA}$zhou2023lima, and our DEFT-UCS models with respect to SARI (a) and ROUGE-L (b) scores.
  • Figure 3: Utilizing hard sampling in UCS results in a best, overall DEFT-UCS model that requires only 32.5% of $D_{CoEDIT}$ to beat 6/8 evaluation datasets considering SARI (a) and ROUGE-L (b) scores.
  • Figure 4: With less $D_{base}$, leveraging hard sampling in our DEFT-UCS leads to better performing models (winning %); as $D_{base}$ increases, random sampling leads to better performing models.
  • Figure 5: Comparing the distribution of task-related data among clusters after performing K-Means when utilizing Sentence-T5 embedding (a), BART CLS embeddings (b) and averaged Flan-T5 word embeddings (c) for sentence representations.
  • ...and 2 more figures