Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models

TL;DR

This paper addresses the challenge of deploying accurate NLP models in low-resource settings by combining fine-tuning of a large teacher LLM with synthetic data generation. The authors fine-tune a 20B GPT-NeoX as a teacher and generate augmented training data, either by annotating unlabeled instances or by producing new input–output pairs, to train compact student models (RoBERTa-Large for classification and BART-Large for generation). Across four text classification tasks and two NLG tasks, both annotation and generation consistently improve downstream performance, with larger gains when original data is scarce; a 125-example (5%) fine-tuning of the teacher can yield multi-point improvements. The study also shows NeoX-20B can outperform GPT-3.5 175B in some settings and that combining annotation and generation provides cost-effective gains. This approach offers a practical data-centric pathway to leverage large LLMs for training smaller, faster models.

Abstract

The in-context learning ability of large language models (LLMs) enables them to generalize to novel downstream tasks with relatively few labeled examples. However, they require enormous computational resources to be deployed. Alternatively, smaller models can solve specific tasks if fine-tuned with enough labeled examples. These examples, however, are expensive to obtain. In pursuit of the best of both worlds, we study synthetic data generation of fine-tuning training data via fine-tuned teacher LLMs to improve the downstream performance of much smaller models. In four text classification and two text generation tasks, we find that both data generation and annotation dramatically improve the respective downstream model's performance, occasionally necessitating only a minor fraction of the original training dataset.

Figures (7)

• Figure 1: Synthetic Data Generation via Fine-Tuning the Teacher LLM: Step 1: We fine-tune (FT) an LLM (GPT-NeoX-20B neox). Step 2: We either annotate unlabeled instances or generate entirely new ones. Step 3: We train a small downstream model on the augmented training dataset.
• Figure 2: Text Classification.
• Figure 3: Natural Language Generation.
• Figure 5: Different amounts of Ours data added to the student model training set. While both the SGD and WebNLG tasks benefit from the synthetic data, we observe diminishing returns.
• Figure 6: Combining Annotation ($Y | X$) and Generation ($X, Y$) equally performs almost as well as annotating the same amount, while the latter assumes access to unlabeled instances, which can be difficult in practice.