BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation

TL;DR

The paper tackles the data bottleneck in training large language models by exploring few-shot synthetic data generation. It reveals a diversity-quality tradeoff: base models provide high diversity but lower quality, while instruction-tuned models offer higher quality with limited diversity. The authors propose Base-Refine (BARE), a two-stage approach that uses a base model to generate diverse data from minimal seeds and then refines each example with an instruction-tuned model to boost realism and correctness. Across multiple domains and tasks, BARE consistently improves downstream performance, achieving state-of-the-art results for several settings with far fewer seed examples than prior methods, and demonstrating practical data efficiency for synthetic training pipelines.

Abstract

As the demand for high-quality data in model training grows, researchers and developers are increasingly generating synthetic data to tune and train LLMs. However, current data generation methods rely on seed sets containing tens of thousands of examples to prompt instruction-tuned models. This reliance can be especially problematic when the curation of high-quality examples is expensive or difficult. In this paper we explore the novel few-shot synthetic data generation setting -- generating a high-quality dataset from a few examples. We show that when working with only a few seed examples, instruction-tuned models used in current synthetic data methods produce insufficient diversity for downstream tasks. In contrast, we show that base models without post-training, largely untapped for synthetic data generation, offer substantially greater output diversity, albeit with lower instruction following abilities. Leveraging this insight, we propose Base-Refine (BARE), a novel two-stage method that combines the diversity of base models with the quality assurance of instruction-tuned models. BARE excels in few-shot synthetic data generation: using only 3 seed examples it generates diverse, high-quality datasets that significantly improve downstream task performance. We show that fine-tuning Llama 3.1 8B with 1,000 BARE-generated samples achieves performance comparable to state-of-the-art similarly sized models on LiveCodeBench tasks. Furthermore, data generated with BARE enables a 101% improvement for a fine-tuned Llama 3.2 1B on GSM8K over data generated by only instruction-models, and an 18.4% improvement for a fine-tuned Llama 3.1 8B over the state-of-the-art RAFT method for RAG data generation.

Figures (10)

• Figure 1: Histogram of pairwise embedding cosine similarity scores for 1000 Llama-3.1-70B-Base vs Instruct generations of grade school math problems with 3 seed examples. The base distribution is further left, indicating lower similarity and hence higher diversity.
• Figure 2: Accuracy of a Llama-3.2-1B-Instruct model fine-tuned on real-world data and 5 different sets of math problems synthetically generated using 3 seed examples, evaluated on a randomly selected $n=500$ subset of GSM8K. Training with BARE-generated data outperforms all other data sources.
• Figure 3: Distribution of pairwise embedding cosine similarity scores for Llama-3.1-70B-Instruct and Llama-3.1-70B-Base generations for Enron spam. Base model distribution have more density in the low-similarity region and less density in the high-similarity region, indicating greater diversity.
• Figure 4: Indistinguishability Rate for Llama-3.1-70B Base and Instruct data generation methods across various datasets. Llama-3.1-70B-Instruct is almost uniformly better at generating examples that appear in-domain when compared alongside real-world data.
• Figure 5: BARE combines base models with instruction tuned models. Instruction-tuned models provide high-quality but low-diversity data, while base models provide low-quality but high-diversity data. With minimal seed examples, BARE independently generates a diverse initial set of data points with a base model and refines each entry individually with an instruction-tuned model to create a high-quality, high-diversity dataset. In this example of a real grade school math problem generation, the Llama-3.1-70B-Base model hallucinates in its answer to its own question. The refiner (Llama-3.1-70B-Instruct) recognizes this and disambiguates the question and corrects the reasoning.