Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models

TL;DR

The paper tackles the high cost of instruction tuning by introducing a learning-percentage based metric (LP) to rank and select training data. It demonstrates that smaller models can autonomously curate high-quality data for larger models across sizes from 1B to 13B, with hard samples driving better generalization and with data hardness transferable across model sizes. An approximate, faster variant (LP_app) matches or exceeds the performance of the full LP method, enabling efficient data selection, and human evaluation corroborates the competitive quality of models trained on a small, difficulty-focused subset. Overall, the approach reduces data and compute requirements for instruction tuning while maintaining or improving model performance.

Abstract

Instruction-tuning language models has become a crucial step in aligning them for general use. Typically, this process involves extensive training on large datasets, incurring high training costs. In this paper, we introduce a novel training data selection based on the learning percentage of the samples. We assert that current language models possess the capability to autonomously select high-quality training data, leading to comparable or improved performance compared to training on the entire dataset. Our experiments span different-sized models, revealing that this characteristic holds for models ranging from 1B (small) to 13B (large) in size. Moreover, we demonstrate an interesting finding that the data hardness transfers across model sizes, and a smaller 350M model can effectively curate high-quality training data with hard samples for a larger 13B model, resulting in an equally or superior instruction-tuned model compared to training on the complete dataset. Utilizing open-sourced OPT and Llama-2 models up to 13B in size, two publicly available instruction-tuning training datasets and evaluated by both automatic metrics & humans, our paper introduces a novel approach to training data selection, showcasing a more efficient alternative.

Figures (11)

• Figure 1: The win rate of OPT-13B model trained on 10% data sub-sampled by smaller OPT models (350M, 1.3B, 2.7B) from Alpaca Data, is compared against the OPT-13B model trained on the full dataset. All win rates exceed 50, indicating even a smaller 350M dataset can curate high-quality data for a larger 13B model.
• Figure 2: We partition datasets into three equal-sized buckets based on their $\mathcal{LP}(1)$ scores. We train a model per bucket and report its win rate against the one trained on the complete dataset. The model used to compute $\mathcal{LP}(1)$ scores and trained is depicted on the X-axis and the win rate on the Y-axis. We observe the model trained on the lowest $\mathcal{LP}(1)$ values (33% Low $\mathcal{LP}(1)$) exhibits superior performance compared to the others.
• Figure 3: We partition Alpaca-Data into three equal-sized buckets based on their $\mathcal{LP}(1)$ scores. The model used to compute $\mathcal{LP}(1)$ scores and trained is on the X-axis and the win rate on the Y-axis. We observe the model trained on the lowest $\mathcal{LP}(1)$ values (33% Low $\mathcal{LP}(1)$) exhibits superior performance.
• Figure 4: We consider Alpaca-Data, vary the percentage of data selected, and plot the win rate of OPT models, trained on the selected data in comparison to models trained on the complete dataset. The minimum percentage of data necessary for each model to surpass the 50% threshold is highlighted with .
• Figure 5: We vary the percentage of selected data to train 13B model and conduct a comparison of win rates obtained when data is self-selected by the 13B model vs selected by smaller models. The smaller model used is mentioned on the X-axis and the win rate is on the Y-axis. We observe that the data hardness transfers from smaller models to 13B, leading to improved or comparable performance compared to 13B model trained on the self-selected data.