RICo: Refined In-Context Contribution for Automatic Instruction-Tuning Data Selection

TL;DR

This work targets data selection for instruction tuning by introducing RICo, a gradient-free framework that quantifies refined sample contributions via task-level and global-level scores derived from perplexity-based metrics with a fairness adjustment. A lightweight, LoRA-based selection paradigm enables linear-time data curation, drastically reducing inference costs while maintaining or improving performance across multiple LLMs and benchmarks. Empirical results show that models trained on a small fraction of RICo-selected data can outperform full-data baselines, with notable gains on LLaMA3.1-8B and cross-dataset generalization to WizardLM data. The study also analyzes the properties of high-contribution samples, highlighting diverse tasks and non-linear difficulty patterns, and discusses limitations related to relying on implicit ICL dynamics as an approximation of full training.

Abstract

Data selection for instruction tuning is crucial for improving the performance of large language models (LLMs) while reducing training costs. In this paper, we propose Refined Contribution Measurement with In-Context Learning (RICo), a novel gradient-free method that quantifies the fine-grained contribution of individual samples to both task-level and global-level model performance. RICo enables more accurate identification of high-contribution data, leading to better instruction tuning. We further introduce a lightweight selection paradigm trained on RICo scores, enabling scalable data selection with a strictly linear inference complexity. Extensive experiments on three LLMs across 12 benchmarks and 5 pairwise evaluation sets demonstrate the effectiveness of RICo. Remarkably, on LLaMA3.1-8B, models trained on 15% of RICo-selected data outperform full datasets by 5.42% points and exceed the best performance of widely used selection methods by 2.06% points. We further analyze high-contribution samples selected by RICo, which show both diverse tasks and appropriate difficulty levels, rather than just the hardest ones.

Figures (5)

• Figure 1: Overview of the RICo Method. The diagram illustrates the two key components of the method: refined contribution quantification with the RICo score and RICo-guided selection paradigm training. This approach enables gradient-free, bias-reduced data selection with scalable inference.
• Figure 2: Pairwise comparison results (win-tie-lose) between models trained on RICo-selected subsets and models trained on the full Alpaca dataset across five evaluation benchmarks: Vicuna, Koala, WizardLM, SInstruct, and LIMA. RICo-selected data enhances instruction-following ability with fewer samples, as evidenced by pairwise comparisons.
• Figure 3: Model performance (average score) on evaluation benchmarks with varying proportions of RICo-selected Alpaca data. The models are trained based on LLaMA3.1-8B. The dataset is Alpaca dataset. As data scale grows, performance generally improves and then declines, with the best result at 15%.
• Figure 4: Visualization using t-SNE on sample embeddings from the Alpaca dataset. Red points represent samples with the top 15% RICo high-contribution scores and gray points represent other samples from the dataset. High-contribution RICo samples exhibit diverse characteristics.
• Figure 5: Difficulty distribution of top 15% high-contribution samples vs. full Alpaca dataset. High-contribution samples (red bars), selected via RICo for instruction tuning, exhibit distinct difficulty patterns compared to the full dataset (gray bars). High-contribution RICo samples tend to fall within an appropriate difficulty range.