ProDS: Preference-oriented Data Selection for Instruction Tuning

TL;DR

This work tackles instruction-tuning data selection by aligning training samples with human preferences rather than solely optimizing instruction-to-response mappings. It introduces ProDS, a three-stage framework that uses direct preference optimization (DPO) to estimate task-specific preferences and a bidirectional preference synthesis (BiPS) to score samples along positive and negative directions, with an annealing-based fusion to select a high-quality subset. The approach is validated against both target-agnostic and targeted baselines, showing data-efficient improvements and the ability to surpass full-dataset performance in some settings, particularly for tasks with rich preference signals. The findings highlight the practical impact of incorporating human-preference signals into data selection for instruction tuning, while noting the offline computational overhead as a limitation and suggesting exploration of alternative preference modeling approaches in the future.

Abstract

Instruction data selection aims to identify a high-quality subset from the training set that matches or exceeds the performance of the full dataset on target tasks. Existing methods focus on the instruction-to-response mapping, but neglect the human preference for diverse responses. In this paper, we propose Preference-oriented Data Selection method (ProDS) that scores training samples based on their alignment with preferences observed in the target set. Our key innovation lies in shifting the data selection criteria from merely estimating features for accurate response generation to explicitly aligning training samples with human preferences in target tasks. Specifically, direct preference optimization (DPO) is employed to estimate human preferences across diverse responses. Besides, a bidirectional preference synthesis strategy is designed to score training samples according to both positive preferences and negative preferences. Extensive experimental results demonstrate our superiority to existing task-agnostic and targeted methods.

Figures (9)

• Figure 1: Illustration of (a) instruction data selection, (b) existing methods, and (c) our ProDS. "HQ Data" denotes the selected high-quality instruction subset. "$S_{pos}$" and "$S_{neg}$" are the consistency of training samples with the positive and negative preferences, respectively.
• Figure 2: Illustration of our proposed ProDS. ProDS first uses the warmed-up models to compute bidirectional preference, and then scores the training samples by synthesizing their consistencies with the obtained preferences.
• Figure 3: Illustration of the validation DPO pair construction process, where "$s_c$" and "$s_b$" are scores given by GPT-4 for the quality of "$r_\text{cmp}$" and "$r_\text{base}$".
• Figure 4: Performance of Llama2-7B fine-tuned on the selected top 10% of Alpaca compared to that fine-tuned on the full Alpaca.
• Figure 5: Average winning scores of Llama2-7B across the 5 test sets at different data scales.