Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities

TL;DR

This work identifies a bias in cross-task influence scores that hampers balancing diverse capabilities during instruction fine-tuning. It introduces BIDS, which combines instance-level normalization and iterative marginal-gain optimization to select data that improves underrepresented tasks. Across Llama-3-8B and Mistral-7B-v0.3 on seven benchmarks, BIDS consistently outperforms both influence-based and non-influence baselines and can surpass full-dataset training with a substantially smaller data budget. The findings highlight how normalized, balanced influential data fosters more robust multi-task learning and offer a practical approach for efficient, high-quality instruction tuning.

Abstract

Selecting appropriate training data is crucial for effective instruction fine-tuning of large language models (LLMs), which aims to (1) elicit strong capabilities, and (2) achieve balanced performance across a diverse range of tasks. Influence-based methods show promise in achieving (1) by estimating the contribution of each training example to the model's predictions, but often struggle with (2). Our systematic investigation reveals that this underperformance can be attributed to an inherent bias where certain tasks intrinsically have greater influence than others. As a result, data selection is often biased towards these tasks, not only hurting the model's performance on others but also, counterintuitively, harms performance on these high-influence tasks themselves. As a remedy, we propose BIDS, a Balanced and Influential Data Selection algorithm. BIDS first normalizes influence scores of the training data, and then iteratively balances data selection by choosing the training example with the highest influence on the most underrepresented task. Experiments with both Llama-3 and Mistral-v0.3 on seven benchmarks spanning five diverse capabilities show that BIDS consistently outperforms both state-of-the-art influence-based algorithms and other non-influence-based selection frameworks. Surprisingly, training on a 15% subset selected by BIDS can even outperform full-dataset training with a much more balanced performance. Our analysis further highlights the importance of both instance-level normalization and iterative optimization of selected data for balanced learning of diverse capabilities.

Figures (10)

• Figure 1: Unnormalized Average Influence Distribution (AID) of the whole UltraInteract dataset yuan2024advancing, showing great disparities in scale for inter-task and intra-task influence.
• Figure 2: Task frequencies with Highest Influence (THI) under the 10% budget. MMLU is obviously oversampled in LESS-selected data.
• Figure 3: A comparison between BIDS and the task-wise max algorithm used by LESS. For convenience, we represent the training set ${\mathcal{D}}$ with its Attribution Matrix (AM), in which the $i$-th row is the $|{\mathcal{V}}|$-dimensional Influence Distribution of the $i$-th training example, ${\bm{t}}_i$, in ${\mathcal{D}}$. BIDS differs from LESS in mainly two aspects. First, it applies a column-wise normalization to the AM. Next, instead of directly selecting top-$B$ examples in influence, BIDS applies an iterative algorithm which, at each iteration, obtains the utility $\Delta^{(i)}$ of each candidate example ${\bm{t}}_i$ by calculating how much improvement in influence it can bring to the current selected subset ${\mathcal{T}}$, and selects candidate ${\bm{t}}_{i^*}$ with the highest utility $\Delta^{(i^*)}$. Please see §\ref{['sec:bids']} for a more detailed walkthrough.
• Figure 4: Comparative analysis of THI under the 10% budget. Both $- \texttt{Iter}$ and BIDS have more balanced task frequencies compared with $- (\texttt{Norm} + \texttt{Iter})$.
• Figure 5: Comparative analysis of normalized AID under the 10% budget. From $- (\texttt{Norm} + \texttt{Iter})$ to $- \texttt{Iter}$ to BIDS, the disparity in AID among different tasks and instances gradually diminishes, with both decreasing upper bounds and increasing lower bounds.