Neural Networks for Learnable and Scalable Influence Estimation of Instruction Fine-Tuning Data

TL;DR

This work tackles the high computational cost of influence-function-based data valuation for instruction fine-tuning by learning a compact InfluenceNetwork (NN-CIFT) that estimates data influence. The method trains the network on a small subset of influence-function outputs and uses it to estimate influences for the remaining data, enabling substantial speedups (up to $99\%$) with networks as small as $2.0\times 10^{5}$ parameters, roughly $10^{-4}$ of a typical LM. Across four state-of-the-art influence functions and multiple models/datasets, NN-CIFT achieves comparable downstream performance to the original methods while dramatically reducing cost, and it does not require retraining for new data. The approach combines a lightweight two-layer network with embedding-based inputs and a principled training regime, supported by extensive ablations and hyperparameter analyses. Overall, NN-CIFT offers a scalable, model-agnostic path to efficient data valuation and subset selection for instruction fine-tuning.

Abstract

Influence functions provide crucial insights into model training, but existing methods suffer from large computational costs and limited generalization. Particularly, recent works have proposed various metrics and algorithms to calculate the influence of data using language models, which do not scale well with large models and datasets. This is because of the expensive forward and backward passes required for computation, substantial memory requirements to store large models, and poor generalization of influence estimates to new data. In this paper, we explore the use of small neural networks -- which we refer to as the InfluenceNetwork -- to estimate influence values, achieving up to 99% cost reduction. Our evaluation demonstrates that influence values can be estimated with models just 0.0027% the size of full language models (we use 7B and 8B versions). We apply our algorithm of estimating influence values (called NN-CIFT: Neural Networks for effiCient Instruction Fine-Tuning) to the downstream task of subset selection for general instruction fine-tuning. In our study, we include four state-of-the-art influence functions and show no compromise in performance, despite large speedups, between NN-CIFT and the original influence functions. We provide an in-depth hyperparameter analyses of NN-CIFT. The code for our method can be found here: https://github.com/agarwalishika/NN-CIFT.

Figures (4)

• Figure 1: Overview of NN-CIFT. The first step consists of using established influence functions to foomyorange collect data for training the InfluenceNetwork. Next, the data from Step (1) is used to train the InfluenceNetwork and, subsequently, foomygreen estimate the influence values for the rest of the data. Finally, the data selection algorithm corresponding to the original influence function is used to select a subset of IFT data to fine-tune a model on.
• Figure 2: MSE versus InfluenceNetwork training data size (u) plotted for 8 different training sizes, broken down by the quadrant. These results are for learning DELIFT influence values. Error rates on each quadrant correspond to losses across different sets: Q1 for training, Q2/Q3 for validation, and Q4 for testing. As shown, the InfluenceNetwork achieves MSE of merely 0.05% starting from $u=0.05$ and always outperforms the baselines.
• Figure 3: Distribution of influence values across each of the methods on the Alpaca dataset. To clarify, "D_F" is $\mathcal{D_F}$ and "D_T" is $\mathcal{D_T}$. The x-axis spans the 15,000 examples from $\mathcal{D_F}$ and the y-axis spans the 5,000 samples from $\mathcal{D_T}$. SelectIT only has an x-axis.
• Figure 4: MSE versus InfluenceNetwork sizes (measured by the number of parameters). We try 1-5 layers with 46 different combinations of hidden layer sizes from {5, 10, 20, 50, 100, 200, 500, 1000, 2000, 3000, 4000, 5000}.