ResoFilter: Fine-grained Synthetic Data Filtering for Large Language Models through Data-Parameter Resonance Analysis

TL;DR

ResoFilter leverages the fine-tuning process to obtain Data-Parameter features for data selection, offering improved interpretability by representing data characteristics through model weights, and exhibits strong generalization across different models and domains.

Abstract

Large language models (LLMs) have shown remarkable effectiveness across various domains, with data augmentation methods utilizing GPT for synthetic data generation becoming prevalent. However, the quality and utility of augmented data remain questionable, and current methods lack clear metrics for evaluating data characteristics. To address these challenges, we propose ResoFilter, a novel method that integrates models, data, and tasks to refine datasets. ResoFilter leverages the fine-tuning process to obtain Data-Parameter features for data selection, offering improved interpretability by representing data characteristics through model weights. Our experiments demonstrate that ResoFilter achieves comparable results to full-scale fine-tuning using only half the data in mathematical tasks and exhibits strong generalization across different models and domains. This method provides valuable insights for constructing synthetic datasets and evaluating high-quality data, offering a promising solution for enhancing data augmentation techniques and improving training dataset quality for LLMs. For reproducibility, we will release our code and data upon acceptance.

Figures (7)

• Figure 1: Workflow for our method. The left side of the figure illustrates the detailed process of calculating parameter changes for individual data samples using the $W_{up}$ matrix from the last n layers of the neural network. The right side demonstrates the application of this method to the entire dataset, including steps for computing parameter change values for each sample, sorting based on these values, filtering out samples with the largest parameter changes, and restoring the remaining data to its original order.
• Figure 2: We analyzed the $W_{up}$ weights of the model from the first layer to the 26th layer. The 25% shows a continuous upward trend, while the 75% fluctuates within a certain range.
• Figure 3: Here we used filtering based on different proportions, and then trained and evaluated using the remaining data. The figure shows the GSM8K scores under different filtering ratios.
• Figure 4: Illustrates the token length distribution comparison across three datasets. Low Diff Value has much larger token numbers than High Diff Value data.
• Figure 5: Illustrates the token Frequency distribution comparison across three datasets. In order to analyze the frequency distribution of tokens, we first perform token statistics on the entire dataset to obtain the word frequency of each token. Subsequently, we calculated the total frequency of all token words in each sample divided by the sample length to obtain the average token frequency. High Diff Value shows more obscure vocabulary is used compared to Low Diff Value.