Threshold Filtering Packing for Supervised Fine-Tuning: Training Related Samples within Packs

TL;DR

Threshold Filtering Packing (TFP) tackles cross-contamination in supervised fine-tuning by forming packs of related yet diverse samples through a greedy, TSP-inspired path and a distance threshold $t$ to enforce intra-pack diversity. It avoids masking by concatenating context-rich packs and integrates smoothly with existing SFT pipelines, while maintaining efficiency as data lengths grow. Empirical results show TFP delivering up to $7\%$ gains on GSM8K and $4\%$ on HumanEval, plus up to $15\%$ accuracy improvements on tabular fairness tasks when applying balanced or resampling strategies within packs. The approach is demonstrated to be effective across text and code reasoning benchmarks, offering a practical, scalable way to improve few-shot performance and fairness in LLM fine-tuning.

Abstract

Packing for Supervised Fine-Tuning (SFT) in autoregressive models involves concatenating data points of varying lengths until reaching the designed maximum length to facilitate GPU processing. However, randomly concatenating data points can lead to cross-contamination of sequences due to the significant difference in their subject matter. The mainstream approaches in SFT ensure that each token in the attention calculation phase only focuses on tokens within its own short sequence, without providing additional learning signals for the preceding context. To address these challenges, we introduce Threshold Filtering Packing (TFP), a method that selects samples with related context while maintaining sufficient diversity within the same pack. Our experiments show that TFP offers a simple-to-implement and scalable approach that significantly enhances SFT performance, with observed improvements of up to 7\% on GSM8K, 4\% on HumanEval. Furthermore, results from bias benchmark datasets highlight TFP's promising performance in improving fairness while also boosting prediction accuracy by 15\%.

Figures (4)

• Figure 1: Overview of TFP. Different from vanilla FT (a), which uses “[PAD]” tokens up to the maximum length, and random packing (b), which places randomly shuffled samples in the same pack and may lead to cross-contamination, TFP (d) places related samples in the same pack while applying a threshold on TSP (c) to ensure that these samples are sufficiently distinct. This approach allows models to learn across diverse contexts and prevent overly similar samples from being grouped together.
• Figure 2: Training time (hrs) of LLaMA3-8B using different methods on GSM8K.
• Figure 3: Performance comparison of different embedding models and different data segments on GSM8K.
• Figure 4: Accuracy of TFP on Llama Models of different distance threshold number $r$.