Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning

TL;DR

This work tackles the data-efficiency challenges of supervised fine-tuning for large language models by diagnosing data utility with the Error–Uncertainty (EU) Plane, which maps samples by perplexity (error) and entropy (uncertainty) into four quadrants. It introduces Generalized Dynamic Data Pruning as a bilevel framework and Quadrant-based Tuning (Q-Tuning) as a two-stage, context-aware approach that jointly prunes samples and tokens: Stage 1 remove harmful/noisy or redundant samples, Stage 2 selectively prune tokens within valuable misconception samples while preserving calibration data in full. The method yields state-of-the-art results across multiple models and benchmarks, achieving remarkable data efficiency (e.g., strong gains with as little as a few percent of data) and robust scalability from 8B to 32B parameter families. Empirically, Q-Tuning demonstrates consistent improvements over full-data SFT and independent pruning baselines, with favorable latency/memory trade-offs and clear interpretations of which tokens are pruned. Overall, the work provides a principled, scalable blueprint for maximizing data utility in budget-constrained LLM fine-tuning through integrated, dynamic data pruning.

Abstract

As supervised fine-tuning (SFT) evolves from a lightweight post-training step into a compute-intensive phase rivaling mid-training in scale, data efficiency has become critical for aligning large language models (LLMs) under tight budgets. Existing data pruning methods suffer from a fragmented design: they operate either at the sample level or the token level in isolation, failing to jointly optimize both dimensions. This disconnect leads to significant inefficiencies--high-value samples may still contain redundant tokens, while token-level pruning often discards crucial instructional or corrective signals embedded in individual examples. To address this bottleneck, we introduce the Error-Uncertainty (EU) Plane, a diagnostic framework that jointly characterizes the heterogeneous utility of training data across samples and tokens. Guided by this insight, we propose Quadrant-based Tuning (Q-Tuning), a unified framework that strategically coordinates sample pruning and token pruning. Q-Tuning employs a two-stage strategy: first, it performs sample-level triage to retain examples rich in informative misconceptions or calibration signals; second, it applies an asymmetric token-pruning policy, using a context-aware scoring mechanism to trim less salient tokens exclusively from misconception samples while preserving calibration samples in their entirety. Our method sets a new state of the art across five diverse benchmarks. Remarkably, on SmolLM2-1.7B, Q-Tuning achieves a +38\% average improvement over the full-data SFT baseline using only 12.5\% of the original training data. As the first dynamic pruning approach to consistently outperform full-data training, Q-Tuning provides a practical and scalable blueprint for maximizing data utilization in budget-constrained LLM SFT.

Figures (9)

• Figure 1: (a) Error--Uncertainty (EU) plane. We partition samples by perplexity and entropy into four regions: Q1 (harmful noise), Q2 (valuable misconceptions), Q3 (redundant knowledge), and Q4 (calibration data). (b) Q-Tuning. Q-Tuning performs joint pruning guided by the EU plane: it drops Q1 and Q3, selectively prunes tokens in Q2, and retains Q4 in full.
• Figure 2: (a) Constructing the Error-Uncertainty (EU) Plane via Bisect Search. We run the base LLM to compute sample-level perplexity (PPL) and entropy, and use bisect search to set thresholds $(\alpha^*_{\text{low}}, \alpha^*_{\text{high}}, \beta^*_{\text{low}}, \beta^*_{\text{high}})$ that partition the EU plane into Q1--Q4. (b) Sample Pruning for Q1 and Q3. Samples in Q1 and Q3 are pruned at the sample level, while those in Q2 and Q4 are retained. (c) Token-Level Pruning for Q2 Samples. For retained Q2 samples, token-level pruning is applied based on both the token’s own perplexity and the average perplexity of its neighboring tokens. Tokens with high local PPL are removed, preserving only the most informative ones.
• Figure 3: Comparison with stronger task-relevant baselines under matched low-budget settings. Top: Sample-pruner baselines with a sample ratio of 12.5% and a token ratio of 50%. Bottom: Sample-pruner baselines with a sample ratio of 12.5% and a token ratio of 50%; the dashed line marks the full-data upper bound. Additional results under more pruning ratios are reported in Appendix \ref{['appendix:add_baseline']}.
• Figure 4: Comparison of Qwen3-series of varying scales (8B, 14B, 32B) across multiple benchmarks and their average. We report performance of Zero-shot, full dataset, and Q-Tuning with 12.5% samples under two token ratios (50% and 70%).
• Figure 5: Effect of varying (a) batch size and (b) neighbor awareness for Mistral-7B under three keep ratio configurations. Additional benchmark results (Avg. of five benchmarks, ARC-E, ARC-C) are provided in the Figure \ref{['fig:ablation_supp']} in Appendix \ref{['app:hyp']}.