InstructDiff: Domain-Adaptive Data Selection via Differential Entropy for Efficient LLM Fine-Tuning

TL;DR

The paper tackles the data-inefficiency of supervised fine-tuning for large language models by proposing InstructDiff, a domain-adaptive data selection framework based on differential entropy between a base model and a lightweight calibration model. It uses a two-stage pipeline—Warmup Calibration to create a calibration model, and Distribution-Aware Selection that computes per-example $\Delta\text{NLL}$ and $\Delta H$, applies bi-directional filtering, and ranks candidates by $\Delta H$ to select a learnable frontier. Across math, general instruction-following, medical QA, and code generation, InstructDiff achieves up to $+52\%$ relative improvement over full-data training while using only 10\% of the data, and scales to large pools with weak-to-strong calibration. This work demonstrates that domain-adaptive entropy dynamics provides a principled signal for data curation in SFT, reducing compute while maintaining or boosting performance.

Abstract

Supervised fine-tuning (SFT) is fundamental to adapting large language models, yet training on complete datasets incurs prohibitive costs with diminishing returns. Existing data selection methods suffer from severe domain specificity: techniques optimized for general instruction-following fail on reasoning tasks, and vice versa. We observe that measuring entropy differences between base models and minimally instruction-tuned calibrated models reveals a pattern -- samples with the lowest differential entropy consistently yield optimal performance across domains, yet this principle manifests domain-adaptively: reasoning tasks favor entropy increase (cognitive expansion), while general tasks favor entropy decrease (cognitive compression). We introduce InstructDiff, a unified framework that operationalizes differential entropy as a domain-adaptive selection criterion through warmup calibration, bi-directional NLL filtering, and entropy-based ranking. Extensive experiments show that InstructDiff achieves 17\% relative improvement over full data training on mathematical reasoning and 52\% for general instruction-following, outperforming prior baselines while using only 10\% of the data.

Figures (9)

• Figure 1: Domain-adaptive entropy dynamics reveal distinct learning patterns. We visualize the entropy difference ($\Delta H = H_{\text{base}} - H_{\text{inst}}$) distributions across three domains. For general instruction-following and medical QA, almost all samples show $H_{\text{base}} > H_{\text{inst}}$, i.e., instruction-tuning compresses uncertainty (entropy decrease, cognitive compression). In contrast, for mathematical reasoning, about half of the samples show $H_{\text{inst}} > H_{\text{base}}$, i.e., instruction-tuning increases entropy (cognitive expansion). This contrast supports our unified but domain-adaptive selection principle based on differential entropy.
• Figure 2: The InstructDiff two-stage selection pipeline.Stage 1: Warmup Calibration. Randomly sample a small warmup subset to lightly instruction-tune the base model, producing a calibration model as reference. Stage 2: Distribution-Aware Selection. For each candidate sample, compute negative log-likelihood difference ($\Delta$NLL) and entropy difference ($\Delta H$) between the base and calibration models. Filter out samples with extreme $\Delta$NLL, then select the lowest $\Delta H$ samples from the learnable range, in a domain-adaptive way. This process can optionally be repeated with the updated model for further gains.
• Figure 3: Entropy Selection Range Analysis. Bottom 10% (low $\Delta H$) consistently outperforms, validating domain-adaptive entropy dynamics.
• Figure 4: Correlation Matrix.$\Delta H$ measures cognitive alignment orthogonal to difficulty metrics.
• Figure 5: Iterative Refinement. Two iterations provide optimal cost-benefit trade-off before diminishing returns.