Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data

Jiapeng Wang, Yiwen Hu, Yanzipeng Gao, Haoyu Wang, Shuo Wang, Hongyu Lu, Jiaxin Mao, Wayne Xin Zhao, Junyi Li, Xiao Zhang

TL;DR

Faced with data scarcity in domain-specific LLM adaptation, the paper shows that repeated exposure during multi-epoch training can yield gains but eventually leads to degradation due to imbalanced token learning. It proposes EntroDrop, an entropy-guided token dropout with a curriculum schedule that selectively perturbs low-entropy tokens in target-domain data while preserving informative high-entropy content. Across 0.6B–8B parameter models, EntroDrop achieves consistent gains on math reasoning and code generation and largely preserves general capabilities, extending the useful training horizon and mitigating overfitting. The work provides a practical pathway to improve data efficiency and domain adaptation for autoregressive LLMs in data-constrained settings.

Abstract

As access to high-quality, domain-specific data grows increasingly scarce, multi-epoch training has become a practical strategy for adapting large language models (LLMs). However, autoregressive models often suffer from performance degradation under repeated data exposure, where overfitting leads to a marked decline in model capability. Through empirical analysis, we trace this degradation to an imbalance in learning dynamics: predictable, low-entropy tokens are learned quickly and come to dominate optimization, while the model's ability to generalize on high-entropy tokens deteriorates with continued training. To address this, we introduce EntroDrop, an entropy-guided token dropout method that functions as structured data regularization. EntroDrop selectively masks low-entropy tokens during training and employs a curriculum schedule to adjust regularization strength in alignment with training progress. Experiments across model scales from 0.6B to 8B parameters show that EntroDrop consistently outperforms standard regularization baselines and maintains robust performance throughout extended multi-epoch training. These findings underscore the importance of aligning regularization with token-level learning dynamics when training on limited data. Our approach offers a promising pathway toward more effective adaptation of LLMs in data-constrained domains.

Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data

TL;DR

Faced with data scarcity in domain-specific LLM adaptation, the paper shows that repeated exposure during multi-epoch training can yield gains but eventually leads to degradation due to imbalanced token learning. It proposes EntroDrop, an entropy-guided token dropout with a curriculum schedule that selectively perturbs low-entropy tokens in target-domain data while preserving informative high-entropy content. Across 0.6B–8B parameter models, EntroDrop achieves consistent gains on math reasoning and code generation and largely preserves general capabilities, extending the useful training horizon and mitigating overfitting. The work provides a practical pathway to improve data efficiency and domain adaptation for autoregressive LLMs in data-constrained settings.

Abstract

As access to high-quality, domain-specific data grows increasingly scarce, multi-epoch training has become a practical strategy for adapting large language models (LLMs). However, autoregressive models often suffer from performance degradation under repeated data exposure, where overfitting leads to a marked decline in model capability. Through empirical analysis, we trace this degradation to an imbalance in learning dynamics: predictable, low-entropy tokens are learned quickly and come to dominate optimization, while the model's ability to generalize on high-entropy tokens deteriorates with continued training. To address this, we introduce EntroDrop, an entropy-guided token dropout method that functions as structured data regularization. EntroDrop selectively masks low-entropy tokens during training and employs a curriculum schedule to adjust regularization strength in alignment with training progress. Experiments across model scales from 0.6B to 8B parameters show that EntroDrop consistently outperforms standard regularization baselines and maintains robust performance throughout extended multi-epoch training. These findings underscore the importance of aligning regularization with token-level learning dynamics when training on limited data. Our approach offers a promising pathway toward more effective adaptation of LLMs in data-constrained domains.
Paper Structure (26 sections, 1 theorem, 17 equations, 4 figures, 5 tables)

This paper contains 26 sections, 1 theorem, 17 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminary Study
  3. RQ1: Is multi-epoch training beneficial for scarce data?
  4. RQ2: Does excessive repetition harm performance?
  5. RQ3: What characterizes the degradation process?
  6. Methodology
  7. Formulation
  8. Entropy-guided Token Targeting
  9. Curriculum-based Dropout Schedule
  10. Discussion
  11. Experiments
  12. Experimental Setup
  13. Main Results
  14. Ablation Analysis
  15. Impact of Dropout Ratio Strategies
  16. ...and 11 more sections

Key Result

Theorem 1

Under assumptions listed in Table tab:assumption, the gradient variance (w.r.t. model parameters $\theta$) of the EntroDrop loss $L^{\mathcal{M}}$ can be bounded by that of the loss $L$ without token dropout: where $\gamma_j$ is the curriculum mask rate in Eq. eq:entropy_mask:prob, $\alpha: = \sum_{t=1}^T g_t / T$ denotes the proportion of low-entropy tokens (with $T$ as the input sequence length

Figures (4)

  • Figure 1: Preliminary analysis of learning dynamics under multi-epoch training with limited data. (a) Under a fixed compute budget, increasing the number of training epochs (up to a certain point) for domain-specific data $D_{math}$ consistently improves accuracy, demonstrating that multi-epoch training is essential for data-constrained scenarios. (b) Excessive repetition eventually leads to performance degradation; accuracy declines as validation loss begins to rise. (c) A fine-grained analysis reveals that loss on low-entropy tokens remains stable and near zero, whereas loss on high-entropy tokens rebounds significantly after an initial decrease.
  • Figure 2: Analysis of learning dynamics. (a) Comparison with standard regularization: Our method (EntroDrop) significantly extends the effective training duration compared to baselines (e.g., NEFTune, weight decay) on Qwen3-0.6B. (b) Impact of masking ratio schedule: The dynamic curriculum schedule matches the baseline's early learning efficiency while preventing collapse in later stages, outperforming a fixed ratio. (c) A representative mathematical reasoning example illustrating that high-entropy tokens often correspond to pivotal semantic steps, while low-entropy tokens represent predictable patterns.
  • Figure 3: Hyper-parameter tuning for baselines.
  • Figure 4: Comparison of Loss Dynamics by Token Entropy.

Theorems & Definitions (4)

  • Theorem 1: Gradient Variance Bound
  • Remark 1: Token Dropout Supplements Data-Level Regularization
  • Remark 2: Gradient Variance Reduction Enhances Generalization
  • proof : Proof of Theorem \ref{['thm:TokenDropout:Gradientbound']}