Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy Gradient with Adaptive Entropy Annealing for Continual Fine-Tuning

Yaqian Zhang, Bernhard Pfahringer, Eibe Frank, Albert Bifet

Abstract

Despite their success, large pretrained vision models remain vulnerable to catastrophic forgetting when adapted to new tasks in class-incremental settings. Parameter-efficient fine-tuning (PEFT) alleviates this by restricting trainable parameters, yet most approaches still rely on cross-entropy (CE) loss, a surrogate for the 0-1 loss, to learn from new data. We revisit this choice and revive the true objective (0-1 loss) through a reinforcement learning perspective. By formulating classification as a one-step Markov Decision Process, we derive an Expected Policy Gradient (EPG) method that directly minimizes misclassification error with a low-variance gradient estimation. Our analysis shows that CE can be interpreted as EPG with an additional sample-weighting mechanism: CE encourages exploration by emphasizing low-confidence samples, while EPG prioritizes high-confidence ones. Building on this insight, we propose adaptive entropy annealing (aEPG), a training strategy that transitions from exploratory (CE-like) to exploitative (EPG-like) learning. aEPG-based methods outperform CE-based methods across diverse benchmarks and with various PEFT modules. More broadly, we evaluate various entropy regularization methods and demonstrate that lower entropy of the output prediction distribution enhances adaptation in pretrained vision models.

Policy Gradient with Adaptive Entropy Annealing for Continual Fine-Tuning

Abstract

Despite their success, large pretrained vision models remain vulnerable to catastrophic forgetting when adapted to new tasks in class-incremental settings. Parameter-efficient fine-tuning (PEFT) alleviates this by restricting trainable parameters, yet most approaches still rely on cross-entropy (CE) loss, a surrogate for the 0-1 loss, to learn from new data. We revisit this choice and revive the true objective (0-1 loss) through a reinforcement learning perspective. By formulating classification as a one-step Markov Decision Process, we derive an Expected Policy Gradient (EPG) method that directly minimizes misclassification error with a low-variance gradient estimation. Our analysis shows that CE can be interpreted as EPG with an additional sample-weighting mechanism: CE encourages exploration by emphasizing low-confidence samples, while EPG prioritizes high-confidence ones. Building on this insight, we propose adaptive entropy annealing (aEPG), a training strategy that transitions from exploratory (CE-like) to exploitative (EPG-like) learning. aEPG-based methods outperform CE-based methods across diverse benchmarks and with various PEFT modules. More broadly, we evaluate various entropy regularization methods and demonstrate that lower entropy of the output prediction distribution enhances adaptation in pretrained vision models.
Paper Structure (21 sections, 3 theorems, 17 equations, 6 figures, 9 tables)

This paper contains 21 sections, 3 theorems, 17 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Classification MDP Formulation
  5. Expected Policy Gradient
  6. Comparative Analysis of CE and EPG
  7. Algorithm: Adaptive Entropy Annealing
  8. Experiments
  9. Results
  10. Ablation Studies
  11. Conclusion
  12. Proofs
  13. Proof of Proposition \ref{['proposition_01']}
  14. Proof of Proposition \ref{['proposition:label noise']}
  15. Proof of Proposition \ref{['proposition:KL']}
  16. ...and 6 more sections

Key Result

Proposition 1

Minimizing the 0-1 loss of classifier $h_\theta$ is equivalent to maximizing the RL objective:

Figures (6)

  • Figure 1: The entropy of the output distribution of the model: EPG leads to higher entropy than CE.
  • Figure 2: Entropy dynamics on Split-ImageNet-R. Results are averaged across 5 independent runs, with the 95% confidence interval shaded. Entropy-reducing methods (EPG, aEPG, EP) achieve higher accuracy than the cross-entropy (CE) baseline, while entropy-increasing methods (focal loss, label smoothing, confidence penalty) result in lower performance. Corresponding results for the Split-Food101 dataset are shown in Appendix Fig. \ref{['fig:focal_food101']}.
  • Figure 3: The effect of $\alpha$ when combining CE and EPG with $\alpha \mathcal{L}_{CE} + (1-\alpha) \mathcal{L}_{EPG}$. Results show the mean and standard deviation across 5 runs.
  • Figure 4: Training CIFAR100 with ResNet50 from scratch. CE-EPG with an alpha value of 0.2 outperforms the standard CE loss (Best test accuracy: 81% vs. 78%.)
  • Figure 5: Entropy dynamics in continual fine-tuning VisionTransformers on Split-Food101. Compared to the cross-entropy loss, Expected Policy Gradient (EPG), adaptive EPG (aEPG), and Entropy Penalty (EP) lead to lower entropy and improved accuracy. In contrast, focal loss, label smoothing, and confidence penalty (CP) lead to higher entropy and worse performance.
  • ...and 1 more figures

Theorems & Definitions (9)

  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Proposition 3
  • proof
  • proof
  • proof
  • proof