Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diffusion-Classifier Synergy: Reward-Aligned Learning via Mutual Boosting Loop for FSCIL

Ruitao Wu, Yifan Zhao, Guangyao Chen, Jia Li

TL;DR

This work tackles Few-Shot Class-Incremental Learning by coupling diffusion-based data generation with an FSCIL classifier through a reward-guided mutual boosting loop. By splitting rewards into feature-level (semantic coherence and diversity via PAMMD and VM) and logits-level (classifier-aware generation via RC and CSCA), the framework iteratively improves both the diffusion outputs and the classifier’s discrimination across sessions. The proposed approach achieves state-of-the-art results on standard FSCIL benchmarks, demonstrating strong knowledge retention for old classes while efficiently incorporating new ones from limited data. This co-evolutionary strategy highlights how incorporating classifier feedback into diffusion guidance can substantially enhance data-efficient continual learning with practical implications for real-world, dynamic environments.

Abstract

Few-Shot Class-Incremental Learning (FSCIL) challenges models to sequentially learn new classes from minimal examples without forgetting prior knowledge, a task complicated by the stability-plasticity dilemma and data scarcity. Current FSCIL methods often struggle with generalization due to their reliance on limited datasets. While diffusion models offer a path for data augmentation, their direct application can lead to semantic misalignment or ineffective guidance. This paper introduces Diffusion-Classifier Synergy (DCS), a novel framework that establishes a mutual boosting loop between diffusion model and FSCIL classifier. DCS utilizes a reward-aligned learning strategy, where a dynamic, multi-faceted reward function derived from the classifier's state directs the diffusion model. This reward system operates at two levels: the feature level ensures semantic coherence and diversity using prototype-anchored maximum mean discrepancy and dimension-wise variance matching, while the logits level promotes exploratory image generation and enhances inter-class discriminability through confidence recalibration and cross-session confusion-aware mechanisms. This co-evolutionary process, where generated images refine the classifier and an improved classifier state yields better reward signals, demonstrably achieves state-of-the-art performance on FSCIL benchmarks, significantly enhancing both knowledge retention and new class learning.

Diffusion-Classifier Synergy: Reward-Aligned Learning via Mutual Boosting Loop for FSCIL

TL;DR

This work tackles Few-Shot Class-Incremental Learning by coupling diffusion-based data generation with an FSCIL classifier through a reward-guided mutual boosting loop. By splitting rewards into feature-level (semantic coherence and diversity via PAMMD and VM) and logits-level (classifier-aware generation via RC and CSCA), the framework iteratively improves both the diffusion outputs and the classifier’s discrimination across sessions. The proposed approach achieves state-of-the-art results on standard FSCIL benchmarks, demonstrating strong knowledge retention for old classes while efficiently incorporating new ones from limited data. This co-evolutionary strategy highlights how incorporating classifier feedback into diffusion guidance can substantially enhance data-efficient continual learning with practical implications for real-world, dynamic environments.

Abstract

Few-Shot Class-Incremental Learning (FSCIL) challenges models to sequentially learn new classes from minimal examples without forgetting prior knowledge, a task complicated by the stability-plasticity dilemma and data scarcity. Current FSCIL methods often struggle with generalization due to their reliance on limited datasets. While diffusion models offer a path for data augmentation, their direct application can lead to semantic misalignment or ineffective guidance. This paper introduces Diffusion-Classifier Synergy (DCS), a novel framework that establishes a mutual boosting loop between diffusion model and FSCIL classifier. DCS utilizes a reward-aligned learning strategy, where a dynamic, multi-faceted reward function derived from the classifier's state directs the diffusion model. This reward system operates at two levels: the feature level ensures semantic coherence and diversity using prototype-anchored maximum mean discrepancy and dimension-wise variance matching, while the logits level promotes exploratory image generation and enhances inter-class discriminability through confidence recalibration and cross-session confusion-aware mechanisms. This co-evolutionary process, where generated images refine the classifier and an improved classifier state yields better reward signals, demonstrably achieves state-of-the-art performance on FSCIL benchmarks, significantly enhancing both knowledge retention and new class learning.

Paper Structure

This paper contains 39 sections, 37 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminary
  4. Few-Shot Class-Incremental Learning
  5. Diffusion Alignment as Sampling (DAS)
  6. Reward-Aligned Learning via Mutual Boosting Loop for FSCIL
  7. Addressing FSCIL Challenges with a Mutual Boosting Loop
  8. Feature-Level Reward for Semantic Coherence and Diversity
  9. Logits-Level Rewards for Robust Discrimination Learning
  10. Experiments
  11. Experimental Setup
  12. Datasets
  13. Evaluation metrics
  14. Implementation details
  15. Comparison with the State of the Art
  16. ...and 24 more sections

Figures (4)

  • Figure 1: (a) Unidirectional knowledge provision in conventional methods results in inefficient generated images. (b) Our approach mitigates this inefficiency via a combined feature-level and logits-level reward, facilitated by a mutual boosting loop between the diffusion model and classifier.
  • Figure 2: Left: Trade-off between semantic fidelity and richness with increasing guidance scale. Right: t-SNE visualization of features from real (dot) and generated (pentagram) images illustrates the restricted distribution of the latter within the classifier's decision space.
  • Figure 3: Training performance comparison. DCS versus the vanilla diffusion model generating images at varying quantities and guidance scales, with the latter underperforming at comparable generation scales.
  • Figure 4: Qualitative comparison of real and generated images on miniImageNet. The first column displays real images from the dataset. The subsequent columns show diverse and semantically correct images generated by our DCS framework for the corresponding classes.