Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Compressing Model with Few Class-Imbalance Samples: An Out-of-Distribution Expedition

Tian-Shuang Wu, Shen-Huan Lyu, Ning Chen, Zhihao Qu, Baoliu Ye

TL;DR

The paper tackles the underexplored issue of class imbalance in few-shot model compression by introducing OE-FSMC, a framework that leverages out-of-distribution (OOD) data to rebalance training during both compression and fine-tuning. It combines a complementary OOD label assignment, class-aware pruning, a joint distillation loss, and a regularization term to prevent overfitting to OOD samples, with early stopping to preserve generalization. Empirical results across CIFAR-10/100 and ILSVRC-2012 show that OE-FSMC improves accuracy for multiple base methods and architectures, especially when data are severely limited, demonstrating the method's generality and practical impact. The work contributes a practical strategy for mitigating minority-class degradation in few-shot compression and opens avenues for applying OOD-assisted rebalancing to broader compression paradigms, including potential extensions to quantization.

Abstract

In recent years, as a compromise between privacy and performance, few-sample model compression has been widely adopted to deal with limited data resulting from privacy and security concerns. However, when the number of available samples is extremely limited, class imbalance becomes a common and tricky problem. Achieving an equal number of samples across all classes is often costly and impractical in real-world applications, and previous studies on few-sample model compression have mostly ignored this significant issue. Our experiments comprehensively demonstrate that class imbalance negatively affects the overall performance of few-sample model compression methods. To address this problem, we propose a novel and adaptive framework named OOD-Enhanced Few-Sample Model Compression (OE-FSMC). This framework integrates easily accessible out-of-distribution (OOD) data into both the compression and fine-tuning processes, effectively rebalancing the training distribution. We also incorporate a joint distillation loss and a regularization term to reduce the risk of the model overfitting to the OOD data. Extensive experiments on multiple benchmark datasets show that our framework can be seamlessly incorporated into existing few-sample model compression methods, effectively mitigating the accuracy degradation caused by class imbalance.

Compressing Model with Few Class-Imbalance Samples: An Out-of-Distribution Expedition

TL;DR

The paper tackles the underexplored issue of class imbalance in few-shot model compression by introducing OE-FSMC, a framework that leverages out-of-distribution (OOD) data to rebalance training during both compression and fine-tuning. It combines a complementary OOD label assignment, class-aware pruning, a joint distillation loss, and a regularization term to prevent overfitting to OOD samples, with early stopping to preserve generalization. Empirical results across CIFAR-10/100 and ILSVRC-2012 show that OE-FSMC improves accuracy for multiple base methods and architectures, especially when data are severely limited, demonstrating the method's generality and practical impact. The work contributes a practical strategy for mitigating minority-class degradation in few-shot compression and opens avenues for applying OOD-assisted rebalancing to broader compression paradigms, including potential extensions to quantization.

Abstract

In recent years, as a compromise between privacy and performance, few-sample model compression has been widely adopted to deal with limited data resulting from privacy and security concerns. However, when the number of available samples is extremely limited, class imbalance becomes a common and tricky problem. Achieving an equal number of samples across all classes is often costly and impractical in real-world applications, and previous studies on few-sample model compression have mostly ignored this significant issue. Our experiments comprehensively demonstrate that class imbalance negatively affects the overall performance of few-sample model compression methods. To address this problem, we propose a novel and adaptive framework named OOD-Enhanced Few-Sample Model Compression (OE-FSMC). This framework integrates easily accessible out-of-distribution (OOD) data into both the compression and fine-tuning processes, effectively rebalancing the training distribution. We also incorporate a joint distillation loss and a regularization term to reduce the risk of the model overfitting to the OOD data. Extensive experiments on multiple benchmark datasets show that our framework can be seamlessly incorporated into existing few-sample model compression methods, effectively mitigating the accuracy degradation caused by class imbalance.

Paper Structure

This paper contains 19 sections, 1 theorem, 9 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Few-sample Model Compression
  4. Class Imbalance Problem
  5. Preliminaries
  6. Method
  7. The Urgent Need to Address Class Imbalance in Few-Sample Model Compression
  8. Feasibility of Using OOD Data
  9. Our Method: A Framework Using OOD Data
  10. OOD Set Handling:
  11. Compression stage:
  12. Fine-tuning Stage:
  13. Experiment
  14. Experiment Setting
  15. Validation and Analysis of the Problem (RQ1)
  16. ...and 4 more sections

Key Result

Theorem 1

When labels are uniformly sampled from the label space within the distribution, augmenting the training set $D_{\mathrm{mix}} = D_{\mathrm{train}} \cup D_{\mathrm{out}}$ does not affect the prediction of the Bayesian classifier: where $P_{\mathrm{mix}}(X, Y)$ represents the data distribution of $D_{\mathrm{mix}}$.

Figures (3)

  • Figure 1: Overview of introduction. Left: Illustration of few-sample model compression workflow for deployment in small hospitals. Right: Class imbalance in disease cases.
  • Figure 2: Illustration of OE-FSMC. Left: Label assignment strategy for OOD data. Right: Framework of OE-FSMC in Compression and Fine-Tuning Process.
  • Figure 3: Results of extra experiments.In this figure: a) Results of ablation study. b) Effect of different sizes of the auxiliary set. c) Sensitivity analysis of $\lambda \in [0, 1]$. d) Sensitivity analysis of $\eta \in [0.1, 5]$.

Theorems & Definitions (1)

  • Theorem 1