Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robustness to distribution shifts of compressed networks for edge devices

Lulan Shen, Ali Edalati, Brett Meyer, Warren Gross, James J. Clark

TL;DR

This work investigates how three common network compression strategies—pruning, knowledge distillation (KD), and post-training quantization (PTQ)—affect robustness to distribution shifts when deploying deep networks on edge devices. Using ResNet variants on the Office-31 domain-shift benchmark, the study finds that compression generally reduces robustness, with larger base models suffering more, but compact models produced via KD or PTQ can retain substantial performance in unseen domains. Notably, 8-bit post-training quantization yields the strongest domain-shift robustness, often with minimal degradation when reducing size to about one-quarter of the original, while pruning tends to degrade adversarial and domain robustness more. The results provide practical guidance: for edge deployments where domain shift robustness is critical, KD- or PTQ-derived compact models—especially PTQ—are preferable to pruning, with KD offering consistent adversarial robustness across compression rates and PTQ delivering superior domain-shift resilience.

Abstract

It is necessary to develop efficient DNNs deployed on edge devices with limited computation resources. However, the compressed networks often execute new tasks in the target domain, which is different from the source domain where the original network is trained. It is important to investigate the robustness of compressed networks in two types of data distribution shifts: domain shifts and adversarial perturbations. In this study, we discover that compressed models are less robust to distribution shifts than their original networks. Interestingly, larger networks are more vulnerable to losing robustness than smaller ones, even when they are compressed to a similar size as the smaller networks. Furthermore, compact networks obtained by knowledge distillation are much more robust to distribution shifts than pruned networks. Finally, post-training quantization is a reliable method for achieving significant robustness to distribution shifts, and it outperforms both pruned and distilled models in terms of robustness.

Robustness to distribution shifts of compressed networks for edge devices

TL;DR

This work investigates how three common network compression strategies—pruning, knowledge distillation (KD), and post-training quantization (PTQ)—affect robustness to distribution shifts when deploying deep networks on edge devices. Using ResNet variants on the Office-31 domain-shift benchmark, the study finds that compression generally reduces robustness, with larger base models suffering more, but compact models produced via KD or PTQ can retain substantial performance in unseen domains. Notably, 8-bit post-training quantization yields the strongest domain-shift robustness, often with minimal degradation when reducing size to about one-quarter of the original, while pruning tends to degrade adversarial and domain robustness more. The results provide practical guidance: for edge deployments where domain shift robustness is critical, KD- or PTQ-derived compact models—especially PTQ—are preferable to pruning, with KD offering consistent adversarial robustness across compression rates and PTQ delivering superior domain-shift resilience.

Abstract

It is necessary to develop efficient DNNs deployed on edge devices with limited computation resources. However, the compressed networks often execute new tasks in the target domain, which is different from the source domain where the original network is trained. It is important to investigate the robustness of compressed networks in two types of data distribution shifts: domain shifts and adversarial perturbations. In this study, we discover that compressed models are less robust to distribution shifts than their original networks. Interestingly, larger networks are more vulnerable to losing robustness than smaller ones, even when they are compressed to a similar size as the smaller networks. Furthermore, compact networks obtained by knowledge distillation are much more robust to distribution shifts than pruned networks. Finally, post-training quantization is a reliable method for achieving significant robustness to distribution shifts, and it outperforms both pruned and distilled models in terms of robustness.
Paper Structure (16 sections, 7 equations, 2 figures, 5 tables)

This paper contains 16 sections, 7 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Model Compression
  4. Adversarial Attacks
  5. Methodology
  6. Model Compression
  7. Pruning
  8. Knowledge Distillation
  9. Quantization
  10. Domain Shifts
  11. Adversarial Attacks
  12. Experimental Setup
  13. Results and discussion
  14. Domain Shifts
  15. Adversarial Attacks
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Subplots (a)-(d) show misclassified images of a speaker from the Amazon domain by ResNet-50 under C&W, DeepFool, and PGD adversarial attacks. Subplots (e)-(h) show the corresponding perturbations generated under attacks, magnified by a factor of 500.
  • Figure 2: Sample images of backpack, bike, and bookcase from Amazon, Webcam, and DSLR domain of the Office-31 Saenko2010AdaptingVC.