Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating the Impact of Quantization on Adversarial Robustness

Qun Li, Yuan Meng, Chen Tang, Jiacheng Jiang, Zhi Wang

TL;DR

This paper addresses the problem of how quantization affects adversarial robustness in neural networks. It identifies that inconsistent findings in prior literature stem from varying pipelines, initialization, and robustness components, and proposes a unified framework to study PTQ and QAT under adversarial evaluations. Using CIFAR-10 with ResNet-20, it compares PTQ with PWLQ and QAT with PACT across different robustness strategies, evaluating under $L_p$ norm attacks with budget $ε$. Key results show activation quantization provides stronger robustness gains than weight quantization, robust initialization helps in certain regimes, and adversarial training in QAT offers robustness at a large computational cost, with transfer learning enabling robust performance with partial data. These findings guide deployment of secure, resource-efficient quantized models in security-critical applications.

Abstract

Quantization is a promising technique for reducing the bit-width of deep models to improve their runtime performance and storage efficiency, and thus becomes a fundamental step for deployment. In real-world scenarios, quantized models are often faced with adversarial attacks which cause the model to make incorrect inferences by introducing slight perturbations. However, recent studies have paid less attention to the impact of quantization on the model robustness. More surprisingly, existing studies on this topic even present inconsistent conclusions, which prompted our in-depth investigation. In this paper, we conduct a first-time analysis of the impact of the quantization pipeline components that can incorporate robust optimization under the settings of Post-Training Quantization and Quantization-Aware Training. Through our detailed analysis, we discovered that this inconsistency arises from the use of different pipelines in different studies, specifically regarding whether robust optimization is performed and at which quantization stage it occurs. Our research findings contribute insights into deploying more secure and robust quantized networks, assisting practitioners in reference for scenarios with high-security requirements and limited resources.

Investigating the Impact of Quantization on Adversarial Robustness

TL;DR

This paper addresses the problem of how quantization affects adversarial robustness in neural networks. It identifies that inconsistent findings in prior literature stem from varying pipelines, initialization, and robustness components, and proposes a unified framework to study PTQ and QAT under adversarial evaluations. Using CIFAR-10 with ResNet-20, it compares PTQ with PWLQ and QAT with PACT across different robustness strategies, evaluating under norm attacks with budget . Key results show activation quantization provides stronger robustness gains than weight quantization, robust initialization helps in certain regimes, and adversarial training in QAT offers robustness at a large computational cost, with transfer learning enabling robust performance with partial data. These findings guide deployment of secure, resource-efficient quantized models in security-critical applications.

Abstract

Quantization is a promising technique for reducing the bit-width of deep models to improve their runtime performance and storage efficiency, and thus becomes a fundamental step for deployment. In real-world scenarios, quantized models are often faced with adversarial attacks which cause the model to make incorrect inferences by introducing slight perturbations. However, recent studies have paid less attention to the impact of quantization on the model robustness. More surprisingly, existing studies on this topic even present inconsistent conclusions, which prompted our in-depth investigation. In this paper, we conduct a first-time analysis of the impact of the quantization pipeline components that can incorporate robust optimization under the settings of Post-Training Quantization and Quantization-Aware Training. Through our detailed analysis, we discovered that this inconsistency arises from the use of different pipelines in different studies, specifically regarding whether robust optimization is performed and at which quantization stage it occurs. Our research findings contribute insights into deploying more secure and robust quantized networks, assisting practitioners in reference for scenarios with high-security requirements and limited resources.
Paper Structure (12 sections, 2 figures, 4 tables)

This paper contains 12 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Robust Quantization Overview
  4. Quantization Pipeline
  5. Evaluate Method
  6. Experiments and Insights
  7. Various Components in Quantization Pipeline
  8. Experimental Setup
  9. Results and Analysis
  10. Conclusion
  11. Attack Strength and Quantization Settings
  12. More Details about Transfer Adversarial Learning

Figures (2)

  • Figure 1: Quantization Pipeline.
  • Figure 2: Accuracy and robustness under different attacks in QAT. The left diagram represents the results of keeping weights in full precision and quantizing activations, while the right diagram represents the results of keeping activations in full precision and quantizing weights.