The Impact of Quantization on the Robustness of Transformer-based Text Classifiers
Seyed Parsa Neshaei, Yasaman Boreshban, Gholamreza Ghassem-Sani, Seyed Abolghasem Mirroshandel
TL;DR
The paper addresses the robustness of transformer-based NLP models under adversarial perturbations and proposes 8-bit dynamic quantization (via ONNXRuntime) as a defense. By quantizing BERT and DistilBERT, it demonstrates substantial improvements in adversarial accuracy across SST-2, Emotion, and MR against TextFooler, PWWS, and PSO, with only minor drops on clean data. Compared to adversarial training, quantization yields larger robustness gains without incurring training overhead, suggesting quantization as a practical robustness amplification method. The work highlights quantization’s potential in enhancing NLP model reliability while outlining future directions for broader model support and ensemble approaches.
Abstract
Transformer-based models have made remarkable advancements in various NLP areas. Nevertheless, these models often exhibit vulnerabilities when confronted with adversarial attacks. In this paper, we explore the effect of quantization on the robustness of Transformer-based models. Quantization usually involves mapping a high-precision real number to a lower-precision value, aiming at reducing the size of the model at hand. To the best of our knowledge, this work is the first application of quantization on the robustness of NLP models. In our experiments, we evaluate the impact of quantization on BERT and DistilBERT models in text classification using SST-2, Emotion, and MR datasets. We also evaluate the performance of these models against TextFooler, PWWS, and PSO adversarial attacks. Our findings show that quantization significantly improves (by an average of 18.68%) the adversarial accuracy of the models. Furthermore, we compare the effect of quantization versus that of the adversarial training approach on robustness. Our experiments indicate that quantization increases the robustness of the model by 18.80% on average compared to adversarial training without imposing any extra computational overhead during training. Therefore, our results highlight the effectiveness of quantization in improving the robustness of NLP models.