TTAQ: Towards Stable Post-training Quantization in Continuous Domain Adaptation

TL;DR

A novel and stable quantization process for test-time adaptation (TTA), dubbed TTAQ, to address the performance degradation of traditional PTQ in dynamically evolving test domains and introduces Adaptive Balanced Loss (ABL) to adjust the logits by taking advantage of the frequency and complexity of the class, which can effectively address the class imbalance caused by unpredictable data streams during optimization.

Abstract

Post-training quantization (PTQ) reduces excessive hardware cost by quantizing full-precision models into lower bit representations on a tiny calibration set, without retraining. Despite the remarkable progress made through recent efforts, traditional PTQ methods typically encounter failure in dynamic and ever-changing real-world scenarios, involving unpredictable data streams and continual domain shifts, which poses greater challenges. In this paper, we propose a novel and stable quantization process for test-time adaptation (TTA), dubbed TTAQ, to address the performance degradation of traditional PTQ in dynamically evolving test domains. To tackle domain shifts in quantizer, TTAQ proposes the Perturbation Error Mitigation (PEM) and Perturbation Consistency Reconstruction (PCR). Specifically, PEM analyzes the error propagation and devises a weight regularization scheme to mitigate the impact of input perturbations. On the other hand, PCR introduces consistency learning to ensure that quantized models provide stable predictions for same sample. Furthermore, we introduce Adaptive Balanced Loss (ABL) to adjust the logits by taking advantage of the frequency and complexity of the class, which can effectively address the class imbalance caused by unpredictable data streams during optimization. Extensive experiments are conducted on multiple datasets with generic TTA methods, proving that TTAQ can outperform existing baselines and encouragingly improve the accuracy of low bit PTQ models in continually changing test domains. For instance, TTAQ decreases the mean error of 2-bit models on ImageNet-C dataset by an impressive 10.1\%.

Figures (3)

• Figure 1: Illustration of the problem: (a) The distribution of the 1st to 10th channels of the activation in ResNet-18's first convolutional layer shows a significant shift. The quantization parameters calibrated based on the original distribution are inaccurate, resulting in performance degradation; (b) A toy example illustrating the bias introduced by class-imbalanced samples. The solid gray line represents the unbiased classifier with the optimal decision boundary, while the dotted line depicts the learned biased classifier on imbalanced streaming samples, which also hinders the accuracy.
• Figure 2: Pipeline of TTAQ. In Post-Traing Quantization stage (left), perturbation consistency reconstruction apply consistency learning by encouraging quantized model can have the same output when there is a slight perturbation between $X^q_{l-1}$ and $\hat{X}^q_{l-1}$. TTAQ also introduce global information by $\mathcal{L}_{logits}$ to alleviate the overfitting problem. In Continual Test-Time Adaption stage (right), adaptive balanced loss adjusts the logits by the frequency and accumulated gradient of each class to enhance the learning of unbiased classifiers.
• Figure 3: T-SNE visualization of learned feature on quantized wider-resnet (W2A4), which reveal discriminative capability of classifier.