R2 Loss: Range Restriction Loss for Model Compression and Quantization

TL;DR

The paper addresses the challenge of ultra-low-bit quantization and compression by focusing on weight outliers that inflate weight ranges. It proposes Range Restriction Loss (R2-Loss) to regularize weights during pre-training, producing quantization- and compression-friendly distributions; three variants—$L_\infty$ R2-Loss, Margin R2-Loss, and Soft-Min-Max R2-Loss—cater to different downstream regimes. Through extensive experiments on ImageNet with architectures like MobileNet-V1/V2 and ResNet-18, the authors demonstrate consistent improvements across PTQ, QAT, and DKM-based compression, including notable gains for 2-bit weight/8-bit activation and 1- to 2-bit compression scenarios, as well as cross-domain results on MobileBERT. The findings suggest that R2-Loss serves as a robust, architecture-agnostic pre-training regularizer that augments state-of-the-art quantization and compression methods, enabling more accurate ultra-low-bit deployments on edge devices. The work highlights practical implications for reducing deployment costs and widening the accessibility of efficient deep learning models without sacrificing essential performance gains.

Abstract

Model quantization and compression is widely used techniques to reduce usage of computing resource at inference time. While state-of-the-art works have been achieved reasonable accuracy with higher bit such as 4bit or 8bit, but still it is challenging to quantize/compress a model further, e.g., 1bit or 2bit. To overcome the challenge, we focus on outliers in weights of a pre-trained model which disrupt effective lower bit quantization and compression. In this work, we propose Range Restriction Loss (R2-Loss) for building lower bit quantization and compression friendly models by removing outliers from weights during pre-training. By effectively restricting range of weights, we mold the overall distribution into a tight shape to ensure high quantization bit resolution, therefore allowing model compression and quantization techniques can to utilize their limited numeric representation powers better. We introduce three different, L-inf R2-Loss, its extension Margin R2-Loss and a new Soft-Min-MaxR2-Loss to be used as an auxiliary loss during full-precision model training. These R2-Loss can be used in different cases such as L-inf and Margin R2-Loss would be effective for symmetric quantization, while Soft-Min-Max R2-Loss shows better performance for model compression. In our experiment, R2-Loss improves lower bit quantization accuracy with state-of-the-art post-training quantization (PTQ), quantization-aware training (QAT), and model compression techniques. With R2-Loss, MobileNet-V2 2bit weight and 8bit activation PTQ, MobileNet-V1 2bit weight and activation QAT, ResNet18 1bit weight compression are improved to 59.49% from 50.66%, 59.05% from 55.96%, and 52.58% from 45.54%, respectively.

Figures (3)

• Figure 1: (Left) Example weight distribution for weight quantization with three bins. (Right) Expected benefit of for low bit quantization/compression.
• Figure 2: Weight distribution of the first five 3x3 convolution layers and FC layer of MobileNet-V2 using L2 norm (baseline), heavy L2 norm (10x heavy L2 norm than baseline) and the proposed (the red dots correspond outliers). KURE: Kurtosis Regularization kure.
• Figure 3: MobileNet-V2 result varying strength of . 4bit weight and 8bit activation PTQ with AdaRound and SQuant.