Dynamic Base model Shift for Delta Compression

TL;DR

Delta compression reduces storage by encoding the difference between finetuned and pre-trained weights, but using a fixed pretrained base can degrade performance at high compression. This work introduces Dynamic Base Model Shift (DBMS), which per task constructs a shifted base $W_{base}^t = W_{pre} + λ_1^t \cdot τ_{base}$ with $τ_{base} = BitDelta\left(\frac{1}{N}\sum_{i=1}^N W_i - W_{pre}\right)$ and then scales the delta with $δ_t' = λ_2^t \cdot C(W_t - W_{base}^t)$; two task-specific scalars $λ_1^t$ and $λ_2^t$ are learned by low-cost training on a 10% unlabeled subset to minimize $L = \mathrm{MSE}(W_t'(x), W_t(x))$, where $W_t' = W_{pre} + λ_1^t \cdot τ_{base} + λ_2^t \cdot C(W_t - W_{pre} - λ_1^t \cdot τ_{base})$. Across language, vision, and multi-modal models, DBMS substantially improves performance under extreme compression, is compatible with both pruning- and quantization-based delta methods, and incurs only modest storage overhead. This suggests a practical path to maintain finetuned-model fidelity while achieving very high compression in diverse downstream settings.

Abstract

Transformer-based models with the pretrain-finetune paradigm bring about significant progress, along with the heavy storage and deployment costs of finetuned models on multiple tasks. Delta compression attempts to lower the costs by reducing the redundancy of delta parameters (i.e., the difference between the finetuned and pre-trained model weights) through pruning or quantization. However, existing methods by default employ the pretrained model as the base model and compress the delta parameters for every task, which may causes significant performance degradation, especially when the compression rate is extremely high. To tackle this issue, we investigate the impact of different base models on the performance of delta compression and find that the pre-trained base model can hardly be optimal. To this end, we propose Dynamic Base Model Shift (DBMS), which dynamically adapts the base model to the target task before performing delta compression. Specifically, we adjust two parameters, which respectively determine the magnitude of the base model shift and the overall scale of delta compression, to boost the compression performance on each task. Through low-cost learning of these two parameters, our DBMS can maintain most of the finetuned model's performance even under an extremely high compression ratio setting, significantly surpassing existing methods. Moreover, our DBMS is orthogonal and can be integrated with a variety of other methods, and it has been evaluated across different types of models including language, vision transformer, and multi-modal models.

Figures (4)

• Figure 1: Comparison between vanilla delta compression methods (left) and our DBMS (right), which first (a) obtains a compressed base vector, then we (b) fuse the base vector with the ajusted magnitude to the pretrained model to realize dynamic base model shift. Finally we (c) further dynamically adjust the scales of delta parameters to boost the performance.
• Figure 2: Performance heatmap of compressing RoBERTa models finetuned on the GLUE wang2018glue benchmark under the setting of different $\lambda_1$ and $\lambda_2$ values. We can observe that the original pre-trained base model paradigm ($\lambda_1=0$ and $\lambda_2=1$, which is marked by the intersection of the green dashed lines) rarely achieves optimality, and the optimal values of $\lambda_1$ and $\lambda_2$ vary across different datasets.
• Figure 3: The loss curve of DBMS using different initaliztion strategies, respectively ours and ones-initialization.
• Figure : Algorithm Flow of DBMS