Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeltaDQ: Ultra-High Delta Compression for Fine-Tuned LLMs via Group-wise Dropout and Separate Quantization

Yanfeng Jiang, Zelan Yang, Bohua Chen, Shen Li, Yong Li, Tao Li

TL;DR

DeltaDQ targets efficient deployment of multiple fine-tuned LLMs by compressing only the delta between each model and a common base model. It leverages Balanced Intermediate Results to justify Group-wise Dropout and a subsequent Separate Quantization stage, enabling ultra-high compression while preserving accuracy. Empirical results demonstrate up to $\8$-fold or more improvements in some regimes and up to $128\times$ for WizardMath-7B and $512\times$ for WizardMath-70B, with competitive or improved task performance. The approach reduces deployment overhead by storing the base model once and reconstructing each delta at inference through a staged, sparse, low-bit representation, offering practical impact for large-scale multi-model serving.

Abstract

Large language models achieve exceptional performance on various downstream tasks through supervised fine-tuning. However, the diversity of downstream tasks and practical requirements makes deploying multiple full-parameter fine-tuned models challenging. Current methods that compress the delta weight struggle to achieve ultra-high compression, failing to minimize the deployment overhead. To address the above issue, we propose a novel distribution-driven delta compression framework DeltaDQ, which utilizes Group-wise Dropout and Separate Quantization to achieve ultra-high compression for the delta weight. We have observed that the matrix-computed intermediate results for the delta weight exhibit extremely small variance and min-max range characteristics, referred to as Balanced Intermediate Results. Exploiting this phenomenon, we introduce Group-wise Dropout to perform dropout on the delta weight using an optimal group size. Furthermore, using Separate Quantization, sparse weights are quantized and decomposed to achieve a lower bit. Experimental results show that DeltaDQ achieves 16x compression with improved accuracy compared to baselines for WizardMath and WizardCoder models across different parameter scales. Moreover, DeltaDQ demonstrates the ability for ultra-high compression ratio, achieving 128x compression for the WizardMath-7B model and 512x compression for the WizardMath-70B model.

DeltaDQ: Ultra-High Delta Compression for Fine-Tuned LLMs via Group-wise Dropout and Separate Quantization

TL;DR

DeltaDQ targets efficient deployment of multiple fine-tuned LLMs by compressing only the delta between each model and a common base model. It leverages Balanced Intermediate Results to justify Group-wise Dropout and a subsequent Separate Quantization stage, enabling ultra-high compression while preserving accuracy. Empirical results demonstrate up to -fold or more improvements in some regimes and up to for WizardMath-7B and for WizardMath-70B, with competitive or improved task performance. The approach reduces deployment overhead by storing the base model once and reconstructing each delta at inference through a staged, sparse, low-bit representation, offering practical impact for large-scale multi-model serving.

Abstract

Large language models achieve exceptional performance on various downstream tasks through supervised fine-tuning. However, the diversity of downstream tasks and practical requirements makes deploying multiple full-parameter fine-tuned models challenging. Current methods that compress the delta weight struggle to achieve ultra-high compression, failing to minimize the deployment overhead. To address the above issue, we propose a novel distribution-driven delta compression framework DeltaDQ, which utilizes Group-wise Dropout and Separate Quantization to achieve ultra-high compression for the delta weight. We have observed that the matrix-computed intermediate results for the delta weight exhibit extremely small variance and min-max range characteristics, referred to as Balanced Intermediate Results. Exploiting this phenomenon, we introduce Group-wise Dropout to perform dropout on the delta weight using an optimal group size. Furthermore, using Separate Quantization, sparse weights are quantized and decomposed to achieve a lower bit. Experimental results show that DeltaDQ achieves 16x compression with improved accuracy compared to baselines for WizardMath and WizardCoder models across different parameter scales. Moreover, DeltaDQ demonstrates the ability for ultra-high compression ratio, achieving 128x compression for the WizardMath-7B model and 512x compression for the WizardMath-70B model.

Paper Structure

This paper contains 14 sections, 12 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model Compression for LLMs
  4. Multi Fine-Tuned Model Deployment
  5. Methodology
  6. Preliminaries: Delta Compression
  7. Balanced Intermediate Results
  8. Group-wise Dropout
  9. Separate Quantization
  10. Experiment
  11. Setup
  12. Main Results
  13. Analysis
  14. Conclusion

Figures (8)

  • Figure 1: Difference between conventional model compression and delta compression.
  • Figure 2: Overview of our delta compression framework DeltaDQ. Our framework is divided into four steps; Step 1: Split Weight; Step 2: Group-wise Dropout; Step 3: Separate Quantization; Step 4: Deployment.
  • Figure 3: Illustration of Separate Computation. Calculating base weight and delta weight separately in the Linear layer, followed by synchronization.
  • Figure 4: Comparison of the variance and min-max range distribution of the intermediate results of the delta weight and fine-tuned weight for each output element matrix computation.
  • Figure 5: Impact of different group sizes on the accuracy of WizardMath-7B at the same compression ratio.
  • ...and 3 more figures