Singular Value Scaling: Efficient Generative Model Compression via Pruned Weights Refinement

TL;DR

This work tackles the inefficiencies in pruning-based generative-model compression by identifying that pruned weights often contain dominant singular vectors that bias fine-tuning. It introduces Singular Value Scaling (SVS), a lightweight refinement that scales pruned weight singular values to balance contributions from all singular vectors, improving trainability without extra training. SVS is architecture-agnostic and demonstrated to enhance compression performance for StyleGAN2/3 and DDPM, yielding faster convergence and better quality metrics (e.g., FID, P&R, D&C, SSIM) across datasets. The approach is validated through extensive experiments, ablations, and implementation details, highlighting its potential as a general, practical tool for cross-model generative-model compression.

Abstract

While pruning methods effectively maintain model performance without extra training costs, they often focus solely on preserving crucial connections, overlooking the impact of pruned weights on subsequent fine-tuning or distillation, leading to inefficiencies. Moreover, most compression techniques for generative models have been developed primarily for GANs, tailored to specific architectures like StyleGAN, and research into compressing Diffusion models has just begun. Even more, these methods are often applicable only to GANs or Diffusion models, highlighting the need for approaches that work across both model types. In this paper, we introduce Singular Value Scaling (SVS), a versatile technique for refining pruned weights, applicable to both model types. Our analysis reveals that pruned weights often exhibit dominant singular vectors, hindering fine-tuning efficiency and leading to suboptimal performance compared to random initialization. Our method enhances weight initialization by minimizing the disparities between singular values of pruned weights, thereby improving the fine-tuning process. This approach not only guides the compressed model toward superior solutions but also significantly speeds up fine-tuning. Extensive experiments on StyleGAN2, StyleGAN3 and DDPM demonstrate that SVS improves compression performance across model types without additional training costs. Our code is available at: https://github.com/LAIT-CVLab/Singular-Value-Scaling.

Figures (11)

• Figure 1: (a) A schematic overview of our compression framework. Unlike existing model compression scheme, we additionally perform pruned weights refining step for efficient model fine-tuning. First, we prune the pre-trained model. Next, we compute singular values of the weights of the pruned model. Then, we refine the pruned weights by scaling down the singular values with large magnitudes and scaling up the singular values with small magnitudes (Singular Value Scaling, SVS). Finally, the refined pruned model are fine-tuned. (b), (c) The singular value distribution before and after singular value scaling in the pruned weights by the method of DCP-GAN and Diff-Prune. The x-axis represents singular values and the y-axis represents the density. When applied SVS, the singular value disparity reduces, balancing the power of each singular vector in the pruned weights.
• Figure 2: FID convergence graph in different StyleGAN2 Architectures compressed by different methods. The x-axis represents the number of images shown to the discriminator. Solid line represents FID with respect to the training images, while dashed line represents the best FID of the compressed model by the corresponding method. "scaled" means that our Singular Value Scaling is applied to the pruned weights.
• Figure 3: Qualitative results on FFHQ and LSUN Church datasets. Samples in each row are generated from same noise vector with StyleGAN2 (small), which is compressed using different compression methods with channel sparsity 70%. "Ours" denotes the compressed model with DCP-GAN refined by Singular Value Scaling.
• Figure 4: (a), (b) FID convergence graph in different StyleGAN2 Architectures compressed by different methods xu2022mindchung2024diversity in LSUN Church dataset, (c) FID convergence graph in StyleGAN3 Architecture compressed by different methods in FFHQ dataset. Solid line represents FID with respect to the training images, while dashed line represents the best FID of the compressed model by the corresponding method. "scaled" means that our Singular Value Scaling is applied to the pruned weights.
• Figure 5: FID convergence graph in DDPM compressed by different methods fang2023structural in CIFAR10 dataset. Solid line represents FID with respect to the training images, while dashed line represents the best FID of the compressed model by the corresponding method. "pr" denotes the channel sparsity (pruning ratio). "scaled" means that our Singular Value Scaling is applied to the pruned weights.