An Empirical Investigation of Matrix Factorization Methods for Pre-trained Transformers

TL;DR

This study investigates factorization-based compression for pretrained transformers, comparing simple Low-Rank Factorization against Monarch and related approaches. It reveals that naïve projection often causes training instability, which can be mitigated by a Staged Factorization strategy that compresses layers progressively. Across six GLUE tasks and multiple pretrained models, Low-Rank Factorization consistently achieves higher accuracy and lower latency than Monarch, challenging claims of Monarch's superiority. The work highlights the prevalence of global redundancies exploitable by simple low-rank forms and shows that stability depends on model and data, with T5 displaying particular robustness to factorization. Overall, the results advocate using simple, stable low-rank factorization for practical compressed transformers.

Abstract

The increasing size of transformer-based models in NLP makes the question of compressing them important. In this work, we present a comprehensive analysis of factorization based model compression techniques. Specifically, we focus on comparing straightforward low-rank factorization against the recently introduced Monarch factorization, which exhibits impressive performance preservation on the GLUE benchmark. To mitigate stability issues associated with low-rank factorization of the matrices in pre-trained transformers, we introduce a staged factorization approach wherein layers are factorized one by one instead of being factorized simultaneously. Through this strategy we significantly enhance the stability and reliability of the compression process. Further, we introduce a simple block-wise low-rank factorization method, which has a close relationship to Monarch factorization. Our experiments lead to the surprising conclusion that straightforward low-rank factorization consistently outperforms Monarch factorization across both different compression ratios and six different text classification tasks.

Figures (6)

• Figure 1: The dotted line represents traditional fine-tuning for the non-factorized model initialized with pretrained weights. The solid line signifies an intermediate step involving the projection of pretrained weights onto factorized matrices, followed by task-specific fine-tuning.
• Figure 2: Comparison of the training loss of the factorized and unfactorized models. The loss curves are shown for an unfactorized BERT model vs factorized models (25% parameters) trained on the MNLI dataset containing three labels with initial learning rate of 2e-5.
• Figure 3: Relative stability of factorization methods with and without staging. The results shown here are for compressing the BERT (base) model.
• Figure 4: Relative stability of factorization methods across low and high fine-tuning data settings.
• Figure 5: Reconstruction Error vs No. of Singular Values. The matrix on the left is the query matrix from the first layer of the pre-trained BERT model( bert-base-uncased), and the right is for the query matrix of the last layer (12th). As can been seen, pre-trained matrices in these pre-trained models are close to full-rank.