Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference

Changwoo Lee, Soo Min Kwon, Qing Qu, Hun-Seok Kim

TL;DR

The efficiency of using the BLAST matrix for compressing both language and vision tasks is demonstrated, showing that for medium-sized models such as ViT and GPT-2, training with BLAST weights boosts performance while reducing complexity by 70% and 40%, respectively.

Abstract

Large-scale foundation models have demonstrated exceptional performance in language and vision tasks. However, the numerous dense matrix-vector operations involved in these large networks pose significant computational challenges during inference. To address these challenges, we introduce the Block-Level Adaptive STructured (BLAST) matrix, designed to learn and leverage efficient structures prevalent in the weight matrices of linear layers within deep learning models. Compared to existing structured matrices, the BLAST matrix offers substantial flexibility, as it can represent various types of structures that are either learned from data or computed from pre-existing weight matrices. We demonstrate the efficiency of using the BLAST matrix for compressing both language and vision tasks, showing that (i) for medium-sized models such as ViT and GPT-2, training with BLAST weights boosts performance while reducing complexity by 70% and 40%, respectively; and (ii) for large foundation models such as Llama-7B and DiT-XL, the BLAST matrix achieves a 2x compression while exhibiting the lowest performance degradation among all tested structured matrices. Our code is available at https://github.com/changwoolee/BLAST.

BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference

TL;DR

The efficiency of using the BLAST matrix for compressing both language and vision tasks is demonstrated, showing that for medium-sized models such as ViT and GPT-2, training with BLAST weights boosts performance while reducing complexity by 70% and 40%, respectively.

Abstract

Large-scale foundation models have demonstrated exceptional performance in language and vision tasks. However, the numerous dense matrix-vector operations involved in these large networks pose significant computational challenges during inference. To address these challenges, we introduce the Block-Level Adaptive STructured (BLAST) matrix, designed to learn and leverage efficient structures prevalent in the weight matrices of linear layers within deep learning models. Compared to existing structured matrices, the BLAST matrix offers substantial flexibility, as it can represent various types of structures that are either learned from data or computed from pre-existing weight matrices. We demonstrate the efficiency of using the BLAST matrix for compressing both language and vision tasks, showing that (i) for medium-sized models such as ViT and GPT-2, training with BLAST weights boosts performance while reducing complexity by 70% and 40%, respectively; and (ii) for large foundation models such as Llama-7B and DiT-XL, the BLAST matrix achieves a 2x compression while exhibiting the lowest performance degradation among all tested structured matrices. Our code is available at https://github.com/changwoolee/BLAST.

Paper Structure

This paper contains 57 sections, 3 theorems, 48 equations, 15 figures, 13 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Notation and Organization.
  3. Block-Level Adaptive Structured (BLAST) Matrix
  4. Low-Rank Matrices as Special Cases of BLAST
  5. Matrix Multiplication
  6. Applications of BLAST Matrices
  7. Training from Scratch using BLAST Matrices
  8. Compressing Weights via BLAST Factorization
  9. BLAST Factorization via Gradient Descent
  10. Blast Factorization via Preconditioned Gradient Descent
  11. Experimental Results
  12. Training from Scratch
  13. Image Classification
  14. Language Model Evaluation
  15. Compression and Re-training
  16. ...and 42 more sections

Key Result

Theorem 1

Let ${\bm{A}}_{i,j}\in\mathbb{R}^{p\times p}$ be a target block and ${\bm{U}}_i^{(k)},{\bm{V}}_j^{(k)}\in\mathbb{R}^{p\times r}$, and ${\bm{s}}_{i,j}^{(k)}\in\mathbb{R}^{r}$ be factors of a block in the BLAST matrix to be optimized. With the step sizes $0<\eta_{{\bm{U}}_i^{(k)}}\le 1/\sigma_1\left(\

Figures (15)

  • Figure 1: Examples of generated images using DiT peebles2023scalable starting from the same noise vectors and a deterministic solver. The original model is compressed by 50% through BLAST or low-rank matrices and re-trained for 10 epochs on ImageNet. The images from the model compressed via BLAST preserve the quality of the images of the original model, whereas the images generated by the low-rank model contain more undesired artifacts.
  • Figure 2: Existing structured matrices and our proposed BLAST matrix. The unique structure of BLAST allows for flexible matrix structures while enabling faster matrix multiplication compared to existing matrices.
  • Figure 3: Convergence of the BLAST factorization with and without the preconditioning steps on noiseless low-rank matrix factorization with rank $r^\star$. Left: The BLAST parameter $r = r^{\star}$, Right: $r > r^{\star}$. When $r>r^*$, the convergence rate of GD without the preconditioning is slowed down, while GD with the preconditioning (PrecGD) can recover the ground truth with small error.
  • Figure 4: CIFAR-10/100 accuracy of ViT-S trained from scratch with different structured matrices.
  • Figure 5: Pre-training result: WikiText 103 test perplexity-FLOPs trade-off curves from GPT-2 with different types of weight matrices.
  • ...and 10 more figures

Theorems & Definitions (6)

  • Theorem 1
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • proof