Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Maestro: Uncovering Low-Rank Structures via Trainable Decomposition

Samuel Horvath, Stefanos Laskaridis, Shashank Rajput, Hongyi Wang

TL;DR

Maestro addresses the escalating cost of large DNNs by embedding a data-driven, trainable low-rank decomposition into training via Low-rank Ordered Decomposition (LoD). It factorizes each layer as $W^i = U^i (V^i)^\top$ with per-layer ranks $r_i$, trained through rank-sampling and hierarchical group-lasso to progressively shrink the model while preserving accuracy. The approach yields theoretical guarantees that LoD recovers classical decompositions (SVD/PCA) in special cases and demonstrates empirical superiority over SVD-based baselines across vision and language tasks, with notable reductions in training cost and model footprint. The results show Maestro enables train-once, deploy-everywhere deployment with graceful accuracy-latency trade-offs, making it attractive for edge devices and constrained environments. Overall, Maestro presents a principled, data-aware framework for compact, high-performance DNNs that integrates compression into the training process.

Abstract

Deep Neural Networks (DNNs) have been a large driver for AI breakthroughs in recent years. However, these models have been getting increasingly large as they become more accurate and safe. This means that their training becomes increasingly costly and time-consuming and typically yields a single model to fit all targets. Various techniques have been proposed in the literature to mitigate this, including pruning, sparsification, or quantization of model weights and updates. While achieving high compression rates, they often incur significant computational overheads at training or lead to non-negligible accuracy penalty. Alternatively, factorization methods have been leveraged for low-rank compression of DNNs. Similarly, such techniques (e.g., SVD) frequently rely on heavy iterative decompositions of layers and are potentially sub-optimal for non-linear models, such as DNNs. We take a further step in designing efficient low-rank models and propose Maestro, a framework for trainable low-rank layers. Instead of iteratively applying a priori decompositions, the low-rank structure is baked into the training process through LoD, a low-rank ordered decomposition. Not only is this the first time importance ordering via sampling is applied on the decomposed DNN structure, but it also allows selecting ranks at a layer granularity. Our theoretical analysis demonstrates that in special cases LoD recovers the SVD decomposition and PCA. Applied to DNNs, Maestro enables the extraction of lower footprint models that preserve performance. Simultaneously, it enables the graceful trade-off between accuracy-latency for deployment to even more constrained devices without retraining.

Maestro: Uncovering Low-Rank Structures via Trainable Decomposition

TL;DR

Maestro addresses the escalating cost of large DNNs by embedding a data-driven, trainable low-rank decomposition into training via Low-rank Ordered Decomposition (LoD). It factorizes each layer as with per-layer ranks , trained through rank-sampling and hierarchical group-lasso to progressively shrink the model while preserving accuracy. The approach yields theoretical guarantees that LoD recovers classical decompositions (SVD/PCA) in special cases and demonstrates empirical superiority over SVD-based baselines across vision and language tasks, with notable reductions in training cost and model footprint. The results show Maestro enables train-once, deploy-everywhere deployment with graceful accuracy-latency trade-offs, making it attractive for edge devices and constrained environments. Overall, Maestro presents a principled, data-aware framework for compact, high-performance DNNs that integrates compression into the training process.

Abstract

Deep Neural Networks (DNNs) have been a large driver for AI breakthroughs in recent years. However, these models have been getting increasingly large as they become more accurate and safe. This means that their training becomes increasingly costly and time-consuming and typically yields a single model to fit all targets. Various techniques have been proposed in the literature to mitigate this, including pruning, sparsification, or quantization of model weights and updates. While achieving high compression rates, they often incur significant computational overheads at training or lead to non-negligible accuracy penalty. Alternatively, factorization methods have been leveraged for low-rank compression of DNNs. Similarly, such techniques (e.g., SVD) frequently rely on heavy iterative decompositions of layers and are potentially sub-optimal for non-linear models, such as DNNs. We take a further step in designing efficient low-rank models and propose Maestro, a framework for trainable low-rank layers. Instead of iteratively applying a priori decompositions, the low-rank structure is baked into the training process through LoD, a low-rank ordered decomposition. Not only is this the first time importance ordering via sampling is applied on the decomposed DNN structure, but it also allows selecting ranks at a layer granularity. Our theoretical analysis demonstrates that in special cases LoD recovers the SVD decomposition and PCA. Applied to DNNs, Maestro enables the extraction of lower footprint models that preserve performance. Simultaneously, it enables the graceful trade-off between accuracy-latency for deployment to even more constrained devices without retraining.
Paper Structure (29 sections, 2 theorems, 18 equations, 15 figures, 16 tables)

This paper contains 29 sections, 2 theorems, 18 equations, 15 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Maestro
  4. Formulation
  5. Layer Factorization
  6. Training Techniques
  7. Train-Once, Deploy-Everywhere
  8. Theoretical Guarantees
  9. Experiments
  10. Experimental Setup
  11. Performance Comparison
  12. Training Behaviour of Maestro
  13. Ablation Study
  14. Accuracy-Latency Trade-Off at Training and Deployment Time
  15. Conclusion and Future Work
  16. ...and 14 more sections

Key Result

Theorem 4.1

Let $A = \tilde{U} \tilde{\Sigma} \tilde{V}^\top$ be a SVD decomposition of $A$. Then, the minimization problem eq:low_rank_optim is equivalent to PCA applied to the transformed dataset $x \rightarrow \tilde{\Sigma} \tilde{V}^\top x$, $x \sim {\cal X}$ projected on the column space of $\tilde{U}$.

Figures (15)

  • Figure 1: Maestro's construction. To obtain low-rank approximation, the given linear map is decomposed and trained with LoD to obtain an ordered representation that can be efficiently pruned.
  • Figure 2: Maestro (Training Process)
  • Figure 3: Maestro (Hyper-parameter optimization)
  • Figure 4: Empirical showcase of theoretical properties of the Maestro's formulation.
  • Figure 5: Maestro vs. baselines on CIFAR10. Spectral-Init results is taken from the original work; For XNOR-Net each weight is quantized from 32 to 1-bit. Thus, we report a compression rate of $3.125\%$; Detailed results are presented in table form in the Appendix \ref{['app:detailed_baselines']}.
  • ...and 10 more figures

Theorems & Definitions (3)

  • Theorem 4.1: Informal
  • Theorem 3.1
  • proof