Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers

Yiming Wang, Jinyu Li

TL;DR

ResidualTransformer tackles memory constraints for speech transformers by sharing a full-rank weight across adjacent layers and adding a per-layer residual composed of a low-rank term and a diagonal matrix. The approach enables substantial encoder compression (roughly 3×) while recovering much of the lost capacity through the residuals, as demonstrated on 10k-hour ASR and speech translation tasks. Key insights include that weight sharing works best when restricted to adjacent layers, and that residual low-rank plus diagonal components significantly close the gap to fully unshared baselines, with rank and diagonal ablations guiding efficient configurations. The results suggest practical, memory-efficient Transformer deployment for large-scale speech systems without requiring specialized hardware or aggressive quantization.

Abstract

Memory constraint of always-on devices is one of the major concerns when deploying speech processing models on these devices. While larger models trained with sufficiently large amount of data generally perform better, making them fit in the device memory is a demanding challenge. In this paper, we aim to reduce model size by reparameterizing model weights across Transformer encoder layers and assuming a special weight composition and structure. More specifically, inspired by ResNet and the more recent LoRA work, we propose an approach named ResidualTransformer, where each weight matrix in a Transformer layer comprises 1) a shared full-rank component with its adjacent layers, and 2) a unique low-rank component to itself. The low-rank matrices only account for a small amount of model size increase. In addition, we add diagonal weight matrices to improve modeling capacity of the low-rank matrices. Experiments of our 10k-hour speech recognition and speech translation tasks show that the Transformer encoder size can be reduced by ~3X with very slight performance degradation.

ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers

TL;DR

ResidualTransformer tackles memory constraints for speech transformers by sharing a full-rank weight across adjacent layers and adding a per-layer residual composed of a low-rank term and a diagonal matrix. The approach enables substantial encoder compression (roughly 3×) while recovering much of the lost capacity through the residuals, as demonstrated on 10k-hour ASR and speech translation tasks. Key insights include that weight sharing works best when restricted to adjacent layers, and that residual low-rank plus diagonal components significantly close the gap to fully unshared baselines, with rank and diagonal ablations guiding efficient configurations. The results suggest practical, memory-efficient Transformer deployment for large-scale speech systems without requiring specialized hardware or aggressive quantization.

Abstract

Memory constraint of always-on devices is one of the major concerns when deploying speech processing models on these devices. While larger models trained with sufficiently large amount of data generally perform better, making them fit in the device memory is a demanding challenge. In this paper, we aim to reduce model size by reparameterizing model weights across Transformer encoder layers and assuming a special weight composition and structure. More specifically, inspired by ResNet and the more recent LoRA work, we propose an approach named ResidualTransformer, where each weight matrix in a Transformer layer comprises 1) a shared full-rank component with its adjacent layers, and 2) a unique low-rank component to itself. The low-rank matrices only account for a small amount of model size increase. In addition, we add diagonal weight matrices to improve modeling capacity of the low-rank matrices. Experiments of our 10k-hour speech recognition and speech translation tasks show that the Transformer encoder size can be reduced by ~3X with very slight performance degradation.
Paper Structure (16 sections, 3 equations, 1 figure, 6 tables)

This paper contains 16 sections, 3 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Transformers
  5. Weight Sharing across Layers
  6. Adding Residual Matrices to Shared Weights
  7. Experiments
  8. Datasets
  9. Models
  10. Training
  11. Results
  12. Weight Sharing
  13. Residual Weights
  14. Rank of Matrices
  15. Ablation Study: Diagonal Matrices
  16. ...and 1 more sections

Figures (1)

  • Figure 1: The proposed weight structure for each weight matrix in a Transformer layer. The layer index $l$ is omitted in the notations. $W$ is a full-rank matrix being shared with other layers, $A$ and $B$ are two low-rank decomposed matrices, and $D$ is a diagonal matrix.