Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hankel Singular Value Regularization for Highly Compressible State Space Models

Paul Schwerdtner, Jules Berman, Benjamin Peherstorfer

TL;DR

To make the proposed Hankel singular value regularization scalable, an algorithm is developed to efficiently compute the Hankel singular values during training iterations by exploiting the specific block-diagonal structure of the system matrices that are used in the state space model parametrization.

Abstract

Deep neural networks using state space models as layers are well suited for long-range sequence tasks but can be challenging to compress after training. We use that regularizing the sum of Hankel singular values of state space models leads to a fast decay of these singular values and thus to compressible models. To make the proposed Hankel singular value regularization scalable, we develop an algorithm to efficiently compute the Hankel singular values during training iterations by exploiting the specific block-diagonal structure of the system matrices that we use in our state space model parametrization. Experiments on Long Range Arena benchmarks demonstrate that the regularized state space layers are up to 10$\times$ more compressible than standard state space layers while maintaining high accuracy.

Hankel Singular Value Regularization for Highly Compressible State Space Models

TL;DR

To make the proposed Hankel singular value regularization scalable, an algorithm is developed to efficiently compute the Hankel singular values during training iterations by exploiting the specific block-diagonal structure of the system matrices that are used in the state space model parametrization.

Abstract

Deep neural networks using state space models as layers are well suited for long-range sequence tasks but can be challenging to compress after training. We use that regularizing the sum of Hankel singular values of state space models leads to a fast decay of these singular values and thus to compressible models. To make the proposed Hankel singular value regularization scalable, we develop an algorithm to efficiently compute the Hankel singular values during training iterations by exploiting the specific block-diagonal structure of the system matrices that we use in our state space model parametrization. Experiments on Long Range Arena benchmarks demonstrate that the regularized state space layers are up to 10 more compressible than standard state space layers while maintaining high accuracy.
Paper Structure (39 sections, 4 theorems, 20 equations, 4 figures, 8 tables, 1 algorithm)

This paper contains 39 sections, 4 theorems, 20 equations, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Compressing state space models
  3. System-theoretic perspective on SSM compressibility
  4. Literature review
  5. Models with state space layers
  6. Compression and distillation of SSMs
  7. Our approach and summary of contributions
  8. Hankel singular value regularization (HSVR)
  9. Parametrization of SSMs for scalable computation of Hankel singular values
  10. Time-discrete SSMs and their Hankel singular values
  11. Parametrization via rotation matrices
  12. Enforcing stability
  13. Scalable training procedure
  14. Leveraging block-diagonal structure for computing singular values
  15. Fast time integration with associative scan
  16. ...and 24 more sections

Key Result

Proposition 1

For any linear time-invariant system of order $n$ there exists an infinitesimal perturbation such that the sequence-to-sequence map $\{\boldsymbol{u}_k\}_{k = 0}^{\infty} \mapsto \{\boldsymbol{y}_k\}_{k = 0}^{\infty}$ of the perturbed model can be described by an SSM with matrices of the form eq:ou

Figures (4)

  • Figure 1: We propose to regularize the Hankel singular values of SSMs so that they become compressible. Left: Regularizing with the Hankel singular values during training leads to SSMs with a fast Hankel singular value (HSV) decay. Middle: SSMs with a fast HSV decay have many low-energy states that only contribute little to the layer output. Right: Compressing the SSM by truncating only such low-energy states changes the corresponding sequence-to-sequence map insignificantly and retains the overall accuracy. Without our regularization, HSVs decay slowly and compression leads to an accuracy deterioration.
  • Figure 2: Regularizing Hankel singular values leads to highly compressible SSMs while maintaining accuracy.
  • Figure 3: Our HSVR approach trains SSMs that have favorably Hankel singular value decay for compression.
  • Figure 4: Comparison of HSVR to all methods in GwakMKP2024Layer-Adaptive

Theorems & Definitions (6)

  • Proposition 1
  • Proposition 2
  • Proposition
  • proof
  • Proposition
  • proof