Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Deep State-Space Model Compression Method using Upper Bound on Output Error

Hiroki Sakamoto, Kazuhiro Sato

TL;DR

This work tackles compressing Deep State-Space Models that internally embed Linear Quadratic-Output (LQO) blocks by deriving a provable upper bound on the output error $e_{\xi}$ and showing it is controlled by per-layer time-limited $H^2$ norms $\|\mathcal{S}^{(i)}-\hat{\mathcal{S}}^{(i)}\|_{h^2_L}$. It introduces a gradient-based Model Order Reduction (MOR) procedure that minimizes the bound $f(\hat{\mathbf{S}})=\sum_{i=1}^{\xi} \tilde{G}_i \|\mathcal{S}^{(i)}-\hat{\mathcal{S}}^{(i)}\|_{h^2_L}$ under stability and structural constraints, thereby preserving inter-layer interactions. The method yields highly compressed models with strong end-to-end performance on the IMDb task within the Long Range Arena, achieving roughly an $80\%$ reduction in trainable parameters with only about a 4–5% drop in accuracy, and in some cases even outperforming retrained baselines. This provides both a theoretical justification for MOR-based compression of Deep SSMs and a practical approach for low-cost deployment under compute constraints.

Abstract

We study deep state-space models (Deep SSMs) that contain linear-quadratic-output (LQO) systems as internal blocks and present a compression method with a provable output error guarantee. We first derive an upper bound on the output error between two Deep SSMs and show that the bound can be expressed via the $h^2$-error norms between the layerwise LQO systems, thereby providing a theoretical justification for existing model order reduction (MOR)-based compression. Building on this bound, we formulate an optimization problem in terms of the $h^2$-error norm and develop a gradient-based MOR method. On the IMDb task from the Long Range Arena benchmark, we demonstrate that our compression method achieves strong performance. Moreover, unlike prior approaches, we reduce roughly 80% of trainable parameters without retraining, with only a 4-5% performance drop.

A Deep State-Space Model Compression Method using Upper Bound on Output Error

TL;DR

This work tackles compressing Deep State-Space Models that internally embed Linear Quadratic-Output (LQO) blocks by deriving a provable upper bound on the output error and showing it is controlled by per-layer time-limited norms . It introduces a gradient-based Model Order Reduction (MOR) procedure that minimizes the bound under stability and structural constraints, thereby preserving inter-layer interactions. The method yields highly compressed models with strong end-to-end performance on the IMDb task within the Long Range Arena, achieving roughly an reduction in trainable parameters with only about a 4–5% drop in accuracy, and in some cases even outperforming retrained baselines. This provides both a theoretical justification for MOR-based compression of Deep SSMs and a practical approach for low-cost deployment under compute constraints.

Abstract

We study deep state-space models (Deep SSMs) that contain linear-quadratic-output (LQO) systems as internal blocks and present a compression method with a provable output error guarantee. We first derive an upper bound on the output error between two Deep SSMs and show that the bound can be expressed via the -error norms between the layerwise LQO systems, thereby providing a theoretical justification for existing model order reduction (MOR)-based compression. Building on this bound, we formulate an optimization problem in terms of the -error norm and develop a gradient-based MOR method. On the IMDb task from the Long Range Arena benchmark, we demonstrate that our compression method achieves strong performance. Moreover, unlike prior approaches, we reduce roughly 80% of trainable parameters without retraining, with only a 4-5% performance drop.
Paper Structure (12 sections, 8 theorems, 42 equations, 2 figures, 1 table, 1 algorithm)

This paper contains 12 sections, 8 theorems, 42 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Discrete-time complex LQO systems
  3. Problem setting
  4. Deep SSMs with Linear Quadratic-Output Systems
  5. MOR-Based Compression of Deep SSMs
  6. Output Error Analysis for Deep SSMs
  7. Model reduction for Deep SSM
  8. Model Order Reduction Problem
  9. Gradients
  10. Gradient-Based Algorithm
  11. Numerical experiments
  12. Summary

Key Result

Lemma 1

Let $u=(u_k)_{k=0}^{L-1}\in \ell^2_{L}$. Then, the following inequality holds:

Figures (2)

  • Figure 1: Comparison between existing MOR methods and the proposed method for $\xi$-layer Deep SSMs. Existing methods perform MOR independently on the linear time-invariant (LTI) subsystems within a Deep SSM and produce $\hat{s}_{\mathrm{out,ex}}$. In contrast, the proposed method constructs a reduced Deep SSM with reduced LQO systems $\{\hat{\mathcal{S}}^{(i)}\}_{i=1}^{\xi}$ that approximates the pretrained Deep SSM’s output $s_{\mathrm{out}}$ while accounting for inter-layer interactions, the nonlinearity $\sigma$, and LayerNorm (LN); in particular, it minimizes $\|s_{\mathrm{out}}-\hat{s}_{\mathrm{out,pro}}\|_{\ell^{\infty}_L}$ for a given input sequence $s_{\mathrm{in}}$.
  • Figure 2: Structure of the Deep SSM considered in this work.

Theorems & Definitions (24)

  • Remark 1
  • Lemma 1
  • proof
  • Theorem 1
  • proof
  • Corollary 1
  • proof
  • Lemma 2
  • proof
  • Remark 2
  • ...and 14 more