Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SaFARi: State-Space Models for Frame-Agnostic Representation

Hossein Babaei, Mel White, Sina Alemohammad, Richard G. Baraniuk

TL;DR

SaFARi extends the HiPPO framework by enabling state-space models for frame-agnostic representations, allowing online approximation with any frame or basis. It introduces scaled and translated uniform measures, derives corresponding SSMs, and provides truncation-based finite-dimensional constructions (ToD and DoT) with a demonstrated optimal DoT variant. The paper offers a rigorous error analysis distinguishing truncation and mixing errors, plus complexity and kernel-based acceleration results, making frame-agnostic SSMs practical for long-range sequence modeling. This framework expands the applicability of SSM-based approaches to diverse bases and paves the way for integrating frame-agnostic representations into architectures like S4 and Mamba, with potential gains in modeling long-range dependencies.

Abstract

State-Space Models (SSMs) have re-emerged as a powerful tool for online function approximation, and as the backbone of machine learning models for long-range dependent data. However, to date, only a few polynomial bases have been explored for this purpose, and the state-of-the-art implementations were built upon the best of a few limited options. In this paper, we present a generalized method for building an SSM with any frame or basis, rather than being restricted to polynomials. This framework encompasses the approach known as HiPPO, but also permits an infinite diversity of other possible "species" within the SSM architecture. We dub this approach SaFARi: SSMs for Frame-Agnostic Representation.

SaFARi: State-Space Models for Frame-Agnostic Representation

TL;DR

SaFARi extends the HiPPO framework by enabling state-space models for frame-agnostic representations, allowing online approximation with any frame or basis. It introduces scaled and translated uniform measures, derives corresponding SSMs, and provides truncation-based finite-dimensional constructions (ToD and DoT) with a demonstrated optimal DoT variant. The paper offers a rigorous error analysis distinguishing truncation and mixing errors, plus complexity and kernel-based acceleration results, making frame-agnostic SSMs practical for long-range sequence modeling. This framework expands the applicability of SSM-based approaches to diverse bases and paves the way for integrating frame-agnostic representations into architectures like S4 and Mamba, with potential gains in modeling long-range dependencies.

Abstract

State-Space Models (SSMs) have re-emerged as a powerful tool for online function approximation, and as the backbone of machine learning models for long-range dependent data. However, to date, only a few polynomial bases have been explored for this purpose, and the state-of-the-art implementations were built upon the best of a few limited options. In this paper, we present a generalized method for building an SSM with any frame or basis, rather than being restricted to polynomials. This framework encompasses the approach known as HiPPO, but also permits an infinite diversity of other possible "species" within the SSM architecture. We dub this approach SaFARi: SSMs for Frame-Agnostic Representation.
Paper Structure (39 sections, 18 theorems, 114 equations, 7 figures, 1 table)

This paper contains 39 sections, 18 theorems, 114 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. State-space models
  4. HiPPO: High-order Polynomial Projection Operators
  5. Limitations of HiPPO
  6. Mathematical Preliminaries
  7. Function representation using frames
  8. SaFARi
  9. Formulation
  10. Uniform scaled measure
  11. Uniform translated measure
  12. SaFARi as generalization of HiPPO
  13. Implementation Considerations
  14. Truncation of frames
  15. Finite-dimensional approximation of SaFARi
  16. ...and 24 more sections

Key Result

Theorem 1

For the representation defined in Eq. Scaling_representation, the partial derivative of $\vec{c}$ with respect to $T$ is where $B$ is the complex conjugate of a vector containing members of the main frame evaluated at $T=1$ so that $B= \{ \overline{\phi}_n (T=1) \}_{n \in \Gamma}$. The $A$ operator can also be described as a matrix

Figures (7)

  • Figure 1: An SSM block-diagram, with the necessary ODE update step included.
  • Figure 2: Two different uniform measures (red) applied to a signal (blue). The red shaded area demonstrates how the measure changes as time evolves and more samples of the input are observed.
  • Figure 3: HiPPO provides a closed-form solution for the scaled Legendre (LegS) SSM. SaFARi provides a computed solution, where the accuracy depends on the discretization of the $N\times L$ frame. Larger $L$ gives a finer discretization of the basis vectors and thus a better numerical result.
  • Figure 4: Illustration of error types in an SSM due to frame truncation. Truncation errors arise from signal energy in coefficients of index $n > N$, while mixing errors result from energy blending during $A \, c$.
  • Figure 5: Examples of the $A$ matrices of the HiPPO SSM for several basis/measure combinations: (a) Scaled Legendre, (b) Translated Legendre, (c) Scaled Fourier, (d) Translated Fourier. The dense non-zero elements in the upper right of (b) explain its poor performance compared to (a). The numerous small nonzero elements above the diagonal in (c) and (d) contribute to mixing errors over long sequences.
  • ...and 2 more figures

Theorems & Definitions (22)

  • Definition 1
  • Theorem 1
  • Definition 2
  • Definition 3
  • Theorem 2
  • Definition 4
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • Theorem 6
  • ...and 12 more