Learning to Dissipate Energy in Oscillatory State-Space Models

TL;DR

The paper extends Linear Oscillatory State-Space Models by introducing Damped-LinOSS (D-LinOSS), which learns damping in addition to frequency, decoupling energy dissipation from oscillation scale. Through spectral analysis, the authors show D-LinOSS can realize the full stable second-order dynamics with eigenvalues filling the unit disk, unlike prior LinOSS variants with restricted spectra. They propose a stable parameterization and initialization strategy, enabling arbitrary eigenvalue distributions and reducing hyperparameter search. Empirically, D-LinOSS achieves state-of-the-art results on eight real-world sequence tasks, with faster convergence and a 50% reduction in hyperparameter space, demonstrating practical benefits for long-range sequence modeling in oscillatory systems.

Abstract

State-space models (SSMs) are a class of networks for sequence learning that benefit from fixed state size and linear complexity with respect to sequence length, contrasting the quadratic scaling of typical attention mechanisms. Inspired from observations in neuroscience, Linear Oscillatory State-Space models (LinOSS) are a recently proposed class of SSMs constructed from layers of discretized forced harmonic oscillators. Although these models perform competitively, leveraging fast parallel scans over diagonal recurrent matrices and achieving state-of-the-art performance on tasks with sequence length up to 50k, LinOSS models rely on rigid energy dissipation ("forgetting") mechanisms that are inherently coupled to the time scale of state evolution. As forgetting is a crucial mechanism for long-range reasoning, we demonstrate the representational limitations of these models and introduce Damped Linear Oscillatory State-Space models (D-LinOSS), a more general class of oscillatory SSMs that learn to dissipate latent state energy on arbitrary time scales. We analyze the spectral distribution of the model's recurrent matrices and prove that the SSM layers exhibit stable dynamics under a simple, flexible parameterization. Without introducing additional complexity, D-LinOSS consistently outperforms previous LinOSS methods on long-range learning tasks, achieves faster convergence, and reduces the hyperparameter search space by 50%.

Figures (4)

• Figure 1: Previous LinOSS models directly couple the frequency and magnitude of discretized system eigenvalues, reducing latent state energy dissipation to a single scale when normalizing time by frequency. Through a simple, complexity-free parametric extension, D-LinOSS can learn system damping on any scale, independent of frequency, expanding the range of expressible internal dynamics. The specific damping behavior depicted in the right diagram is selected arbitrarily.
• Figure 2: Validation metric convergence for the adding task of different sequence lengths.
• Figure 3: Reachable eigenvalue sets for the different oscillatory SSMs. The eigenvalue $\lambda=0.8$ of the exponential decay experiment from Section \ref{['sec:motivation']} only lies in the spectral range of D-LinOSS.
• Figure 4: Initialization study varying intervals of eigenvalue magnitude and methods of sampling.

Theorems & Definitions (18)

• Proposition 3.1
• proof
• Proposition 3.2
• proof
• Proposition 3.3
• proof
• Proposition 3.4
• proof
• Proposition A.1
• proof