Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Oscillators Are All You Need: Irregular Time Series Modelling via Damped Harmonic Oscillators with Closed-Form Solutions

Yashas Shende, Aritra Das, Reva Laxmi Chauhan, Arghya Pathak, Debayan Gupta

TL;DR

This work tackles irregular time-series modelling by replacing NODE-based attention with a closed-form damped harmonic oscillator surrogate, OsciFormer. Keys and values are modelled as damped oscillators driven by learned forces, while queries are expressed in a sinusoidal basis to realize a resonance-based attention mechanism with a closed-form solution. The authors prove that a fixed bank of oscillators can approximate ContiFormer-style attention to arbitrary precision and demonstrate strong empirical gains: state-of-the-art performance on irregular benchmarks, significant memory savings, and substantial speedups over numerical solvers. The approach combines solid theoretical guarantees with practical scalability, enabling efficient continuous-time attention for irregularly sampled data and offering pathways to broader physics-inspired representations in ML systems.

Abstract

Transformers excel at time series modelling through attention mechanisms that capture long-term temporal patterns. However, they assume uniform time intervals and therefore struggle with irregular time series. Neural Ordinary Differential Equations (NODEs) effectively handle irregular time series by modelling hidden states as continuously evolving trajectories. ContiFormers arxiv:2402.10635 combine NODEs with Transformers, but inherit the computational bottleneck of the former by using heavy numerical solvers. This bottleneck can be removed by using a closed-form solution for the given dynamical system - but this is known to be intractable in general! We obviate this by replacing NODEs with a novel linear damped harmonic oscillator analogy - which has a known closed-form solution. We model keys and values as damped, driven oscillators and expand the query in a sinusoidal basis up to a suitable number of modes. This analogy naturally captures the query-key coupling that is fundamental to any transformer architecture by modelling attention as a resonance phenomenon. Our closed-form solution eliminates the computational overhead of numerical ODE solvers while preserving expressivity. We prove that this oscillator-based parameterisation maintains the universal approximation property of continuous-time attention; specifically, any discrete attention matrix realisable by ContiFormer's continuous keys can be approximated arbitrarily well by our fixed oscillator modes. Our approach delivers both theoretical guarantees and scalability, achieving state-of-the-art performance on irregular time series benchmarks while being orders of magnitude faster.

Oscillators Are All You Need: Irregular Time Series Modelling via Damped Harmonic Oscillators with Closed-Form Solutions

TL;DR

This work tackles irregular time-series modelling by replacing NODE-based attention with a closed-form damped harmonic oscillator surrogate, OsciFormer. Keys and values are modelled as damped oscillators driven by learned forces, while queries are expressed in a sinusoidal basis to realize a resonance-based attention mechanism with a closed-form solution. The authors prove that a fixed bank of oscillators can approximate ContiFormer-style attention to arbitrary precision and demonstrate strong empirical gains: state-of-the-art performance on irregular benchmarks, significant memory savings, and substantial speedups over numerical solvers. The approach combines solid theoretical guarantees with practical scalability, enabling efficient continuous-time attention for irregularly sampled data and offering pathways to broader physics-inspired representations in ML systems.

Abstract

Transformers excel at time series modelling through attention mechanisms that capture long-term temporal patterns. However, they assume uniform time intervals and therefore struggle with irregular time series. Neural Ordinary Differential Equations (NODEs) effectively handle irregular time series by modelling hidden states as continuously evolving trajectories. ContiFormers arxiv:2402.10635 combine NODEs with Transformers, but inherit the computational bottleneck of the former by using heavy numerical solvers. This bottleneck can be removed by using a closed-form solution for the given dynamical system - but this is known to be intractable in general! We obviate this by replacing NODEs with a novel linear damped harmonic oscillator analogy - which has a known closed-form solution. We model keys and values as damped, driven oscillators and expand the query in a sinusoidal basis up to a suitable number of modes. This analogy naturally captures the query-key coupling that is fundamental to any transformer architecture by modelling attention as a resonance phenomenon. Our closed-form solution eliminates the computational overhead of numerical ODE solvers while preserving expressivity. We prove that this oscillator-based parameterisation maintains the universal approximation property of continuous-time attention; specifically, any discrete attention matrix realisable by ContiFormer's continuous keys can be approximated arbitrarily well by our fixed oscillator modes. Our approach delivers both theoretical guarantees and scalability, achieving state-of-the-art performance on irregular time series benchmarks while being orders of magnitude faster.
Paper Structure (70 sections, 11 theorems, 226 equations, 8 figures, 11 tables)

This paper contains 70 sections, 11 theorems, 226 equations, 8 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Harmonic Oscillator based modelling
  4. Averaged attention: decomposition.
  5. Steady-state contribution $\mathcal{I}^{\mathrm{(ss)}}_i$:
  6. Transient contribution $\mathcal{I}^{\mathrm{(tr)}}_i$.
  7. Final result.
  8. Harmonic Approximation Theorem
  9. Computational Complexity
  10. Numerical continuous-time realization (baseline)
  11. Closed-form realization
  12. Comparison
  13. Representative instance
  14. Architecture
  15. Experiments
  16. ...and 55 more sections

Key Result

Theorem 1

Let $q\in C([a,b];\mathbb{R}^{d_k})$ and continuous keys $\{k_i\}_{i=1}^N$ with $k_i:[t_i,b]\to\mathbb{R}^{d_k}$. For any $\varepsilon>0$ there exists an integer $M$ (depending on $\varepsilon$ and the keys) and a single shared oscillator bank on the fixed grid $\{\omega_n\}_{n=0}^{M}$ with $\gamma_ where $\tilde{k}_i(t):=C e^{A(t-t_i)}z_{i,0}$ is the bank-generated key. Consequently, for all $j\g

Figures (8)

  • Figure 1: Architecture Pipeline
  • Figure 2: Trajectories and Training Time Visualisations
  • Figure 3:
  • Figure 4:
  • Figure 5: Phase–frequency attention $\alpha(\omega,\varphi)$ for a representative key. The bright ridge in the $(\omega,\varphi)$ plane indicates the resonance region.
  • ...and 3 more figures

Theorems & Definitions (26)

  • Theorem 1
  • proof
  • Corollary 1
  • Definition 1: Averaged inner-product logit
  • Definition 2: Masked pre-softmax CT attention and softmax
  • Definition 3: Fixed oscillator bank and readout
  • Remark 1
  • Definition 4: Fejér kernel and means
  • Lemma 1: Basic properties of $K_N$
  • proof
  • ...and 16 more