Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Alternators With Noise Models

Mohammad R. Rezaei, Adji Bousso Dieng

TL;DR

Alternator++ extends the Alternator framework by introducing trainable noise models for both observed and latent trajectories, coupling diffusion inspired noise modeling with a noise matching objective. This yields a low dimensional latent state with enhanced capacity to capture stochastic, multimodal time series dynamics, enabling faster sampling and improved density estimation, imputation, and forecasting. Empirical results show robust gains over strong baselines such as Mamba, ScoreGrad, and Dyffusion across multiple real datasets, including SST forecasting, while highlighting practical considerations like tuning noise schedules. The work offers a principled path to combine the efficiency of low dimensional latent representations with the flexibility of diffusion style noise modeling for time series analysis.

Abstract

Alternators have recently been introduced as a framework for modeling time-dependent data. They often outperform other popular frameworks, such as state-space models and diffusion models, on challenging time-series tasks. This paper introduces a new Alternator model, called Alternator++, which enhances the flexibility of traditional Alternators by explicitly modeling the noise terms used to sample the latent and observed trajectories, drawing on the idea of noise models from the diffusion modeling literature. Alternator++ optimizes the sum of the Alternator loss and a noise-matching loss. The latter forces the noise trajectories generated by the two noise models to approximate the noise trajectories that produce the observed and latent trajectories. We demonstrate the effectiveness of Alternator++ in tasks such as density estimation, time series imputation, and forecasting, showing that it outperforms several strong baselines, including Mambas, ScoreGrad, and Dyffusion.

Alternators With Noise Models

TL;DR

Alternator++ extends the Alternator framework by introducing trainable noise models for both observed and latent trajectories, coupling diffusion inspired noise modeling with a noise matching objective. This yields a low dimensional latent state with enhanced capacity to capture stochastic, multimodal time series dynamics, enabling faster sampling and improved density estimation, imputation, and forecasting. Empirical results show robust gains over strong baselines such as Mamba, ScoreGrad, and Dyffusion across multiple real datasets, including SST forecasting, while highlighting practical considerations like tuning noise schedules. The work offers a principled path to combine the efficiency of low dimensional latent representations with the flexibility of diffusion style noise modeling for time series analysis.

Abstract

Alternators have recently been introduced as a framework for modeling time-dependent data. They often outperform other popular frameworks, such as state-space models and diffusion models, on challenging time-series tasks. This paper introduces a new Alternator model, called Alternator++, which enhances the flexibility of traditional Alternators by explicitly modeling the noise terms used to sample the latent and observed trajectories, drawing on the idea of noise models from the diffusion modeling literature. Alternator++ optimizes the sum of the Alternator loss and a noise-matching loss. The latter forces the noise trajectories generated by the two noise models to approximate the noise trajectories that produce the observed and latent trajectories. We demonstrate the effectiveness of Alternator++ in tasks such as density estimation, time series imputation, and forecasting, showing that it outperforms several strong baselines, including Mambas, ScoreGrad, and Dyffusion.
Paper Structure (19 sections, 11 equations, 2 figures, 6 tables, 1 algorithm)

This paper contains 19 sections, 11 equations, 2 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Diffusion Models
  4. Alternators
  5. Alternator++
  6. Generative Process
  7. Training Objective
  8. Sampling and Encoding New Sequences
  9. Experiments
  10. Density estimation
  11. Time Series Imputation
  12. Sea surface temperature forecasting
  13. Related Work
  14. Conclusion
  15. Datasets Details
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Comparing the distributions learned by various models against the ground truth distribution. Alternator++ captures multimodal distributions better than Alternator, Mamba, and ScoreGrad.
  • Figure 2: Performance on missing data imputation across several datasets, evaluated in terms of MAE, MSE, and CC. Results are averaged over missing rates ranging from 10% to 90%. Alternator++ generally outperforms the baselines in terms of MSE and CC. However, for MAE, it faces challenges on the Covid dataset, where Alternator and Mamba perform better.