ARMA Cell: A Modular and Effective Approach for Neural Autoregressive Modeling

TL;DR

This work introduces ARMA cell, a modular neural-network unit that encodes ARMA($p$,$q$) dynamics within recurrent architectures to bridge classical time-series modeling and deep learning. It extends to VARMA and ConvARMA for multivariate and tensor-variate data, enabling end-to-end learning with regular neural components. Empirical results show that ARMA cells achieve competitive or superior performance compared to LSTM/GRU baselines across univariate, multivariate, and tensor-valued time series, while offering greater training stability and modularity. The proposed framework supports easy integration into existing architectures and provides a practical path for hybrid linear and nonlinear modeling with robust optimization. The paper also offers an open-source TensorFlow implementation to foster adoption and systematic comparisons in practice.

Abstract

The autoregressive moving average (ARMA) model is a classical, and arguably one of the most studied approaches to model time series data. It has compelling theoretical properties and is widely used among practitioners. More recent deep learning approaches popularize recurrent neural networks (RNNs) and, in particular, Long Short-Term Memory (LSTM) cells that have become one of the best performing and most common building blocks in neural time series modeling. While advantageous for time series data or sequences with long-term effects, complex RNN cells are not always a must and can sometimes even be inferior to simpler recurrent approaches. In this work, we introduce the ARMA cell, a simpler, modular, and effective approach for time series modeling in neural networks. This cell can be used in any neural network architecture where recurrent structures are present and naturally handles multivariate time series using vector autoregression. We also introduce the ConvARMA cell as a natural successor for spatially-correlated time series. Our experiments show that the proposed methodology is competitive with popular alternatives in terms of performance while being more robust and compelling due to its simplicity

Figures (9)

• Figure 1: Left: Graphical visualizations of how predictions are computed in a univariate ARMA($2$,$2$) cell using the time series values $x$ from the current and previous time points as well as past model predictions $\hat{x}$. Right: Zooming in on the rightmost model cell from the left picture to show the computations of the ARMA cell with parameters as defined in \ref{['eq:arma_reform']}.
• Figure 2: Visualization of an ARMA cell with multiple units representing a mixture of linear and non-linear ARMA models by using different activation functions (left) and a network with stacked ARMA cells creating a more complex model class by transforming inputs by subsequent ARMA cells (right).
• Figure 3: Exemplary visualization of a single-filter ConvARMA cell processing matrix-variate time series (with a single channel) with three lags (upper left) and matrix-variate predictions with three lags (bottom left) using convolutions and combining the results into a single matrix prediction (bottom/top right) with additional bias term $b$ and activation function $\sigma$ (center right).
• Figure 4: Exemplary optimization paths for a single linear ARMA($2$,$1$) cell using stochastic gradient descent. After around 30 iterations, the models converge to the maximum likelihood coefficients.
• Figure 5: The ARMA cell and the Elman model converging to the same coefficients for an ARMA($1$,$1$) process.