Nonlinear Dynamic Factor Analysis With a Transformer Network

TL;DR

This paper introduces an Encoder-Encoder Transformer tailored for nonlinear dynamic factor analysis in macro time series, producing one latent factor per lag within a context window of $P=9$ with about $26{,}000$ parameters. A key innovation is a prior-information regularizer that pulls the Transformer toward a conventional linear factor model (e.g., Kalman filter), improving performance on small datasets while preserving nonlinear flexibility. The attention mechanism provides interpretable insights into which variables and lags drive the factor estimate, and it also enables analysis of regime switches via time-varying attention patterns. In Monte Carlo experiments, the Transformer often surpasses a linear Kalman filter when data deviate from linear-Gaussian assumptions, and it is applied to construct a coincident index of U.S. real activity that aligns with NBER recessions and responds to major crises. Overall, the study establishes a foundation for using Transformers in macroeconomic latent-state estimation with transparent attention-based diagnostics and a practicable pathway for small-sample applications.

Abstract

The paper develops a Transformer architecture for estimating dynamic factors from multivariate time series data under flexible identification assumptions. Performance on small datasets is improved substantially by using a conventional factor model as prior information via a regularization term in the training objective. The results are interpreted with Attention matrices that quantify the relative importance of variables and their lags for the factor estimate. Time variation in Attention patterns can help detect regime switches and evaluate narratives. Monte Carlo experiments suggest that the Transformer is more accurate than the linear factor model, when the data deviate from linear-Gaussian assumptions. An empirical application uses the Transformer to construct a coincident index of U.S. real economic activity.

Figures (57)

• Figure 1: The State Encoder stack on the left constructs a factor estimate from the data. The Measurement Encoder stack on the right is forced to rely heavily on the factor estimate when predicting the observable variables. The factor estimate is conditioned on data during training. Performance on small datasets is improved by using the linear factor model with the Kalman filter as prior information regularizer in the loss function with a weight $\lambda$. Figure by the author, adapted from an MIT-licensed https://github.com/negrinho/sane_tikz/blob/fd6f291d9815613594d724678cb91ac9d412fbb7/examples/transformer.tex.
• Figure 2: Example of pre-processing: Matrix of input data is vectorized into granular tokens, which are then embedded with representation vectors.
• Figure 3: The effectiveness of the training in improving the factor accuracy can be analyzed by visualizing the loss-accuracy pairs as points from different epochs, and fitting linear curves over them. Each line in the figure represents a separate training run with different initial parameters on the same data.
• Figure 4: The simulated datasets in the Monte Carlo study of this study are divided into training, validation and testing sets. The out-of-sample test set contains more data than training and validation sets combined, to guarantee that the results generalize.
• Figure 5: The training, validation and test sets are partitioned into P--lag context windows using a rolling window with a stride of 1.