Generalized Factor Neural Network Model for High-dimensional Regression

TL;DR

The paper tackles high-dimensional regression with latent low-dimensional structure by integrating nonparametric regression, factor models, and neural networks. It introduces differentiable PCA and Soft PCA layers, along with Additive Layers, to build Generalized Factor Neural Networks (GFANN and GFADNN) capable of learning nonlinear factor effects in $p$-dimensional data with small latent dimension $k$. Through simulations and empirical studies on SPY ETF forecasting and FRED-MD macro data, the proposed architectures consistently outperform FAR-NN, FAST-NN, and standard neural networks, especially under nonlinear factor-observation relationships. The work demonstrates that a factor-then-additive architecture efficiently captures hierarchical compositional structure, offering scalable, interpretable, and robust performance for high-dimensional regression tasks with practical impact in finance and macroeconomics.

Abstract

We tackle the challenges of modeling high-dimensional data sets, particularly those with latent low-dimensional structures hidden within complex, non-linear, and noisy relationships. Our approach enables a seamless integration of concepts from non-parametric regression, factor models, and neural networks for high-dimensional regression. Our approach introduces PCA and Soft PCA layers, which can be embedded at any stage of a neural network architecture, allowing the model to alternate between factor modeling and non-linear transformations. This flexibility makes our method especially effective for processing hierarchical compositional data. We explore ours and other techniques for imposing low-rank structures on neural networks and examine how architectural design impacts model performance. The effectiveness of our method is demonstrated through simulation studies, as well as applications to forecasting future price movements of equity ETF indices and nowcasting with macroeconomic data.

Figures (13)

• Figure 1: A visual illustration of integrating multiple PCA layers into neural networks. The layers in red denote the PCA layers.
• Figure 2: Training and validation error when training a Neural Network model with a PCA layer in the middle.
• Figure 3: Training and validation error when training a Neural Network model with a PCA layer in the middle, with an operating schedule.
• Figure 4: Training and Validation error with an operating schedule of [1,2,...40] and the corresponding percentage variance explained by the first $k$ components
• Figure 5: An illustration of the General Factor Additive Neural Network Model. The red and blue layers represent generalized factor layers, while the final layers are additive layers. The input to the additive layers is already (partially) decorrelated before being passed through them.

Theorems & Definitions (1)

• Definition 1: Diversified projection matrix