A Statistical-Modelling Approach to Feedforward Neural Network Model Selection

TL;DR

A novel model selection method, which performs both input- and hidden-node selection, is proposed using the Bayesian information criterion (BIC) for FNNs, which leads to an increased probability of recovering the true model, while parsimoniously achieving favourable out-of-sample performance.

Abstract

Feedforward neural networks (FNNs) can be viewed as non-linear regression models, where covariates enter the model through a combination of weighted summations and non-linear functions. Although these models have some similarities to the approaches used within statistical modelling, the majority of neural network research has been conducted outside of the field of statistics. This has resulted in a lack of statistically-based methodology, and, in particular, there has been little emphasis on model parsimony. Determining the input layer structure is analogous to variable selection, while the structure for the hidden layer relates to model complexity. In practice, neural network model selection is often carried out by comparing models using out-of-sample performance. However, in contrast, the construction of an associated likelihood function opens the door to information-criteria-based variable and architecture selection. A novel model selection method, which performs both input- and hidden-node selection, is proposed using the Bayesian information criterion (BIC) for FNNs. The choice of BIC over out-of-sample performance as the model selection objective function leads to an increased probability of recovering the true model, while parsimoniously achieving favourable out-of-sample performance. Simulation studies are used to evaluate and justify the proposed method, and applications on real data are investigated.

Figures (19)

• Figure 1: Neural network architecture with $p$ input nodes and $q$ hidden nodes.
• Figure 2: Model selection schematic. Nodes coloured grey are being considered in current phase. Nodes coloured gold represent optimal nodes in that phase to be brought forward to the next phase.
• Figure 3: Simulation 1: boxplots for TNR (the true negative rate for the input variables) for each method by sample size.
• Figure 4: Simulation 1: boxplots for $q$ (the number of hidden nodes selected) for each method by sample size. Median value highlighted in red. Dashed line indicates the true value of $q$.
• Figure 5: Simulation 2: boxplots for OOS Test for the models selected by each objective function; for comparison, the results for the true model (with inputs $x_1, x_2, x_3$ and $q = 3$) and the full model (with inputs $x_1, x_2, \dotsc, x_{13}$ and $q=10$).