A least distance estimator for a multivariate regression model using deep neural networks

TL;DR

The paper presents a deep neural network-based least distance estimator (DNN-LD) for multivariate regression, designed to capture dependencies among multiple responses and provide robustness to outliers. It extends the model with adaptive group Lasso penalties (GDNN-LD and AGDNN-LD) to perform variable selection in high-dimensional settings, using a quadratic smoothing approach to optimize the non-differentiable LD loss. Through simulations and a real-data application on concrete slump data, the authors show that DNN-LD and its penalized variants deliver improved predictive accuracy, especially when response correlations are strong, and can yield sparse, interpretable models. The work offers a flexible, robust alternative to standard least squares for multivariate nonlinear regression and points to future theoretical development and deeper network extensions.

Abstract

We propose a deep neural network (DNN) based least distance (LD) estimator (DNN-LD) for a multivariate regression problem, addressing the limitations of the conventional methods. Due to the flexibility of a DNN structure, both linear and nonlinear conditional mean functions can be easily modeled, and a multivariate regression model can be realized by simply adding extra nodes at the output layer. The proposed method is more efficient in capturing the dependency structure among responses than the least squares loss, and robust to outliers. In addition, we consider $L_1$-type penalization for variable selection, crucial in analyzing high-dimensional data. Namely, we propose what we call (A)GDNN-LD estimator that enjoys variable selection and model estimation simultaneously, by applying the (adaptive) group Lasso penalty to weight parameters in the DNN structure. For the computation, we propose a quadratic smoothing approximation method to facilitate optimizing the non-smooth objective function based on the least distance loss. The simulation studies and a real data analysis demonstrate the promising performance of the proposed method.

Figures (7)

• Figure 1: Extensible property of a neural network for being multivariate regression model from a univariate one.
• Figure 2: An example of how group Lasso penalization works in the fully connected DNN architecture of the two hidden layers. Weights between the first and the second layer of rounded with black dashed line indicate the group of the weights to be shrunk towards zero vector Ho2020.
• Figure 3: Fully connected neural network architecture for a multivariate nonlinear regression with aforementioned notations. All bias nodes are set to one.
• Figure 4: The left panel is for viewing the characteristic of LD loss over LS loss and the right panel shows the quadratic smoothing approximation function suggested in \ref{['smoothing']}.
• Figure 5: The two true cases for Example 1 are displayed. The left panel supports that case 1 of a high level of correlation among the response variables, and the right panel, shows a weak relationship among the response variables.