Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Efficient Approach to Regression Problems with Tensor Neural Networks

Yongxin Li, Yifan Wang, Zhongshuo Lin, Hehu Xie

Abstract

This paper introduces a tensor neural network (TNN) to address nonparametric regression problems, leveraging its distinct sub-network structure to effectively facilitate variable separation and enhance the approximation of complex, high-dimensional functions. The TNN demonstrates superior performance compared to conventional Feed-Forward Networks (FFN) and Radial Basis Function Networks (RBN) in terms of both approximation accuracy and generalization capacity, even with a comparable number of parameters. A significant innovation in our approach is the integration of statistical regression and numerical integration within the TNN framework. This allows for efficient computation of high-dimensional integrals associated with the regression function and provides detailed insights into the underlying data structure. Furthermore, we employ gradient and Laplacian analysis on the regression outputs to identify key dimensions influencing the predictions, thereby guiding the design of subsequent experiments. These advancements make TNN a powerful tool for applications requiring precise high-dimensional data analysis and predictive modeling.

An Efficient Approach to Regression Problems with Tensor Neural Networks

Abstract

This paper introduces a tensor neural network (TNN) to address nonparametric regression problems, leveraging its distinct sub-network structure to effectively facilitate variable separation and enhance the approximation of complex, high-dimensional functions. The TNN demonstrates superior performance compared to conventional Feed-Forward Networks (FFN) and Radial Basis Function Networks (RBN) in terms of both approximation accuracy and generalization capacity, even with a comparable number of parameters. A significant innovation in our approach is the integration of statistical regression and numerical integration within the TNN framework. This allows for efficient computation of high-dimensional integrals associated with the regression function and provides detailed insights into the underlying data structure. Furthermore, we employ gradient and Laplacian analysis on the regression outputs to identify key dimensions influencing the predictions, thereby guiding the design of subsequent experiments. These advancements make TNN a powerful tool for applications requiring precise high-dimensional data analysis and predictive modeling.
Paper Structure (16 sections, 1 theorem, 22 equations, 10 figures, 6 tables)

This paper contains 16 sections, 1 theorem, 22 equations, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. The Nonparametric Regression Problem
  4. The Feed-forward Networks
  5. The Architecture of Tensor Neural Networks
  6. Integration and Prediction
  7. Gradient and Laplacian Analysis for Experimental Design
  8. Gradient Representation
  9. Laplacian Representation
  10. Normalization of Inputs and Fine-Tuning the Output Approximation
  11. Training of TNN
  12. Numerical Examples
  13. Example of 8-Dimensional Sum Function
  14. Example of 8-Dimensional Product Function
  15. Modeling of Strength of High-Performance Concrete
  16. ...and 1 more sections

Key Result

Theorem 2.1

wang2023tensor Assume that each $\Omega_i$ is a bounded open interval in $\mathbb R$ for $i=1, \cdots, d$, $\Omega=\Omega_1\times\cdots\times\Omega_d$, and the function $f(x)\in H^m(\Omega)$. For any tolerance $\varepsilon>0$, there exist a positive integer $p$ and a corresponding TNN defined by (de

Figures (10)

  • Figure 1: Schematic of a feed-forward network, showing data propagation from the Input Layer to the Sum Layer.
  • Figure 2: Schematic of the tensor neural network, illustrating unidirectional data propagation from the Input Layer to the Sum Layer.
  • Figure 3: Training set MSE for different depth settings of subnetworks in the approximation of the continuous function. The convergence speed of the TNN network is influenced by network depth, with increased network width reducing this correlation.
  • Figure 4: Training set MSE for different width settings of subnetworks in the approximation of the continuous function. The depth can also influence the approximation rate of TNN networks, especially at a depth of 4.
  • Figure 5: Training set MSE for different depth setting of subnetworks in the approximation of the continuous function.
  • ...and 5 more figures

Theorems & Definitions (1)

  • Theorem 2.1