Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universal Approximation Theorem for Input-Connected Multilayer Perceptrons

Vugar Ismailov

TL;DR

The paper introduces Input-Connected MLPs (IC--MLPs), where each hidden layer directly receives an affine projection of the raw input, enabling depthwise input influence. It proves a sharp universality criterion: deep IC--MLPs approximate any continuous function on compact sets if and only if the activation is nonlinear, first for univariate inputs and then extending to multivariate inputs. The results rely on mollification and classical density arguments (Weierstrass/Stone–Weierstrass) and leverage closure under linear combinations enabled by the input-connected design. This work provides a transparent, minimal condition for universality, clarifying the expressive power of IC--MLPs relative to standard MLPs and highlighting future questions on rates and depth-width tradeoffs.

Abstract

We introduce the Input-Connected Multilayer Perceptron (IC-MLP), a feedforward neural network architecture in which each hidden neuron receives, in addition to the outputs of the preceding layer, a direct affine connection from the raw input. We first study this architecture in the univariate setting and give an explicit and systematic description of IC-MLPs with an arbitrary finite number of hidden layers, including iterated formulas for the network functions. In this setting, we prove a universal approximation theorem showing that deep IC-MLPs can approximate any continuous function on a closed interval of the real line if and only if the activation function is nonlinear. We then extend the analysis to vector-valued inputs and establish a corresponding universal approximation theorem for continuous functions on compact subsets of $\mathbb{R}^n$.

Universal Approximation Theorem for Input-Connected Multilayer Perceptrons

TL;DR

The paper introduces Input-Connected MLPs (IC--MLPs), where each hidden layer directly receives an affine projection of the raw input, enabling depthwise input influence. It proves a sharp universality criterion: deep IC--MLPs approximate any continuous function on compact sets if and only if the activation is nonlinear, first for univariate inputs and then extending to multivariate inputs. The results rely on mollification and classical density arguments (Weierstrass/Stone–Weierstrass) and leverage closure under linear combinations enabled by the input-connected design. This work provides a transparent, minimal condition for universality, clarifying the expressive power of IC--MLPs relative to standard MLPs and highlighting future questions on rates and depth-width tradeoffs.

Abstract

We introduce the Input-Connected Multilayer Perceptron (IC-MLP), a feedforward neural network architecture in which each hidden neuron receives, in addition to the outputs of the preceding layer, a direct affine connection from the raw input. We first study this architecture in the univariate setting and give an explicit and systematic description of IC-MLPs with an arbitrary finite number of hidden layers, including iterated formulas for the network functions. In this setting, we prove a universal approximation theorem showing that deep IC-MLPs can approximate any continuous function on a closed interval of the real line if and only if the activation function is nonlinear. We then extend the analysis to vector-valued inputs and establish a corresponding universal approximation theorem for continuous functions on compact subsets of .
Paper Structure (7 sections, 2 theorems, 63 equations, 2 figures)

This paper contains 7 sections, 2 theorems, 63 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Scalar IC--MLP Architecture
  3. Universal Approximation Theorem for Univariate Functions
  4. Vector-Valued Inputs and Multivariate Universal Approximation
  5. IC--MLP Architecture with Vector Inputs
  6. Universal Approximation Theorem for Multivariate Functions
  7. Discussion and Conclusions

Key Result

Theorem 1

Let $\sigma:\mathbb{R}\to\mathbb{R}$ be continuous. The following statements are equivalent: In other words, deep IC--MLP networks can approximate continuous functions on compact sets of $\mathbb{R}$ if and only if the activation function $\sigma$ is nonlinear.

Figures (2)

  • Figure 1: Scalar Input-Connected Multilayer Perceptron (IC--MLP). The scalar input $x$ is fed directly to every hidden layer and to the output neuron (dashed arrows), in addition to the standard feedforward connections between successive layers (solid arrows).
  • Figure 2: Input-Connected Multilayer Perceptron (IC--MLP) with vector input. The input $(x_1,\dots,x_n)$ is fed directly to every hidden layer and to the output neuron (dashed arrows), in addition to the standard feedforward connections between successive layers (solid arrows).

Theorems & Definitions (4)

  • Theorem 1
  • proof
  • Theorem 2
  • proof