Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximation Theory of Tree Tensor Networks: Tensorized Univariate Functions -- Part I

Mazen Ali, Anthony Nouy

TL;DR

This work introduces a tensorization-based framework to study one-dimensional function approximation on $[0,1)$ using tree tensor networks (TNs) and their rank-structured representations. It identifies $L^p$ spaces with tensor-product spaces via the map $T_{b,d}$, defines tensor subspaces $ extbf{V}_{b,d,S}$ and the TT format, and constructs three approximation tools with measures of complexity—$ ext{compl}_{ ext{N}}$, $ ext{compl}_{ ext{C}}$, and $ ext{compl}_{ ext{S}}$—leading to the approximation classes $N_q^ ext{α}$, $C_q^ ext{α}$, and $S_q^ ext{α}$. The main result is that these classes are quasi-normed spaces with continuous embeddings $C_q^ ext{α} o S_q^ ext{α} o N_q^ ext{α} o C_q^{ ext{α}/2}$, and Part II will connect these TN spaces to Besov smoothness, including the (non-)embedding results. The work also frames tensorization as a feature step implementable in neural networks, drawing parallels to ReLU networks and highlighting the expressive power of depth and sparsity in TN architectures. Altogether, the paper typifies a bridge between TN-based approximation theory and neural-network expressivity, with Part II clarifying Besov embeddings and inverse-embedding phenomena.

Abstract

We study the approximation of functions by tensor networks (TNs). We show that Lebesgue $L^p$-spaces in one dimension can be identified with tensor product spaces of arbitrary order through tensorization. We use this tensor product structure to define subsets of $L^p$ of rank-structured functions of finite representation complexity. These subsets are then used to define different approximation classes of tensor networks, associated with different measures of complexity. These approximation classes are shown to be quasi-normed linear spaces. We study some elementary properties and relationships of said spaces. In part II of this work, we will show that classical smoothness (Besov) spaces are continuously embedded into these approximation classes. We will also show that functions in these approximation classes do not possess any Besov smoothness, unless one restricts the depth of the tensor networks. The results of this work are both an analysis of the approximation spaces of TNs and a study of the expressivity of a particular type of neural networks (NN) -- namely feed-forward sum-product networks with sparse architecture. The input variables of this network result from the tensorization step, interpreted as a particular featuring step which can also be implemented with a neural network with a specific architecture. We point out interesting parallels to recent results on the expressivity of rectified linear unit (ReLU) networks -- currently one of the most popular type of NNs.

Approximation Theory of Tree Tensor Networks: Tensorized Univariate Functions -- Part I

TL;DR

This work introduces a tensorization-based framework to study one-dimensional function approximation on using tree tensor networks (TNs) and their rank-structured representations. It identifies spaces with tensor-product spaces via the map , defines tensor subspaces and the TT format, and constructs three approximation tools with measures of complexity—, , and —leading to the approximation classes , , and . The main result is that these classes are quasi-normed spaces with continuous embeddings , and Part II will connect these TN spaces to Besov smoothness, including the (non-)embedding results. The work also frames tensorization as a feature step implementable in neural networks, drawing parallels to ReLU networks and highlighting the expressive power of depth and sparsity in TN architectures. Altogether, the paper typifies a bridge between TN-based approximation theory and neural-network expressivity, with Part II clarifying Besov embeddings and inverse-embedding phenomena.

Abstract

We study the approximation of functions by tensor networks (TNs). We show that Lebesgue -spaces in one dimension can be identified with tensor product spaces of arbitrary order through tensorization. We use this tensor product structure to define subsets of of rank-structured functions of finite representation complexity. These subsets are then used to define different approximation classes of tensor networks, associated with different measures of complexity. These approximation classes are shown to be quasi-normed linear spaces. We study some elementary properties and relationships of said spaces. In part II of this work, we will show that classical smoothness (Besov) spaces are continuously embedded into these approximation classes. We will also show that functions in these approximation classes do not possess any Besov smoothness, unless one restricts the depth of the tensor networks. The results of this work are both an analysis of the approximation spaces of TNs and a study of the expressivity of a particular type of neural networks (NN) -- namely feed-forward sum-product networks with sparse architecture. The input variables of this network result from the tensorization step, interpreted as a particular featuring step which can also be implemented with a neural network with a specific architecture. We point out interesting parallels to recent results on the expressivity of rectified linear unit (ReLU) networks -- currently one of the most popular type of NNs.

Paper Structure

This paper contains 34 sections, 36 theorems, 218 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Approximation of Functions
  3. Neural Networks
  4. Tensor Networks
  5. Tensor vs. Neural Networks
  6. Main Results
  7. Outline
  8. Tensorization of Functions
  9. The Tensorization Map
  10. Ranks and Minimal Subspaces
  11. Measures, Lebesgue and Smoothness Spaces
  12. Lebesgue Spaces
  13. Sobolev Spaces
  14. Besov Spaces
  15. Tensor Subspaces and Corresponding Function Spaces
  16. ...and 19 more sections

Key Result

Lemma 2.1

The conversion map $\tbd$ defines a linear bijection from the set $I_{b}^d\times[0,1)$ to the interval $[0,1)$, with inverse defined for $x\in [0,1)$ by

Figures (5)

  • Figure 1: Example of an artificial neural network. On the left we have the input nodes marked in red that represent input data to the neural system. The yellow nodes are the neurons that perform some simple operations on the input. The edges between the nodes represent synapses or connections that transfer (after possibly applying an affine linear transformation) the output of one node into the input of another. The final green nodes are the output nodes. In this particular example the number of layers$L$ is three, with two hidden layers.
  • Figure 2: Examples of tensor networks. The vertices in \ref{['fig:tns']} represent the low-dimensional functions in the decomposition, such as $v^1, \hdots,v^d$ in \ref{['eq:TTex']}. The edges between the vertices represent summation over an index (contraction) between two functions, such as summation over $k_\nu$ in \ref{['eq:TTex']}. The free edges represent input variables $x_1,\hdots,x_d$ in \ref{['eq:TTex']}.
  • Figure 3: A function $f:[0,1)\rightarrow\mathbb{R}$ and partial evaluations of $\boldsymbol{f}^\nu \in \mathbf{V}_{b,d}$ for $b=d=2$.
  • Figure 4: Functions $\sigma$ and $\tilde{\sigma}$
  • Figure 5: Representation of some features and their products for $b=2$.

Theorems & Definitions (102)

  • Lemma 2.1
  • Definition 2.2: Tensorization Map
  • Proposition 2.3
  • Lemma 2.4
  • proof
  • Corollary 2.5
  • Lemma 2.6
  • proof
  • Definition 2.7: $\beta$-rank
  • Definition 2.8: $\beta$-minimal subspace
  • ...and 92 more