Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximation Theory of Tree Tensor Networks: Tensorized Univariate Functions -- Part II

Mazen Ali, Anthony Nouy

TL;DR

This work uses classical interpolation theory to show that a scale of interpolated smoothness spaces is continuously embedded into the scale of TT approximation spaces and shows that the TT approximation spaces are, in a sense, much larger than smoothness spaces when the depth of the tensor network is not restricted but are embedded into a scale of interpolated smoothness spaces if one restricts the depth.

Abstract

We study the approximation by tensor networks (TNs) of functions from classical smoothness classes. The considered approximation tool combines a tensorization of functions in $L^p([0,1))$, which allows to identify a univariate function with a multivariate function (or tensor), and the use of tree tensor networks (the tensor train format) for exploiting low-rank structures of multivariate functions. The resulting tool can be interpreted as a feed-forward neural network, with first layers implementing the tensorization, interpreted as a particular featuring step, followed by a sum-product network with sparse architecture. In part I of this work, we presented several approximation classes associated with different measures of complexity of tensor networks and studied their properties. In this work (part II), we show how classical approximation tools, such as polynomials or splines (with fixed or free knots), can be encoded as a tensor network with controlled complexity. We use this to derive direct (Jackson) inequalities for the approximation spaces of tensor networks. This is then utilized to show that Besov spaces are continuously embedded into these approximation spaces. In other words, we show that arbitrary Besov functions can be approximated with optimal or near to optimal rate. We also show that an arbitrary function in the approximation class possesses no Besov smoothness, unless one limits the depth of the tensor network.

Approximation Theory of Tree Tensor Networks: Tensorized Univariate Functions -- Part II

TL;DR

This work uses classical interpolation theory to show that a scale of interpolated smoothness spaces is continuously embedded into the scale of TT approximation spaces and shows that the TT approximation spaces are, in a sense, much larger than smoothness spaces when the depth of the tensor network is not restricted but are embedded into a scale of interpolated smoothness spaces if one restricts the depth.

Abstract

We study the approximation by tensor networks (TNs) of functions from classical smoothness classes. The considered approximation tool combines a tensorization of functions in , which allows to identify a univariate function with a multivariate function (or tensor), and the use of tree tensor networks (the tensor train format) for exploiting low-rank structures of multivariate functions. The resulting tool can be interpreted as a feed-forward neural network, with first layers implementing the tensorization, interpreted as a particular featuring step, followed by a sum-product network with sparse architecture. In part I of this work, we presented several approximation classes associated with different measures of complexity of tensor networks and studied their properties. In this work (part II), we show how classical approximation tools, such as polynomials or splines (with fixed or free knots), can be encoded as a tensor network with controlled complexity. We use this to derive direct (Jackson) inequalities for the approximation spaces of tensor networks. This is then utilized to show that Besov spaces are continuously embedded into these approximation spaces. In other words, we show that arbitrary Besov functions can be approximated with optimal or near to optimal rate. We also show that an arbitrary function in the approximation class possesses no Besov smoothness, unless one limits the depth of the tensor network.

Paper Structure

This paper contains 27 sections, 37 theorems, 212 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Recalling Results of Part I
  3. Tensorization
  4. Ranks
  5. Tensor Subspaces
  6. Tensor Train Format and Corresponding Approximation Tool
  7. Approximation Spaces
  8. Main Results of This Work
  9. Encoding Classical Approximation Tools
  10. Polynomials
  11. Fixed Knot Splines
  12. Free Knot Splines
  13. Multi-Resolution Analysis
  14. Piecewise Polynomial MRA
  15. $S$ Contains MRA Generators
  16. ...and 12 more sections

Key Result

Theorem 2.2

For any $0 < p \le \infty$, define Then, $\Tbd$ is a linear isometry from $L^{p}([0,1))$ to $\mathbf{V}_{b,d,L^p}$ equipped with the (quasi-)norm $\Vert \cdot\Vert_p$ defined by for $p<\infty$, or

Figures (3)

  • Figure 1: DeVore diagram of smoothness spaces DeVore98. The Sobolev embedding line is the diagonal with the points $(1/\tau,\alpha)$ and $(1/\mu, r)$
  • Figure 2: "Sawtooth" function.
  • Figure 3: Visual representation of different partitioning levels of the interval $[0,1)$, with $b=2$ and $d=4$.

Theorems & Definitions (89)

  • Definition 2.1: Tensorization Map
  • Theorem 2.2: Tensorization is an $L^{p}$-Isometry
  • Definition 2.3: $\beta$-rank
  • Definition 2.4: $(\beta,d)$-rank
  • Lemma 2.5
  • Definition 2.6: Closed under Dilation
  • Lemma 2.7
  • Lemma 2.8: Local projection ranks
  • Definition 2.9: Tensor Train Format
  • Theorem 2.10
  • ...and 79 more