Neural Approximation and Its Applications

Abstract

Multivariate function approximation is a fundamental problem in machine learning. Classic multivariate function approximations rely on hand-crafted basis functions (e.g., polynomial basis and Fourier basis), which limits their approximation ability and data adaptation ability, resulting in unsatisfactory performance. To address these challenges, we introduce the neural basis function by leveraging an untrained neural network as the basis function. Equipped with the proposed neural basis function, we suggest the neural approximation (NeuApprox) paradigm for multivariate function approximation. Specifically, the underlying multivariate function behind the multi-dimensional data is decomposed into a sum of block terms. The clear physically-interpreted block term is the product of expressive neural basis functions and their corresponding learnable coefficients, which allows us to faithfully capture distinct components of the underlying data and also flexibly adapt to new data by readily fine-tuning the neural basis functions. Attributed to the elaborately designed block terms, the suggested NeuApprox enjoys strong approximation ability and flexible data adaptation ability over the hand-crafted basis function-based methods. We also theoretically prove that NeuApprox can approximate any multivariate continuous function to arbitrary accuracy. Extensive experiments on diverse multi-dimensional datasets (including multispectral images, light field data, videos, traffic data, and point cloud data) demonstrate the promising performance of NeuApprox in terms of both approximation capability and adaptability.

Figures (10)

• Figure 1: Comparison of multivariate function approximation behind multi-dimensional data (i.e., MSI Painting and MSI Toy with SR = 0.1) based on different basis function. For different data, the hand-crafted basis functions (e.g., polynomial, Fourier, and Gaussian basis functions) are predefined, while the proposed neural basis functions are data-adaptive. The enlarged patch and corresponding residuals are displayed under the recovered results.
• Figure 2: Different block terms of the proposed NeuApprox and their corresponding Fourier spectrum images (bottom) on MSI Toy with SR = 0.1. (a)-(c) first, second, and third block terms of the proposed NeuApprox with corresponding Fourier spectrum images.
• Figure 3: Illustration of the proposed NeuApprox for a three-variable function $f(x,y,z)$. Leveraging multiple block terms to approximate the target function behind the multi-dimensional data. Each block term, which can inherently capture different components of the target function, is formed by the product of a neural basis function and its corresponding coefficient. From top to bottom: multi-dimensional data, cross-section of the three-variable functions behind the multi-dimensional data, and the flowchart of our proposed NeuApprox. Here, $(x,y,z)$ is the coordinate; $f_{\theta_{1}^{j}}(\cdot)$, $f_{\theta_{2}^{j}}(\cdot)$, and $f_{\theta_{3}^{j}}(\cdot)$ are the suggested neural basis functions for block term index $j=1,2,\cdots,T$.
• Figure 4: The results of multi-dimensional image inpainting with SR=0.15 by different methods on MSIs (i.e., Toys and Painting), videos (i.e., Foreman and Carphone), and light field data (i.e., Greek and Origami).
• Figure 5: The results of traffic data completion by different methods. From top to bottom: the 58-th slice of PEMS07, 40-th slice of Seattle, and 148-th slice of Seattle.

Theorems & Definitions (5)

• Theorem 1: cf. bf_sn
• Theorem 2: Universal Approximation Theorem inr_th
• Definition 1
• Theorem 3
• proof