Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Function Extrapolation with Neural Networks and Its Application for Manifolds

Guy Hay, Nir Sharon

TL;DR

This work tackles the challenge of estimating a function on an extrapolation domain $\Xi$ from samples on a separate domain $\Omega$, possibly lying on manifolds. It introduces the Neural Extrapolation Technique (NExT), which learns a projection of functions onto a learned space by mapping observed data to coefficient vectors and optimizing an extrapolation-focused loss $E_\Xi$. A key contribution is the extrapolation error bound and the condition number $\kappa = d\frac{M_\Xi}{m_\Omega}$ that quantifies difficulty and explains why minimizing data error alone can be suboptimal for extrapolation. The framework accommodates anchor functions and can incorporate manifold bases (e.g., spherical harmonics), showing strong empirical gains over LS and standard DL methods, including for far-domain and noisy extrapolation. This approach yields practical extrapolation improvements across diverse domains and reveals a transparent theoretical link between data-domain fitting and extrapolation performance.

Abstract

This paper addresses the problem of accurately estimating a function on one domain when only its discrete samples are available on another domain. To answer this challenge, we utilize a neural network, which we train to incorporate prior knowledge of the function. In addition, by carefully analyzing the problem, we obtain a bound on the error over the extrapolation domain and define a condition number for this problem that quantifies the level of difficulty of the setup. Compared to other machine learning methods that provide time series prediction, such as transformers, our approach is suitable for setups where the interpolation and extrapolation regions are general subdomains and, in particular, manifolds. In addition, our construction leads to an improved loss function that helps us boost the accuracy and robustness of our neural network. We conduct comprehensive numerical tests and comparisons of our extrapolation versus standard methods. The results illustrate the effectiveness of our approach in various scenarios.

Function Extrapolation with Neural Networks and Its Application for Manifolds

TL;DR

This work tackles the challenge of estimating a function on an extrapolation domain from samples on a separate domain , possibly lying on manifolds. It introduces the Neural Extrapolation Technique (NExT), which learns a projection of functions onto a learned space by mapping observed data to coefficient vectors and optimizing an extrapolation-focused loss . A key contribution is the extrapolation error bound and the condition number that quantifies difficulty and explains why minimizing data error alone can be suboptimal for extrapolation. The framework accommodates anchor functions and can incorporate manifold bases (e.g., spherical harmonics), showing strong empirical gains over LS and standard DL methods, including for far-domain and noisy extrapolation. This approach yields practical extrapolation improvements across diverse domains and reveals a transparent theoretical link between data-domain fitting and extrapolation performance.

Abstract

This paper addresses the problem of accurately estimating a function on one domain when only its discrete samples are available on another domain. To answer this challenge, we utilize a neural network, which we train to incorporate prior knowledge of the function. In addition, by carefully analyzing the problem, we obtain a bound on the error over the extrapolation domain and define a condition number for this problem that quantifies the level of difficulty of the setup. Compared to other machine learning methods that provide time series prediction, such as transformers, our approach is suitable for setups where the interpolation and extrapolation regions are general subdomains and, in particular, manifolds. In addition, our construction leads to an improved loss function that helps us boost the accuracy and robustness of our neural network. We conduct comprehensive numerical tests and comparisons of our extrapolation versus standard methods. The results illustrate the effectiveness of our approach in various scenarios.
Paper Structure (19 sections, 3 theorems, 24 equations, 11 figures, 6 tables, 1 algorithm)

This paper contains 19 sections, 3 theorems, 24 equations, 11 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem formulation and prior work
  3. Problem formulation
  4. Neural networks
  5. The standard least squares approach
  6. Deep learning approaches for function extrapolation
  7. Our extrapolation method
  8. Error analysis
  9. An overview of the NExT framework
  10. Extrapolating with anchor functions
  11. Experimental results
  12. Noisy function samples
  13. Monotonic functions with noisy samples
  14. Extrapolating with anchor functions experiments
  15. Deep learning experimental results
  16. ...and 4 more sections

Key Result

Theorem 3.4

Let $\{\phi_k\}_{k=1}^d$ be a sequence of real-valued functions, defined both in $\Omega$ and $\Xi$. The following conditions hold: Where $M_\Xi$, and $m_\Omega$ are defined in Definition def_least_squares_extrapolation_condition_number, and $E_\Omega$ is the error in eq_ie_mse_objective_2 for $\Omega$ instead of $\Xi$.

Figures (11)

  • Figure 1: A comparison between our method (left) and an LS-based extrapolation (right). The space consists of a 5th-degree Chebyshev polynomial where the LS achieves an RMSE score of 0.633, while NExT outperforms it with 0.015, achieving a 97.7% error reduction rate. The LS model clearly overfits the training data domain $\Omega$, where the data is given, while NExT manages to focus on minimizing \ref{['eq_ie_mse_objective_2']} over $\Xi$.
  • Figure 2: A comparison between our model (left) and an LS-based extrapolation (right) for a specific 7-degree monotonic Chebyshev polynomial. LS achieves an RMSE score of 0.190, while NExT outperforms it with 0.015, achieving a 92.3% error reduction rate. The LS model clearly overfits the training data domain $\Omega$, where the data is given, while NExT manages to focus on minimizing \ref{['eq_ie_mse_objective_2']} over $\Xi$. In addition, although both didn't predict a true monotonic function, NExT managed to predict a monotonic function in $\Xi$, indicating it learned to predict this kind of functions, at least in the desired domain $\Xi$.
  • Figure 3: Anchor functions with decaying functions. Plotted next to the wished extrapolation function $f$. $f(x)+\frac{2}{x+1}$ (left), $f(x)+\frac{3 \sin(x)}{x+1}$ (middle), $f(x)+0.9^x$ (right), with RMSE scores of 0.310, 0.345, and 0.562 respectively.
  • Figure 4: Anchor functions with non-decaying functions. Plotted next to the wished extrapolation function $f$. $f(x)+\frac{x}{10}$ (left), $f(x)+\sin^2(x)$ (middle), and $f(x)+\frac{\log^2(x+1)}{5}$ (right), with RMSE scores of 0.552, 0.613, and 0.702 respectively.
  • Figure 5: Results for extrapolating \ref{['eq_extrapolation_function_1']} for LS (top) and NExT (bottom) using a frame. Left figures use the 3 decaying functions only, and right figures use the 3 decaying functions with an additional 7 basis elements from the trigonometric functions. LS RMSE scores are 1.02 and 2.399, respectively, for without and with filler functions, and NExT scores are 0.844 and 0.131, respectively.
  • ...and 6 more figures

Theorems & Definitions (19)

  • Remark 2.1: More general function spaces
  • Definition 2.1: Extrapolation problem
  • Definition 2.2: Anchor functions
  • Definition 2.3: Anchored extrapolation problem
  • Remark 3.1
  • Remark 3.2
  • Definition 3.1: Extrapolation condition number
  • Remark 3.3
  • Theorem 3.4
  • Lemma 3.5
  • ...and 9 more