Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the study of frequency control and spectral bias in Wavelet-Based Kolmogorov Arnold networks: A path to physics-informed KANs

Juan Daniel Meshir, Abel Palafox, Edgar Alejandro Guerrero

TL;DR

The paper tackles spectral bias in differential-equation solving networks by introducing Wavelet Kolmogorov Arnold Networks (Wav-KANs) and analyzing them through the Neural Tangent Kernel (NTK). It shows that the decay of NTK eigenvalues can be controlled by the mother wavelet frequency, such as the Morlet wavelet with parameter $b$, enabling balanced learning of low- and high-frequency components. The authors provide theoretical propositions and empirical demonstrations that adjusting $b$ or increasing hidden units mitigates spectral bias, and extend the approach to physics-informed neural networks as Wav-KINNs, which solve Poisson, Heat, Wave, and Helmholtz equations with fewer parameters and without domain decomposition. They report robust performance for several PDE types but note loss-term imbalance can hinder certain problems, indicating future work on loss balancing and a full NTK theory for wavelet-based networks.

Abstract

Spectral bias, the tendency of neural networks to prioritize learning low-frequency components of functions during the initial training stages, poses a significant challenge when approximating solutions with high-frequency details. This issue is particularly pronounced in physics-informed neural networks (PINNs), widely used to solve differential equations that describe physical phenomena. In the literature, contributions such as Wavelet Kolmogorov Arnold Networks (Wav-KANs) have demonstrated promising results in capturing both low- and high-frequency components. Similarly, Fourier features (FF) are often employed to address this challenge. However, the theoretical foundations of Wav-KANs, particularly the relationship between the frequency of the mother wavelet and spectral bias, remain underexplored. A more in-depth understanding of how Wav-KANs manage high-frequency terms could offer valuable insights for addressing oscillatory phenomena encountered in parabolic, elliptic, and hyperbolic differential equations. In this work, we analyze the eigenvalues of the neural tangent kernel (NTK) of Wav-KANs to enhance their ability to converge on high-frequency components, effectively mitigating spectral bias. Our theoretical findings are validated through numerical experiments, where we also discuss the limitations of traditional approaches, such as standard PINNs and Fourier features, in addressing multi-frequency problems.

On the study of frequency control and spectral bias in Wavelet-Based Kolmogorov Arnold networks: A path to physics-informed KANs

TL;DR

The paper tackles spectral bias in differential-equation solving networks by introducing Wavelet Kolmogorov Arnold Networks (Wav-KANs) and analyzing them through the Neural Tangent Kernel (NTK). It shows that the decay of NTK eigenvalues can be controlled by the mother wavelet frequency, such as the Morlet wavelet with parameter , enabling balanced learning of low- and high-frequency components. The authors provide theoretical propositions and empirical demonstrations that adjusting or increasing hidden units mitigates spectral bias, and extend the approach to physics-informed neural networks as Wav-KINNs, which solve Poisson, Heat, Wave, and Helmholtz equations with fewer parameters and without domain decomposition. They report robust performance for several PDE types but note loss-term imbalance can hinder certain problems, indicating future work on loss balancing and a full NTK theory for wavelet-based networks.

Abstract

Spectral bias, the tendency of neural networks to prioritize learning low-frequency components of functions during the initial training stages, poses a significant challenge when approximating solutions with high-frequency details. This issue is particularly pronounced in physics-informed neural networks (PINNs), widely used to solve differential equations that describe physical phenomena. In the literature, contributions such as Wavelet Kolmogorov Arnold Networks (Wav-KANs) have demonstrated promising results in capturing both low- and high-frequency components. Similarly, Fourier features (FF) are often employed to address this challenge. However, the theoretical foundations of Wav-KANs, particularly the relationship between the frequency of the mother wavelet and spectral bias, remain underexplored. A more in-depth understanding of how Wav-KANs manage high-frequency terms could offer valuable insights for addressing oscillatory phenomena encountered in parabolic, elliptic, and hyperbolic differential equations. In this work, we analyze the eigenvalues of the neural tangent kernel (NTK) of Wav-KANs to enhance their ability to converge on high-frequency components, effectively mitigating spectral bias. Our theoretical findings are validated through numerical experiments, where we also discuss the limitations of traditional approaches, such as standard PINNs and Fourier features, in addressing multi-frequency problems.

Paper Structure

This paper contains 14 sections, 4 theorems, 53 equations, 13 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Wavelet Kolmogorov Arnold Networks and Spectral Bias
  4. Wavelet Kolmogorov-Arnold networks (Wav-KANs)
  5. On the study of spectral bias through the lens of the Neural Tangent Kernel
  6. Controlling NTK eigenvalues via mother wavelet frequency
  7. Controlling NTK eigenvalues via hidden units
  8. Extending to Physics-Informed Neural Networks
  9. Discussion
  10. Conclusion
  11. Proof Proposition 1
  12. Proof Proposition 2
  13. Proof Proposition 3
  14. Study of different mother wavelets

Key Result

Theorem 1

For any continuous function $f: [0,1]^n\to \mathbb{R}$, there exist $2n+1$ functions $\Psi_q: \mathbb{R}\to \mathbb{R}$ and $2n+1\times n$ functions $\psi_{q,p}: [0,1]\to \mathbb{R}$, all univariate and continuous, such that

Figures (13)

  • Figure 1: Kolmogorov-Arnold representation as KAN of [2,5,1] layers, where each is a learnable function, and each node is a simple summation.
  • Figure 2: Example of a Wav-KAN used to approximate the function $f(x)=4x^5$ over the interval $[-1,1]$ with a [1, 3, 1] layer structure. Each edge represents a scaled and shifted instance of the mother wavelet, chosen as the Morlet wavelet function, defined by $e^{-\frac{1}{2}x^2}\cos(2x)$.
  • Figure 3: Analyzing Wav-KAN behavior through the NTK. a) Approximation of function \ref{['eq:solpoissoneq']} with high- and low-frequency components using a two-layer Wav-KAN $[1,35,1]$ and Morlet wavelet $\psi(x) = e^{-x^2}\cos(bx)$ for different $b$ values: $1, 5, 10,$ and $15$. b) NTK eigenvalues in descending order for these values of $b$. c) Loss evolution for varying $b$ values.
  • Figure 4: NTK eigenvalues in descending order for $b=15$ and $b=25$.
  • Figure 5: a) Function approximation for \ref{['eq:solpoissoneq']} with $b=15$. b) Function approximation for \ref{['eq:solpoissoneq']} with $b=25$. c) Loss evolution for $b=15$. d) Loss evolution for $b=25$.
  • ...and 8 more figures

Theorems & Definitions (10)

  • Theorem 1: Kolmogorov-Arnold representation
  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Proposition 3
  • proof
  • proof
  • proof
  • proof