Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cauchy activation function and XNet

Xin Li, Zhihong Xia, Hongkun Zhang

TL;DR

This work introduces the Cauchy activation function, derived from the Cauchy integral theorem, and the XNet architecture to achieve high-precision function approximation with shallower networks. The authors prove a Cauchy Approximation Theorem and a General Approximation Theorem, establishing $O(m^{-k})$ convergence for any $k>0$ and extending universal approximation to higher dimensions. Empirically, Cauchy activation improves performance on MNIST and CIFAR-10, and delivers substantial advantages in solving low- and high-dimensional PDEs, including high-dimensional Allen–Cahn problems, often outperforming PINNs and standard activations while reducing network depth and training time. The results imply significant potential for accurate, efficient scientific computing and CV tasks, where high-order local approximation and rapid convergence are valuable. Overall, the Cauchy activation enables higher precision with simpler architectures, suggesting broad applicability across image processing and computational mathematics.

Abstract

We have developed a novel activation function, named the Cauchy Activation Function. This function is derived from the Cauchy Integral Theorem in complex analysis and is specifically tailored for problems requiring high precision. This innovation has led to the creation of a new class of neural networks, which we call (Comple)XNet, or simply XNet. We will demonstrate that XNet is particularly effective for high-dimensional challenges such as image classification and solving Partial Differential Equations (PDEs). Our evaluations show that XNet significantly outperforms established benchmarks like MNIST and CIFAR-10 in computer vision, and offers substantial advantages over Physics-Informed Neural Networks (PINNs) in both low-dimensional and high-dimensional PDE scenarios.

Cauchy activation function and XNet

TL;DR

This work introduces the Cauchy activation function, derived from the Cauchy integral theorem, and the XNet architecture to achieve high-precision function approximation with shallower networks. The authors prove a Cauchy Approximation Theorem and a General Approximation Theorem, establishing convergence for any and extending universal approximation to higher dimensions. Empirically, Cauchy activation improves performance on MNIST and CIFAR-10, and delivers substantial advantages in solving low- and high-dimensional PDEs, including high-dimensional Allen–Cahn problems, often outperforming PINNs and standard activations while reducing network depth and training time. The results imply significant potential for accurate, efficient scientific computing and CV tasks, where high-order local approximation and rapid convergence are valuable. Overall, the Cauchy activation enables higher precision with simpler architectures, suggesting broad applicability across image processing and computational mathematics.

Abstract

We have developed a novel activation function, named the Cauchy Activation Function. This function is derived from the Cauchy Integral Theorem in complex analysis and is specifically tailored for problems requiring high precision. This innovation has led to the creation of a new class of neural networks, which we call (Comple)XNet, or simply XNet. We will demonstrate that XNet is particularly effective for high-dimensional challenges such as image classification and solving Partial Differential Equations (PDEs). Our evaluations show that XNet significantly outperforms established benchmarks like MNIST and CIFAR-10 in computer vision, and offers substantial advantages over Physics-Informed Neural Networks (PINNs) in both low-dimensional and high-dimensional PDE scenarios.
Paper Structure (17 sections, 2 theorems, 40 equations, 21 figures, 9 tables)

This paper contains 17 sections, 2 theorems, 40 equations, 21 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Algorithm Development
  3. Enhanced Neural Network Efficiency with Cauchy Activation Function: High-Order Approximation and Beyond
  4. Approximation Theorems
  5. Examples
  6. Regression Task
  7. High-Order Approximation Analysis
  8. Comparison with ReLU under Noisy Conditions
  9. Handwriting Recognition: MNIST with XNet
  10. CIFAR-10
  11. PDE: Heat Function
  12. Poisson Equation with Dirichelet Boundary Condition
  13. Burger's equation
  14. Modifications for Experiments
  15. Results
  16. ...and 2 more sections

Key Result

Theorem 1

Let $f(z_1, z_2, \ldots z_N)$ be an analytic function in an open domain $U \subset \mathbb{C}^N$ and let $M \subset U$ be a compact subset in $U$. Given any $\epsilon >0$, there is a list of points $(\xi_1^k, \ldots, \xi_N^k)$, for $k=1, 2, \ldots, m$, in $U$ and corresponding parameters $\lambda_1, for all points $(z_1, z_2, \ldots z_N) \in M$.

Figures (21)

  • Figure 1: ReLU
  • Figure 2: Sigmoid
  • Figure 3: 2 terms of Cauchy activation, with $\lambda_1=\lambda_2=d=1$
  • Figure 4: Visualization of the two terms of the Cauchy activation function under different parameter settings.
  • Figure 5: Training Loss for [100].
  • ...and 16 more figures

Theorems & Definitions (2)

  • Theorem 1: Cauchy Approximation Theorem
  • Theorem 2: General Approximation Theorem