MetaSymNet: A Tree-like Symbol Network with Adaptive Architecture and Activation Functions

TL;DR

MetaSymNet reframes symbolic regression as real-time numerical optimization on a tree-like network whose internal nodes use PANGU metafunctions and leaves use Variable metafunctions, enabling adaptive growth and contraction to match task complexity. The activation functions evolve during training to yield concise, interpretable expressions, with an entropy-based loss promoting decisive symbol selection. Across extensive benchmarks, MetaSymNet achieves strong fit ($R^2$), robustness to noise, and efficient inference, while producing compact expressions compared to state-of-the-art baselines. These results suggest practical impact for extracting explainable mathematical models in science and engineering, where understanding the learned formulas is as important as predictive accuracy.

Abstract

Mathematical formulas serve as the means of communication between humans and nature, encapsulating the operational laws governing natural phenomena. The concise formulation of these laws is a crucial objective in scientific research and an important challenge for artificial intelligence (AI). While traditional artificial neural networks (MLP) excel at data fitting, they often yield uninterpretable black box results that hinder our understanding of the relationship between variables x and predicted values y. Moreover, the fixed network architecture in MLP often gives rise to redundancy in both network structure and parameters. To address these issues, we propose MetaSymNet, a novel neural network that dynamically adjusts its structure in real-time, allowing for both expansion and contraction. This adaptive network employs the PANGU meta function as its activation function, which is a unique type capable of evolving into various basic functions during training to compose mathematical formulas tailored to specific needs. We then evolve the neural network into a concise, interpretable mathematical expression. To evaluate MetaSymNet's performance, we compare it with four state-of-the-art symbolic regression algorithms across more than 10 public datasets comprising 222 formulas. Our experimental results demonstrate that our algorithm outperforms others consistently regardless of noise presence or absence. Furthermore, we assess MetaSymNet against MLP and SVM regarding their fitting ability and extrapolation capability, these are two essential aspects of machine learning algorithms. The findings reveal that our algorithm excels in both areas. Finally, we compared MetaSymNet with MLP using iterative pruning in network structure complexity. The results show that MetaSymNet's network structure complexity is obviously less than MLP under the same goodness of fit.

Figures (6)

• Figure 1: Flowchart of the MetaSymNet. (i) First randomly initialize a network, where the internal node $S$ is the PANGU meta-function and the leaf node $S_x$ is the Variable meta-function. (ii) A numerical optimization algorithm is used to optimize the parameters. In this process, the amplitude parameter $\mathcal{W}$ and the bias parameter $\mathcal{B}$ are optimized first, and then the selection parameters $\mathbb{Z}$ and $\mathbb{D}$ of the PANGU meta-functions and Variable meta-functions are optimized. Iterate several times. (iii) After parameter optimization, we determine the basic candidate symbols to which each PANGU metafunction and Variable metafunction should evolve based on the selection parameters $\mathbb{Z}$ and $\mathbb{D}$. (iv) When the network evolves into an expression, we further refine the constants of the expression and calculate the loss and $R^2$. The iteration stops when $R^2$ reaches the specified threshold. Otherwise, We replace the internal nodes (operation symbols) of the obtained expression binary tree with PANGU metafunctions and leaf nodes (variable symbols) with Variable metafunctions. The network structure is then supplemented with Variable meta-functions such that each PANGU meta-function has two children. The iteration continues.
• Figure 2: The structure of PANGU metafunctions and Variable metafunctions. The figure depicts the detailed internal structure diagram of the PANGU metafunction (top) and Variable metafunction (bottom). Note: The variable metafunctions are chosen from two variables at a time.
• Figure 3: Analysis on performance. \ref{['fig3a']} demonstrates the trends of $R^2$ values between MetaSymNet and five baseline methods across varying noise levels. \ref{['fig3b']} Provides a $R^2$-time Pareto plot of the various algorithms on individual datasets, showing optimization performance.
• Figure 4: The figure shows the full recovery rate of each algorithm on different datasets. Note: we ran each expression 50 times.
• Figure 5: MetaSymNet outperforms many baselines. Compared with SOTA (GP-based method), MetasymNet has significantly improved inference speed. We compared MetaSymNet's average test performance and inference time to baselines provided by the SRbench benchmark on the Feynman SR problem and black-box regression problem. Color is used to distinguish between three model families: deep-learning-based SR, genetic programming-based SR, and classical machine learning approaches which do not provide symbolic solutions.