A Functional Analysis Approach to Symbolic Regression

TL;DR

This work reframes symbolic regression as a norm-minimization problem in a Hilbert space $\mathbb{F}$ of functions on the training set and introduces Fourier Tree Growing (FTG), an FA-inspired SR algorithm. FTG iteratively builds a linearly independent set of atom functions $v_i$ from elementary operators and computes coefficients $\alpha_i$ by projecting onto their span using Gram-matrix computations, yielding a Fourier-like expansion of the target function $F$. Empirical results show FTG substantially outperforming traditional tree-based GP on classical one-dimensional SR benchmarks and on a large-scale polynomial benchmark (LSP), while identifying Gram-matrix conditioning as a key numerical bottleneck; these insights motivate future FA-based refinements and hybrid GP-FTG approaches. The work thus provides a novel theoretical lens and practical algorithmic tool for advancing SR and explainable ML through functional-analytic methods.

Abstract

Symbolic regression (SR) poses a significant challenge for randomized search heuristics due to its reliance on the synthesis of expressions for input-output mappings. Although traditional genetic programming (GP) algorithms have achieved success in various domains, they exhibit limited performance when tree-based representations are used for SR. To address these limitations, we introduce a novel SR approach called Fourier Tree Growing (FTG) that draws insights from functional analysis. This new perspective enables us to perform optimization directly in a different space, thus avoiding intricate symbolic expressions. Our proposed algorithm exhibits significant performance improvements over traditional GP methods on a range of classical one-dimensional benchmarking problems. To identify and explain limiting factors of GP and FTG, we perform experiments on a large-scale polynomials benchmark with high-order polynomials up to degree 100. To the best of the authors' knowledge, this work represents the pioneering application of functional analysis in addressing SR problems. The superior performance of the proposed algorithm and insights into the limitations of GP open the way for further advancing GP for SR and related areas of explainable machine learning.

Figures (3)

• Figure 1: Visualization of the first (a) and second (b) steps that FTG can make for a one-dimensional domain.
• Figure 2: Comparison of success rate (a) and median of function evaluations to reach the given tolerance (b) of FTG with three standard GP algorithms: $(1+1)$-GP, $(1+\lambda)$-GP, Canonical-GP across a range of benchmark problems and tolerance levels.
• Figure 3: Performance Comparison of Canonical-GP (grey) and $(1+\lambda)$-GP (light red) and FTG (red) on the proposed Large-Scale Polynomial Benchmarks.

Theorems & Definitions (12)

• Definition 1: Metric Space
• Definition 2: Normed Vector Space
• Definition 3: Cauchy Sequence
• Definition 4: Complete Space
• Definition 5: Inner Product Space
• Definition 6: Hilbert Space
• Theorem 1: Projection Theorem kantorovich2016functional
• Definition 7: Linear Independence
• Definition 8
• Theorem 2