Symbolic Regression with Multimodal Large Language Models and Kolmogorov Arnold Networks

TL;DR

Symbolic regression is tackled by merging multimodal large language models with FunSearch-inspired genetic search and Kolmogorov–Arnold representations. The authors introduce LLM-LEx for univariate tasks and extend to multivariate problems with KAN-LEx, which decomposes multivariate functions into sums of univariate edge functions and then simplifies the composed expression. Benchmarks show LLM-LEx can frequently recover exact expressions better than traditional engines on random function sets, while KAN-LEx demonstrates the feasibility of solving multivariate cases through edge-wise regression and graph-based simplification. The work highlights practical viability with modest code and outlines future directions, including stronger genetic algorithms, vision transformers for higher dimensions, and domain-specific prompt engineering, all complemented by publicly available code.

Abstract

We present a novel approach to symbolic regression using vision-capable large language models (LLMs) and the ideas behind Google DeepMind's Funsearch. The LLM is given a plot of a univariate function and tasked with proposing an ansatz for that function. The free parameters of the ansatz are fitted using standard numerical optimisers, and a collection of such ansätze make up the population of a genetic algorithm. Unlike other symbolic regression techniques, our method does not require the specification of a set of functions to be used in regression, but with appropriate prompt engineering, we can arbitrarily condition the generative step. By using Kolmogorov Arnold Networks (KANs), we demonstrate that ``univariate is all you need'' for symbolic regression, and extend this method to multivariate functions by learning the univariate function on each edge of a trained KAN. The combined expression is then simplified by further processing with a language model.

Figures (8)

• Figure 1: An example function to learn
• Figure 2: The structure of LLM-LEx.
• Figure 3: Four example functions found by LLM-LEx with various scores. The functions returned by the algorithm and their scores are given in Table \ref{['tab:Compare']} and Table \ref{['tab:CompareScores']}, respectively. The number of points used to produce the plots are the same that we used to provide the graph of the function to LLM-LEx, which leads to the aliasing effect in the highly oscillatory part of the last function.
• Figure 4: Dense vs. Sparse sampling of $x$ values for $f(x) = 4.67315 + \cos(\exp(x))$.
• Figure 5: Compute times and scores of LLM-LEx across different LLMs. Below the dashed red line, the genetic algorithm is stopped early due to high performance.