In Context Learning and Reasoning for Symbolic Regression with Large Language Models

TL;DR

This work investigates using large language models (LLMs) for symbolic regression by combining in-context learning with data- and theory-driven prompts and an external optimization loop. The proposed two-prompt, scratchpad-enabled workflow allows LLMs to propose symbolic expressions, which are then numerically optimized by SciPy and fed back for refinement, enabling iterative improvement toward accurate and parsimonious equations. Across multiple scientific datasets, GPT-4 and GPT-4o can rediscover several known laws, with performance boosted by including scientific context and stepwise scratchpad reasoning, though they struggle on noisier, larger datasets and do not outperform specialized SR methods on more complex targets. The study demonstrates that natural-language interfaces can facilitate theory-data integration and broaden accessibility to symbolic regression, while highlighting trade-offs related to data quality, computational cost, and prompt design; future work may leverage retrieval augmentation and more robust evaluation to enhance scalability and reliability.

Abstract

Large Language Models (LLMs) are transformer-based machine learning models that have shown remarkable performance in tasks for which they were not explicitly trained. Here, we explore the potential of LLMs to perform symbolic regression -- a machine-learning method for finding simple and accurate equations from datasets. We prompt GPT-4 to suggest expressions from data, which are then optimized and evaluated using external Python tools. These results are fed back to GPT-4, which proposes improved expressions while optimizing for complexity and loss. Using chain-of-thought prompting, we instruct GPT-4 to analyze the data, prior expressions, and the scientific context (expressed in natural language) for each problem before generating new expressions. We evaluated the workflow in rediscovery of five well-known scientific equations from experimental data, and on an additional dataset without a known equation. GPT-4 successfully rediscovered all five equations, and in general, performed better when prompted to use a scratchpad and consider scientific context. We demonstrate how strategic prompting improves the model's performance and how the natural language interface simplifies integrating theory with data. We also observe how theory can sometimes offset noisy data and, in other cases, data can make up for poor context. Although this approach does not outperform established SR programs where target equations are more complex, LLMs can nonetheless iterate toward improved solutions while following instructions and incorporating scientific context in natural language.

Figures (20)

• Figure 1: Illustration of GPT-4 attempting symbolic regression. GPT-4 predicts expressions with optimized coefficients when passed a dataset for nitrogen adsorption on mica langmuir_adsorption_1918. The Python code snippet from Prompt 2 output has been truncated to keep the figure concise. Note that the actual parameter values produced by running the code differ from what GPT-4 generates. Figure \ref{['gpt3.5Test1']} in SI shows the output from GPT-3.5 turbo which produces similar results.
• Figure 2: Workflow for using LLMs for SR. First, the dataset is sent to LLMs (with or without context) which is instructed to suggest expressions without optimizing parameters. The generated expressions are then parsed by Python and optimized using SciPy (Nelder-Mead nelder1965simplex method with basin-hopping as the numerical optimizer wales1997global). Results for each expression are stored in a Python dictionary, and added to a list of results from prior expressions. The top performing expressions are sent as feedback to LLMs, which is asked to suggest better expressions optimizing for both complexity and loss. The feedback loop is run for a set number of iterations.
• Figure 3: Bias from examples GPT-4 generated exact expressions from the prompt examples that were given to illustrate the output syntax. The revised prompts (Prompt 2 and 3) eliminate this bias from the equation prediction process.
• Figure 4: Illustration of scratchpad approach. We observe substantial, qualitative improvements in the predicted expressions after implementing the scratchpad technique (Prompt 2). The suggested expressions for (a) Langmuir's and (b) Kepler's Law dataset include operators ($/$ and $\text{sqrt}$, respectively) present in the target models ($y = \frac{c_1*x}{c_2+x}$ and $y^2 = c_1x^{\frac{3}{2}}$, respectively)
• Figure 5: Illustration of GPT-4 outputs for SR-similar expressions.