Towards Scaling Laws for Symbolic Regression

TL;DR

This work investigates whether symbolic regression (SR) follows scaling laws analogous to those observed in language models by training a scalable end-to-end transformer SR pipeline on synthetic data with expressions limited to two variables and integer constants. It demonstrates that both validation loss and solved-rate follow power-law dependences on compute across five model sizes, and identifies compute-dependent trends for optimal learning rate, batch size, and a token-to-parameter ratio around $ ext{$oxed{15}$}$ that yields good efficiency in the studied regime. The findings provide a principled, compute-driven design framework for next-generation SR models and suggest that scaling compute, data, and model size in a coordinated way can yield predictable, substantial gains. Overall, the work lays foundational scaling laws for SR and highlights practical guidelines for developing more capable symbolic regression systems.

Abstract

Symbolic regression (SR) aims to discover the underlying mathematical expressions that explain observed data. This holds promise for both gaining scientific insight and for producing inherently interpretable and generalizable models for tabular data. In this work we focus on the basics of SR. Deep learning-based SR has recently become competitive with genetic programming approaches, but the role of scale has remained largely unexplored. Inspired by scaling laws in language modeling, we present the first systematic investigation of scaling in SR, using a scalable end-to-end transformer pipeline and carefully generated training data. Across five different model sizes and spanning three orders of magnitude in compute, we find that both validation loss and solved rate follow clear power-law trends with compute. We further identify compute-optimal hyperparameter scaling: optimal batch size and learning rate grow with model size, and a token-to-parameter ratio of $\approx$15 is optimal in our regime, with a slight upward trend as compute increases. These results demonstrate that SR performance is largely predictable from compute and offer important insights for training the next generation of SR models.

Figures (8)

• Figure 1: Overview of our two-step data generation. In the first step, we recursively generate a set of base expressions by applying a set of binary operators (BOP) and a set of unary operators (UOP). In the second step, we sample expression-dataset pairs from our base expressions.
• Figure 2: Overview of our model architecture.
• Figure 3: $\text{Acc}_{\text{solved}}$ scales as a power law of training compute. Each marker corresponds to a trained model and depicts the mean perfect-solved ratio over three random seeds. Plot design inspired by franke2025learning.
• Figure 4: Results of our hyperparameter grid search. For each model size we trained different combinations of batch size and learning rate using a token-to-parameter ratio of 20, until we found an optimum. The stars indicate the runs with the lowest loss for each model size and batch size. Plot design inspired by wortsman2023small.
• Figure 5: Compute-optimal batch size and learning rate as functions of model size $N$.