Deep Symbolic Regression for Recurrent Sequences

TL;DR

This work introduces transformer-based symbolic regression for recurrent sequences, enabling a model to infer the underlying recurrence from a short initial segment of a sequence. It develops dual data-generation and encoding pipelines for integer and float sequences, compares symbolic recurrence discovery with numeric extrapolation, and demonstrates strong in-domain performance alongside meaningful out-of-domain insights on OEIS data and out-of-vocabulary tokens. Key contributions include a beam-ranking strategy that leverages initial terms, explicit handling of recurrence trees in prefix form, and evidence that the approach can approximate complex constants and function expressions asymptotically. The results suggest practical potential for exact recurrence discovery and principled approximations, while also outlining limitations in distribution shifts, noise robustness, and the need for extensions to broader mathematical structures.

Abstract

Symbolic regression, i.e. predicting a function from the observation of its values, is well-known to be a challenging task. In this paper, we train Transformers to infer the function or recurrence relation underlying sequences of integers or floats, a typical task in human IQ tests which has hardly been tackled in the machine learning literature. We evaluate our integer model on a subset of OEIS sequences, and show that it outperforms built-in Mathematica functions for recurrence prediction. We also demonstrate that our float model is able to yield informative approximations of out-of-vocabulary functions and constants, e.g. $\operatorname{bessel0}(x)\approx \frac{\sin(x)+\cos(x)}{\sqrt{πx}}$ and $1.644934\approx π^2/6$. An interactive demonstration of our models is provided at https://symbolicregression.metademolab.com.

Figures (9)

• Figure 1: The symbolic model extrapolates further and with higher precision than the numeric model. From left to right, we vary the tolerance $\tau$, the number of predictions $n_{pred}$, the number of operators $o$, the recurrence degree $d$ and the number of input terms $l$. In each plot, we use the following defaults for quantities which are not varied: $\tau=10^{-10}$, $n_{pred}=10$, $o\in[\![1,10]\!]$, $d\in[\![1,6]\!]$, $l\in[\![5,30]\!]$.
• Figure 2: Accuracy of our models on various in-domain and out-of-domain groups. We set $\tau=10^{-10}$, $n_{pred}=10$.
• Figure 3: The number embeddings reveal intriguing mathematical structure. We represented the t-SNE of the embeddings of the integer model and the exponent embeddings of the float model. We depicted the first 100 integer embeddings (10,000 in the model), and the exponent embeddings -40 to 40 (-100 to 100 in the model).
• Figure 4: Simplification reduces the training loss, but does not bring any improvement in test accuracy. We displayed the first 40 epochs of training of our symbolic models.
• Figure 5: Our models only see a small fraction of the possible expressions during training. We report the number of possible expressions for each number of operators (skeleton refers to an expression with the choice of leaves factored out). Even after a hundred epochs, our models have only seen a fraction of the possible expressions with more than 4 operators.