Can Test-time Computation Mitigate Memorization Bias in Neural Symbolic Regression?

TL;DR

This paper investigates memorization bias as a core limitation of Transformer-based neural symbolic regression (NSR). Through controlled experiments and a theoretical analysis, it shows that Transformers tend to memorize training expressions and struggle to compose novel formulas that also accurately fit data. It then evaluates test-time strategies—large-beam decoding, TPSR, and NSR-gvs—that provide additional information at inference to mitigate bias, finding that such strategies can increase novelty but do not necessarily improve or may even harm numerical accuracy. The results illuminate a fundamental tension between generalization and memorization in NSR and motivate future work on robust, compositional symbolic regression methods. Overall, the findings offer a foundation for designing NSR approaches that balance bias reduction with numerical fidelity in scientific applications.

Abstract

Symbolic regression aims to discover mathematical equations that fit given numerical data. It has been applied in various fields of scientific research, such as producing human-readable expressions that explain physical phenomena. Recently, Neural symbolic regression (NSR) methods that involve Transformers pre-trained on large-scale synthetic datasets have gained attention. While these methods offer advantages such as short inference time, they suffer from low performance, particularly when the number of input variables is large. In this study, we hypothesized that this limitation stems from the memorization bias of Transformers in symbolic regression. We conducted a quantitative evaluation of this bias in Transformers using a synthetic dataset and found that Transformers rarely generate expressions not present in the training data. Additional theoretical analysis reveals that this bias arises from the Transformer's inability to construct expressions compositionally while verifying their numerical validity. We finally examined if tailoring test-time strategies can lead to reduced memorization bias and better performance. We empirically demonstrate that providing additional information to the model at test time can significantly mitigate memorization bias. On the other hand, we also find that reducing memorization bias does not necessarily correlate with improved performance. These findings contribute to a deeper understanding of the limitations of NSR approaches and offer a foundation for designing more robust, generalizable symbolic regression methods. Code is available at https://github.com/Shun-0922/Mem-Bias-NSR .

Figures (6)

• Figure 1: NeSymReS exhibits strong fitting performance on the baseline dataset but performs poorly on expressions from the not_included dataset, whose tree structures are absent from the training data.
• Figure 2: Overview of NSR-gvs's inference procedure. We first sample subtrees from the candidate pool, then supply them to the model together with numerical data. Then, the generated prediction is numerically verified and the self-verification feedback is used to update the candidate pool. This procedure is performed repeatedly to generate better predictions over time.
• Figure 3: Evaluation of test-time strategies on the not_included dataset. The x-axis represents the percentage of expressions generated that were not included in the training data. The y-axis shows the proportion of expressions that exceeded the $R^2$ thresholds of $0.5$, $0.9$, $0.95$, $0.99$, $0.999$, $0.9999$ and $0.99999$, respectively.
• Figure 4: Comparison of test-time strategies under practical settings. The figure on the left shows the performance on the AI-Feynman dataset, and the figure on the right presents results on the only_five_variables_nc dataset.
• Figure 5: Trade-off between performance and computational cost for different test-time strategies. We varied the beam sizes for each model as follows: $\{1, 5, 50, 100, 150\}$ for NeSymReS, and $\{1, 3, 5\}$ for both NeSymReS+TPSR and NSR-gvs. For NSR-gvs+TPSR, we only experimented with beam size set to $1$. The left figure shows the trade-off between the ability to generate expressions and computational cost, while the right figure shows the trade-off between numerical accuracy and computational cost.

Theorems & Definitions (11)

• Definition 1: Last-token prediction problem
• Theorem 1
• Definition 2: $(D,d)$-bounded log-precision Transformer
• Lemma 1: Circuit simulation merrill2023parallelism
• Definition 3: Boolean formula value problem
• Lemma 2: Bracket parsing feng2023towards
• Theorem 2: Bounded log-precision Transformer lower bound
• proof
• Theorem 3: informal
• Theorem 4: PAC guarantee