Advancing Symbolic Discovery on Unsupervised Data: A Pre-training Framework for Non-degenerate Implicit Equation Discovery

TL;DR

The paper tackles the challenge of discovering implicit equations from unsupervised data, where the governing relation is $f(\mathbf{x})=0$. It introduces PIE, a pre-trained neural symbolic model that treats implicit equation discovery as translating data into an equation skeleton and then refines constants via optimization, thereby avoiding degenerate solutions. A CL-FEM-based fitness guides constant optimization to suppress degeneracy, enhancing robustness. Empirical results on synthetic in-domain data and the AI-Feynman out-of-domain set show PIE achieving state-of-the-art performance and strong generalization, signaling a meaningful step toward unsupervised scientific discovery with symbolic interpretability.

Abstract

Symbolic regression (SR) -- which learns symbolic equations to describe the underlying relation from input-output pairs -- is widely used for scientific discovery. However, a rich set of scientific data from the real world (e.g., particle trajectories and astrophysics) are typically unsupervised, devoid of explicit input-output pairs. In this paper, we focus on symbolic implicit equation discovery, which aims to discover the mathematical relation from unsupervised data that follows an implicit equation $f(\mathbf{x}) =0$. However, due to the dense distribution of degenerate solutions (e.g., $f(\mathbf{x})=x_i-x_i$) in the discrete search space, most existing SR approaches customized for this task fail to achieve satisfactory performance. To tackle this problem, we introduce a novel pre-training framework -- namely, Pre-trained neural symbolic model for Implicit Equation (PIE) -- to discover implicit equations from unsupervised data. The core idea is that, we formulate the implicit equation discovery on unsupervised scientific data as a translation task and utilize the prior learned from the pre-training dataset to infer non-degenerate skeletons of the underlying relation end-to-end. Extensive experiments shows that, leveraging the prior from a pre-trained language model, PIE effectively tackles the problem of degenerate solutions and significantly outperforms all the existing SR approaches. PIE shows an encouraging step towards general scientific discovery on unsupervised data.

Figures (5)

• Figure 1: Three scientific discovery examples on unsupervised data following specific implicit equations. Specifically, Figure \ref{['fig: example1']} is a hyperbola equation for particle trajectories, Figure \ref{['fig: example2']} is a elliptical equation for celestial motion, and Figure \ref{['fig: example3']} is a inverse proportion equation. See the legend for their complete forms. Compare the learned implicit equations, we conclude that PIE effectively tackles the degenerate solutions and significantly outperforms all the other baselines.
• Figure 2: Visualize the limitation of vanilla SR approaches and the intuition of our pre-training framework.
• Figure 3: Illustration of the framework of PIE for symbolic implicit equation discovery. Part 1 shows the generation process of the pre-training dataset $\mathcal{D}_{\text{pre-train}}=\{(\mathcal{D}_k, \tilde{f}_k)\}_{k=1}^{K}$; Part 2 shows the model architecture, which contains an embedding layer to handle raw floating-point data nesymbres and a Set Transformer set-transformer to translate the input to the skeleton; Part 3 shows the inference process, which combines the beam search with the CL-FEM fitness cl-fem to further avoid degenerate solutions.
• Figure 4: Evaluate the robustness under noisy data and different number of data points on Synthetic. Results show that PIE is highly robust under different noise level and number of input data points.
• Figure 5: We evaluate different hyperparameters we used for training and inference and report their sensitivity in terms of different metrics. Results show that the performance of PIE is sensitive to the size of the pre-training dataset, while it is relatively insensitive to the number of the input data points during training, the size of beam search during inference, and the tolerance level during inference.