SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training

TL;DR

SNIP introduces Symbolic-Numeric Integrated Pre-training with dual Transformer encoders for symbolic and numeric data, trained via a symmetric contrastive objective to align cross-modal embeddings expressed as $\bm{Z}_S$ and $\bm{Z}_V$. Pre-training on about $60$ million synthetic paired examples enables cross-modal property prediction and symbolic regression by leveraging a latent space that encodes both symbolic structure and numeric behavior. The approach yields strong performance in low-data regimes, interpretable latent spaces, and a generative latent space that supports latent-space optimization for symbolic discovery on SRBench. This work provides a scalable foundation for bridging symbolic mathematics with numeric observations, enabling cross-modal reasoning and efficient symbolic regression through pre-trained shared representations.

Abstract

In an era where symbolic mathematical equations are indispensable for modeling complex natural phenomena, scientific inquiry often involves collecting observations and translating them into mathematical expressions. Recently, deep learning has emerged as a powerful tool for extracting insights from data. However, existing models typically specialize in either numeric or symbolic domains, and are usually trained in a supervised manner tailored to specific tasks. This approach neglects the substantial benefits that could arise from a task-agnostic multi-modal understanding between symbolic equations and their numeric counterparts. To bridge the gap, we introduce SNIP, a Symbolic-Numeric Integrated Pre-training model, which employs contrastive learning between symbolic and numeric domains, enhancing their mutual similarities in the embeddings. By performing latent space analysis, we observe that SNIP provides cross-domain insights into the representations, revealing that symbolic supervision enhances the embeddings of numeric data and vice versa. We evaluate SNIP across diverse tasks, including symbolic-to-numeric mathematical property prediction and numeric-to-symbolic equation discovery, commonly known as symbolic regression. Results show that SNIP effectively transfers to various tasks, consistently outperforming fully supervised baselines and competing strongly with established task-specific methods, especially in the low data regime scenarios where available data is limited. Code and model are available at: https://github.com/deep-symbolic-mathematics/Multimodal-Math-Pretraining

Figures (14)

• Figure 1: The SNIP Framework: A schematic representation of the dual-encoder pre-training scheme for mutual learning between symbolic equations and their numerical observations. Both symbolic and numeric encoders work in tandem, capturing the paired similarities and essence of their respective modalities.
• Figure 2: 2D t-SNE representations of the encoded vectors across three model variants, colored for (top) Non-Convexity Ratio and (bottom) Function Upwardness prediction tasks.
• Figure 3: $R^2$ scores for NCR property prediction task vs. the number of training samples.
• Figure 4: Using SNIP for Symbolic Regression: (a) Training includes adding an expression generation module atop SNIP's numeric encoder; (b) Inference aims to enhance expressions by optimizing within SNIP's interpolatable latent space.
• Figure 5: Interpolatability of SNIP numeric latent space.