MMSR: Symbolic Regression is a Multi-Modal Information Fusion Task

TL;DR

Experimental results show that compared with multiple large-scale pre-training baselines, MMSR achieves the most advanced results on multiple mainstream datasets including SRBench.

Abstract

Mathematical formulas are the crystallization of human wisdom in exploring the laws of nature for thousands of years. Describing the complex laws of nature with a concise mathematical formula is a constant pursuit of scientists and a great challenge for artificial intelligence. This field is called symbolic regression (SR). Symbolic regression was originally formulated as a combinatorial optimization problem, and Genetic Programming (GP) and Reinforcement Learning algorithms were used to solve it. However, GP is sensitive to hyperparameters, and these two types of algorithms are inefficient. To solve this problem, researchers treat the mapping from data to expressions as a translation problem. And the corresponding large-scale pre-trained model is introduced. However, the data and expression skeletons do not have very clear word correspondences as the two languages do. Instead, they are more like two modalities (e.g., image and text). Therefore, in this paper, we proposed MMSR. The SR problem is solved as a pure multi-modal problem, and contrastive learning is also introduced in the training process for modal alignment to facilitate later modal feature fusion. It is worth noting that to better promote the modal feature fusion, we adopt the strategy of training contrastive learning loss and other losses at the same time, which only needs one-step training, instead of training contrastive learning loss first and then training other losses. Because our experiments prove training together can make the feature extraction module and feature fusion module wearing-in better. Experimental results show that compared with multiple large-scale pre-training baselines, MMSR achieves the most advanced results on multiple mainstream datasets including SRBench. Our code is open source at https://github.com/1716757342/MMSR

Figures (9)

• Figure 1: Multi-model-based Symbolic Regression task versus multi-model-based Image Caption task. (a) Multi-model-based Symbolic regression task. In symbolic Regression task, an expression is expected to get from a set of input data [X,Y]. It can be considered as a multi-modal task, where the expression and the data are two different modalities. (b) Multi-model-based Image Caption task. The text modality is output to caption an image.
• Figure 2: Expression binary tree. A binary tree representation of the expression $y = 4.2*x_2 + \sin(x_1)$ and a preorder traversal (skeleton) of its binary tree.
• Figure 3: The framework of MMSR model. Among them, the fig. (b) shows the overall algorithm flowchart of MMSR. The fig. (a) shows the structure of the skeleton encoder of MMSR, and the fig. (c) shows the structure of the decoder, note that in the decoder, we use the cross-attention mechanism in the Multi-head cross-attention module for information fusion.
• Figure 4: The feature fusion process of MMSR model. The fused features are calculated from the feature information embedded implicitly in the data and expression skeleton.
• Figure 5: Radar charts demonstrating the fitting ability ($R^2$). It can be found that MMSR achieved the best fitting performance among all the comparison methods.