Bootstrapping OTS-Funcimg Pre-training Model (Botfip) -- A Comprehensive Symbolic Regression Framework

TL;DR

Botfip tackles the SR bottleneck by introducing a multimodal framework that aligns function-image representations with operation-tree sequences using a BLIP-inspired encoder-decoder. It combines a ViT-based Funcimg encoder with a BERT-like OTS encoder/decoder, shared weights, and contrastive/matching pretraining plus an L-BFGS refinement step for constants, achieving strong in-domain SR performance on simpler expressions. The approach demonstrates robustness and efficiency benefits for small-to-moderate SR problems, while highlighting extrapolation limitations and the need for more data and error-correction strategies to scale to complex expressions and broader AI-for-Math tasks. Overall, Botfip presents a promising direction for integrating multimodal signals into symbolic regression, potentially enabling larger models and broader scientific computing applications with improved generalization and speed.

Abstract

In the field of scientific computing, many problem-solving approaches tend to focus only on the process and final outcome, even in AI for science, there is a lack of deep multimodal information mining behind the data, missing a multimodal framework akin to that in the image-text domain. In this paper, we take Symbolic Regression(SR) as our focal point and, drawing inspiration from the BLIP model in the image-text domain, propose a scientific computing multimodal framework based on Function Images (Funcimg) and Operation Tree Sequence (OTS), named Bootstrapping OTS-Funcimg Pre-training Model (Botfip). In SR experiments, we validate the advantages of Botfip in low-complexity SR problems, showcasing its potential. As a MED framework, Botfip holds promise for future applications in a broader range of scientific computing problems.

Figures (4)

• Figure 1: The Botfip framework demonstration, where the left blue part shows the main structure of various components of Botfip (including Funcimg Encoder, OTS Encoder/Decoder) under different pre-training tasks, and also demonstrates the different tasks (FOC, FOM, etc.) in the entire pre-training process. The right green part displays the inference process that model first generate OTS, then output OTS and corresponding constants array obtained through random initialization and iterative optimization using the LBFGS algorithm together.
• Figure 2: The architecture of the operation tree generation system, the process of generating Funcimg-OTS pairs, and the dataset formulation procedure.
• Figure 3: Validation results visualization after OTS fine-tune modeling task. Specifically, Fig. \ref{['fig:validation_acc_node_num']} presents the numerical curve of the model's validation loss across datasets composed of operation trees with varying numbers of nodes. Fig.\ref{['fig:validation_acc_noise']} shows the curve of validation results on a five nodes dataset with input noise. The dashed and solid lines respectively represent the trained model with and without input noise during the pre-training and fine-tuning phases.
• Figure 4: Comparison of Constants Array Update Results between AdamW and LBFGS Optimizers.