Pretrained Language Models are Symbolic Mathematics Solvers too!

TL;DR

This work demonstrates that pretrained language models trained on translation tasks can effectively solve symbolic mathematics, notably symbolic integration, with substantially fewer labeled examples than prior symbolic DL methods. By framing symbolic math as Seq2Seq translation and leveraging $m$BART$/$Marian-MT architectures, the approach achieves competitive integration accuracy using about $1.5$ orders of magnitude fewer data, while differential equations remain more demanding due to higher-order recursions. The authors propose a theory based on Anna Karenina Principle and Lotka–Volterra dynamics to explain generalization and feature survival during fine-tuning, and they extensively study robustness to distribution shifts including generation-type and equation-type shifts. Overall, the study suggests that language pretraining can endow symbolic math solvers with strong data efficiency and robustness, with some limitations in high-recursion tasks like higher-order ODEs.

Abstract

Solving symbolic mathematics has always been of in the arena of human ingenuity that needs compositional reasoning and recurrence. However, recent studies have shown that large-scale language models such as transformers are universal and surprisingly can be trained as a sequence-to-sequence task to solve complex mathematical equations. These large transformer models need humongous amounts of training data to generalize to unseen symbolic mathematics problems. In this paper, we present a sample efficient way of solving the symbolic tasks by first pretraining the transformer model with language translation and then fine-tuning the pretrained transformer model to solve the downstream task of symbolic mathematics. We achieve comparable accuracy on the integration task with our pretrained model while using around $1.5$ orders of magnitude less number of training samples with respect to the state-of-the-art deep learning for symbolic mathematics. The test accuracy on differential equation tasks is considerably lower comparing with integration as they need higher order recursions that are not present in language translations. We propose the generalizability of our pretrained language model from Anna Karenina Principle (AKP). We pretrain our model with different pairs of language translations. Our results show language bias in solving symbolic mathematics tasks. Finally, we study the robustness of the fine-tuned model on symbolic math tasks against distribution shift, and our approach generalizes better in distribution shift scenarios for the function integration.

Figures (1)

• Figure 1: The accuracy of our mBART language model and the LC model when trained on different training sample sizes. Panels (a), (b), and (c) are for the integration task.