MathLearner: A Large Language Model Agent Framework for Learning to Solve Mathematical Problems

TL;DR

MathLearner tackles the limited mathematical reasoning of large language models by embedding inductive learning principles into a two-module agent framework that learns from examples and retrieves learned strategies. It combines Parsel-based program generation with a feature-based retrieval system backed by a vector database to reuse solved methods and guide code generation for new problems. In experiments on the MATH dataset, MathLearner improves global accuracy from 0.4133 to 0.50, with a profitability of 0.2096, precision accuracy of 0.5155, and a target achievement rate of 0.1754, solving 75 of 150 problems. The work demonstrates enhanced generalization to unseen problems and holds promise as a personalized educational tool, leveraging inductive reasoning and retrieval-augmented generation to democratize access to high-quality math tutoring.

Abstract

With the development of artificial intelligence (AI), large language models (LLM) are widely used in many fields. However, the reasoning ability of LLM is still very limited when it comes to mathematical reasoning. Mathematics plays an important role in all aspects of human society and is a technical guarantee in the fields of healthcare, transport and aerospace, for this reason, the development of AI big language models in the field of mathematics has great potential significance. To improve the mathematical reasoning ability of large language models, we proposed an agent framework for learning to solve mathematical problems based on inductive reasoning. By emulating the human learning process of generalization of learned information and effective application of previous knowledge in new reasoning tasks, this framework has great performance in the mathematical reasoning process. It improves global accuracy over the baseline method (chain-of-thought) by 20.96% and solves 17.54% of the mathematical problems that the baseline cannot solve. Benefiting from the efficient RETRIEVAL method, our model improves the ability of large language models to efficiently use external knowledge, i.e., the mathematical computation of the model can be based on written procedures. In education, our model can be used as a personalised learning aid, thus reducing the inequality of educational resources.

Figures (4)

• Figure 1: a case study on using MathLearner, which performs two main steps: extracting feature values from the input problem and finding similar solution steps in the database based on these feature values. If similar solution steps exist in the database, we send these steps along with the new input problem to GPT so that GPT can generate Python code to solve the problem. If similar problem-solving steps do not exist in the database, we send the problem directly to GPT, allowing it to generate the Python code that solves it. This process makes the problem-solving process more efficient and precise.
• Figure 2: An overview of the pipeline of Learning Module. From left to right, the large language model 1) receives examples, 2) generates modified solutions, 3) verifies and modifies possible errors in the solution, and 4) generates features for questions. Then, the features and solution will be saved into a database.
• Figure 3: An overview of the pipeline of Application Module. From left to right, the large language model 1) receives questions, 2) generates features for entered questions, 3) retrieves the closest solution, and 4) generates new solutions for entered questions.
• Figure 4: The category of the questions ($U = C\&R + C\&\neg R + \neg C\&R + \neg C\& \neg R$)