MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactions

TL;DR

MathChat exposes a critical gap in current math-focused LLMs: strong single-turn problem solving does not guarantee capability in multi-turn, instruction-following mathematical dialogues. The authors introduce a GSM8K-derived benchmark with four tasks to probe multiturn reasoning and open-ended generation, and propose MathChat_sync, a synthetic dialogue dataset for supervised fine-tuning. Experimental results show existing math-specialized LLMs struggle on MathChat, while SFT with MathChat_sync improves open-ended task performance and can boost, without severely harming, direct problem solving. This work offers a concrete benchmark and a practical SFT approach to advancing interactive, real-world mathematical problem solving in LLMs.

Abstract

Large language models (LLMs) have demonstrated impressive capabilities in mathematical problem solving, particularly in single turn question answering formats. However, real world scenarios often involve mathematical question answering that requires multi turn or interactive information exchanges, and the performance of LLMs on these tasks is still underexplored. This paper introduces MathChat, a comprehensive benchmark specifically designed to evaluate LLMs across a broader spectrum of mathematical tasks. These tasks are structured to assess the models' abilities in multiturn interactions and open ended generation. We evaluate the performance of various SOTA LLMs on the MathChat benchmark, and we observe that while these models excel in single turn question answering, they significantly underperform in more complex scenarios that require sustained reasoning and dialogue understanding. To address the above limitations of existing LLMs when faced with multiturn and open ended tasks, we develop MathChat sync, a synthetic dialogue based math dataset for LLM finetuning, focusing on improving models' interaction and instruction following capabilities in conversations. Experimental results emphasize the need for training LLMs with diverse, conversational instruction tuning datasets like MathChatsync. We believe this work outlines one promising direction for improving the multiturn mathematical reasoning abilities of LLMs, thus pushing forward the development of LLMs that are more adept at interactive mathematical problem solving and real world applications.

Figures (17)

• Figure 1: Taxonomy of MathChat. The inner ring represents the task categories involved in MathChat. The intermediate ring lists the evaluation tasks in MathChat. The outer ring shows the tested capabilities in our tasks beyond simple math problem solving. See detailed descriptions in Section \ref{['dataset_overview']}.
• Figure 2: The performance comparison among various LLMs. Math LLMs (e.g., Deepseek-Math) have great performance on single-round QA dataset GSM8K, achieving similar performance to GPT-3.5. However, they significantly underperform GPT3.5 on MathChat, which requires more advanced reasoning abilities. We average the evaluation metrics in each task and scale all values into 0-1 for better visibility.
• Figure 3: The examples of the four tasks in our MathChat benchmark. We place all tasks under a scenario of a dialogue between the user and the assistant. The part with a italics font is the input to the LLMs and the highlighted parts are generated by LLMs and to be evaluated.
• Figure 4: The Round3 answer quality in follow-up QA task.
• Figure 5: The answer quality in Error Analysis task.