LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languages

TL;DR

This work tackles the challenge of evaluating LLMs' ability to perform mathematical modeling by proposing a solver-based, process-oriented framework and introducing the Mamo benchmark. By mapping natural-language problems to mathematical models and validating the resulting solutions with automated solvers, the authors shift focus from end results to the modeling process itself. The benchmark covers $ODE$ and optimization ($LP$/$MILP$) tasks, revealing that larger models improve performance on complex problems while open-source models compete on easier tasks. The approach enables automatic, ground-truth-aligned assessment of mathematical reasoning in LLMs and points to future research directions for expanding modeling domains and mitigating formatting influence. Overall, the paper provides a principled methodology and dataset to advance rigorous measurement of LLMs' mathematical modeling capabilities.

Abstract

Large Language Models (LLMs) have demonstrated strong performance across various natural language processing tasks, yet their proficiency in mathematical reasoning remains a key challenge. Addressing the gap between natural and mathematical language requires advanced reasoning capabilities, approaching those of Artificial General Intelligence (AGI). However, the evaluation remains challenging, as perfectly representing reality is inherently elusive, and traditional methods like manual or direct comparison of mathematical statements (Ramamonjison et al., 2023) are insufficient for assessing true modeling ability. We propose a process-oriented framework to evaluate LLMs' ability to construct mathematical models, using solvers to compare outputs with ground truth. Introducing Mamo, a benchmark with 1,209 questions covering ordinary differential equations, linear programming, and mixed-integer linear programming, we enable automatic evaluation of modeling accuracy. The results show that existing LLMs struggle with complex mathematical modeling tasks, with larger models demonstrating superior performance, while open-source models remain competitive in simpler cases but still fall short of proprietary models in more challenging problems.

Figures (20)

• Figure 1: $Q$ is the natural language problem, while $M_1$,$M_2$ are different mathematical models.
• Figure 2: The pipeline to use exact answer verification via an additional solver.
• Figure 3: $Q$ is the natural language problem (may be ODE problem or optimization problem), where y is a solution function in ODE problems.
• Figure 4: Examples illustrating the application of the General Comparison Criterion
• Figure 5: The testing pipeline involving code modifier.