A Survey on Mathematical Reasoning and Optimization with Large Language Models

TL;DR

This survey surveys how Large Language Models (LLMs) and pre-trained language models have evolved to perform mathematical calculation, reasoning, theorem proving, and symbolic computation, with a focus on their integration into optimization and control tasks. It categorizes approaches into PLM-based (autoreg/dis autoregressive) methods and LLM-driven reasoning strategies, including instruction learning, tool augmentation, and advanced Chain-of-Thought techniques, while highlighting the challenges of numerical precision, consistency, and verification. The work discusses optimization-centric applications such as mixed-integer programming, linear-quadratic control, and multi-agent LP, including case studies like Battery Energy Storage Systems and water-tank control, and showcases how LLMs can assist problem formulation, constraint generation, and solution refinement. It also surveys datasets, performance benchmarks, model comparisons, time-series applications, and the open-vs-closed access landscape, offering actionable directions for future research to improve interpretability, robustness, and solver integration in AI-driven mathematical reasoning. Overall, the paper emphasizes hybrid neural-symbolic reasoning and tool-augmented inference as promising avenues to achieve reliable, scalable, and practically impactful AI systems for engineering, finance, and scientific research.

Abstract

Mathematical reasoning and optimization are fundamental to artificial intelligence and computational problem-solving. Recent advancements in Large Language Models (LLMs) have significantly improved AI-driven mathematical reasoning, theorem proving, and optimization techniques. This survey explores the evolution of mathematical problem-solving in AI, from early statistical learning approaches to modern deep learning and transformer-based methodologies. We review the capabilities of pretrained language models and LLMs in performing arithmetic operations, complex reasoning, theorem proving, and structured symbolic computation. A key focus is on how LLMs integrate with optimization and control frameworks, including mixed-integer programming, linear quadratic control, and multi-agent optimization strategies. We examine how LLMs assist in problem formulation, constraint generation, and heuristic search, bridging theoretical reasoning with practical applications. We also discuss enhancement techniques such as Chain-of-Thought reasoning, instruction tuning, and tool-augmented methods that improve LLM's problem-solving performance. Despite their progress, LLMs face challenges in numerical precision, logical consistency, and proof verification. Emerging trends such as hybrid neural-symbolic reasoning, structured prompt engineering, and multi-step self-correction aim to overcome these limitations. Future research should focus on interpretability, integration with domain-specific solvers, and improving the robustness of AI-driven decision-making. This survey offers a comprehensive review of the current landscape and future directions of mathematical reasoning and optimization with LLMs, with applications across engineering, finance, and scientific research.