Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Solving for X and Beyond: Can Large Language Models Solve Complex Math Problems with More-Than-Two Unknowns?

Kuei-Chun Kao, Ruochen Wang, Cho-Jui Hsieh

TL;DR

The Formulate-and-Solve strategy, a generalized prompting approach that effectively handles problems with an arbitrary number of unknowns, is proposed, and is revealed to enhance LLM performance on the BeyondX benchmark but also provides deeper insights into the computational limits of LLMs when faced with more complex mathematical challenges.

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance in solving math problems, a hallmark of human intelligence. Despite high success rates on current benchmarks; however, these often feature simple problems with only one or two unknowns, which do not sufficiently challenge their reasoning capacities. This paper introduces a novel benchmark, BeyondX, designed to address these limitations by incorporating problems with multiple unknowns. Recognizing the challenges in proposing multi-unknown problems from scratch, we developed BeyondX using an innovative automated pipeline that progressively increases complexity by expanding the number of unknowns in simpler problems. Empirical study on BeyondX reveals that the performance of existing LLMs, even those fine-tuned specifically on math tasks, significantly decreases as the number of unknowns increases - with a performance drop of up to 70\% observed in GPT-4. To tackle these challenges, we propose the Formulate-and-Solve strategy, a generalized prompting approach that effectively handles problems with an arbitrary number of unknowns. Our findings reveal that this strategy not only enhances LLM performance on the BeyondX benchmark but also provides deeper insights into the computational limits of LLMs when faced with more complex mathematical challenges.

Solving for X and Beyond: Can Large Language Models Solve Complex Math Problems with More-Than-Two Unknowns?

TL;DR

The Formulate-and-Solve strategy, a generalized prompting approach that effectively handles problems with an arbitrary number of unknowns, is proposed, and is revealed to enhance LLM performance on the BeyondX benchmark but also provides deeper insights into the computational limits of LLMs when faced with more complex mathematical challenges.

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance in solving math problems, a hallmark of human intelligence. Despite high success rates on current benchmarks; however, these often feature simple problems with only one or two unknowns, which do not sufficiently challenge their reasoning capacities. This paper introduces a novel benchmark, BeyondX, designed to address these limitations by incorporating problems with multiple unknowns. Recognizing the challenges in proposing multi-unknown problems from scratch, we developed BeyondX using an innovative automated pipeline that progressively increases complexity by expanding the number of unknowns in simpler problems. Empirical study on BeyondX reveals that the performance of existing LLMs, even those fine-tuned specifically on math tasks, significantly decreases as the number of unknowns increases - with a performance drop of up to 70\% observed in GPT-4. To tackle these challenges, we propose the Formulate-and-Solve strategy, a generalized prompting approach that effectively handles problems with an arbitrary number of unknowns. Our findings reveal that this strategy not only enhances LLM performance on the BeyondX benchmark but also provides deeper insights into the computational limits of LLMs when faced with more complex mathematical challenges.
Paper Structure (52 sections, 6 figures, 21 tables, 1 algorithm)

This paper contains 52 sections, 6 figures, 21 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. C1: BeyondX - The first multi-unknown algebraic benchmark.
  3. C2: Existing LLMs struggles with increasing unknowns.
  4. C3: Formulate-and-Solve A prompting method to tackle multi-unknown problems.
  5. Related Work
  6. Math Word Problem Generation
  7. Math Word Problem Solver
  8. Math Reasoning with LLMs
  9. Automatic Generation of Multi-Unknown Algebra Problems via Progressive Expansion
  10. Challenges for Constructing Multi-Unknown Datasets
  11. Generating new problems with LLMs.
  12. Limitations of naive generation.
  13. Generating New Problems via Progress Expansion
  14. Pipeline overview.
  15. Multi-step problem expansion.
  16. ...and 37 more sections

Figures (6)

  • Figure 1: An example question of multi-unknown algebra problem generation and its corresponding reasoning steps. The prompts used for each step can be found Appendix \ref{['tab:mwp_generation_instruction']}.
  • Figure 2: Preliminary study of different LLMs and prompting methods on multi-unknown algebra datasets.
  • Figure 3: The overview of Automatic Solver of Algebra Problems.
  • Figure 4: The performance of different existing open-source models.
  • Figure 5: The performance on different numbers of unknown.
  • ...and 1 more figures