Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Language Models Rival Mathematics Students? Evaluating Mathematical Reasoning through Textual Manipulation and Human Experiments

Andrii Nikolaiev, Yiannos Stathopoulos, Simone Teufel

TL;DR

This study probes whether modern large language models (LLMs) match or exceed human mathematical reasoning in combinatorics. It introduces the Combi-Puzzles dataset, comprising 125 problem variants across five forms to test generalisation while preserving core combinatorial structure. In two parallel experiments, GPT-4-Turbo outperforms open-source LLMs and human participants overall, with peak performance in the Mathematical form ($94\%$) but notable sensitivity to problem framing; humans show relatively stable performance across forms. The results highlight GPT-4’s strong but form-sensitive reasoning and raise questions about generalisation of mathematical reasoning in LLMs, motivating dataset design and methods to dissect cognitive differences in future work.

Abstract

In this paper we look at the ability of recent large language models (LLMs) at solving mathematical problems in combinatorics. We compare models LLaMA-2, LLaMA-3.1, GPT-4, and Mixtral against each other and against human pupils and undergraduates with prior experience in mathematical olympiads. To facilitate these comparisons we introduce the Combi-Puzzles dataset, which contains 125 problem variants based on 25 combinatorial reasoning problems. Each problem is presented in one of five distinct forms, created by systematically manipulating the problem statements through adversarial additions, numeric parameter changes, and linguistic obfuscation. Our variations preserve the mathematical core and are designed to measure the generalisability of LLM problem-solving abilities, while also increasing confidence that problems are submitted to LLMs in forms that have not been seen as training instances. We found that a model based on GPT-4 outperformed all other models in producing correct responses, and performed significantly better in the mathematical variation of the problems than humans. We also found that modifications to problem statements significantly impact the LLM's performance, while human performance remains unaffected.

Can Language Models Rival Mathematics Students? Evaluating Mathematical Reasoning through Textual Manipulation and Human Experiments

TL;DR

This study probes whether modern large language models (LLMs) match or exceed human mathematical reasoning in combinatorics. It introduces the Combi-Puzzles dataset, comprising 125 problem variants across five forms to test generalisation while preserving core combinatorial structure. In two parallel experiments, GPT-4-Turbo outperforms open-source LLMs and human participants overall, with peak performance in the Mathematical form () but notable sensitivity to problem framing; humans show relatively stable performance across forms. The results highlight GPT-4’s strong but form-sensitive reasoning and raise questions about generalisation of mathematical reasoning in LLMs, motivating dataset design and methods to dissect cognitive differences in future work.

Abstract

In this paper we look at the ability of recent large language models (LLMs) at solving mathematical problems in combinatorics. We compare models LLaMA-2, LLaMA-3.1, GPT-4, and Mixtral against each other and against human pupils and undergraduates with prior experience in mathematical olympiads. To facilitate these comparisons we introduce the Combi-Puzzles dataset, which contains 125 problem variants based on 25 combinatorial reasoning problems. Each problem is presented in one of five distinct forms, created by systematically manipulating the problem statements through adversarial additions, numeric parameter changes, and linguistic obfuscation. Our variations preserve the mathematical core and are designed to measure the generalisability of LLM problem-solving abilities, while also increasing confidence that problems are submitted to LLMs in forms that have not been seen as training instances. We found that a model based on GPT-4 outperformed all other models in producing correct responses, and performed significantly better in the mathematical variation of the problems than humans. We also found that modifications to problem statements significantly impact the LLM's performance, while human performance remains unaffected.

Paper Structure

This paper contains 16 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Related work
  3. Problems complexity.
  4. Data contamination.
  5. Human evaluation.
  6. Combi-Puzzles: Dataset Construction
  7. Methods
  8. Experiment with LLMs
  9. Experiment with Human participants
  10. Experimental Results and Discussion
  11. Conclusion
  12. Limitations
  13. Ethics Statement
  14. Marking Scheme
  15. Problems Poorly Solved
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Table represents human groups' scores on the problem set. Each cell represents a problem variation: common (grey), mathematical (blue), adversarial (red), parameterisation (green), linguistic obfuscation (purple).
  • Figure 2: Individual problem scores and the percentage of problems solved correctly are shown for the GPT-4 (left) and human participants (right).