CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities

TL;DR

CHAMP introduces a competition-level, concept- and hint-annotated math problem dataset to probe LLMs' finer-grained mathematical reasoning and solution-verification abilities. By combining problem statements with general concepts, problem-specific hints, and first-wrong-step annotations, the authors evaluate across 10 models with 17 prompt designs, showing that final-answer accuracy often overstates true reasoning quality. The study demonstrates that hints can help some models, but misleading concepts can hinder others, and that full solution verification remains challenging despite high final-answer performance. A second experiment systematically assesses solution verification, revealing that most models struggle to reliably identify errors or validate reference solutions, underscoring the need for multi-faceted evaluation frameworks in mathematical reasoning tasks and the value CHAMP adds for benchmarking future models.

Abstract

Recent large language models (LLMs) have shown indications of mathematical reasoning ability on challenging competition-level problems, especially with self-generated verbalizations of intermediate reasoning steps (i.e., chain-of-thought prompting). However, current evaluations mainly focus on the end-to-end final answer correctness, and it is unclear whether LLMs can make use of helpful side information such as problem-specific hints. In this paper, we propose a challenging benchmark dataset for enabling such analyses. The Concept and Hint-Annotated Math Problems (CHAMP) consists of high school math competition problems, annotated with concepts, or general math facts, and hints, or problem-specific tricks. These annotations allow us to explore the effects of additional information, such as relevant hints, misleading concepts, or related problems. This benchmark is difficult, with the best model only scoring 58.1% in standard settings. With concepts and hints, performance sometimes improves, indicating that some models can make use of such side information. Furthermore, we annotate model-generated solutions for their correctness. Using this corpus, we find that models often arrive at the correct final answer through wrong reasoning steps. In addition, we test whether models are able to verify these solutions, and find that most models struggle.

Figures (3)

• Figure 1: Overview of our dataset and experiment contribution. [fill color=black,inner color=white,]A [fill color=black,inner color=white,]A : We collect 270 challenging, high-school math competition problems (e.g., Find all positive integer solutions to the equation $x^3+3=4y(y+1)$). For each problem, we write the relevant and helpful Concepts (e.g., $a^3\pm b^3=(a+b)(a^2\pm ab+b^2)$), and Hints (e.g, Express $x^3$ as the product of two factors involving $y$). [fill color=black,inner color=white,]B [fill color=black,inner color=white,]B : In our experiments, to investigate a model's ability to understand and use the additional C & H information, we design 17 prompts to evaluate ten models: GPT-3.5 / 4 / 4 Turbo, PaLM 2 Medium, Llama 2 7B / 70B, Llama 3 8B / 70B, Mistral 7B and Mixtral 8x22B. [fill color=black,inner color=white,]C [fill color=black,inner color=white,]C : For each problem, we manually judge two model-generated solutions on their correctness, and further annotate the first wrong step of the reasoning (red highlights), if present. [fill color=black,inner color=white,]D [fill color=black,inner color=white,]D : This corpus thus serves as a novel dataset for benchmarking and evaluating the solution verification ability of LLMs.
• Figure 2: The distribution of dataset statistics in CHAMP. Problems in CHAMP require a nontrivial number of reasoning steps (6.0 on average). Each problem is linked to an average of 1.4 concepts and 1.7 hints.
• Figure 3: The prompt for model verification evaluation. Text in black is given as the default, and we experiment with several variations. [fill color=black,inner color=white,]A [fill color=black,inner color=white,]A : we choose to give or withhold the reference solution in the prompt, where the blue italic texts are not provided in the latter case. [fill color=black,inner color=white,]B [fill color=black,inner color=white,]B : we evaluate model solutions for two prompts -- problem only and problem with concept and hint list. [fill color=black,inner color=white,]C [fill color=black,inner color=white,]C : the corresponding solution is given as the "Student Answer".