SuperCLUE-Math6: Graded Multi-Step Math Reasoning Benchmark for LLMs in Chinese
Liang Xu, Hang Xue, Lei Zhu, Kangkang Zhao
TL;DR
This work introduces SC-Math6, a native Chinese benchmark designed to evaluate multi-step mathematical reasoning in LLMs and to address GSM8K's English-centric limitations. It presents 1072 problems (2144 items with follow-ups) sourced from Chinese contexts, accompanied by natural-language solution walkthroughs and a transparent scoring framework that yields a 1–5 Reasoning Level through a composite of Reasoning Steps Score and Overall Accuracy. The authors detail data collection quality controls, a step-weighted scoring scheme, and comprehensive benchmarking across 13 Chinese models, finding clear stratification by reasoning ability and that SC-Math6 is more challenging than GSM8K. The results highlight the importance of instruction compliance and response length in performance and establish a solid benchmark to spur development of more human-like mathematical reasoning in Chinese language models.
Abstract
We introduce SuperCLUE-Math6(SC-Math6), a new benchmark dataset to evaluate the mathematical reasoning abilities of Chinese language models. SC-Math6 is designed as an upgraded Chinese version of the GSM8K dataset with enhanced difficulty, diversity, and application scope. It consists of over 2000 mathematical word problems requiring multi-step reasoning and providing natural language solutions. We propose an innovative scheme to quantify the reasoning capability of large models based on performance over problems with different reasoning steps. Experiments on 13 representative Chinese models demonstrate a clear stratification of reasoning levels, with top models like GPT-4 showing superior performance. SC-Math6 fills the gap in Chinese mathematical reasoning benchmarks and provides a comprehensive testbed to advance the intelligence of Chinese language models.