Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations

Jiaxing Sun, Weiquan Huang, Jiang Wu, Chenya Gu, Wei Li, Songyang Zhang, Hang Yan, Conghui He

TL;DR

CHARM introduces the first comprehensive Chinese commonsense reasoning benchmark, covering global and Chinese-specific domains, and combines closely interconnected reasoning and memorization tasks to study how memorization supports reasoning in LLMs. By evaluating 19 LLMs with five prompting strategies, the study shows that model orientation and task domain significantly affect prompting effectiveness, and that memorization plays a foundational role in integrated reasoning, with clear distinctions across model types. The work reveals domain- and language-dependent patterns in prompt strategy success (e.g., XLT for English-oriented LLMs and ZH-CoT for Chinese-oriented LLMs) and provides two methods (FRMM and MIB) to isolate memorization-independent reasoning, yielding actionable insights for improving Chinese commonsense reasoning. CHARM thus offers a rigorous, two-domain framework that can guide optimization of non-English LLMs and serves as a reference for cross-domain evaluation in related fields, with the CHARM repository released at https://github.com/opendatalab/CHARM.

Abstract

We introduce CHARM, the first benchmark for comprehensively and in-depth evaluating the commonsense reasoning ability of large language models (LLMs) in Chinese, which covers both globally known and Chinese-specific commonsense. We evaluated 7 English and 12 Chinese-oriented LLMs on CHARM, employing 5 representative prompt strategies for improving LLMs' reasoning ability, such as Chain-of-Thought. Our findings indicate that the LLM's language orientation and the task's domain influence the effectiveness of the prompt strategy, which enriches previous research findings. We built closely-interconnected reasoning and memorization tasks, and found that some LLMs struggle with memorizing Chinese commonsense, affecting their reasoning ability, while others show differences in reasoning despite similar memorization performance. We also evaluated the LLMs' memorization-independent reasoning abilities and analyzed the typical errors. Our study precisely identified the LLMs' strengths and weaknesses, providing the clear direction for optimization. It can also serve as a reference for studies in other fields. We will release CHARM at https://github.com/opendatalab/CHARM .

Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations

TL;DR

CHARM introduces the first comprehensive Chinese commonsense reasoning benchmark, covering global and Chinese-specific domains, and combines closely interconnected reasoning and memorization tasks to study how memorization supports reasoning in LLMs. By evaluating 19 LLMs with five prompting strategies, the study shows that model orientation and task domain significantly affect prompting effectiveness, and that memorization plays a foundational role in integrated reasoning, with clear distinctions across model types. The work reveals domain- and language-dependent patterns in prompt strategy success (e.g., XLT for English-oriented LLMs and ZH-CoT for Chinese-oriented LLMs) and provides two methods (FRMM and MIB) to isolate memorization-independent reasoning, yielding actionable insights for improving Chinese commonsense reasoning. CHARM thus offers a rigorous, two-domain framework that can guide optimization of non-English LLMs and serves as a reference for cross-domain evaluation in related fields, with the CHARM repository released at https://github.com/opendatalab/CHARM.

Abstract

We introduce CHARM, the first benchmark for comprehensively and in-depth evaluating the commonsense reasoning ability of large language models (LLMs) in Chinese, which covers both globally known and Chinese-specific commonsense. We evaluated 7 English and 12 Chinese-oriented LLMs on CHARM, employing 5 representative prompt strategies for improving LLMs' reasoning ability, such as Chain-of-Thought. Our findings indicate that the LLM's language orientation and the task's domain influence the effectiveness of the prompt strategy, which enriches previous research findings. We built closely-interconnected reasoning and memorization tasks, and found that some LLMs struggle with memorizing Chinese commonsense, affecting their reasoning ability, while others show differences in reasoning despite similar memorization performance. We also evaluated the LLMs' memorization-independent reasoning abilities and analyzed the typical errors. Our study precisely identified the LLMs' strengths and weaknesses, providing the clear direction for optimization. It can also serve as a reference for studies in other fields. We will release CHARM at https://github.com/opendatalab/CHARM .
Paper Structure (34 sections, 12 figures, 13 tables)

This paper contains 34 sections, 12 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CHARM
  4. Commonsense Domain
  5. Reasoning Tasks
  6. Construction of Reasoning Tasks
  7. Memorization Tasks
  8. Quality Assurance
  9. Experimental Setup
  10. Language Models
  11. Prompt Strategies
  12. Results and Analysis
  13. Integrated Reasoning Performance
  14. Prompt Strategy Selection
  15. Integrated Reasoning vs Memorization
  16. ...and 19 more sections

Figures (12)

  • Figure 1: Construction of CHARM. CHARM encompasses both global and Chinese-specific commonsense. CHARM consists closely-interconnected reasoning and memorization tasks.
  • Figure 2: Distribution of CHARM construction.
  • Figure 3: Averaged accuracy across the 4 MRI tasks in the Chinese commonsense domain.
  • Figure 4: Distribution of the memorization-independent reasoning errors
  • Figure 5: Entity and question examples of the commonsense aspects.
  • ...and 7 more figures