SCoRE: Benchmarking Long-Chain Reasoning in Commonsense Scenarios

TL;DR

SCoRE introduces a bilingual benchmark for long-chain commonsense reasoning in scenario-based contexts, addressing the limitations of prior datasets that emphasize short chains and high annotation costs. It uses a knowledge-driven synthesis pipeline that combines manually constructed knowledge bases with a rule-based inference engine to automatically generate $100{,}000$ multi-hop questions (from $2$ to $11$ hops) across four domains plus a mix domain, accompanied by explicit reasoning traces and fine-grained labels. A dedicated 6,000-question test set in Chinese and English enables diagnostic evaluation across domains, with explicit chain lengths and seven label dimensions to trace performance to specific knowledge points. Evaluation across 13 contemporary reasoning LLMs shows a ceiling of $69.78\%$ accuracy overall and $47.91\%$ on the hard subset, highlighting persistent challenges in rare knowledge, logical consistency, and over-interpretation, while revealing systematic error patterns and opportunities to guide future model design and training. Overall, SCoRE provides a scalable, interpretable framework for diagnosing long-chain commonsense reasoning and serves as a benchmark to drive targeted improvements in reasoning capabilities of LLMs.

Abstract

Currently, long-chain reasoning remains a key challenge for large language models (LLMs) because natural texts lack sufficient explicit reasoning data. However, existing benchmarks suffer from limitations such as narrow coverage, short reasoning paths, or high construction costs. We introduce SCoRE (Scenario-based Commonsense Reasoning Evaluation), a benchmark that synthesizes multi-hop questions from scenario schemas of entities, relations, and logical rules to assess long-chain commonsense reasoning. SCoRE contains 100k bilingual (Chinese-English) multiple-choice questions whose reasoning chains span 2-11 hops and are grouped into various difficulty levels. Each question is accompanied by fine-grained knowledge labels, explicit reasoning chains, and difficulty levels for diagnostic evaluation. Evaluation results on cutting-edge LLMs such as o3-mini and Deepseek R1 shows that even the best model attains only 69.78% accuracy on SCoRE (even only 47.91% on the hard set), with errors often stemming from rare knowledge, logical inconsistency, and over-interpretation of simple questions. SCoRE offers a scalable, extensible framework for evaluating and diagnosing the long-chain commonsense reasoning abilities of LLMs and guiding future advances in model design and training.

Figures (26)

• Figure 1: The framework of knowledge base. Illustrated with scenarios in the space domain and entities in the nature domain.
• Figure 2: Overall process of data synthesis. The pipeline consists of three stages: (1) Scenario Definition: selecting entities/events from the knowledge base and constructing natural language descriptions; (2) Inference Data Generation: applying the Reasoner to iteratively generate a fact base via rule-based logic and select a minimal statement set that uniquely determines all entity positions; (3) Question Design: using the statement set and answer key to generate different question types (e.g., precise, vague, true/false) with appropriate options. This process ensures verifiable reasoning chains, rich logical structures, and natural language fluency.
• Figure 3: The average performance of LLMs on Chinese and English questions on different levels.
• Figure 4: The performance of LLMs on Chinese and English questions with different different knowledge attributes and question types. Here, if the answer to a question involves only a single entity, it is termed "precise." If it involves multiple entities, it is termed "vague".
• Figure 5: The English and Chinese prompts used in the evaluations.