Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering

Francesco Maria Molfese, Luca Moroni, Ciro Porcaro, Simone Conia, Roberto Navigli

TL;DR

ReTraceQA introduces a first gold benchmark for evaluating the validity and localization of reasoning traces in commonsense QA. By annotating 2,421 traces with earliest-error points and error types, it reveals that many models reach correct answers via flawed reasoning and that reasoning-aware evaluation can reduce apparent performance by up to ~25% compared to answer-only metrics. The study assesses both LLM-based judges and process reward models, finding that while judges can detect overall trace correctness, precise error localization remains challenging, and cross-domain transfer from math-oriented PRMs is limited. Across four commonsense datasets, results emphasize the need for reasoning-aware evaluation to better reflect true capabilities and to guide future model development beyond STEM domains.

Abstract

While Small Language Models (SLMs) have demonstrated promising performance on an increasingly wide array of commonsense reasoning benchmarks, current evaluation practices rely almost exclusively on the accuracy of their final answers, neglecting the validity of the reasoning processes that lead to those answers. To address this issue, we introduce ReTraceQA, a novel benchmark that introduces process-level evaluation for commonsense reasoning tasks. Our expert-annotated dataset reveals that in a substantial portion of instances (14-24%), SLMs provide correct final answers despite flawed reasoning processes, suggesting that the capabilities of SLMs are often overestimated by evaluation metrics that focus only on comparing the final answer with the ground truth. Indeed, we show that when employing strong Large Language Models (LLMs) as automated judges for reasoning-aware evaluation rather than answer-only metrics, SLM performance drops significantly across all models and datasets, with scores decreasing by up to 25%.

ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering

TL;DR

ReTraceQA introduces a first gold benchmark for evaluating the validity and localization of reasoning traces in commonsense QA. By annotating 2,421 traces with earliest-error points and error types, it reveals that many models reach correct answers via flawed reasoning and that reasoning-aware evaluation can reduce apparent performance by up to ~25% compared to answer-only metrics. The study assesses both LLM-based judges and process reward models, finding that while judges can detect overall trace correctness, precise error localization remains challenging, and cross-domain transfer from math-oriented PRMs is limited. Across four commonsense datasets, results emphasize the need for reasoning-aware evaluation to better reflect true capabilities and to guide future model development beyond STEM domains.

Abstract

While Small Language Models (SLMs) have demonstrated promising performance on an increasingly wide array of commonsense reasoning benchmarks, current evaluation practices rely almost exclusively on the accuracy of their final answers, neglecting the validity of the reasoning processes that lead to those answers. To address this issue, we introduce ReTraceQA, a novel benchmark that introduces process-level evaluation for commonsense reasoning tasks. Our expert-annotated dataset reveals that in a substantial portion of instances (14-24%), SLMs provide correct final answers despite flawed reasoning processes, suggesting that the capabilities of SLMs are often overestimated by evaluation metrics that focus only on comparing the final answer with the ground truth. Indeed, we show that when employing strong Large Language Models (LLMs) as automated judges for reasoning-aware evaluation rather than answer-only metrics, SLM performance drops significantly across all models and datasets, with scores decreasing by up to 25%.

Paper Structure

This paper contains 40 sections, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Process-Based Evaluation Approaches.
  4. Benchmarks for Reasoning Evaluation.
  5. Limitations and Research Gaps.
  6. ReTraceQA
  7. Task Definition
  8. Dataset Selection
  9. Solution Generation
  10. Solution Reformatting
  11. Human Annotation
  12. Benchmark Statistics
  13. Experimental Setup
  14. Models
  15. LLM-as-a-judge.
  16. ...and 25 more sections

Figures (3)

  • Figure 1: The ReTraceQA pipeline resulting in 2,421 reasoning traces annotated with error information.
  • Figure 2: Comparison of step error positions: (a) human annotation and (b) o1-mini employed as a judge.
  • Figure 3: Process Error Rate (%): The proportion of incorrect reasoning traces that reach the correct final answer, calculated across the annotated subsets of our benchmark for each model.