Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Study of Reasoning Steps in Thinking Code LLMs

Haoran Xue, Gias Uddin, Song Wang

TL;DR

The paper systematically evaluates six thinking LLMs on 100 BigCodeBench tasks to study explicit reasoning traces in code generation. It characterizes reasoning as step counts and verbosity, tests causal effects of changing reasoning budgets, and pairs these with a comprehensive human evaluation of reasoning quality (efficiency, logic consistency, completeness). A taxonomy of faulty reasoning patterns reveals completeness—especially edge-case handling—as the most frequent deficiency, with task complexity amplifying its impact. The results show non-monotonic, model-specific relationships between reasoning depth and success, and demonstrate that small reductions in reasoning often preserve standard-task performance but degrade hard-task results; the study also demonstrates stability and self-correction capabilities in several models. Overall, the work highlights fundamental gaps in current thinking LLMs for reliable, edge-case aware software engineering, and provides data, tooling, and metrics to guide future improvements.

Abstract

Thinking Large Language Models (LLMs) generate explicit intermediate reasoning traces before final answers, potentially improving transparency, interpretability, and solution accuracy for code generation. However, the quality of these reasoning chains remains underexplored. We present a comprehensive empirical study examining the reasoning process and quality of thinking LLMs for code generation. We evaluate six state-of-the-art reasoning LLMs (DeepSeek-R1, OpenAI-o3-mini, Claude-3.7-Sonnet-Thinking, Gemini-2.0-Flash-Thinking, Gemini-2.5-Flash, and Qwen-QwQ) across 100 code generation tasks of varying difficulty from BigCodeBench. We quantify reasoning-chain structure through step counts and verbosity, conduct controlled step-budget adjustments, and perform a 21-participant human evaluation across three dimensions: efficiency, logical correctness, and completeness. Our step-count interventions reveal that targeted step increases can improve resolution rates for certain models/tasks, while modest reductions often preserve success on standard tasks, rarely on hard ones. Through systematic analysis, we develop a reasoning-problematic taxonomy, identifying completeness as the dominant failure mode. Task complexity significantly impacts reasoning quality; hard problems are substantially more prone to incompleteness than standard tasks. Our stability analysis demonstrates that thinking LLMs maintain consistent logical structures across computational effort levels and can self-correct previous errors. This study provides new insights into the strengths and limitations of current thinking LLMs in software engineering.

An Empirical Study of Reasoning Steps in Thinking Code LLMs

TL;DR

The paper systematically evaluates six thinking LLMs on 100 BigCodeBench tasks to study explicit reasoning traces in code generation. It characterizes reasoning as step counts and verbosity, tests causal effects of changing reasoning budgets, and pairs these with a comprehensive human evaluation of reasoning quality (efficiency, logic consistency, completeness). A taxonomy of faulty reasoning patterns reveals completeness—especially edge-case handling—as the most frequent deficiency, with task complexity amplifying its impact. The results show non-monotonic, model-specific relationships between reasoning depth and success, and demonstrate that small reductions in reasoning often preserve standard-task performance but degrade hard-task results; the study also demonstrates stability and self-correction capabilities in several models. Overall, the work highlights fundamental gaps in current thinking LLMs for reliable, edge-case aware software engineering, and provides data, tooling, and metrics to guide future improvements.

Abstract

Thinking Large Language Models (LLMs) generate explicit intermediate reasoning traces before final answers, potentially improving transparency, interpretability, and solution accuracy for code generation. However, the quality of these reasoning chains remains underexplored. We present a comprehensive empirical study examining the reasoning process and quality of thinking LLMs for code generation. We evaluate six state-of-the-art reasoning LLMs (DeepSeek-R1, OpenAI-o3-mini, Claude-3.7-Sonnet-Thinking, Gemini-2.0-Flash-Thinking, Gemini-2.5-Flash, and Qwen-QwQ) across 100 code generation tasks of varying difficulty from BigCodeBench. We quantify reasoning-chain structure through step counts and verbosity, conduct controlled step-budget adjustments, and perform a 21-participant human evaluation across three dimensions: efficiency, logical correctness, and completeness. Our step-count interventions reveal that targeted step increases can improve resolution rates for certain models/tasks, while modest reductions often preserve success on standard tasks, rarely on hard ones. Through systematic analysis, we develop a reasoning-problematic taxonomy, identifying completeness as the dominant failure mode. Task complexity significantly impacts reasoning quality; hard problems are substantially more prone to incompleteness than standard tasks. Our stability analysis demonstrates that thinking LLMs maintain consistent logical structures across computational effort levels and can self-correct previous errors. This study provides new insights into the strengths and limitations of current thinking LLMs in software engineering.

Paper Structure

This paper contains 58 sections, 5 equations, 28 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Study Setup
  3. Studied Code Generation Tasks
  4. Studied Reasoning LLMs
  5. Data Collection
  6. Assessing the Reasoning Process in Thinking Code LLMs (RQ1)
  7. Experiment Approach
  8. RQ1.1 Step Count Analysis
  9. Approach
  10. Results
  11. RQ1.2 Step Count Effect
  12. RQ1.2.1 Think Deeper
  13. Approach
  14. Results
  15. RQ1.2.2 Reduce Steps
  16. ...and 43 more sections

Figures (28)

  • Figure 1: Processing flow comparison: Non-thinking vs. LLMs with CoT vs. Thinking LLMs
  • Figure 2: Overview of the Study Setup
  • Figure 3: Average reasoning steps (success/failure) and resolution rate for each model, split by difficulty. Both left y-axes share the same range and tick interval (0, 5, 10, 15). Resolution rate uses the right y-axis.
  • Figure 4: Side-by-side raw chains for BigCodeBench/889. Showing the verbatim steps clarifies how the +50% step increase makes error handling, numeric-only selection, and the empty-CSV branch explicit, yielding a correct solution.
  • Figure 5: Resolution rate by model across step count percentage increase levels. Colors match the reference plot. Top: Full split. Bottom: Hard split.
  • ...and 23 more figures