Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dual-Process Scaffold Reasoning for Enhancing LLM Code Debugging

Po-Chung Hsieh, Chin-Po Chen, Jeng-Lin Li, Ming-Ching Chang

TL;DR

Problem: balancing reasoning depth and computational efficiency in LLM-based code debugging. Approach: introduce Scaffold Reasoning (SR) with three streams—Scaffold, Analytic, and Integration—to combine abstract guidance with code-level analysis in a model-agnostic framework. Findings: on DebugBench, SR achieves 88.91% pass rate and 5.36 seconds per problem, with ablation analyses confirming the necessity of the integrated dual-stream design and the critical role of the $S^2$ step. Significance: results align with human cognitive processes and point to practical improvements in debugging efficiency and accuracy, with potential extensions to training and cross-domain reasoning studies.

Abstract

Recent LLMs have demonstrated sophisticated problem-solving capabilities on various benchmarks through advanced reasoning algorithms. However, the key research question of identifying reasoning steps that balance complexity and computational efficiency remains unsolved. Recent research has increasingly drawn upon psychological theories to explore strategies for optimizing cognitive pathways. The LLM's final outputs and intermediate steps are regarded as System 1 and System 2, respectively. However, an in-depth exploration of the System 2 reasoning is still lacking. Therefore, we propose a novel psychologically backed Scaffold Reasoning framework for code debugging, which encompasses the Scaffold Stream, Analytic Stream, and Integration Stream. The construction of reference code within the Scaffold Stream is integrated with the buggy code analysis results produced by the Analytic Stream through the Integration Stream. Our framework achieves an 88.91% pass rate and an average inference time of 5.36 seconds per-problem on DebugBench, outperforming other reasoning approaches across various LLMs in both reasoning accuracy and efficiency. Further analyses elucidate the advantages and limitations of various cognitive pathways across varying problem difficulties and bug types. Our findings also corroborate the alignment of the proposed Scaffold Reasoning framework with human cognitive processes.

Dual-Process Scaffold Reasoning for Enhancing LLM Code Debugging

TL;DR

Problem: balancing reasoning depth and computational efficiency in LLM-based code debugging. Approach: introduce Scaffold Reasoning (SR) with three streams—Scaffold, Analytic, and Integration—to combine abstract guidance with code-level analysis in a model-agnostic framework. Findings: on DebugBench, SR achieves 88.91% pass rate and 5.36 seconds per problem, with ablation analyses confirming the necessity of the integrated dual-stream design and the critical role of the step. Significance: results align with human cognitive processes and point to practical improvements in debugging efficiency and accuracy, with potential extensions to training and cross-domain reasoning studies.

Abstract

Recent LLMs have demonstrated sophisticated problem-solving capabilities on various benchmarks through advanced reasoning algorithms. However, the key research question of identifying reasoning steps that balance complexity and computational efficiency remains unsolved. Recent research has increasingly drawn upon psychological theories to explore strategies for optimizing cognitive pathways. The LLM's final outputs and intermediate steps are regarded as System 1 and System 2, respectively. However, an in-depth exploration of the System 2 reasoning is still lacking. Therefore, we propose a novel psychologically backed Scaffold Reasoning framework for code debugging, which encompasses the Scaffold Stream, Analytic Stream, and Integration Stream. The construction of reference code within the Scaffold Stream is integrated with the buggy code analysis results produced by the Analytic Stream through the Integration Stream. Our framework achieves an 88.91% pass rate and an average inference time of 5.36 seconds per-problem on DebugBench, outperforming other reasoning approaches across various LLMs in both reasoning accuracy and efficiency. Further analyses elucidate the advantages and limitations of various cognitive pathways across varying problem difficulties and bug types. Our findings also corroborate the alignment of the proposed Scaffold Reasoning framework with human cognitive processes.

Paper Structure

This paper contains 8 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. METHOD
  3. Scaffold Reasoning Framework
  4. Prompt Implementation
  5. Experiments
  6. Ablation Results of Reasoning Path and Reasoning Steps
  7. Reasoning Effects on Problem Difficulties and Bug Types
  8. Conclusion

Figures (2)

  • Figure 1: Scaffold Reasoning (SR) Framework applied to the problem of create-components-with-same-value: (left) SR reasoning steps; (middle) a simplified code reasoning process illustrating each function of the framework. The Scaffold Stream produces reference code that is distinct yet logically aligned, supporting the Integration Stream when compared with the buggy code.
  • Figure 2: Prompts used in the Scaffold Reasoning Framework.