SEER: Enhancing Chain-of-Thought Code Generation through Self-Exploring Deep Reasoning

TL;DR

SEER reframes code generation with Chain-of-Thought as a decision-making problem and tackles three limitations: limited reasoning-path diversity, lack of intermediate-step quality assessment, and overthinking. It introduces a three-stage framework combining diverse path exploration via MCTS with path perturbation/refinement, reasoning quality-aware policy and value training, and adaptive CoT that toggles between direct coding and stepwise reasoning. Empirical results on MBPP, HumanEval, and LiveCodeBench show consistent gains across model sizes, with smaller models benefiting most and even outperforming some larger proprietary baselines. The approach includes ablations, parameter studies, and cross-language tests, and the authors release code, datasets, and models to support replication and further research.

Abstract

Code generation, the task of creating executable programs from natural language requirements, has recently seen tremendous advances through Chain-of-Thought (CoT) reasoning, which enables Large Language Models (LLMs) to develop high-level reasoning plans before writing code. Recent research has proposed various methods to enhance models' CoT reasoning for code generation such as prompt engineering and supervised fine-tuning. However, existing approaches still face three critical limitations: (1) limited exploration of diverse reasoning paths, which constrains generalization across various programming scenarios, (2) lack of quality assessment for intermediate reasoning steps, which hampers the reliability of the generated plans and code, and (3) the potential negative impact of "overthinking", potentially leading to unnecessarily complex and incorrect solutions. To address these limitations, we frame CoT code generation as a decision making problem and present SEER, a SElf-Exploring deep Reasoning framework that enables accurate and adaptive reasoning for code generation. SEER introduces three key components: (1) Diverse reasoning path exploration, which aims at exploring diverse reasoning paths and annotating intermediate steps without relying on manual experts or closed-source proprietary models; (2) Reasoning quality-aware model training, which trains a policy model for generating candidate reasoning steps and a value model for assessing their quality; and (3) Adaptive CoT reasoning, which dynamically switches between direct generation and step-by-step reasoning for different problems.

Figures (7)

• Figure 1: An illustration of chain-of-thought code generation (a) and multiple reasoning paths (b). The green and red node denote correct and incorrect steps, respectively. The dashed lines and circles represent unexplored paths.
• Figure 2: An example of "overthinking" in code generation: Reasoning improves overall accuracy but can also lead to errors (a). While solution (b) includes detailed reasoning, it opts for a complex implementation and fails to handle zero input. In contrast, solution (c) takes a direct approach using the built-in bin() function and correctly handles all inputs.
• Figure 3: An overview of the four key operations in MCTS.
• Figure 4: The overall framework of SEER.
• Figure 5: Parameter analysis of $B_1$ and $B_2$.