Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Critic

TL;DR

This paper introduces Critic-CoT, a framework that shifts LLM self-critique toward System-2-like, stepwise reasoning by using Chain-of-Thought Critique and weak supervision. It presents a two-stage training procedure to teach the model to critique and refine its own outputs, plus inference-time strategies (Iterative Refine and Critic As Filter) to leverage the critique during decoding. Empirical results on GSM8K and MATH show notable gains in in-domain math reasoning, with further improvements in out-of-domain tasks (StrategyQA, AGIEval, HumanEval) and evidence of a mutual reinforcement between critique ability and task solving. The work also includes extensive ablations, confirming the importance of stepwise CoT critique, data composition, and the advantage over simple distillation baselines, and discusses limitations and avenues for future enhancement of self-critic frameworks.

Abstract

Self-critic has become a crucial mechanism for enhancing the reasoning performance of LLMs. However, current approaches mainly involve basic prompts for intuitive instance-level feedback, which resembles System-1 processes and limits the reasoning capabilities. Moreover, there is a lack of in-depth investigations into the relationship between LLM's ability to criticize and its task-solving performance. To address these issues, we propose Critic-CoT, a novel framework that pushes LLMs toward System-2-like critic capability. Through a step-wise CoT reasoning paradigm and the automatic construction of distant-supervision data without human annotation, Critic-CoT enables LLMs to engage in slow, analytic self-critique and refinement, thereby improving their reasoning abilities. Experiments on GSM8K and MATH demonstrate that our enhanced model significantly boosts task-solving performance by filtering out invalid solutions or iterative refinement. Furthermore, we investigate the intrinsic correlation between critique and task-solving abilities within LLMs, discovering that these abilities can mutually reinforce each other rather than conflict.

Figures (6)

• Figure 1: Illustration of Critic-CoT: Previous instance-level critic methods attempt to identify errors directly without any prior analysis, and restart from the beginning during refinement. In contrast, our proposed Critic-CoT framework performs a step-wise examination using the Chain-of-Thought approach. When refining, rather than starting from scratch, our method makes the correction from the specific error step with the help of the corresponding critique.
• Figure 2: The Process of Critic-CoT during training (a) and inference (b). For training, we collect the critic-refine data on the generator's samples via weak supervision (Section \ref{['section:Chain-of-Thought Critique']}). Through fine-tuning, we enable the target model to criticize and refine its own reasoning process. Then, during inference, we can leverage the capabilities via Iterative Refine or Critic As Filter (Section \ref{['section:Inference']}).
• Figure A1: Performance of majority vote on GSM8K and MATH500 Datasets
• Figure A2: Performance group by difficulty level, on GSM8K and MATH500 Datasets
• Figure A3: Criticize and refine a problem in the GSM8K test set.