DeepCritic: Deliberate Critique with Large Language Models

TL;DR

DeepCritic introduces a two-stage framework to endow LLMs with deliberate, stepwise math critiques that go beyond shallow feedback. By first generating a curated seed of long-form critiques and fine-tuning (Stage 1), and then applying reinforcement learning with both human-annotated and automatically generated data (Stage 2), the approach achieves strong error-identification and substantially better guidance for generators. It demonstrates superior performance on multiple math-critique benchmarks compared to PRMs and existing LLM critics, and exhibits favorable test-time scaling for both the critic and the generator via majority voting and critique-guided refinement. The results underscore the potential of deliberate reasoning and scalable oversight to elevate mathematical reasoning in LLMs, with practical implications for automated feedback and self-improvement workflows.

Abstract

As Large Language Models (LLMs) are rapidly evolving, providing accurate feedback and scalable oversight on their outputs becomes an urgent and critical problem. Leveraging LLMs as critique models to achieve automated supervision is a promising solution. In this work, we focus on studying and enhancing the math critique ability of LLMs. Current LLM critics provide critiques that are too shallow and superficial on each step, leading to low judgment accuracy and struggling to offer sufficient feedback for the LLM generator to correct mistakes. To tackle this issue, we propose a novel and effective two-stage framework to develop LLM critics that are capable of deliberately critiquing on each reasoning step of math solutions. In the first stage, we utilize Qwen2.5-72B-Instruct to generate 4.5K long-form critiques as seed data for supervised fine-tuning. Each seed critique consists of deliberate step-wise critiques that includes multi-perspective verifications as well as in-depth critiques of initial critiques for each reasoning step. Then, we perform reinforcement learning on the fine-tuned model with either existing human-labeled data from PRM800K or our automatically annotated data obtained via Monte Carlo sampling-based correctness estimation, to further incentivize its critique ability. Our developed critique model built on Qwen2.5-7B-Instruct not only significantly outperforms existing LLM critics (including the same-sized DeepSeek-R1-distill models and GPT-4o) on various error identification benchmarks, but also more effectively helps the LLM generator refine erroneous steps through more detailed feedback.

Figures (6)

• Figure 1: Comparison of critiques generated by current LLM critics and our developed critic. The red highlights in the solution steps represent the erroneous part. The critiques of current LLM critics (e.g., Qwen2.5-7B-Instruct) are overly superficial, primarily consisting of declarative statements rather than in-depth analysis or critical evaluation. In contrast, our critic can generate a deliberate reasoning process before making a judgment, incorporating iterative evaluation, multi-perspective verification, and meta-critiquing.
• Figure 2: The two-stage pipeline of training our deep critique models. In Stage 1, we first utilize Qwen2.5-72B-Instruct to generate an initial step-wise critique for each step in the solution, followed by an in-depth critique of the initial critique. Then, we use Qwen2.5-72B-Instruct to merge these two critiques into one deliberate critique in the long-CoT form. Finally, we perform SFT on above created critique data to teach the model the format of deliberately critiquing. In Stage 2, we perform RL to the SFT model on either existing human-annotated data or auto-labeled data via Monte Carlo sampling-based correctness estimation, to further stimulate the critique ability of the critic.
• Figure 3: Majority voting results (Maj@8) of each model across all benchmarks. Pass@1 results are from Table \ref{['tab: main results']}. "PB" denotes ProcessBench.
• Figure 4: Verified majority voting results of Qwen2.5-7B/72B-Instruct on MATH500 and AIME2024-2025 by taking different models as verifiers.
• Figure 5: Training hyper-parameters in SFT.