Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training LLMs for Divide-and-Conquer Reasoning Elevates Test-Time Scalability

Xiao Liang, Zhong-Zhi Li, Zhenghao Lin, Eric Hancheng Jiang, Hengyuan Zhang, Yelong Shen, Kai-Wei Chang, Ying Nian Wu, Yeyun Gong, Weizhu Chen

TL;DR

The paper targets the limits of chain-of-thought (CoT) reasoning on challenging tasks and identifies a misalignment with DAC-style inference when models are only post-trained for general purposes. It introduces DAC-RL, an end-to-end reinforcement learning framework that trains LLMs to perform divide-and-conquer reasoning by jointly optimizing problem division and conquering, with a unified objective $\mathcal{J}(\theta)$. Empirically, DAC-RL yields higher reasoning ceilings and better test-time scalability than CoT, with average gains on competition-level benchmarks and deep DAC variants providing further improvements; Mix-RL demonstrates that DAC training can also enhance CoT performance on complex problems. The work includes extensive ablations and analyses of integration with CoT, test-time configurations, compression vs. exploration, cold-start distillation, and format constraints, providing practical guidance for deploying DAC-based reasoning in frontier tasks.

Abstract

Large language models (LLMs) have demonstrated strong reasoning capabilities through step-by-step chain-of-thought (CoT) reasoning. Nevertheless, at the limits of model capability, CoT often proves insufficient, and its strictly sequential nature constrains test-time scalability. A potential alternative is divide-and-conquer (DAC) reasoning, which decomposes a complex problem into subproblems to facilitate more effective exploration of the solution. Although promising, our analysis reveals a fundamental misalignment between general-purpose post-training and DAC-style inference, which limits the model's capacity to fully leverage this potential. To bridge this gap and fully unlock LLMs' reasoning capabilities on the most challenging tasks, we propose an end-to-end reinforcement learning (RL) framework to enhance their DAC-style reasoning capacity. At each step, the policy decomposes a problem into a group of subproblems, solves them sequentially, and addresses the original one conditioned on the subproblem solutions, with both decomposition and solution integrated into RL training. Under comparable training, our DAC-style framework endows the model with a higher performance ceiling and stronger test-time scalability, surpassing CoT by 8.6% in Pass@1 and 6.3% in Pass@32 on competition-level benchmarks.

Training LLMs for Divide-and-Conquer Reasoning Elevates Test-Time Scalability

TL;DR

The paper targets the limits of chain-of-thought (CoT) reasoning on challenging tasks and identifies a misalignment with DAC-style inference when models are only post-trained for general purposes. It introduces DAC-RL, an end-to-end reinforcement learning framework that trains LLMs to perform divide-and-conquer reasoning by jointly optimizing problem division and conquering, with a unified objective . Empirically, DAC-RL yields higher reasoning ceilings and better test-time scalability than CoT, with average gains on competition-level benchmarks and deep DAC variants providing further improvements; Mix-RL demonstrates that DAC training can also enhance CoT performance on complex problems. The work includes extensive ablations and analyses of integration with CoT, test-time configurations, compression vs. exploration, cold-start distillation, and format constraints, providing practical guidance for deploying DAC-based reasoning in frontier tasks.

Abstract

Large language models (LLMs) have demonstrated strong reasoning capabilities through step-by-step chain-of-thought (CoT) reasoning. Nevertheless, at the limits of model capability, CoT often proves insufficient, and its strictly sequential nature constrains test-time scalability. A potential alternative is divide-and-conquer (DAC) reasoning, which decomposes a complex problem into subproblems to facilitate more effective exploration of the solution. Although promising, our analysis reveals a fundamental misalignment between general-purpose post-training and DAC-style inference, which limits the model's capacity to fully leverage this potential. To bridge this gap and fully unlock LLMs' reasoning capabilities on the most challenging tasks, we propose an end-to-end reinforcement learning (RL) framework to enhance their DAC-style reasoning capacity. At each step, the policy decomposes a problem into a group of subproblems, solves them sequentially, and addresses the original one conditioned on the subproblem solutions, with both decomposition and solution integrated into RL training. Under comparable training, our DAC-style framework endows the model with a higher performance ceiling and stronger test-time scalability, surpassing CoT by 8.6% in Pass@1 and 6.3% in Pass@32 on competition-level benchmarks.
Paper Structure (25 sections, 1 theorem, 14 equations, 13 figures, 3 tables)

This paper contains 25 sections, 1 theorem, 14 equations, 13 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Preliminaries
  4. Task Formalization
  5. Subproblem Division
  6. Subproblem and Original Conquering
  7. Experiments
  8. Settings
  9. Main Results
  10. Analysis
  11. Integrating CoT- and DAC-RL Training
  12. Test-time Scalability and Configurations
  13. Concise and Diverse Reasoning with DAC
  14. Cold-Start Distillation for DAC-RL
  15. The Effects of a Subproblem-Solving Format Constraint
  16. ...and 10 more sections

Key Result

Lemma 2.1

Let $s_i\in\{0,1\}$ indicate whether subproblem $i$ is solved correctly and $\mathbf{s}=(s_1,\ldots,s_m)$. Let $C\in\{0,1\}$ indicate whether the original problem is solved correctly. If the division-conquer pipeline induces the causal direction $\mathbf{s}\to C$, we have:

Figures (13)

  • Figure 1: Overview of the LLM post-training pipeline, comparison of the DAC and CoT-style inference, and our proposed DAC post-training. Without dedicated training, DAC inference remains ineffective, whereas DAC-specific post-training makes this advanced reasoning paradigm robust across downstream tasks.
  • Figure 2: We evaluate the CoT and DAC Pass@32 performance on four competition-level benchmarks (Table \ref{['tab:main_results']}) for both general instruction and reasoning post-trained models. The right panel presents Pass@32 performance for the https://huggingface.co/Qwen/Qwen2.5-7B-Instruct and https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507 models before and after task-specific RL training.
  • Figure 3: An overview of the DAC-style inference and reward assignments in training, illustrated with a case study. The policy decomposes the original problem into a group of subproblems, samples candidate conquering solutions in parallel, and leverages their correctness to compute division rewards for optimizing problem decomposition.
  • Figure 4: Intermediate evaluations across all four benchmarks during DAC-RL and CoT-RL training. Middle: For experiments using https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507, the training set is updated to the difficult subset after the 400th iteration, as detailed in Section \ref{['sec:settings']}. Right: Results of our Deep DAC training experiments, additionally including a comparison with the CoT-RL baseline using 32 rollouts in training.
  • Figure 5: An overall comparison among standard CoT-RL, Mix-RL, and DAC-RL trained on all problems. Left: Pass@1 accuracy on all benchmarks using CoT inference for both CoT-RL and Mix-RL. Middle: AIME score under the same settings as in Left. Right: Average performance across four competition-level benchmarks, where the CoT-trained policy uses CoT-style inference, while the other two employ DAC-style inference.
  • ...and 8 more figures

Theorems & Definitions (1)

  • Lemma 2.1: Final-answer reward positively associates with subproblem correctness