Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoning

TL;DR

This work tackles the latency bottleneck of long chain-of-thought reasoning in large language models by introducing DC-CoT, which trains a director to identify parallelizable subtasks and assigns them to workers for concurrent execution. The approach combines a parallel inference architecture with a multi-stage RL curriculum and data filtering to maintain accuracy while lowering the longest path length, a proxy for wall-clock latency. Across benchmarks like AIME 2024 and HMMT 2025, DC-CoT achieves similar or better accuracy than strong baselines while reducing latency by about 35–40%, with further gains when using high-length penalties or majority voting. The results demonstrate that explicitly teaching parallel thinking via RL on top of a long CoT base can meaningfully accelerate reasoning without sacrificing performance, and the authors provide public code and datasets for reproducibility.

Abstract

Long chain-of-thought reasoning (Long CoT) is now fundamental to state-of-the-art LLMs, especially in mathematical reasoning. However, LLM generation is highly sequential, and long CoTs lead to a high latency. We propose to train Divide-and-Conquer CoT (DC-CoT) to reduce the latency. With DC-CoT, the model can act as a director that identifies distinct subtasks that can be performed in parallel in its reasoning process, and then spawns workers to execute the subtasks. Our goal is to achieve high accuracy, with a low longest path length, which is a theoretical measure of the latency needed for the response. We start with a long CoT base model (DeepScaleR-1.5B-Preview), and first use SFT with a small curated demonstration set to initialize its ability to spawn workers in a certain format. Because SFT degrades the accuracy significantly, we design a multi-stage RL algorithm, with various data filtering strategies, to recover the accuracy while decreasing the longest path length. Across several benchmarks including AIME 2024 and HMMT 2025, DC-CoT achieves similar accuracy as DeepScaleR-1.5B-Preview while decreasing longest path length by 35-40%. Our code, SFT dataset and models are publicly available at https://github.com/amahankali10/DC_CoT_RL_for_Low_Latency_CoT_with_Parallel_Reasoning.

Figures (12)

• Figure 1: Accuracy (pass@1) and longest path length for DC-CoT and baselines on AIME 2024. DSR-32K and DSR-12K denote DeepScaleR-1.5B-Preview evaluated with response lengths $32,768$ and $12,000$ respectively. DC-CoT-HLP denotes DC-CoT trained with additional RL with a high length penalty, and DSR-HLP-24K and DSR-HLP-12K denotes baselines trained similarly, starting from DeepScaleR-1.5B-Preview. Maj-DSR-3 denotes DSR-12K evaluated using majority voting (maj@3), and DC-CoT-Maj denotes DC-CoT evaluated using maj@3. DC-CoT achieves higher accuracy than DSR-32K while requiring less longest path length.
• Figure 2: Our parallel inference procedure for DC-CoT. The model performs some reasoning before assigning tasks to workers (Director --- Initial Phase). Then, the workers perform the assigned tasks (Worker 1, Worker 2, Worker 3). The director then reads the workers' outputs (Director --- Read Workers' Thoughts). Finally, the director may either finalize its answer (Director --- Final Answer) or launch another round of parallel generation (Director --- Re-spawn Workers).
• Figure 3: Attention mask at training time. Blocks which are included in the attention computation are shown in green, while blocks which are excluded are shown in red. The attention mask is also causal --- entries above the diagonal are masked out. Here, $W_{123}^{\textup{(pre)}}$ refers to the concatenation of $W_{1}^{\textup{(prefill)}}$, $W_{2}^{\textup{(prefill)}}$ and $W_{3}^{\textup{(prefill)}}$.
• Figure 4: Accuracy (pass@1) and longest path length for all methods on all benchmarks.
• Figure 5: Accuracy and longest path length of DC-CoT on AIME 2024 during all stages of RL.