Recursive Decomposition with Dependencies for Generic Divide-and-Conquer Reasoning

TL;DR

This work addresses the scalability of reasoning with large language models by introducing Recursive Decomposition with Dependencies (RDD), a generic framework that decomposes problems into subtasks with optional dependencies, forming a directed acyclic graph (DAG) and enabling parallel execution. RDD operates through three generic steps—decompose, unit-solve, and merge—guided by a scheduler (BFS for decomposition, DFS for solving/merging) and a fixed pool of meta-prompts, with information flow designed to minimize ancestor context and support error recovery during merging. Empirical evaluation across two benchmarks with six difficulty levels and both task-specific and generic settings shows that RDD outperforms state-of-the-art baselines in compute-matched scenarios as task complexity increases, while also reducing execution time and context length. The results demonstrate the practicality and generality of DAG-based reasoning for real-world LLM systems, though simpler problems may still favor traditional prompting, and future work could improve unit-problem classification and further accelerate parallel execution.

Abstract

Reasoning tasks are crucial in many domains, especially in science and engineering. Although large language models (LLMs) have made progress in reasoning tasks using techniques such as chain-of-thought and least-to-most prompting, these approaches still do not effectively scale to complex problems in either their performance or execution time. Moreover, they often require additional supervision for each new task, such as in-context examples. In this work, we introduce Recursive Decomposition with Dependencies (RDD), a scalable divide-and-conquer method for solving reasoning problems that requires less supervision than prior approaches. Our method can be directly applied to a new problem class even in the absence of any task-specific guidance. Furthermore, RDD supports sub-task dependencies, allowing for ordered execution of sub-tasks, as well as an error recovery mechanism that can correct mistakes made in previous steps. We evaluate our approach on two benchmarks with six difficulty levels each and in two in-context settings: one with task-specific examples and one without. Our results demonstrate that RDD outperforms other methods in a compute-matched setting as task complexity increases, while also being more computationally efficient.

Figures (37)

• Figure 1: The decomposition methodology pipeline: decomposing, unit-solving, and merging. Nodes in gray represent unsolved problems, while nodes in green represent solved problems.
• Figure 2: An example of the decomposition graph generated by the RDD method.
• Figure 3: An evaluation of RDD against CoT weiChainofThoughtPromptingElicits2022 and LtM zhouLeasttoMostPromptingEnables2022 with self-consistency (SC; wangSelfConsistencyImprovesChain2022) on the letter concatenation benchmark in the task-specific few-shot setting. Our system uses LtM at the unit-solving step; we refer to it as RDD+LtM.
• Figure 4: An evaluation of RDD against CoT with self-consistency (SC) the generic few-shot setting. Our system uses CoT at the unit-solving step; we refer to it as RDD+CoT.
• Figure 5: The error sources of the recursive decomposition approach in the letter concatenation benchmark with respect to $n_0$ (the size of the list in the problem) for task-specific in-context (a) and generic (b) experiments . $\phi_{\text{d}}$ corresponds to the observed success rate in the decomposition step, $\phi_{\text{m}}$ in the merging step and $\phi_{\text{u}}$ in the unit-case. The values are computed using all problem classes $c_i$ and within-class difficulties $n_i$ appearing in the decomposition graph. $\phi_{\text{RDD}}$ is the end-to-end accuracy.

Theorems & Definitions (4)

• Theorem 1: Decomposition and merging requirement
• proof
• Theorem 2: Unit case requirement
• proof