Atom of Thoughts for Markov LLM Test-Time Scaling

TL;DR

AoT introduces a memoryless Markovian reasoning framework that decomposes problems into independent subquestions via a DAG during a decomposition phase and contracts them into next-state questions, thereby reducing test-time reasoning complexity while preserving answer equivalence. The two-phase transition (decomposition and contraction) enables modular integration with existing reasoning methods and supports scalable exploration through tree search and reflective refinement, leading to emergent atomic reasoning units. Empirical results across math, coding, and multi-hop QA show AoT provides strong performance and efficiency gains, with notable results on HotpotQA and other benchmarks, and ablations confirm the importance of explicit DAG-guided decomposition. The work suggests AoT as a versatile plug-in for enhancing various test-time inference frameworks, with potential extensions to training-time objectives and safety-aware deployment.

Abstract

Large Language Models (LLMs) achieve superior performance through training-time scaling, and test-time scaling further enhances their capabilities by conducting effective reasoning during inference. However, as the scale of reasoning increases, existing test-time scaling methods suffer from accumulated historical information, which not only wastes computational resources but also interferes with effective reasoning. To address this issue, we observe that complex reasoning can be achieved by solving a series of independent and self-contained subquestions. These subquestions are essentially \textit{atomic questions}, exhibiting the memoryless property similar to Markov processes. Based on this observation, we propose Atom of Thoughts (\our), where each state transition consists of decomposing the current question into a dependency-based directed acyclic graph and contracting its subquestions, forming a simplified question that maintains answer equivalence with the original problem. This answer preservation enables the iterative \textit{decomposition-contraction} process to naturally form a meaningful Markov reasoning process. Furthermore, these atomic states can be seamlessly integrated into existing test-time scaling methods, enabling \our to serve as a plug-in enhancement for improving reasoning capabilities. Experiments across six benchmarks demonstrate the effectiveness of \our both as a standalone framework and a plug-in enhancement. Notably, on HotpotQA, when applied to gpt-4o-mini, \our achieves an \textbf{80.6\%} F1 score, surpassing o3-mini by \textbf{3.4\%} and DeepSeek-R1 by \textbf{10.6\%}. The code is available at \href{https://github.com/qixucen/atom}{https://github.com/qixucen/atom}.

Figures (7)

• Figure 1: Token Allocation Comparison in Reasoning Frameworks.Figure (a) demonstrates the differences between thoughts and states, where the red-highlighted text in thoughts reflects dependencies on historical information, whereas states maintain answer-equivalence with the initial problem while progressively reducing execution complexity. Figure (b) illustrates differences in the number of prompt tokens and completion tokens for CoT, ToT, GoT, and the state-based AoT. For simplicity, we assume each thought consists of the same number of tokens, with an average of $O(n)$ thoughts required to express a solution. While ToT maintains $b$ branches, resulting in a fixed number of $b$ invocations per expansion stage, GoT's settings can be flexibly adjusted depending on the scenario and are thus denoted as $O(b)$.
• Figure 2: Overview of AoT. The Markov reasoning framework iteratively derives states $Q_{i+1}$ from predecessors $Q_{i}$ using DAG decomposition and contraction. The left part shows this iterative process, while the right part highlights the integration with existing methods. Any intermediate state $Q_i$ can act as an entry point $Q_0$ for other methods, ensuring flexible composition while preserving answer equivalence to the original question. This allows AoT to operate independently or as a preprocessing module to optimize the performance or efficiency of existing approaches.
• Figure 3: A comparison of performance and cost of various methods and ablation methods on the dataset, with GPT-4o-mini as the backbone. Each node in the curves represents an AoT (or ablation variants) iteration result, where increasing token consumption indicates deeper iterations. Due to relatively poor AR performance leading to scattered data points, AR data points are excluded.
• Figure 4: The process involves gradually enhancing integration for scaling up at test time. ToT uses three branches, while FoT employs two, four, and eight trees, respectively.
• Figure 5: Distribution of solution depths across questions. Darker orange bars indicate depths that appear more frequently in the dataset.