Thinker: Training LLMs in Hierarchical Thinking for Deep Search via Multi-Turn Interaction

TL;DR

Thinker tackles the challenge of making LLM-based deep search both effective and verifiable by introducing hierarchical thinking that decomposes problems into atomic sub-problems. Each sub-problem is represented in dual forms—natural-language steps and logical actions—allowing structured retrieval and reasoning across breadth decomposition and depth solving, guided by a knowledge boundary detector to prune unnecessary searches. The approach is reinforced by multi-turn supervised training and four logical forms (Retrieval, Deduce, Math, Output), with additional enhancements from the KAG framework and potential gains from RL. Empirical results show strong sample efficiency (hundreds of examples suffice) and consistent outperformance over baselines across multiple datasets and model sizes, with reduced retrieval overhead and improved logical coherence.

Abstract

Efficient retrieval of external knowledge bases and web pages is crucial for enhancing the reasoning abilities of LLMs. Previous works on training LLMs to leverage external retrievers for solving complex problems have predominantly employed end-to-end reinforcement learning. However, these approaches neglect supervision over the reasoning process, making it difficult to guarantee logical coherence and rigor. To address these limitations, we propose Thinker, a hierarchical thinking model for deep search through multi-turn interaction, making the reasoning process supervisable and verifiable. It decomposes complex problems into independently solvable sub-problems, each dually represented in both natural language and an equivalent logical function to support knowledge base and web searches. Concurrently, dependencies between sub-problems are passed as parameters via these logical functions, enhancing the logical coherence of the problem-solving process. To avoid unnecessary external searches, we perform knowledge boundary determination to check if a sub-problem is within the LLM's intrinsic knowledge, allowing it to answer directly. Experimental results indicate that with as few as several hundred training samples, the performance of Thinker is competitive with established baselines. Furthermore, when scaled to the full training set, Thinker significantly outperforms these methods across various datasets and model sizes. The source code is available at https://github.com/OpenSPG/KAG-Thinker.

Figures (7)

• Figure 1: Typical problems with deep search methods based on reinforcement learning training.
• Figure 2: Overview of the Thinker model that uses hierarchical thinking through a multi-turn reasoning process. During problem breadth decomposition, all sub-problems are obtained in a single decomposition pass, where each sub-problem is an atomic problem that can be solved independently. Herein, the terms Step and Action maintain semantic consistency, both denoting such a sub-problem. Within problem breadth decomposition, Step employs $\#n$ for answer propagation of the $n$-th sub-problem, while Action binds variables in logical function (e.g., $o_n$, $s_n$) for variable transmission. By determining the knowledge boundary of the sub-problem, it is decided whether to utilize the base model's answer or to generate a deep retrieval. During depth solving of sub-problems, the system sequentially executes retrieval, focusing, and reasoning in iterative processes until either the sub-problem answer is obtained or the maximum solving attempt threshold is activated.
• Figure 3: The probability distribution of the answer tokens. The left shows the low confidence answers, and the right shows the high confidence answers.
• Figure 4: Performance of Thinker with Reinforcement Learning vs. Training Steps. (a) Average reward during the RL training. (b) Average model response length. (c) Average EM score across seven datasets. (d) Average F1 score across seven datasets.
• Figure 5: Overview of the data construction framework.