From Token to Action: State Machine Reasoning to Mitigate Overthinking in Information Retrieval

TL;DR

This work tackles overthinking in chain-of-thought prompting for information retrieval by introducing State Machine Reasoning (SMR), a transition-based framework that models reasoning as discrete actions over structured states $(q_t, D_t)$. By restricting reasoning to IR-specific actions—Refine, Rerank, and Stop—SMR achieves targeted, incremental improvements with early stopping, reducing token usage while maintaining or boosting retrieval quality. Empirical results on BRIGHT and BEIR show that SMR outperforms strong CoT baselines and compressed-CoT methods, with up to +3.4% ndcg@10 gains on BRIGHT and about a 74% reduction in inference tokens, and it generalizes across retrievers and LLMs without task-specific tuning. The approach offers a practical, tunable alternative to traditional CoT reasoning, enabling interpretable control over the retrieval process and efficient deployment in real-world IR systems.

Abstract

Chain-of-Thought (CoT) prompting enables complex reasoning in large language models (LLMs), including applications in information retrieval (IR). However, it often leads to overthinking, where models produce excessively long and semantically redundant traces with little or no benefit. We identify two key challenges in IR: redundant trajectories that revisit similar states and misguided reasoning that diverges from user intent. To address these, we propose State Machine Reasoning (SMR), a transition-based reasoning framework composed of discrete actions (Refine, Rerank, Stop) that support early stopping and fine-grained control. Experiments on the BEIR and BRIGHT benchmarks show that SMR improves retrieval performance (nDCG@10) by 3.4% while reducing token usage by 74.4%. It generalizes across LLMs and retrievers without requiring task-specific tuning, offering a practical alternative to conventional CoT reasoning. The code and details are available at https://github.com/ldilab/SMR.

Figures (5)

• Figure 1: Conceptual comparison between standard CoT reasoning, compressed CoT reasoning, and our proposed SMR framework. (a) Standard CoT performs token-level generation in a single forward pass, often leading to redundant intermediate states. (b) Compressed CoT shortens trajectories via reinforcement learning, at the risk of misaligned outputs. (c) SMR decomposes reasoning into structured state transitions $(q, D)$ guided by IR-specific actions.
• Figure 2: Illustration of our proposed reasoning framework (SMR). Beginning from an initial query and its retrieved documents, SMR transitions through structured states via three actions: Refine (query rewriting), Rerank (document reordering), and Stop (termination). At each step, the LLM selects an action with justification, and the document list is updated accordingly.
• Figure 3: Distribution of reasoning actions selected by SMR on the BRIGHT benchmark. Red bars indicate the proportion of Refine actions, and blue bars indicate Rerank actions.
• Figure 4: Inference token usage across five representative datasets in the BRIGHT benchmark. SMR (green bars) achieves significantly lower token consumption than Rank1, Rank-R1, and O1-Pruner, while improving retrieval performance. Full results including all datasets are presented in Appendix \ref{['appendix:token_efficiency']}.
• Figure 5: Transition statistics of SMR on the BRIGHT benchmark. The blue bars indicate the number of reasoning steps across all queries, computed cumulatively per transition depth. The red curve shows the average retrieval performance (nDCG@10) at each step.