Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thinking Machines: A Survey of LLM based Reasoning Strategies

Dibyanayan Bandyopadhyay, Soham Bhattacharjee, Asif Ekbal

TL;DR

The paper systematically surveys how LLMs achieve reasoning beyond surface-level language ability, organizing approaches into reinforcement learning, test-time compute, and self-training. It covers foundational concepts (LM objectives, RL, PO, MCTS), and surveys concrete strategies such as PPO/GRPO, DPO, various MCTS-based planning, and self-training regimes like STaR and ReFT. It highlights the shift from pure scaling to inference-time and data-generation strategies, and discusses critical challenges in automation, computation, and data requirements. The work aims to provide a concise, beginner-friendly taxonomy and practical guidance for researchers seeking to advance reasoning in LLMs.

Abstract

Large Language Models (LLMs) are highly proficient in language-based tasks. Their language capabilities have positioned them at the forefront of the future AGI (Artificial General Intelligence) race. However, on closer inspection, Valmeekam et al. (2024); Zecevic et al. (2023); Wu et al. (2024) highlight a significant gap between their language proficiency and reasoning abilities. Reasoning in LLMs and Vision Language Models (VLMs) aims to bridge this gap by enabling these models to think and re-evaluate their actions and responses. Reasoning is an essential capability for complex problem-solving and a necessary step toward establishing trust in Artificial Intelligence (AI). This will make AI suitable for deployment in sensitive domains, such as healthcare, banking, law, defense, security etc. In recent times, with the advent of powerful reasoning models like OpenAI O1 and DeepSeek R1, reasoning endowment has become a critical research topic in LLMs. In this paper, we provide a detailed overview and comparison of existing reasoning techniques and present a systematic survey of reasoning-imbued language models. We also study current challenges and present our findings.

Thinking Machines: A Survey of LLM based Reasoning Strategies

TL;DR

The paper systematically surveys how LLMs achieve reasoning beyond surface-level language ability, organizing approaches into reinforcement learning, test-time compute, and self-training. It covers foundational concepts (LM objectives, RL, PO, MCTS), and surveys concrete strategies such as PPO/GRPO, DPO, various MCTS-based planning, and self-training regimes like STaR and ReFT. It highlights the shift from pure scaling to inference-time and data-generation strategies, and discusses critical challenges in automation, computation, and data requirements. The work aims to provide a concise, beginner-friendly taxonomy and practical guidance for researchers seeking to advance reasoning in LLMs.

Abstract

Large Language Models (LLMs) are highly proficient in language-based tasks. Their language capabilities have positioned them at the forefront of the future AGI (Artificial General Intelligence) race. However, on closer inspection, Valmeekam et al. (2024); Zecevic et al. (2023); Wu et al. (2024) highlight a significant gap between their language proficiency and reasoning abilities. Reasoning in LLMs and Vision Language Models (VLMs) aims to bridge this gap by enabling these models to think and re-evaluate their actions and responses. Reasoning is an essential capability for complex problem-solving and a necessary step toward establishing trust in Artificial Intelligence (AI). This will make AI suitable for deployment in sensitive domains, such as healthcare, banking, law, defense, security etc. In recent times, with the advent of powerful reasoning models like OpenAI O1 and DeepSeek R1, reasoning endowment has become a critical research topic in LLMs. In this paper, we provide a detailed overview and comparison of existing reasoning techniques and present a systematic survey of reasoning-imbued language models. We also study current challenges and present our findings.

Paper Structure

This paper contains 15 sections, 6 equations, 11 figures.

Table of Contents

  1. Introduction
  2. Preliminary Concepts
  3. Language Model Objectives and Sampling
  4. Basics of Reinforcement Learning for LLMs
  5. Policy Optimization
  6. Preference Optimization
  7. Monte Carlo Tree Search (MCTS)
  8. Methods
  9. Reinforcement Learning
  10. Test Time Compute
  11. Feedback Guided Improvement
  12. Scaling Test-Time Computation
  13. Self-Training Methods
  14. Challenges
  15. Conclusion

Figures (11)

  • Figure 1: Basic Reinforcement learning setup consisting of an environment and an LLM.
  • Figure 2: Taxonomy of methods for reasoning.
  • Figure 3: Taxonomy of Reasoning with Reinforcement Learning.
  • Figure 4: Verbal reinforcement objective that relies on generation followed by NL feedback. Often the feedback consists of just observing the impact of actor LLM's own action to the environment, as in yao2023reactsynergizingreasoningacting
  • Figure 5: Left: Monte Carlo tree search where the state values are evaluated using a value estimator network which is trained using RL via preference data or monte-carlo estimate of the reward. Right: World model as a feedback mechanism for tree-search.
  • ...and 6 more figures