A Survey of Inductive Reasoning for Large Language Models

TL;DR

Inductive reasoning enables LLMs to generalize from specific observations to broad rules, but a systematic understanding has been lacking. This survey categorizes enhancement methods into post-training, test-time scaling, and data augmentation, and proposes a unified sandbox-based evaluation with an observation coverage metric. It catalogs benchmarks spanning diverse domains and analyzes inductive sources such as induction heads and the roles of model parameters, architecture, and data in shaping inductive bias. The work provides a foundational framework to guide future research and practical improvements in inductive reasoning for LLMs.

Abstract

Reasoning is an important task for large language models (LLMs). Among all the reasoning paradigms, inductive reasoning is one of the fundamental types, which is characterized by its particular-to-general thinking process and the non-uniqueness of its answers. The inductive mode is crucial for knowledge generalization and aligns better with human cognition, so it is a fundamental mode of learning, hence attracting increasing interest. Despite the importance of inductive reasoning, there is no systematic summary of it. Therefore, this paper presents the first comprehensive survey of inductive reasoning for LLMs. First, methods for improving inductive reasoning are categorized into three main areas: post-training, test-time scaling, and data augmentation. Then, current benchmarks of inductive reasoning are summarized, and a unified sandbox-based evaluation approach with the observation coverage metric is derived. Finally, we offer some analyses regarding the source of inductive ability and how simple model architectures and data help with inductive tasks, providing a solid foundation for future research.

Figures (7)

• Figure 1: Two examples of inductive reasoning. They generalize from specific observations or cases to derive general conclusions. There may be more than one such conclusion that meets all the observations.
• Figure 2: Taxonomy of the survey about the inductive reasoning for LLMs.
• Figure 3: The demonstration of the post-training method for inductive reasoning.
• Figure 4: The demonstration of the test-time scaling method for inductive reasoning.
• Figure 5: The demonstration of the data augmentation method for inductive reasoning.