Generating Diverse Hypotheses for Inductive Reasoning

TL;DR

The paper addresses the problem of redundant hypotheses produced when using IID sampling for inductive reasoning with LLMs. It shows that increasing temperature boosts diversity and accuracy only up to a saturation point limited by text degeneration, and introduces Mixture of Concepts (MoC), a two stage approach that first proposes diverse, non redundant concepts and then generates hypotheses conditioned on each concept. Across four datasets and multiple base LLMs, MoC yields sizable improvements in accuracy and hypothesis diversity, outperforming vanilla IID sampling and hypothesis refinement in many settings. The findings highlight that diversity driven by concept based prompting can substantially enhance automatic inductive reasoning while reducing compute, with implications for scalable program synthesis and reasoning tasks.

Abstract

Inductive reasoning - the process of inferring general rules from a small number of observations - is a fundamental aspect of human intelligence. Recent works suggest that large language models (LLMs) can engage in inductive reasoning by sampling multiple hypotheses about the rules and selecting the one that best explains the observations. However, due to the IID sampling, semantically redundant hypotheses are frequently generated, leading to significant wastage of compute. In this paper, we 1) demonstrate that increasing the temperature to enhance the diversity is limited due to text degeneration issue, and 2) propose a novel method to improve the diversity while maintaining text quality. We first analyze the effect of increasing the temperature parameter, which is regarded as the LLM's diversity control, on IID hypotheses. Our analysis shows that as temperature rises, diversity and accuracy of hypotheses increase up to a certain point, but this trend saturates due to text degeneration. To generate hypotheses that are more semantically diverse and of higher quality, we propose a novel approach inspired by human inductive reasoning, which we call Mixture of Concepts (MoC). When applied to several inductive reasoning benchmarks, MoC demonstrated significant performance improvements compared to standard IID sampling and other approaches.

Figures (7)

• Figure 1: A motivation for MoC approach. IID sampling frequently generates redundant hypotheses (top). Increasing the temperature leads to frequent occurrences of text degeneration (middle). MoC allows for the generation of diverse hypotheses without a decline in hypothesis quality (bottom).
• Figure 2: Ratio (%) of degenerate responses.
• Figure 3: GPT-4o-mini hypothesis diversity on two domains. For the temperature 1.67 and 2.0, we used top-$p$ sampling with $p=0.95$. Results are averaged over 5 runs.
• Figure 4: GPT-4o-mini performance on two domains. For the temperature 1.67 and 2.0, we used top-$p$ sampling with $p=0.95$. Results are averaged over 5 runs.
• Figure 5: An overview of our Mixture of Concepts approach. We generate $K$ distinct concepts (left) and feed them into the LLM separately for hypothesis generation (right).