On the Role of Model Prior in Real-World Inductive Reasoning

TL;DR

This paper investigates the role of task-specific priors versus in-context demonstrations in real-world inductive reasoning by LLMs. Through three hypothesis-generation baselines evaluated on five multimodal datasets and three models, the authors show that priors dominantly drive hypothesis generation, with demonstrations contributing minimal loss in quality when removed. The study uses hypothesis-based, LLM-based, and human evaluations to triangulate findings and demonstrates robustness across label formats and model families. The results challenge the necessity of demonstrations for effective hypothesis generation and suggest focusing on harnessing priors for practical inductive reasoning tasks.

Abstract

Large Language Models (LLMs) show impressive inductive reasoning capabilities, enabling them to generate hypotheses that could generalize effectively to new instances when guided by in-context demonstrations. However, in real-world applications, LLMs' hypothesis generation is not solely determined by these demonstrations but is significantly shaped by task-specific model priors. Despite their critical influence, the distinct contributions of model priors versus demonstrations to hypothesis generation have been underexplored. This study bridges this gap by systematically evaluating three inductive reasoning strategies across five real-world tasks with three LLMs. Our empirical findings reveal that, hypothesis generation is primarily driven by the model's inherent priors; removing demonstrations results in minimal loss of hypothesis quality and downstream usage. Further analysis shows the result is consistent across various label formats with different label configurations, and prior is hard to override, even under flipped labeling. These insights advance our understanding of the dynamics of hypothesis generation in LLMs and highlight the potential for better utilizing model priors in real-world inductive reasoning tasks.

Figures (8)

• Figure 1: Prompt template for hypothesis generation.
• Figure 2: Accuracy difference comparison of single hypothesis-based classification under different label settings: Accuracy difference (accuracy of different label settings - accuracy without demos) across five datasets with IO-Prompting.
• Figure 3: LLM-based Pairwise Comparison: Pairwise win rate (%) of three baselines. The left plot shows the comparison of Helpfulness, while the right plot presents Novelty. The dashed line indicates a tie where "w/ demos" and "w/o demos" perform equally well.
• Figure 4: Human pairwise comparison results on three datasets, showing preferences for hypotheses with, without demos, and cases where it was hard to tell the difference.
• Figure 5: Difference of predictions between correct label and flipped label demos: Adverse Correction Rate (ACR) and Beneficial Correction Rate (BCR) values under multiple hypotheses-based classification.