Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMs

TL;DR

The paper distinguishes deductive and inductive reasoning in LLMs and introduces SolverLearner, a two-stage framework that learns a mapping $y = f_w(x)$ from in-context examples and executes it via external interpreters to isolate inductive reasoning. Empirically, LLMs show weak deductive performance, particularly on counterfactual tasks, while SolverLearner enables near-perfect inductive reasoning across four tasks and multiple models, with GPT-4 outperforming GPT-3.5. The work demonstrates that disentangling learning of the mapping from its application is crucial to reveal pure inductive capabilities, and it provides cross-task, cross-model evidence using arithmetic, syntax, spatial, and cipher decryption tasks. These findings highlight the potential of external execution to isolate and study inductive reasoning in LLMs, with implications for designing more reliable reasoning systems and understanding the limits of deductive capabilities under pretraining regimes.

Abstract

Reasoning encompasses two typical types: deductive reasoning and inductive reasoning. Despite extensive research into the reasoning capabilities of Large Language Models (LLMs), most studies have failed to rigorously differentiate between inductive and deductive reasoning, leading to a blending of the two. This raises an essential question: In LLM reasoning, which poses a greater challenge - deductive or inductive reasoning? While the deductive reasoning capabilities of LLMs, (i.e. their capacity to follow instructions in reasoning tasks), have received considerable attention, their abilities in true inductive reasoning remain largely unexplored. To investigate into the true inductive reasoning capabilities of LLMs, we propose a novel framework, SolverLearner. This framework enables LLMs to learn the underlying function (i.e., $y = f_w(x)$), that maps input data points $(x)$ to their corresponding output values $(y)$, using only in-context examples. By focusing on inductive reasoning and separating it from LLM-based deductive reasoning, we can isolate and investigate inductive reasoning of LLMs in its pure form via SolverLearner. Our observations reveal that LLMs demonstrate remarkable inductive reasoning capabilities through SolverLearner, achieving near-perfect performance with ACC of 1 in most cases. Surprisingly, despite their strong inductive reasoning abilities, LLMs tend to relatively lack deductive reasoning capabilities, particularly in tasks involving ``counterfactual'' reasoning.

Figures (5)

• Figure 1: We have designed a set of comparative experiments that utilize a consistent task across different contexts, each emphasizing either deductive (i.e., methods (a) and (b)) or inductive reasoning (i.e., methods (c) and (d)). As we move from left to right across the figure, the methods gradually transition their primary focus from deductive reasoning to inductive reasoning. Specifically, method (a) is designed to demonstrate the LLMs' deductive reasoning in its pure form. Conversely, method (c) utilizes Input-Output (IO) prompting strategies, which are prevalent for probing the inductive reasoning skills of LLMs. However, we can observe that methods (c) cannot fully disentangle inductive reasoning from deductive reasoning as their learning process directly moves from observations to specific instances, blurring the lines between the two. To exclusively focus on and examine inductive reasoning, we introduce a novel framework called SolverLearner, positioned at the far right of the spectrum.
• Figure 2: An overview of our framework SolverLearner for inductive reasoning. SolverLearner follows a two-step process to segregate the learning of input-output mapping functions from the application of these functions for inference. Specifically, functions are applied through external code interpreters, to avoid incorporating LLM-based deductive reasoning.
• Figure 3: Comparison of the deductive reasoning abilities of LLMs across various tasks. Different methods are illustrated through color-coded bars: blue bars indicate the results achieved using Zero-shot, while orange bars show the performance of 8-IO w/ Mapping Function (MF).
• Figure 4: Comparison of the inductive reasoning abilities of LLMs across various tasks. Different methods are illustrated through color-coded bars: blue bars indicate the results achieved using our proposed SolverLearner, while orange bars show the performance of 8-IO w/o Mapping Function (MF).
• Figure 5: Comparison of the inductive reasoning abilities versus deductive reasoning abilities of LLMs across various tasks. Different methods are illustrated through color-coded bars: blue bars indicate the results achieved using our proposed SolverLearner for inductive reasoning, while orange bars show the performance of Zero-shot for deductive reasoning.