ItD: Large Language Models Can Teach Themselves Induction through Deduction

TL;DR

This work introduces ItD, a framework that enables Large Language Models to teach themselves induction through deduction by first generating deduced task data with a Deductive Data Generation module and then fine-tuning and decoding with Naive Bayesian Induction. ItD demonstrates substantial improvements over prior post-processing approaches on Instruction Induction and List Function benchmarks, validating both components’ contributions and showing gains across multiple model sizes and deductors. The results suggest that leveraging deduction to produce high-quality training data, combined with Bayesian-style aggregation during decoding, can significantly bolster inductive capabilities in LLMs. Limitations include somewhat weaker performance on symbolic tasks and a greedy decoding scheme, with future work aimed at richer deduction signals and more sophisticated decoding strategies.

Abstract

Although Large Language Models (LLMs) are showing impressive performance on a wide range of Natural Language Processing tasks, researchers have found that they still have limited ability to conduct induction. Recent works mainly adopt ``post processes'' paradigms to improve the performance of LLMs on induction (e.g., the hypothesis search & refinement methods), but their performance is still constrained by the inherent inductive capability of the LLMs. In this paper, we propose a novel framework, Induction through Deduction (ItD), to enable the LLMs to teach themselves induction through deduction. The ItD framework is composed of two main components: a Deductive Data Generation module to generate induction data and a Naive Bayesian Induction module to optimize the fine-tuning and decoding of LLMs. Our empirical results showcase the effectiveness of ItD on two induction benchmarks, achieving relative performance improvement of 36% and 10% compared with previous state-of-the-art, respectively. Our ablation study verifies the effectiveness of two key modules of ItD. We also verify the effectiveness of ItD across different LLMs and deductors. The data and code of this paper can be found at https://anonymous.4open.science/r/ItD-E844.

Figures (5)

• Figure 1: Task of Induction. The tested model observes a batch of input-output $(x,y)$ pairs and needs to predict the latent transformation $f$ shared by these $(x,y)$ pairs.
• Figure 2: Comparison of ItD and Previous Methods. Previous hypothesis search & refinement methods are essentially "post processes" to the raw induction results of LLMs, leaving LLMs as Weak Inductors. By contrast, ItD fine-tune the LLMs and propose a novel decoding algorithm to make them Strong Inductors.
• Figure 3: The framework of ItD. ItD includes two main parts, i.e. Deductive Data Generation and Naive Bayesian Induction. Given the induction set $\mathcal{D}_{in}$, ItD will first leverage the deductive capability of LLMs to generate data that closely resembles the distribution of the task data. Then Naive Bayesian Induction is adopted to optimize the use of each observed sample while achieving better performances with the increase in the number of samples.
• Figure 4: The prompt used for Deduction with In-Context Learning. LLMs will generate multiple samples $(x,y)$ for each $f$ in a deductive behavior.
• Figure 5: Naive Bayesian Induction can benefit from the increase in the number of observed samples.