RLIE: Rule Generation with Logistic Regression, Iterative Refinement, and Evaluation for Large Language Models

TL;DR

RLIE tackles the challenge of learning robust, interpretable rule sets by integrating LLM-based rule generation with probabilistic rule weighting. It introduces four stages—Rule Generation, Logistic Regression, Iterative Refinement, and Evaluation—to produce a compact, calibrated set of weighted natural-language rules, and it systematically compares several inference strategies. A key finding is that direct use of the learned weighted rules via a linear classifier often outperforms LLM-based inference that ingests the rules, highlighting the complementary strengths of LLMs in semantic generation and the reliability of classical probabilistic aggregation for precise inference. The framework demonstrates strong, stable performance across six HypoBench tasks and provides practical guidance on when to leverage LLMs for local rule interpretation versus global probabilistic reasoning, advancing neuro-symbolic approaches to inductive reasoning.

Abstract

Large Language Models (LLMs) can propose rules in natural language, sidestepping the need for a predefined predicate space in traditional rule learning. Yet many LLM-based approaches ignore interactions among rules, and the opportunity to couple LLMs with probabilistic rule learning for robust inference remains underexplored. We present RLIE, a unified framework that integrates LLMs with probabilistic modeling to learn a set of weighted rules. RLIE has four stages: (1) Rule generation, where an LLM proposes and filters candidates; (2) Logistic regression, which learns probabilistic weights for global selection and calibration; (3) Iterative refinement, which updates the rule set using prediction errors; and (4) Evaluation, which compares the weighted rule set as a direct classifier with methods that inject rules into an LLM. We evaluate multiple inference strategies on real-world datasets. Applying rules directly with their learned weights yields superior performance, whereas prompting LLMs with the rules, weights, and logistic-model outputs surprisingly degrades accuracy. This supports the view that LLMs excel at semantic generation and interpretation but are less reliable for precise probabilistic integration. RLIE clarifies the potential and limitations of LLMs for inductive reasoning and couples them with classic probabilistic rule combination methods to enable more reliable neuro-symbolic reasoning.

Figures (8)

• Figure 1: The pipeline of the RLIE framework. (1) Rule Generation: An LLM proposes an initial set of rules from a small sample of training data. (2) Logistic Regression: A regularized logistic regression model is trained on the full training set to learn probabilistic weights for the current rules. (3) Iterative Refinement: Prediction errors from the regression model are used to mine hard examples, which are then used to prompt the LLM to refine or generate new rules. This cycle continues until performance on a validation set converges. (4) Evaluation: The learnt rules as well as weights are evaluated with different inference strategies. Detailed explanations of different strategies are in Section \ref{['evaluation']}.
• Figure 2: Prompt for providing observations.
• Figure 3: Prompt for the first iteration of rule generation.
• Figure 4: Prompt for iterative refinement.
• Figure 5: Prompt for single hypothesis inference.