Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Teaching Probabilistic Logical Reasoning to Transformers

Aliakbar Nafar, Kristen Brent Venable, Parisa Kordjamshidi

TL;DR

This work addresses probabilistic logical reasoning under uncertainty in text by introducing Probabilistic Constraint Training (PCT), a training-time constraint mechanism that enforces probabilistic reasoning rules without constraining inference. It pairs PCT with RuleTaker-pro, a dataset featuring instance-specific probabilistic rules, to study generalization, transfer, and explainability in transformer models like RoBERTa Large. Empirical results show that PCT improves intrinsic reasoning, intermediate-inference consistency (constraint satisfaction), and transfer to higher depths and new domains, while large language models struggle with exact probabilistic inferences despite few-shot prompts. The findings highlight the value of integrating probabilistic constraints during training for robust, explainable reasoning in uncertain domains and point to future work in better utilizing textual rules and prompting for LLMs.

Abstract

In this paper, we evaluate the capability of transformer-based language models in making inferences over uncertain text that includes uncertain rules of reasoning. We cover both Pre-trained Language Models (PLMs) and generative Large Language Models (LLMs). Our evaluation results show that both generations of language models struggle with reasoning over uncertain text. We propose a novel end-to-end fine-tuning approach, Probabilistic Constraint Training (PCT), that utilizes probabilistic logical rules as constraints in the fine-tuning phase without relying on these rules in the inference stage. To assess the effectiveness of PCT, we utilize the related corpora and, additionally, create a new and more challenging benchmark that, unlike the previous ones, uses instance-specific rules. Our study demonstrates that PCT improves the transformer-based language model's intrinsic reasoning and makes their probabilistic logical reasoning process more explicit and explainable. Furthermore, PCT equips these models to effectively handle novel situations, including higher reasoning depth, new domains, and complex probabilistic structures.

Teaching Probabilistic Logical Reasoning to Transformers

TL;DR

This work addresses probabilistic logical reasoning under uncertainty in text by introducing Probabilistic Constraint Training (PCT), a training-time constraint mechanism that enforces probabilistic reasoning rules without constraining inference. It pairs PCT with RuleTaker-pro, a dataset featuring instance-specific probabilistic rules, to study generalization, transfer, and explainability in transformer models like RoBERTa Large. Empirical results show that PCT improves intrinsic reasoning, intermediate-inference consistency (constraint satisfaction), and transfer to higher depths and new domains, while large language models struggle with exact probabilistic inferences despite few-shot prompts. The findings highlight the value of integrating probabilistic constraints during training for robust, explainable reasoning in uncertain domains and point to future work in better utilizing textual rules and prompting for LLMs.

Abstract

In this paper, we evaluate the capability of transformer-based language models in making inferences over uncertain text that includes uncertain rules of reasoning. We cover both Pre-trained Language Models (PLMs) and generative Large Language Models (LLMs). Our evaluation results show that both generations of language models struggle with reasoning over uncertain text. We propose a novel end-to-end fine-tuning approach, Probabilistic Constraint Training (PCT), that utilizes probabilistic logical rules as constraints in the fine-tuning phase without relying on these rules in the inference stage. To assess the effectiveness of PCT, we utilize the related corpora and, additionally, create a new and more challenging benchmark that, unlike the previous ones, uses instance-specific rules. Our study demonstrates that PCT improves the transformer-based language model's intrinsic reasoning and makes their probabilistic logical reasoning process more explicit and explainable. Furthermore, PCT equips these models to effectively handle novel situations, including higher reasoning depth, new domains, and complex probabilistic structures.
Paper Structure (31 sections, 5 equations, 3 figures, 20 tables, 2 algorithms)

This paper contains 31 sections, 5 equations, 3 figures, 20 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Background
  4. Problem Definition
  5. Base Model
  6. Deep Learning with Logical Constraints
  7. Training with Probabilistic Constraints
  8. Constraint Integration
  9. Training and Inference
  10. Dataset Creation
  11. Experiments
  12. Q1: Effect of Rules in Textual Format
  13. Q2: Effectiveness of PCT
  14. Q3: Transferability Analysis
  15. Q4: LLM Results
  16. ...and 16 more sections

Figures (3)

  • Figure 1: The CS25 of intermediate inferred facts over 6 Epochs of training for M5.
  • Figure 2: In the given example, the fact "The rabbit visits the lion." can be inferred from the context with a probability of 1.00 at depth 2. Both the base model and the PCT model accurately predicted the probability of this fact. However, only the PCT model took into account the additional bold rule in the text, which led to an 0.85 probability for the hypothesis.
  • Figure :