Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Explainable Rule Application via Structured Prompting: A Neural-Symbolic Approach

Albert Sadowski, Jarosław A. Chudziak

TL;DR

This work tackles the challenge of applying complex, rule-based reasoning in legal contexts by combining neural and symbolic methods through a structured prompting framework. The approach decomposes reasoning into three steps—entity identification, predicate extraction, and rule application via SMT verification—and externalizes task predicates to enable domain refinement without architectural changes. Empirical results on the LegalBench hearsay task show substantial improvements for OpenAI o-family models when using structured decomposition with complementary predicates, demonstrating improved explainability and reduced hallucination risks. The study highlights the practical potential of transparent neural-symbolic reasoning for legal AI and outlines avenues for extending the framework to richer logics, automated predicate design, and cross-domain applications.

Abstract

Large Language Models (LLMs) excel in complex reasoning tasks but struggle with consistent rule application, exception handling, and explainability, particularly in domains like legal analysis that require both natural language understanding and precise logical inference. This paper introduces a structured prompting framework that decomposes reasoning into three verifiable steps: entity identification, property extraction, and symbolic rule application. By integrating neural and symbolic approaches, our method leverages LLMs' interpretive flexibility while ensuring logical consistency through formal verification. The framework externalizes task definitions, enabling domain experts to refine logical structures without altering the architecture. Evaluated on the LegalBench hearsay determination task, our approach significantly outperformed baselines, with OpenAI o-family models showing substantial improvements - o1 achieving an F1 score of 0.929 and o3-mini reaching 0.867 using structured decomposition with complementary predicates, compared to their few-shot baselines of 0.714 and 0.74 respectively. This hybrid neural-symbolic system offers a promising pathway for transparent and consistent rule-based reasoning, suggesting potential for explainable AI applications in structured legal reasoning tasks.

Explainable Rule Application via Structured Prompting: A Neural-Symbolic Approach

TL;DR

This work tackles the challenge of applying complex, rule-based reasoning in legal contexts by combining neural and symbolic methods through a structured prompting framework. The approach decomposes reasoning into three steps—entity identification, predicate extraction, and rule application via SMT verification—and externalizes task predicates to enable domain refinement without architectural changes. Empirical results on the LegalBench hearsay task show substantial improvements for OpenAI o-family models when using structured decomposition with complementary predicates, demonstrating improved explainability and reduced hallucination risks. The study highlights the practical potential of transparent neural-symbolic reasoning for legal AI and outlines avenues for extending the framework to richer logics, automated predicate design, and cross-domain applications.

Abstract

Large Language Models (LLMs) excel in complex reasoning tasks but struggle with consistent rule application, exception handling, and explainability, particularly in domains like legal analysis that require both natural language understanding and precise logical inference. This paper introduces a structured prompting framework that decomposes reasoning into three verifiable steps: entity identification, property extraction, and symbolic rule application. By integrating neural and symbolic approaches, our method leverages LLMs' interpretive flexibility while ensuring logical consistency through formal verification. The framework externalizes task definitions, enabling domain experts to refine logical structures without altering the architecture. Evaluated on the LegalBench hearsay determination task, our approach significantly outperformed baselines, with OpenAI o-family models showing substantial improvements - o1 achieving an F1 score of 0.929 and o3-mini reaching 0.867 using structured decomposition with complementary predicates, compared to their few-shot baselines of 0.714 and 0.74 respectively. This hybrid neural-symbolic system offers a promising pathway for transparent and consistent rule-based reasoning, suggesting potential for explainable AI applications in structured legal reasoning tasks.

Paper Structure

This paper contains 16 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Prompting Techniques for Complex Reasoning
  4. Neural-Symbolic Integration Approaches
  5. Legal Rule Application Challenges
  6. Structured Decomposition Framework
  7. Neural-Symbolic Three-Step Architecture
  8. Externalized Task Predicate Definitions
  9. Case Study: Hearsay Determination
  10. Experimental Setup and Evaluation
  11. Implementation Details
  12. Results and Analysis
  13. Impact of Framework Components and Ablation Studies
  14. Structural Benefits for Explainability
  15. Discussion
  16. ...and 1 more sections

Figures (1)

  • Figure 1: Structured Decomposition Framework. The three-step process flows from Term Identification through Predicate Extraction to Rule Application via SMT solving. Yellow box represents domain expert input (task definition, i.e., terms, predicates, rules), blue boxes show automated LLM/solver processing, and arrows indicate data flow between components.