Structured Decomposition for LLM Reasoning: Cross-Domain Validation and Semantic Web Integration

TL;DR

The paper addresses the challenge of applying auditable rule-based reasoning to unstructured natural language, where LLMs offer interpretation but lack guarantees while symbolic systems provide guarantees but require structured inputs. It proposes a neural-symbolic integration pattern that uses LLMs to populate an ABox from text according to expert-defined TBox specifications, with SWRL-based reasoning (via Pellet) delivering deterministic classifications. The authors formalise three tasks (hearsay determination, method application, clinical trial eligibility) in OWL 2 ontologies, evaluate 11 language models under six experimental conditions, and show that Structured Decomposition with symbolic verification yields statistically significant improvements over few-shot and chain-of-thought prompting, while complementary predicates can decrease performance. The approach yields inspectable reasoning traces, integrates with standard semantic-web tooling, and demonstrates cross-domain generalisability, positioning semantic-web–based reasoning as a scalable, auditable layer atop flexible LLM interpretation for high-stakes domains.

Abstract

Rule-based reasoning over natural language input arises in domains where decisions must be auditable and justifiable: clinical protocols specify eligibility criteria in prose, evidence rules define admissibility through textual conditions, and scientific standards dictate methodological requirements. Applying rules to such inputs demands both interpretive flexibility and formal guarantees. Large language models (LLMs) provide flexibility but cannot ensure consistent rule application; symbolic systems provide guarantees but require structured input. This paper presents an integration pattern that combines these strengths: LLMs serve as ontology population engines, translating unstructured text into ABox assertions according to expert-authored TBox specifications, while SWRL-based reasoners apply rules with deterministic guarantees. The framework decomposes reasoning into entity identification, assertion extraction, and symbolic verification, with task definitions grounded in OWL 2 ontologies. Experiments across three domains (legal hearsay determination, scientific method-task application, clinical trial eligibility) and eleven language models validate the approach. Structured decomposition achieves statistically significant improvements over few-shot prompting in aggregate, with gains observed across all three domains. An ablation study confirms that symbolic verification provides substantial benefit beyond structured prompting alone. The populated ABox integrates with standard semantic web tooling for inspection and querying, positioning the framework for richer inference patterns that simpler formalisms cannot express.

Figures (4)

• Figure 1: The NL2FOL pipeline for translating natural language to first-order logic (reproduced from Lalwani et al. Lalwani2024). Module A decomposes input text through successive extraction stages, claim parsing, entity identification, relation classification, and property extraction, before formulating FOL expressions augmented with background knowledge. Module B compiles the logical formula to SMT format, and Module C interprets solver outputs as natural language counterexamples. This stepwise architecture exemplifies the neural-symbolic integration pattern our framework adopts, though we replace ad-hoc FOL translation with OWL ontology population and SMT verification with SWRL-based reasoning.
• Figure 2: Structured Decomposition Framework with OWL/SWRL integration. Yellow box represents domain expert input (task ontology defining classes, properties, and SWRL rules); blue boxes show automated processing by LLMs (entity identification, assertion extraction) and reasoner (rule application); arrows indicate data flow. Beyond the classification outcome, the framework yields a populated ABox that externalises every extraction decision, providing an inspectable record of how the input text was interpreted.
• Figure 3: Performance shift from few-shot baseline (blue) to structured decomposition (red) across eleven models and three tasks. Green lines indicate improvement; red lines indicate regression. Method Application shows the most consistent gains.
• Figure 4: Precision-recall trade-offs by prompting strategy. Dashed curves show F1 iso-lines (0.6--0.9). Structured decomposition (triangles) achieves higher recall, with strongest performance on Method Application, supporting generalisation beyond the legal domain.