Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-Augmented Symbolic NLU System for More Reliable Continuous Causal Statement Interpretation

Xin Lian, Kenneth D. Forbus

TL;DR

This work addresses the reliability of natural language understanding by combining the broad coverage of large language models (LLMs) with the interpretability of symbolic NLU. It introduces a hybrid system where LLMs perform text rephrasing and lexicon expansion, while symbolic NLU generates structured qualitative representations (QP frames) for reasoning about continuous causal statements. Evaluated on QuaRTz, the hybrid approach significantly outperforms a symbolic-only baseline and approaches the performance of an LLM-only pipeline, demonstrating improved coverage and debuggable reasoning. The results suggest that calibrated LLM assistance can extend symbolic NLU without sacrificing transparency, enabling more reliable qualitative reasoning in open-domain texts.

Abstract

Despite the broad applicability of large language models (LLMs), their reliance on probabilistic inference makes them vulnerable to errors such as hallucination in generated facts and inconsistent output structure in natural language understanding (NLU) tasks. By contrast, symbolic NLU systems provide interpretable understanding grounded in curated lexicons, semantic resources, and syntactic & semantic interpretation rules. They produce relational representations that can be used for accurate reasoning and planning, as well as incremental debuggable learning. However, symbolic NLU systems tend to be more limited in coverage than LLMs and require scarce knowledge representation and linguistics skills to extend and maintain. This paper explores a hybrid approach that integrates the broad-coverage language processing of LLMs with the symbolic NLU capabilities of producing structured relational representations to hopefully get the best of both approaches. We use LLMs for rephrasing and text simplification, to provide broad coverage, and as a source of information to fill in knowledge gaps more automatically. We use symbolic NLU to produce representations that can be used for reasoning and for incremental learning. We evaluate this approach on the task of extracting and interpreting quantities and causal laws from commonsense science texts, along with symbolic- and LLM-only pipelines. Our results suggest that our hybrid method works significantly better than the symbolic-only pipeline.

LLM-Augmented Symbolic NLU System for More Reliable Continuous Causal Statement Interpretation

TL;DR

This work addresses the reliability of natural language understanding by combining the broad coverage of large language models (LLMs) with the interpretability of symbolic NLU. It introduces a hybrid system where LLMs perform text rephrasing and lexicon expansion, while symbolic NLU generates structured qualitative representations (QP frames) for reasoning about continuous causal statements. Evaluated on QuaRTz, the hybrid approach significantly outperforms a symbolic-only baseline and approaches the performance of an LLM-only pipeline, demonstrating improved coverage and debuggable reasoning. The results suggest that calibrated LLM assistance can extend symbolic NLU without sacrificing transparency, enabling more reliable qualitative reasoning in open-domain texts.

Abstract

Despite the broad applicability of large language models (LLMs), their reliance on probabilistic inference makes them vulnerable to errors such as hallucination in generated facts and inconsistent output structure in natural language understanding (NLU) tasks. By contrast, symbolic NLU systems provide interpretable understanding grounded in curated lexicons, semantic resources, and syntactic & semantic interpretation rules. They produce relational representations that can be used for accurate reasoning and planning, as well as incremental debuggable learning. However, symbolic NLU systems tend to be more limited in coverage than LLMs and require scarce knowledge representation and linguistics skills to extend and maintain. This paper explores a hybrid approach that integrates the broad-coverage language processing of LLMs with the symbolic NLU capabilities of producing structured relational representations to hopefully get the best of both approaches. We use LLMs for rephrasing and text simplification, to provide broad coverage, and as a source of information to fill in knowledge gaps more automatically. We use symbolic NLU to produce representations that can be used for reasoning and for incremental learning. We evaluate this approach on the task of extracting and interpreting quantities and causal laws from commonsense science texts, along with symbolic- and LLM-only pipelines. Our results suggest that our hybrid method works significantly better than the symbolic-only pipeline.
Paper Structure (20 sections, 4 equations, 2 figures, 1 table)

This paper contains 20 sections, 4 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. CNLU
  4. QP Theory and Linguistic Frames
  5. QuaRTz
  6. Previous Work in Differential Qualitative Analysis
  7. Augmenting LLMs with Symbolic Systems
  8. Task Description
  9. Methodology
  10. Lexicon Diagnosis and Expansion
  11. Rephrased Texts Generation
  12. Semtrans Diagnosis
  13. New Semtrans Construction
  14. Determining Frame-Quantity Relevance
  15. Pipelines
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Procedure for diagnosing an existing semtrans for a comparative adjective, together with its associated grounding fact under inspection. The diagnosis results in either (1) identifying at least one relevant semtrans from the existing knowledge base, or (2) constructing a new semtrans if: no semtrans exists for the adjective, no existing semtrans encodes a comparison event, or no existing semtrans is sufficiently relevant to the adjective as used in the given text. The dark green magnifying glass icon in the upper-right corner of a step indicates that an LLM assists the step.
  • Figure 2: Procedure for constructing a new semtrans for a comparative adjective deemed insufficiently relevant to the given grounding fact. The system begins by searching the relevant semtranses of the adjective's root form (red path), followed by the relevant semtranses of its antonyms (yellow path), and finally the relevant semtranses of each antonym's root form (blue path). If any search yields semtranses relevant to the grounding fact, the procedure stops and returns the identified quantity type(s) and frame type(s), along with the corresponding influence sign(s) obtained via the LLM. Using these results, the system generates new semtrans(es). If no relevant semtranses are found, the procedure terminates without generating new ones. A dark green magnifying glass icon in the upper-right of any step indicates LLM assistance.