Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Structured Event Reasoning with Large Language Models

Li Zhang

TL;DR

The work investigates how large language models can be made reliable and effective at event reasoning by pairing them with structured representations. It introduces three forms of structure: a language based event relation schema, a semi symbolic event entity schema, and a fully symbolic world model that can be executed by solvers. Across tasks including intent detection, next event prediction, script construction, and planning in procedural and interactive settings, structured approaches consistently outperform end to end LLM reasoning and improve interpretability. The results highlight the benefits of neurosymbolic reasoning, with code like prompts and PDDL based planning enabling verifiable and adaptable AI systems for complex real world reasoning. The work also shows the importance of dataset design, cross domain transfer, and open domain planning challenges for future progress in structured event reasoning with LLMs.

Abstract

Reasoning about real-life events is a unifying challenge in AI and NLP that has profound utility in a variety of domains, while fallacy in high-stake applications could be catastrophic. Able to work with diverse text in these domains, large language models (LLMs) have proven capable of answering questions and solving problems. However, I show that end-to-end LLMs still systematically fail to reason about complex events, and they lack interpretability due to their black-box nature. To address these issues, I propose three general approaches to use LLMs in conjunction with a structured representation of events. The first is a language-based representation involving relations of sub-events that can be learned by LLMs via fine-tuning. The second is a semi-symbolic representation involving states of entities that can be predicted and leveraged by LLMs via few-shot prompting. The third is a fully symbolic representation that can be predicted by LLMs trained with structured data and be executed by symbolic solvers. On a suite of event reasoning tasks spanning common-sense inference and planning, I show that each approach greatly outperforms end-to-end LLMs with more interpretability. These results suggest manners of synergy between LLMs and structured representations for event reasoning and beyond.

Structured Event Reasoning with Large Language Models

TL;DR

The work investigates how large language models can be made reliable and effective at event reasoning by pairing them with structured representations. It introduces three forms of structure: a language based event relation schema, a semi symbolic event entity schema, and a fully symbolic world model that can be executed by solvers. Across tasks including intent detection, next event prediction, script construction, and planning in procedural and interactive settings, structured approaches consistently outperform end to end LLM reasoning and improve interpretability. The results highlight the benefits of neurosymbolic reasoning, with code like prompts and PDDL based planning enabling verifiable and adaptable AI systems for complex real world reasoning. The work also shows the importance of dataset design, cross domain transfer, and open domain planning challenges for future progress in structured event reasoning with LLMs.

Abstract

Reasoning about real-life events is a unifying challenge in AI and NLP that has profound utility in a variety of domains, while fallacy in high-stake applications could be catastrophic. Able to work with diverse text in these domains, large language models (LLMs) have proven capable of answering questions and solving problems. However, I show that end-to-end LLMs still systematically fail to reason about complex events, and they lack interpretability due to their black-box nature. To address these issues, I propose three general approaches to use LLMs in conjunction with a structured representation of events. The first is a language-based representation involving relations of sub-events that can be learned by LLMs via fine-tuning. The second is a semi-symbolic representation involving states of entities that can be predicted and leveraged by LLMs via few-shot prompting. The third is a fully symbolic representation that can be predicted by LLMs trained with structured data and be executed by symbolic solvers. On a suite of event reasoning tasks spanning common-sense inference and planning, I show that each approach greatly outperforms end-to-end LLMs with more interpretability. These results suggest manners of synergy between LLMs and structured representations for event reasoning and beyond.
Paper Structure (95 sections, 9 equations, 40 figures, 29 tables)

This paper contains 95 sections, 9 equations, 40 figures, 29 tables.

Table of Contents

  1. INTRODUCTION
  2. Thesis Statement
  3. Publications Presented
  4. RELATED WORK
  5. Events and Procedures
  6. Events
  7. Procedures
  8. Large Language Models
  9. A Brief History of Language Models
  10. LLMs are Black-Boxes
  11. Machine Reasoning
  12. Reasoning Frameworks
  13. Reasoning Tasks
  14. Interplay
  15. EVENT-RELATION SCHEMA: A NATURAL LANGUAGE REPRESENTATION
  16. ...and 80 more sections

Figures (40)

  • Figure 1: An overview of work discussed in this thesis.
  • Figure 2: Four approaches of using LLMs for structured reasoning: (from upper to lower) end-to-end usage (no structure), fine-tuning with structure, prompting with structure, and neurosymbolic usage.
  • Figure 3: The evolution of language models zhao2023survey.
  • Figure 4: An illustration of my proposed pipeline leveraging a natural language representation of entities. The LLM is fine-tuned with data of that representation.
  • Figure 5: A fine-grained procedure representation (schema) proposed by zhang-etal-2012-automatically.
  • ...and 35 more figures