Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Plausible-Parrots @ MSP2023: Enhancing Semantic Plausibility Modeling using Entity and Event Knowledge

Chong Shen, Chenyue Zhou

TL;DR

The paper investigates enhancing a large language model with fine-grained entity and event knowledge to improve semantic plausibility judgments for simple events, addressing ambiguities in subject-verb-object and mismatch with natural language inputs. It combines ultra-fine-grained entity typing (UFET), event-type detection, and template-based prompt engineering, augmented by data-balancing techniques and a knowledge base (WikiData) to include definitions. Empirical results show significant gains on the PEP-3K dataset when injecting both event-type and entity-type knowledge, while results on PAP reveal limitations for more abstract phrasing, underscoring the need to handle higher-level abstraction in plausibility modeling. The work demonstrates practical benefits for knowledge-grounded prompting in semantic reasoning tasks and outlines future work on more complex events and multi-modal data.

Abstract

In this work, we investigate the effectiveness of injecting external knowledge to a large language model (LLM) to identify semantic plausibility of simple events. Specifically, we enhance the LLM with fine-grained entity types, event types and their definitions extracted from an external knowledge base. These knowledge are injected into our system via designed templates. We also augment the data to balance the label distribution and adapt the task setting to real world scenarios in which event mentions are expressed as natural language sentences. The experimental results show the effectiveness of the injected knowledge on modeling semantic plausibility of events. An error analysis further emphasizes the importance of identifying non-trivial entity and event types.

Plausible-Parrots @ MSP2023: Enhancing Semantic Plausibility Modeling using Entity and Event Knowledge

TL;DR

The paper investigates enhancing a large language model with fine-grained entity and event knowledge to improve semantic plausibility judgments for simple events, addressing ambiguities in subject-verb-object and mismatch with natural language inputs. It combines ultra-fine-grained entity typing (UFET), event-type detection, and template-based prompt engineering, augmented by data-balancing techniques and a knowledge base (WikiData) to include definitions. Empirical results show significant gains on the PEP-3K dataset when injecting both event-type and entity-type knowledge, while results on PAP reveal limitations for more abstract phrasing, underscoring the need to handle higher-level abstraction in plausibility modeling. The work demonstrates practical benefits for knowledge-grounded prompting in semantic reasoning tasks and outlines future work on more complex events and multi-modal data.

Abstract

In this work, we investigate the effectiveness of injecting external knowledge to a large language model (LLM) to identify semantic plausibility of simple events. Specifically, we enhance the LLM with fine-grained entity types, event types and their definitions extracted from an external knowledge base. These knowledge are injected into our system via designed templates. We also augment the data to balance the label distribution and adapt the task setting to real world scenarios in which event mentions are expressed as natural language sentences. The experimental results show the effectiveness of the injected knowledge on modeling semantic plausibility of events. An error analysis further emphasizes the importance of identifying non-trivial entity and event types.
Paper Structure (24 sections, 1 equation, 5 figures, 5 tables)

This paper contains 24 sections, 1 equation, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semantic Plausibility Modeling
  4. Ultra Fine-grained Entity Typing
  5. Event Detection
  6. Task Definition
  7. Methods
  8. Data Augmentation
  9. Ultra Fine-grained Entity Typing
  10. Event Detection
  11. Template-based Prompt Engineering
  12. Experiments
  13. Data
  14. Preliminary Study
  15. Model Implementation
  16. ...and 9 more sections

Figures (5)

  • Figure 1: Simple (s,v,o)-event enhanced by fine-grained entity types for the subject and object and event type for the verb, accompanied with their definitions.
  • Figure 2: System architecture.
  • Figure 3: Word clouds of the most frequent words associated with the labels in PEP-3K train split.
  • Figure 4: Word similarity between top plausible words and implausible words in PAP
  • Figure 5: Word similarity between top plausible words and implausible words in PEP-3K