Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SememeLM: A Sememe Knowledge Enhanced Method for Long-tail Relation Representation

Shuyi Li, Shaojuan Wu, Xiaowang Zhang, Zhiyong Feng

TL;DR

SememeLM tackles the problem of representing long-tail word-pair relations without relying on context by injecting sememe-level knowledge from HowNet into a graph-augmented language-model framework. It builds a directed sememe relation graph, encodes it with a Graph Attention Network, and aligns graph-derived signals with LM representations through a Consistency Alignment Module, complemented by supervised contrastive training. The approach yields improved relation representations across seven analogy benchmarks and shows competitive performance against large language models, while offering a cost-effective alternative for relation reasoning and potential benefits for knowledge graph completion. This work demonstrates the viability of integrating structured sememe knowledge with pre-trained LMs to capture nuanced, low-frequency relations beyond the training corpus.

Abstract

Recognizing relations between two words is a fundamental task with the broad applications. Different from extracting relations from text, it is difficult to identify relations among words without their contexts. Especially for long-tail relations, it becomes more difficult due to inadequate semantic features. Existing approaches based on language models (LMs) utilize rich knowledge of LMs to enhance the semantic features of relations. However, they capture uncommon relations while overlooking less frequent but meaningful ones since knowledge of LMs seriously relies on trained data where often represents common relations. On the other hand, long-tail relations are often uncommon in training data. It is interesting but not trivial to use external knowledge to enrich LMs due to collecting corpus containing long-tail relationships is hardly feasible. In this paper, we propose a sememe knowledge enhanced method (SememeLM) to enhance the representation of long-tail relations, in which sememes can break the contextual constraints between wors. Firstly, we present a sememe relation graph and propose a graph encoding method. Moreover, since external knowledge base possibly consisting of massive irrelevant knowledge, the noise is introduced. We propose a consistency alignment module, which aligns the introduced knowledge with LMs, reduces the noise and integrates the knowledge into the language model. Finally, we conducted experiments on word analogy datasets, which evaluates the ability to distinguish relation representations subtle differences, including long-tail relations. Extensive experiments show that our approach outperforms some state-of-the-art methods.

SememeLM: A Sememe Knowledge Enhanced Method for Long-tail Relation Representation

TL;DR

SememeLM tackles the problem of representing long-tail word-pair relations without relying on context by injecting sememe-level knowledge from HowNet into a graph-augmented language-model framework. It builds a directed sememe relation graph, encodes it with a Graph Attention Network, and aligns graph-derived signals with LM representations through a Consistency Alignment Module, complemented by supervised contrastive training. The approach yields improved relation representations across seven analogy benchmarks and shows competitive performance against large language models, while offering a cost-effective alternative for relation reasoning and potential benefits for knowledge graph completion. This work demonstrates the viability of integrating structured sememe knowledge with pre-trained LMs to capture nuanced, low-frequency relations beyond the training corpus.

Abstract

Recognizing relations between two words is a fundamental task with the broad applications. Different from extracting relations from text, it is difficult to identify relations among words without their contexts. Especially for long-tail relations, it becomes more difficult due to inadequate semantic features. Existing approaches based on language models (LMs) utilize rich knowledge of LMs to enhance the semantic features of relations. However, they capture uncommon relations while overlooking less frequent but meaningful ones since knowledge of LMs seriously relies on trained data where often represents common relations. On the other hand, long-tail relations are often uncommon in training data. It is interesting but not trivial to use external knowledge to enrich LMs due to collecting corpus containing long-tail relationships is hardly feasible. In this paper, we propose a sememe knowledge enhanced method (SememeLM) to enhance the representation of long-tail relations, in which sememes can break the contextual constraints between wors. Firstly, we present a sememe relation graph and propose a graph encoding method. Moreover, since external knowledge base possibly consisting of massive irrelevant knowledge, the noise is introduced. We propose a consistency alignment module, which aligns the introduced knowledge with LMs, reduces the noise and integrates the knowledge into the language model. Finally, we conducted experiments on word analogy datasets, which evaluates the ability to distinguish relation representations subtle differences, including long-tail relations. Extensive experiments show that our approach outperforms some state-of-the-art methods.
Paper Structure (23 sections, 9 equations, 4 figures, 4 tables)

This paper contains 23 sections, 9 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Our approach
  3. HowNet
  4. Sememe relation Graph
  5. Graph Encoding
  6. Consistency Alignment Module
  7. Training
  8. Experiments and Results
  9. Experimental Setup
  10. Datasets
  11. Baselines
  12. Implementation Details
  13. Performance Analysis
  14. Small-sized Language Models
  15. Large Language Models
  16. ...and 8 more sections

Figures (4)

  • Figure 1: The long-tail relation problem and our solution
  • Figure 2: An example of the HowNet structure.
  • Figure 3: The overall flow of training our model when given word pairs.
  • Figure 4: We analyzed the results of our approach, RelBERT, Chen$_{RoBERTa}$ and ChatGPT$_0$ on two examples in E-KAR dataset. The kitten represents the choice made by Chen$_{RoBERTa}$, the fish represents the choice made by RelBERT, while the puppy illustrates the choice made by our approach.