Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Embracing Ambiguity: Improving Similarity-oriented Tasks with Contextual Synonym Knowledge

Yangning Li, Jiaoyan Chen, Yinghui Li, Tianyu Yu, Xi Chen, Hai-Tao Zheng

TL;DR

PICSO, a flexible framework that supports the injection of contextual synonym knowledge from multiple domains into PLMs via a novel entity-aware Adapter which focuses on the semantics of the entities (synonyms) in the contexts, is proposed.

Abstract

Contextual synonym knowledge is crucial for those similarity-oriented tasks whose core challenge lies in capturing semantic similarity between entities in their contexts, such as entity linking and entity matching. However, most Pre-trained Language Models (PLMs) lack synonym knowledge due to inherent limitations of their pre-training objectives such as masked language modeling (MLM). Existing works which inject synonym knowledge into PLMs often suffer from two severe problems: (i) Neglecting the ambiguity of synonyms, and (ii) Undermining semantic understanding of original PLMs, which is caused by inconsistency between the exact semantic similarity of the synonyms and the broad conceptual relevance learned from the original corpus. To address these issues, we propose PICSO, a flexible framework that supports the injection of contextual synonym knowledge from multiple domains into PLMs via a novel entity-aware Adapter which focuses on the semantics of the entities (synonyms) in the contexts. Meanwhile, PICSO stores the synonym knowledge in additional parameters of the Adapter structure, which prevents it from corrupting the semantic understanding of the original PLM. Extensive experiments demonstrate that PICSO can dramatically outperform the original PLMs and the other knowledge and synonym injection models on four different similarity-oriented tasks. In addition, experiments on GLUE prove that PICSO also benefits general natural language understanding tasks. Codes and data will be public.

Embracing Ambiguity: Improving Similarity-oriented Tasks with Contextual Synonym Knowledge

TL;DR

PICSO, a flexible framework that supports the injection of contextual synonym knowledge from multiple domains into PLMs via a novel entity-aware Adapter which focuses on the semantics of the entities (synonyms) in the contexts, is proposed.

Abstract

Contextual synonym knowledge is crucial for those similarity-oriented tasks whose core challenge lies in capturing semantic similarity between entities in their contexts, such as entity linking and entity matching. However, most Pre-trained Language Models (PLMs) lack synonym knowledge due to inherent limitations of their pre-training objectives such as masked language modeling (MLM). Existing works which inject synonym knowledge into PLMs often suffer from two severe problems: (i) Neglecting the ambiguity of synonyms, and (ii) Undermining semantic understanding of original PLMs, which is caused by inconsistency between the exact semantic similarity of the synonyms and the broad conceptual relevance learned from the original corpus. To address these issues, we propose PICSO, a flexible framework that supports the injection of contextual synonym knowledge from multiple domains into PLMs via a novel entity-aware Adapter which focuses on the semantics of the entities (synonyms) in the contexts. Meanwhile, PICSO stores the synonym knowledge in additional parameters of the Adapter structure, which prevents it from corrupting the semantic understanding of the original PLM. Extensive experiments demonstrate that PICSO can dramatically outperform the original PLMs and the other knowledge and synonym injection models on four different similarity-oriented tasks. In addition, experiments on GLUE prove that PICSO also benefits general natural language understanding tasks. Codes and data will be public.
Paper Structure (36 sections, 3 equations, 3 figures, 11 tables)

This paper contains 36 sections, 3 equations, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Injecting Structure Knowledge into PLMs
  4. Injecting Synonym Knowledge into PLMs
  5. Similarity-oriented Tasks
  6. Methodology
  7. Overall Framework
  8. Entity-Aware Adapter
  9. Aggregator
  10. Pre-training Objective
  11. Construct Pre-training Corpus
  12. Experiments
  13. Experiment Setup
  14. Downstream Tasks.
  15. Baselines.
  16. ...and 21 more sections

Figures (3)

  • Figure 1: An example demonstrating the importance of synonym knowledge for entity alignment. Some normal PLM such as BERT tend to encode entities with common tokens to more similar spaces ignoring synonymic semantics, such as Elephantiasis for Elephantiasis graecorum and Lymphatic filariasis (6.62 vs 9.17 we measured in BERT feature space), which causes misalignment.
  • Figure 2: Frameworks of injecting synonym knowledge into PLMs. Top: the previous methods LIBERT and SAPBERT. Bottom: our method PICSO.
  • Figure 3: Structure of one layer of our entity-aware Adapter.