Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Word Sense Induction with Hierarchical Clustering and Mutual Information Maximization

Hadi Abdine, Moussa Kamal Eddine, Michalis Vazirgiannis, Davide Buscaldi

TL;DR

This paper proposes a novel unsupervised method based on hierarchical clustering and invariant information clustering (IIC), which is used to train a small model to optimize the mutual information between two vector representations of a target word occurring in a pair of synthetic paraphrases.

Abstract

Word sense induction (WSI) is a difficult problem in natural language processing that involves the unsupervised automatic detection of a word's senses (i.e. meanings). Recent work achieves significant results on the WSI task by pre-training a language model that can exclusively disambiguate word senses, whereas others employ previously pre-trained language models in conjunction with additional strategies to induce senses. In this paper, we propose a novel unsupervised method based on hierarchical clustering and invariant information clustering (IIC). The IIC is used to train a small model to optimize the mutual information between two vector representations of a target word occurring in a pair of synthetic paraphrases. This model is later used in inference mode to extract a higher quality vector representation to be used in the hierarchical clustering. We evaluate our method on two WSI tasks and in two distinct clustering configurations (fixed and dynamic number of clusters). We empirically demonstrate that, in certain cases, our approach outperforms prior WSI state-of-the-art methods, while in others, it achieves a competitive performance.

Word Sense Induction with Hierarchical Clustering and Mutual Information Maximization

TL;DR

This paper proposes a novel unsupervised method based on hierarchical clustering and invariant information clustering (IIC), which is used to train a small model to optimize the mutual information between two vector representations of a target word occurring in a pair of synthetic paraphrases.

Abstract

Word sense induction (WSI) is a difficult problem in natural language processing that involves the unsupervised automatic detection of a word's senses (i.e. meanings). Recent work achieves significant results on the WSI task by pre-training a language model that can exclusively disambiguate word senses, whereas others employ previously pre-trained language models in conjunction with additional strategies to induce senses. In this paper, we propose a novel unsupervised method based on hierarchical clustering and invariant information clustering (IIC). The IIC is used to train a small model to optimize the mutual information between two vector representations of a target word occurring in a pair of synthetic paraphrases. This model is later used in inference mode to extract a higher quality vector representation to be used in the hierarchical clustering. We evaluate our method on two WSI tasks and in two distinct clustering configurations (fixed and dynamic number of clusters). We empirically demonstrate that, in certain cases, our approach outperforms prior WSI state-of-the-art methods, while in others, it achieves a competitive performance.
Paper Structure (19 sections, 5 equations, 3 figures, 5 tables)

This paper contains 19 sections, 5 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Dataset Setup
  5. BART
  6. Generating randomly perturbated replicates
  7. Vectors Extraction
  8. Loss Function
  9. Invariant Information Clustering Loss
  10. Match Loss
  11. Sense Embedding: Getting New Word Vectors
  12. Clustering
  13. Evaluation
  14. SemEval-2010 task 14:
  15. SemEval-2013 task 13:
  16. ...and 4 more sections

Figures (3)

  • Figure 1: An illustration represents the different sense-based clusters of the word bank with the most frequent words used in the corresponding contexts. These clusters are obtained using agglomerative clustering on a set of RoBERTa vectors of the word bank extracted from 3000 sentences collected from Wikipedia. The centre of each cluster is the 2D PCA vector of the average 'bank' vectors of the cluster. The size of the points is proportional to the frequency of its appearance in the context of each sense-based cluster.
  • Figure 2: The pipeline of our method: For the word "live" chosen as target, a list of sentences is provided. BART is used to generate their corresponding paraphrases. The hidden representation X$_{live}^{l}$ of the target word is extracted from the layer l of a pretrained language model. Dashed line denotes shared parameters.
  • Figure 3: The AVG scores of SemEval-2010 and SemEval-2013 WSI tasks using agglomerative clustering on all the layers of different pretrained models.