A Comparative Study of Transformers on Word Sense Disambiguation

TL;DR

This work systematically compares nine Transformer models on Word Sense Disambiguation by applying a kNN classifier to contextualized embeddings from each model, evaluated on SensEval-2 and SensEval-3. The authors implement improvements over prior BERT-based WSD methods, including lemma-based data collection and using only the final BERT layer for embeddings, and report new state-of-the-art results for both datasets. The results show that BERT, DistilBERT, and ALBERT provide the strongest sense disambiguation signals, while XLNet, Transformer-XL, and several GPT-based models underperform, with t-SNE analyses corroborating strong sense separation for the top models. The study highlights the practical implications of model choice for sense-aware representations and suggests POS-aware enhancements and targeted fine-tuning as fruitful directions for further improvement.

Abstract

Recent years of research in Natural Language Processing (NLP) have witnessed dramatic growth in training large models for generating context-aware language representations. In this regard, numerous NLP systems have leveraged the power of neural network-based architectures to incorporate sense information in embeddings, resulting in Contextualized Word Embeddings (CWEs). Despite this progress, the NLP community has not witnessed any significant work performing a comparative study on the contextualization power of such architectures. This paper presents a comparative study and an extensive analysis of nine widely adopted Transformer models. These models are BERT, CTRL, DistilBERT, OpenAI-GPT, OpenAI-GPT2, Transformer-XL, XLNet, ELECTRA, and ALBERT. We evaluate their contextualization power using two lexical sample Word Sense Disambiguation (WSD) tasks, SensEval-2 and SensEval-3. We adopt a simple yet effective approach to WSD that uses a k-Nearest Neighbor (kNN) classification on CWEs. Experimental results show that the proposed techniques also achieve superior results over the current state-of-the-art on both the WSD tasks

Figures (1)

• Figure 1: t-SNE plots of different senses of 'bank' and their contextualized embeddings. The legend(shown separately in Sub-figure (j)) shows a short description of the respective WordNet sense and the frequency of occurrence in the training data. We used the SensEval-3 training dataset for obtaining these plots.