Investigating the Contextualised Word Embedding Dimensions Specified for Contextual and Temporal Semantic Changes

TL;DR

Comparing pre-trained CWEs and their fine-tuned versions on contextual and temporal semantic change benchmarks under Principal Component Analysis (PCA) and Independent Component Analysis (ICA) transformations finds that PCA to better represent semantic changes than ICA within the top 10% of axes.

Abstract

The sense-aware contextualised word embeddings (SCWEs) encode semantic changes of words within the contextualised word embedding (CWE) spaces. Despite the superior performance of SCWEs in contextual/temporal semantic change detection (SCD) benchmarks, it remains unclear as to how the meaning changes are encoded in the embedding space. To study this, we compare pre-trained CWEs and their fine-tuned versions on contextual and temporal semantic change benchmarks under Principal Component Analysis (PCA) and Independent Component Analysis (ICA) transformations. Our experimental results reveal (a) although there exist a smaller number of axes that are specific to semantic changes of words in the pre-trained CWE space, this information gets distributed across all dimensions when fine-tuned, and (b) in contrast to prior work studying the geometry of CWEs, we find that PCA to better represent semantic changes than ICA within the top 10% of axes. These findings encourage the development of more efficient SCD methods with a small number of SCD-aware dimensions. Source code is available at https://github.com/LivNLP/svp-dims .

Figures (168)

• Figure 1: Pre-trained CWE, Raw
• Figure 2: Pre-trained CWE, PCA
• Figure 3: Pre-trained CWE, ICA
• Figure 4: Fine-tuned SCWE, Raw
• Figure 5: Fine-tuned SCWE, PCA