The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations

TL;DR

An intrinsic evaluation of embeddings from different models on semantic similarity tasks involving OOV words reveals that the quality of representations of words that are split is often, but not always, worse than that of the embeddings of known words.

Abstract

When deriving contextualized word representations from language models, a decision needs to be made on how to obtain one for out-of-vocabulary (OOV) words that are segmented into subwords. What is the best way to represent these words with a single vector, and are these representations of worse quality than those of in-vocabulary words? We carry out an intrinsic evaluation of embeddings from different models on semantic similarity tasks involving OOV words. Our analysis reveals, among other interesting findings, that the quality of representations of words that are split is often, but not always, worse than that of the embeddings of known words. Their similarity values, however, must be interpreted with caution.

Figures (4)

• Figure 1: Example of one of our settings where we calculate the cosine similarity between the representations of an OOV word and a known word. We test different ways of creating one embedding for an OOV word (§ \ref{['sec:experimental_setup']}), such as AVG and LNG, on two similarity tasks (§ \ref{['sec:data']}).
• Figure 2: BERT AVG results by layer and split-type on every split-sim subset.
• Figure 3: Distribution of predicted similarity values by BERT ( AVG) across split-types in split-sim.
• Figure 4: Average similarity values obtained on WiC ( bal) with the AVG strategy.