Static Word Embeddings for Sentence Semantic Representation

TL;DR

This work addresses the need for cost-efficient, fixed-length sentence representations by introducing static word embeddings (SWEs) optimized for sentence semantics. The method extracts SWEs from a pre-trained Sentence Transformer, refines them with sentence-level PCA using ABTT, and further improves quality via knowledge distillation or contrastive learning. Across monolingual and cross-lingual tasks, the proposed SWEs substantially outperform existing static models and even rival basic Sentence Transformer baselines on English benchmarks, while achieving strong cross-lingual translation retrieval performance. Analyses reveal that ABTT removes language-identity components and that the learned norms weight words by their semantic influence, offering practical benefits for CPU-friendly, scalable sentence representation.

Abstract

We propose new static word embeddings optimised for sentence semantic representation. We first extract word embeddings from a pre-trained Sentence Transformer, and improve them with sentence-level principal component analysis, followed by either knowledge distillation or contrastive learning. During inference, we represent sentences by simply averaging word embeddings, which requires little computational cost. We evaluate models on both monolingual and cross-lingual tasks and show that our model substantially outperforms existing static models on sentence semantic tasks, and even surpasses a basic Sentence Transformer model (SimCSE) on a text embedding benchmark. Lastly, we perform a variety of analyses and show that our method successfully removes word embedding components that are not highly relevant to sentence semantics, and adjusts the vector norms based on the influence of words on sentence semantics.

Figures (6)

• Figure 1: Scatter plots of our cross-lingual SWEs in English (blue) and Japanese (orange). The leftmost shows the 1st and 2nd principal components, and the rest are t-SNE visualisation before and after applying PCA wo/w ABTT.
• Figure 2: Comparison of word embedding norms obtained in Sections \ref{['sec_decontext']}, \ref{['sec_pca']}, and \ref{['sec_kd']}. For normalisation, norms of each POS tag are divided by the maximum value of all tags (i.e. the norm of Proper Noun).
• Figure 3: The t-SNE visualisation of sentence embeddings produced by mGTE-base at the 1st, 4th, 8th, and 12th (= last) layers of Transformer for English (blue) and Japanese (orange) sentences. The encoded sentences are 1,000 pairs of translations sampled from a parallel corpus.
• Figure 4: The t-SNE visualisation of sentence embeddings produced by mGTE-MLM-base (the backbone masked language model of mGTE before fine-tuning) at the 1st, 4th, 8th, and 12th (= last) layers of Transformer for English (blue) and Japanese (orange) sentences. The encoded sentences are 1,000 pairs of translations sampled from a parallel corpus.
• Figure 5: Scatter plots of our cross-lingual SWEs in English (blue) and Chinese (orange). The leftmost shows the 1st and 2nd principal components, and the rest are t-SNE visualisation before and after applying PCA with/without ABTT.